Good catch; fixed now; ty.

I think there's a typo; the text refers to "Poltergeist Pummelers" but the input data says "Phantom Pummelers".

Good catch; fixed now; thank you.

The Conservative party in the UK are also called "Tories".

Carla is harder: I think she's some combination of Carl (literally, "free man": appropriate for someone who wants to avoid tyranny) and Karl (as in Karl Marx), but I wouldn't be surprised if there were a more prosaic explanation.

Caplan has been saying this intermittently for the past two years.

I think this is plausibly the best scenario either of us have made to date.

The basic game was very good, layering simple rules on top of each other to create a complex system which was challenging to detangle but easy to understand once you know the rules. I was particularly impressed by the fact that you managed the (imo, near-impossible) feat of making an enjoyable D&D.Sci where handling data starvation was a key part of the problem: most players (including me) seem to have had the (sensible) thought "okay, let's filter for only potions with Onyx and Bone", and success past that point was predicated on realizing there weren't quite enough rows to justify being that picky.

The twist struck me as fair, funny and fun. It provided an object lesson in noticing when things don't quite add up, and letting the qualitative shade the quantitative; it also expanded the scope of the genre in ways I realize I've been neglecting to.

All that said, I have some (minor, petty) criticisms . . . not of the game itself, but how it was presented. Namely:

.This entry was billed as "relatively simple", but I think it was about median difficulty by the standards of D&D.Sci; pretty sure it was harder than (for example) The Sorceror's Personal Shopper.

."STORY (skippable)" was kind of misleading this time: the flavortext had a lot of little hints that the Archmage wasn't on the level, so someone who didn't read it (or failed to read between the lines, like me) would be at a (small) disadvantage.

."Archmage Anachronos is trying to brew Barkskin Potion" was A) the GM saying something false directly to the players, and B) a missed opportunity: if you'd written something like "Your goal is to help Archmage Anachronos brew Barkskin Potion", that would have been a subtle confirmation that giving him exactly what he asked for would lead to the best outcome (vs more aggressive / galaxy-brained forms of sabotage, or refusing to cast judgement on his pursuit of immortality, or any other reaction).

After some consideration (and reading other people's answers, in particular simon's) I've come to the conclusion that the best answer to give is actually

Wait . . . actually, if we're in the mood for galaxy-brained moves, we could go one better and try to

I think your theory about

is completely correct and I feel dumb for not thinking of it; ditto your proposed reaction. On reflection, I suspect that this is because

so your right-ness is a lot more important than it might seem at first glance. Good catch!

Thanks for running this!

Unless I made some trivial mistake,

should work.

Explanation:

Reputational: D&D.Sci.

Experimental: D&D.Sci, with a consistent limit on time & resources used.

Organizational: D&D.Sci, with a consistent limit on time & resources used, using freshly-baked scenarios you know no-one has ever played before.

Limitations:

• Takes several hours to play most scenarios.
• Requires generic coding/spreadsheeting/data-science-ing skills in addition to Rationality; people who are good at those skills get an unfair(?) advantage.
• Getting familiar with the genre gives an unfair(!) advantage.

Misc. addl. reflections on the topic:

• Starting from zero is a valid approach, but looking at existing tests and thinking "okay but what if this was better/harder/about slightly different skills" is also sensible. Figuring out how clever and effective people are is a big industry! We should take inspiration from tests employers give job applicants, and any test any gatekeepers give anyone. (Especially if that means we get to subsidize development of rationality-tests by selling them to HR departments.)
• . . . are there any ways to test rationality which don't rely on complementary skills? Even written tests test your ability to read the questions.
• Videogames could be so good for this if they weren't optimized for fun and accessibility.

stared daggers

This has connotations of being angry, which I don't think is what you're going for. (Unless Peter is getting mad at Oskar for potentially revealing his scheme to his bosses by doing something too similar, or he's irritated that a kindred spirit isn't recognising him fast enough, or unless I've completely misunderstood the implication here.)

I think 0th-order, 2nd-order and 3rd-order forecasting should be called threecasting, fivecasting and sixcasting respectively. This easily lets speakers differentiate between layers; also, imo, names which are bad puns tend to stick.

Second link is broken.

Response to clarifying question:

Yes. The Duke has learned the hard way that his architects' guesses as to how much their projects will end up costing are consistently worse than useless; if you want to optimize on cost as well as impossibility, that's another thing you'll have to deduce from the record of finished projects.

Reflections on my performance:

I'm pleasantly surprised by the effectiveness of my reasoning, and of my meta-reasoning. Not only did my loadout do well, but my calibration was impressively close: the final decision I pegged at a "~95%" success rate got 94.4%, and most of the alternative strategies I mentioned in my post were similarly on-the-nose.

(Unfortunately, my meta-meta-reasoning could still use some work. I figured out that this was a "linear-ish logistic success model with some interactions on top" kind of problem, took this as an opportunity to test that library I made, created a good predictor with a bunch of pretty/informative graphs . . . and then found myself thinking "only need one minigun? doesn't sound right to me", "why would Tyrants/Artillery and Scarabs/Minigun-or-Flamethrowers be so much stronger than every other potential feature interaction?", and "I'm totally gonna turn out to have screwed up and wish I'd handled this with XGBoost, better not even mention how I built my model". If I'd been more calibrated about how calibrated I ended up being, this could have been a really good chance to show off by calling in advance that my unconventional ML approach would succeed here.)

Reflections on the challenge:

This was the 2D performance thing I tried to pull off in Boojumologist, but with better conceptual underpinning and flawless execution. I'm proud, gladdened and envious: can't think of a single way to improve this scenario.

(I, uh, may be biased by how well I happened to do: please take this feedback with a grain of salt.)

By imitating other players

As Jay Bailey mentioned, you can look at how other players approached challenges, and copy the approaches that worked. Pablo Repetto’s playthroughs of three early .scis seem particularly worthwhile given your situation, both because of how comprehensive & well-written they are, and because they were made by someone in the process of learning to use code on data science problems (the first playthrough was done in pure Excel, the other two were handled in Python).

By following a sensible strategy

Below is my standard plan for investigating a dataset, synthetic or otherwise (cribbed from an otherwise-mediocre Udacity course I took most of a decade ago, and still worth following).

-

Univariate Analysis: How is each feature distributed when considered in isolation? You should probably make a histogram for each column.

Bivariate Analysis: Construct and check the correlation matrix between all features. Are there clusters? Create scatterplots (or equivalent) for any pair of features which correlate unusually strongly, any pair of features where at least one is a response variable, and any pair of features you find yourself curious about.

Feature Derivation: Based on what you’ve seen so far – and/or common sense – are there meaningful features you can create from what you’ve been provided? (i.e., if you’re given "Number of Wizards", "Number of Sorcerors" and "Number of Druids" for each row, it might be worth creating a “Total Number of Magic Users” column.) Investigate how these features interact with others.

ML Modelling: If you can, and it seems like a good idea, build an ML model predicting the important/unknown features from those you have. If constructed successfully, this becomes an oracle you can ask about the outcome of any possible choice you could make. (XGBoost and similar tools are extremely versatile, and have pretty good performance on most problems.)

-

(The above is just a rough guide for what to do when you don’t know what to do. If you follow it, you should pretty quickly find yourself with a list of rabbitholes to fall down; you should probably err on the side of dropping everything and deviating from the path as soon as you find something interesting.)

By playing easier D&D.Scis

Difficulty of D&D.Sci games tends to be both high and high-variance; it’s usually assumed that players will have both data-manipulation and model-building skills. For what it’s worth, I can confirm that two relatively-approachable scenarios where not-using-ML won't put you at a disadvantage are (spoilered because this technically leaks information about them):

I'm glad you liked it!

(. . . could you spoiler your strategy and win chance? I know this challenge is three years old, and what you mention here isn't game-breaking info, but I want to keep it possible for people looking/playing through the archives to seek clarifications in the comments without unwittingly/unwillingly learning anything else about the scenario.)

That makes sense, ty.

The leaderboard will track how well you've done relative to random/best play at the # of soldiers you chose to bring.

Could you elaborate on this? I think I'd do better relative to best play with

and do better relative to random play with

so it's not clear which way I should lean; also, I don't know how you plan to quantify "relative to".

What we're facing:

Relevant Weapons:

Current strategies per number of soldiers:

If I have to pick one strategy:

Description of an investigative cul-de-sac:

I notice that

I reason that

But then

From all this I conclude

Like, conceptually it's absolutely unpredictable

That's exactly what I was going for; I wanted phenomena which couldn't have been predicted without using the dataset.

Misc. prelim notes:

Thanks for running this when my one was going to be late, and thanks for checking with me beforehand.

(Also, thanks for the scenario, like, in general: it looks like a fun one!)

I (to my own surprise) got an "above average" score when I took this test a few years back, which I attribute mostly to the lack of emotional and circumstantial 'noise' in the images. I don't think being able to tell what is being emoted by a professional actor told to display exactly one (1) emotion, with no mediating factors, has much connection with being able to read actual people.

(. . . though a level-2 version with tags like "excited but hesitant" or "proud and angry" or "cheerful; unrelatedly, lowkey seasick" could actually be extremely useful, now I think on it.)

Typos:

"Al gore"->"Al Gore"

"newpaper"->"newspaper"

"south park"->"South Park"

"scott alexander"->"Scott Alexander"

"a littler deeper"->"a little deeper"

"Ai"->"AI"

(. . . I'm now really curious as to why you keep decapitalizing names and proper nouns.)

Regarding the actual content of the post: appreciated, approved, and strong-upvoted. Thank you.

This, linked at "Never." in the OP.

an alliance socialist nations

an alliance of socialist nations

I didn't like this post, but I did very much like the "insight porn" post it linked to. (Unfortunately LW doesn't let you simultaneously downvote and strong-upvote a post, so consider my weak-upvote as a sum-of-vibes.)

If someone says ‘What’s for supper?’ a beginner will desperately try to think up something original. He will carefully evaluate dozens of options in his mind.

“Is this funny?” “Will this not reveal something weird about myself?”

It will take him ages to come up with something and eventually he will say something “fried mermaid”.

An improv pro would simply respond “fish”.

Taken - almost verbatim, without attribution - from Impro, by Keith Johnstone. (I don't know whether LW would consider this plagiarism, or consider that to be bad.)

taking it out early and letting it sit

What I actually usually do is move it from the freezer to the refrigerator like 15min before I eat it, so the change in temperature is more predictable and evenly distributed (instead of some parts being melted while others stay too cold).

Is the point that it's initially too hard to scoop?

That and it being too cold to properly enjoy the taste.

(The votes on my original comment make me think most people are less concerned about their dessert-that's-supposed-to-be-cold being too cold. Typical-mind strikes again, I guess.)

I enjoyed the exercise, thanks! 

You're welcome, and thank you for playing.

(I wrote a custom loss function for the NN)

I'm curious how you defined that. (i.e. was it "gradient = x for rows where predicted>actual, gradient = -8x for rows where actual>predicted", or something finickier?)

I think this is because

(I would have said this sooner but helping a player while the challenge was still running seemed like a bad look.)

I suspect a large (possibly not dominant) part of the ice cream effect is required preptime triggering myopic discounting. If eating ice cream at home, you need to take it out of the freezer at least a few minutes before eating it; this means that if your comfort food of choice is ice cream, you'll only eat it if it seems like a legitimately good idea ('a moment of weakness' becomes 'like 10min of weakness', a higher bar for cravings to clear).

Note: I'll be unavoidably and unexpectedly busy at the start of next week, and so will have to delay resolution of this challenge until either Tuesday or Wednesday (probably Tuesday). I'd apologise for the inconvenience but I'm pretty sure no-one minds.

Clarifications:

The Tyrant will weigh his Precious Beasts with the same level of diligence you would: no more, no less.

You can predict weights with as fine a granularity as you like; if you want to claim a turtle has a weight of 12.345678lb, that's fine.

"Mu": Japanese word roughly translateable as 'absence'.

"Kami": Japanese word roughly translateable as 'god'.

"-sama": Japanese honorific for referring to someone whose status/position is much higher than yours.

Mukami-sama, the God of Atheism

I found this disproportionately charming.

I think the commentary on the state of Givewell's evidence - in particular, that worryingly large parts of it come down to "we called a mid-ranking employee once and they claimed they were doing X and we thought they had good vibes" - was good, correct, novel and important: strong upvote for that alone.

 (I disagree that you should blame Givewell for that: they're not hiding their flaws, and AFAICT the only people other than this author who are openly discussing Givewell's limitations are Givewell themselves. Most of their alleged sins IMO come down to the way people insist on treating them, and the bizarre dearth of competitor/successor organisations aiming for "Givewell but >10x more rigorous/demanding".)

I think almost everything else the author says is some combination of incoherent, incorrect, mean-spirited and fnordful. But in the marketplace of ideas, one bullseye is worth any number of missed shots.

Disrecommending Slay The Spire. While it's a great game and it fits the rest of your criteria like a glove, it has very little hidden information in a practical sense (one of the more innovative things about it is that you can almost always see what the enemy will do next turn), and as such has basically no places where explore/exploit tradeoffs and VOI calculations would be relevant (I assume that this isn't a negotiable part of what you're asking for; if not, yeah I also recommend StS).

Tentative recommendation of Slipways; the VOI part isn't as central as I suspect you'd like, but sending out probes sure does cost time and money you could use for settling planets and forming trade routes; and while it's easy enough to survive to the end of your term, it gives what you're asking for if you choose to define 'victory' as 'get 5+ stars on Tough'.

The objective of rationality is to become right instead of wrong.

I think this is technically false, in a subtle but important way. If I gained [knowledge of whether every six-digit number is prime] in exchange for [knowledge of whether wandering out into open traffic is a good idea], I'd have gleaned a net 899999 bits of right-ness, but it still wouldn't have been a worthwhile deal, or made me more rational in any practical sense. The missing gears are becoming right about important && relevant things, bothering to apply that knowledge, and - conditional on applying it at all - applying it well.

I think this project is good (Like, unusually good! It's a step forward! I enjoyed it, and I commend you for your service to the Cause!), but I notice a lack of emphasis on changing actions vs changing minds, both in this post and in the videos I watched, and I want to make sure you've noticed that too.

(And yes, I do recognize the irony of me pointing out a true thing about [pointing out true things without having an associated practical outcome] without having an associated practical outcome. Still think it's worth saying!)

Sorry about that, reality got in the way; also, ended up scrapping my concept for the next one and my backup concept for it; no idea when it'll end up actually made (not necessarily this month), except that I plan to release on a Friday to do the standard "10 days with a choice of weekend" thing.

Damn! Mea culpa; I'll edit the original post so anyone going through the archives won't have the same problem.

Also, strong-upvoted for asking "so, with X years of hindsight, how did this pan out?" on an old post. More people should do that.

Before circumstances let me answer that question, the client got bought out by a bigger company, which was (and is) a lot more cagey about both hiring contractors and sharing internal details with outsiders; last I heard, the client's absorbed remnants are still sometimes using my modelling approach, but I have no idea how much they're using it, how much they're relying on it, or to what extent it's benefiting them.

There are no time effects in the data; past trends can in generality be assumed to exist in the present.

(Good question!)

The same way it does everything: in a weird, non-Euclidean manner which defies human intuition.

For the unreleased challenge, b) isn't for sale: making something intended to (eventually) be played by humans on LW and then using it solely as LLM-fodder would just be too sad. And I'm guessing you wouldn't want a) without b); if so, so much for that.

. . . if the "it must never be released to the public internet" constraint really is that stringent, I might be better advised to make D&D.Sci-style puzzles specifically for your purposes. The following questions then become relevant:

.How closely am I allowed to copy existing work? (This gets easier the more I can base it on something I've already done.)

.How many challenges are you likely to want, and how similar can they be to each other? (Half the difficulty on my end would be getting used to the requirements, format etc; I'd be more inclined to try this if I knew I could get paid for many challenges built along similar lines.)

.Is there a deadline? (When are you likely to no longer need challenges like this?) (Conversely, would I get anything extra for delivering a challenge within the next week or so?)

This seems like a natural fit for D&D.Sci games. All the ones I made are public domain, so you can use them freely (and I bet the other people who made some would give you permission if you asked them nicely), they've been publicly played by clever humans with a variety of skill levels and associated outcomes, and they're obscure enough that I doubt an LLM would have memorized the solutions (and if not you could tweak the names and data-generation hyperparameters to flatfoot them).

. . . I happen to have a completed-but-unreleased D&D.Sci game, which I was planning to put on LW early next month, after everyone got back from their holidays. Would it be helpful if I sent it to you and delayed the release until Feb, so you and yours could let LLMs try it first?