D&D.Sci Alchemy: Archmage Anachronos and the Supply Chain Issues Evaluation & Ruleset
post by aphyer · 2024-06-17T21:29:08.778Z · LW · GW · 11 commentsContents
RULESET STEP 1: MAGICAL POTENCY STEP 2: DIRECTION STRATEGY BONUS OBJECTIVE DATASET GENERATION LEADERBOARD REFLECTION & FEEDBACK REQUEST None 11 comments
This is a follow-up to last week's D&D.Sci scenario [LW · GW]: if you intend to play that, and haven't done so yet, you should do so now before spoiling yourself.
There is a web interactive here you can use to test your answer, and generation code available here if you're interested, or you can read on for the ruleset and scores.
RULESET
There are two steps to brewing a potion:
STEP 1: MAGICAL POTENCY
Any ingredient that doesn't exist in the mundane world is Magical, while any ingredient that exists in the mundane world is not:
| Magical | Not Magical |
| Angel Feather | Badger Skull |
| Beholder Eye | Beech Bark |
| Demon Claw | Crushed Diamond |
| Dragon Scale | Crushed Onyx |
| Dragon Spleen | Crushed Ruby |
| Dragon Tongue | Crushed Sapphire |
| Dragon's Blood | Eye of Newt |
| Ectoplasm | Ground Bone |
| Faerie Tears | Oaken Twigs |
| Giant's Toe | Powdered Silver |
| Troll Blood | Quicksilver |
| Vampire Fang | Redwood Sap |
The first step of potion-brewing is to dissolve the magical potency out of the Magical Ingredients to empower your potion. This requires the right amount of Magical Ingredients: too few, and nothing magical will happen and you will produce Inert Glop, while too many and there will be an uncontrolled Magical Explosion.
If you include:
- 0-1 Magical Ingredients: 100% chance of Inert Glop.
- 2 Magical Ingredients: 50% chance of Inert Glop, 50% chance OK.
- 3 Magical Ingredients: 100% chance OK.
- 4 Magical Ingredients: 50% chance OK, 50% chance Magical Explosion.
- 5+ Magical Ingredients: 100% chance Magical Explosion.
If your potion got past this step OK, move on to:
STEP 2: DIRECTION
Some ingredients are used to direct the magical power into the desired resulting potion. Each potion has two required Key Ingredients, both of which must be included to make it:
| Potion | Key Ingredient 1 | Key Ingredient 2 |
| Barkskin Potion* | Crushed Onyx | Ground Bone |
| Farsight Potion | Beholder Eye | Eye of Newt |
| Fire Breathing Potion | Dragon Spleen | Dragon's Blood |
| Fire Resist Potion | Crushed Ruby | Dragon Scale |
| Glibness Potion | Dragon Tongue | Powdered Silver |
| Growth Potion | Giant's Toe | Redwood Sap |
| Invisibility Potion | Crushed Diamond | Ectoplasm |
| Necromantic Power Potion* | Beech Bark | Oaken Twigs |
| Rage Potion | Badger Skull | Demon Claw |
| Regeneration Potion | Troll Blood | Vampire Fang |
*Well. Sort of. See the Bonus Objective section below.
Some ingredients (Angel Feather, Crushed Sapphire, Faerie Tears and Quicksilver) aren't Key Ingredients for any potion in the dataset. Angel Feather and Faerie Tears are nevertheless useful - as magical ingredients that don't risk creating any clashing potion, they're good ways to add magical potential to a recipe. Crushed Sapphire and Quicksilver have no effect, including them is entirely wasteful.
If you've gotten through Step 1, the outcome depends on how many potions you've included both the Key Ingredients of:
- 0 potions: with nothing to direct it, the magical potential dissolves into an Acidic Slurry.
- 1 potion: you successfully produce that potion.
- 2 or more potions:
- Sometimes (1/n of the time, where n is # of potions you included) a random one of the potions will dominate, and you will produce that one.
- The rest of the time, the clashing directions will produce Mutagenic Ooze.
So, for example, if you brew a potion with:
- Dragon Spleen, Dragon Scale, Dragon Tongue and Dragon's Blood:
- You have included 4 magical ingredients, and the Key Ingredients of one potion (Fire Breathing).
- 50% of the time you will get a Magical Explosion, 50% of the time you will get a Fire Breathing Potion.
- Badger Skull, Demon Claw, Giant's Toe, Redwood Sap.
- You have included 2 magical ingredients, and the Key Ingredients of two potions (Rage and Growth).
- 50% of the time you will get Inert Glop, 25% of the time Mutagenic Ooze, 12.5% of the time Growth Potion, and 12.5% of the time Rage Potion.
- Dragon's Tongue, Powdered Silver, Angel Feather and Faerie Tears:
- You have included 3 magical ingredients, and the Key Ingredients of one potion (Glibness).
- 100% of the time, you will successfully brew a Glibness Potion.
STRATEGY
For best success, you needed to:
- Include Crushed Onyx and Ground Bone, the Key Ingredients for...the desired potion.
- Not include both Key Ingredients of any other potion.
- Include exactly 3 Magical Ingredients.
This was made trickier by the fact that you had only 4 Magical Ingredients available (Demon Claw, Giant's Toe, Troll Blood and Vampire Fang), and that Troll Blood and Vampire Fang were the Key Ingredients of a Regeneration Potion (and hence you could not include both of them).
Nevertheless, you could get a 100% success rate by:
- Including Crushed Onyx and Ground Bone (Key Ingredients).
- Including Demon Claw and Giant's Toe (Magical Ingredients that don't make another potion with one another)
- Including one of Troll Blood and Vampire Fang (third Magical Ingredient).
- Not including Badger Skull (makes Rage Potion with Demon Claw) or Redwood Sap (makes Growth Potion with Giant's Toe).
- Not including both Beech Bark and Oaken Twigs (Key Ingredients of...the wrong potion).
- You could include Crushed Diamond or Quicksilver if you wanted, but it's wasteful and has no effect.
BONUS OBJECTIVE
There was a secret bonus objective hidden in the dataset. It could not be found purely with math on the dataset, it required you to think about the ingredient names and the patterns you were seeing.
On examination of the Key Ingredients of each potion, most potions had Key Ingredients that seemed in some way vaguely metaphorically linked to the potion's effect:
- The Key Ingredients of a Farsight Potion are Beholder Eye and Eye of Newt (both are eyes, to help you see).
- The Key Ingredients of a Fire Breathing Potion are Dragon Spleen and Dragon's Blood (both parts of a fire-breathing dragon).
- The Key Ingredients of a Growth Potion are Giant's Toe and Redwood Sap (both from things well known for growing large).
There were two apparent exceptions to this, however.
- The Key Ingredients of a Barkskin Potion appeared to be Crushed Onyx and Ground Bone.
- The Key Ingredients of a Necromantic Power Potion appeared to be Beech Bark and Oaken Twigs.
This was not, in fact, true.
The Key Ingredients of a Barkskin Potion are in fact Beech Bark and Oaken Twigs, and the Key Ingredients of a Necromantic Power Potion are Crushed Onyx and Ground Bone.
Archmage Anachronos is not trying to brew a Barkskin Potion. He is trying to brew a Necromantic Power Potion in order to conduct a Dread Necromantic Ritual during the upcoming magical conjunction.
Not wanting to reveal this, he's swapped around 'Barkskin Potion' and 'Necromantic Power Potion' in his records of results, telling you that every recipe that actually produced a Barkskin Potion produced a Necromantic Power Potion and vice versa.
simon realized this, I don't think anyone else figured it out independently (as opposed to seeing his comment mentioning it).
DATASET GENERATION
Archmage Anachronos's algorithm for deciding which ingredients to use is as follows:
- 20% of brews are experiments. He grabs 1d6+2 random ingredients to test together, but if they seem too expensive he reconsiders and grabs a different 1d6+2 ingredients (causing a bias in his experiments towards both smaller numbers of ingredients and cheaper ingredients).
- 80% of brews are attempts to make a specific potion. He chooses a random potion to try to produce. Then he picks a random recipe that he has successfully made that potion with in the past, and repeats that recipe. (He does not have a good enough recollection to try a recipe that's had high odds of success at brewing that potion in the past, he'll try any recipe that has ever brewed that potion).
The main relevance of this algorithm to players was that some combinations of ingredients, especially ones that could produce potions and especially ones that could produce multiple different potions, had been brewed many more times than would happen under pure randomness.
LEADERBOARD
This scenario was intended to be relatively straightforward, and bearing this out we had several perfect scores:
- simon fully deciphered the algorithm, got a 100% success rate and also solved the Bonus Objective.
- abstractapplic and Lorxus also both got a 100% success rate at brewing 'Barkskin Potion', but did not independently solve the Bonus Objective until they saw simon's solution to it.
- Unnamed, Yonge, and qwertyasdef all got a 50% success rate:
- Yonge and Unnamed looked at past brewing and tried Crushed Onyx, Demon Claw, Ground Bone and Vampire Fang (50% to succeed, but low on magic).
- qwertyasdef instead used Crushed Onyx, Demon Claw, Ground Bone, Quicksilver, and Troll Blood based on an analysis of which available ingredients often appeared in 'Barkskin Potion', but had the same result of 50% success due to low magic.
Congratulations to all players! Particular congratulations to simon for figuring out the Bonus Objective! Maniacal cackling to all other players!
REFLECTION & FEEDBACK REQUEST
The goal I was shooting for with this scenario was to reward people for actually paying human attention to patterns, rather than just hoping for [YOUR FAVORITE ALGORITHM HERE] to solve all your problems for you.
The Bonus Objective was obviously written to require out-of-context thought. I also tried to set up the scenario to be amenable to thinking about what patterns might mean even with relatively little math: bearing this out, it looks like Lorxus managed to get a perfect score with relatively little in the way of complicated methods/tools just by e.g. thinking about what it might mean that including lots of ingredients led to Magical Explosions and including few ingredients led to Inert Glop (and documented his thought process very well, thank you Lorxus!).
How did this feel from the player end?
As usual, I'm also interested to hear more general feedback on what people thought of this scenario. If you played it, what did you like and what did you not like? If you might have played it but decided not to, what drove you away? What would you like to see more of/less of in future? Do you think the scenario was more complicated than you would have liked? Or too simple to have anything interesting/realistic to uncover? Or both at once? Do you have any other feedback?
11 comments
Comments sorted by top scores.
comment by abstractapplic · 2024-06-18T18:55:27.328Z · LW(p) · GW(p)
I think this is plausibly the best scenario either of us have made to date.
The basic game was very good, layering simple rules on top of each other to create a complex system which was challenging to detangle but easy to understand once you know the rules. I was particularly impressed by the fact that you managed the (imo, near-impossible) feat of making an enjoyable D&D.Sci where handling data starvation was a key part of the problem: most players (including me) seem to have had the (sensible) thought "okay, let's filter for only potions with Onyx and Bone", and success past that point was predicated on realizing there weren't quite enough rows to justify being that picky.
The twist struck me as fair, funny and fun. It provided an object lesson in noticing when things don't quite add up, and letting the qualitative shade the quantitative; it also expanded the scope of the genre in ways I realize I've been neglecting to.
All that said, I have some (minor, petty) criticisms . . . not of the game itself, but how it was presented. Namely:
.This entry was billed as "relatively simple", but I think it was about median difficulty by the standards of D&D.Sci; pretty sure it was harder than (for example) The Sorceror's Personal Shopper.
."STORY (skippable)" was kind of misleading this time: the flavortext had a lot of little hints that the Archmage wasn't on the level, so someone who didn't read it (or failed to read between the lines, like me) would be at a (small) disadvantage.
."Archmage Anachronos is trying to brew Barkskin Potion" was A) the GM saying something false directly to the players, and B) a missed opportunity: if you'd written something like "Your goal is to help Archmage Anachronos brew Barkskin Potion", that would have been a subtle confirmation that giving him exactly what he asked for would lead to the best outcome (vs more aggressive / galaxy-brained forms of sabotage, or refusing to cast judgement on his pursuit of immortality, or any other reaction).
Replies from: aphyer↑ comment by aphyer · 2024-06-18T19:39:26.814Z · LW(p) · GW(p)
I'm glad you liked it, thank you!
This entry was billed as "relatively simple", but I think it was about median difficulty by the standards of D&D.Sci; pretty sure it was harder than (for example) The Sorceror's Personal Shopper.
I guess that's fair. There's a complication here in that...uh...almost all of my scenarios have been above-median complexity and almost all of yours have been below-median. (I should probably write down my thoughts on this at some point). I agree that this one wasn't simpler than most of yours, but I think that it was still a much more approachable entry point than e.g. Duels & D.Sci, or League of Defenders.
(It's possible we should try to standardize a 1-10 complexity scale or some such so that we can stick a difficulty rating on the top of each scenario.)
"STORY (skippable)" was kind of misleading this time
Fair enough, I can tweak that for anyone who finds the scenario in future.
I intended that the story should not provide much help...the intent was not for players to notice that Anachronos was suspicious in-story, the intent was for them to notice from the data, and for the hints in the story to be just some quiet confirmation for a player who realized the twist from the data and then went back to reread the story.
On the other hand, I was expecting more players to get the twist, and thought that I'd only really catch players who ignored the ingredient names entirely and just fed the data into an ML algorithm, so I'm clearly not very well calibrated on this. I was really quite surprised by how many players analyzed the data well enough to say "Barkskin potion requires Crushed Onyx and Ground Bone, Necromantic Power Potion requires Beech Bark and Oaken Twigs" and then went on to say "this sounds reasonable, I have no further questions." (Maybe the onyx-necromancy connection is more D&D lore than most players knew? But I thought that the bone-necromancy and bark-barkskin connections would be obvious even without that).
"Archmage Anachronos is trying to brew Barkskin Potion" was A) the GM saying something false directly to the players
I...think I'm in general allowed to say false things directly to the players as a D&D GM? If the Big Bad is disguised as your innkeeper while the real innkeeper is tied up in the cellar, I think I can say 'The innkeeper tells you it'll be six silver for a room', I don't think I need to say 'The man who introduced himself to you as the innkeeper.'
(Also, you are a Data Scientist. Sense Motive is not a class skill for you. Clearly you failed a Sense Motive check and so believed him!)
...I'll think about whether I want to tweak that line for potential future players.
Replies from: simon, Lorxus↑ comment by simon · 2024-06-19T16:22:39.158Z · LW(p) · GW(p)
I thought the flavour text was just right - I got it from the data, not the flavour text, and saw the flavour text as confirmation, as you intended.
I was really quite surprised by how many players analyzed the data well enough to say "Barkskin potion requires Crushed Onyx and Ground Bone, Necromantic Power Potion requires Beech Bark and Oaken Twigs" and then went on to say "this sounds reasonable, I have no further questions." (Maybe the onyx-necromancy connection is more D&D lore than most players knew? But I thought that the bone-necromancy and bark-barkskin connections would be obvious even without that).
Illusion of transparency I think, hints are harder than anyone making them thinks.
When I looked at the ingredients for a "barkskin potion", as far as I knew at this point the ingredients were arbitrary, so in fact I don't recall finding it suspicious at all. Then later I remember looking at the ingredients for a "necromantic power potion" and thinking something like... "uh... maybe wood stuff is used for wands or something to do necromancy?". It was only when I explicitly made a list of the ingredients for each potion type, rather than looking at each potion individually, and could see that everything else make sense, that I realized the twist.
↑ comment by Lorxus · 2024-06-19T03:45:13.721Z · LW(p) · GW(p)
If the Big Bad is disguised as your innkeeper while the real innkeeper is tied up in the cellar, I think I can say 'The innkeeper tells you it'll be six silver for a room', I don't think I need to say 'The man who introduced himself to you as the innkeeper.'
Perhaps, but you could also simply say "Yeah, the guy at the counter tells you the room will be 6 silver."
comment by Lorxus · 2024-06-18T03:43:58.785Z · LW(p) · GW(p)
...bearing this out, it looks like Lorxus managed to get a perfect score with relatively little actual Data Science just by thinking about what it might mean that including lots of ingredients led to Magical Explosions and including few ingredients led to Inert Glop.
Not quite true! That's where I started to break through, but after that I noticed the Mutagenic Ooze issue as well. It also took me a lot of very careful graceful use of pivot tables. Gods beyond, that table chugged. (And if I can pull the same Truth from the void with less powerful tools, should that not mark me as more powerous in the Art? :P)
I guess I'm not clear on what "actual Data Science" would involve, if not making hypotheses and then conducting observational-experiments? I figured out the MO mechanic specifically by looking at brews that coded for pairs of potions, for the major example. The only thing that would have changed if I'd known SQL would be speed, I suspect.
...and documented his thought process very well, thank you Lorxus!
Always a pleasure! I had a lot of fun with this one. I was a little annoyed by the undeclared bonus objective - I would have wanted any indication at all in the problem statement that anything was not as it appeared. I did notice the correspondence in (i.a.) the Farsight Potion but in the absence of any reason to suspect that the names were anything but fluff, I abstracted away anything past the ingredients being a set of names. Maybe be minimally more obvious? At any rate I'd be happy to be just as detailed in future, if that's something you want.
Replies from: simon, Lorxus↑ comment by simon · 2024-06-19T16:53:01.514Z · LW(p) · GW(p)
I liked the bonus objective myself, but maybe I'm biased about that...
As a someone who is also not a "data scientist" (but just plays one on lesswrong), I also don't know what exactly actual "data science" is, but I guess it's likely intended to mean using more advanced techniques?
(And if I can pull the same Truth from the void with less powerful tools, should that not mark me as more powerous in the Art? :P)
Perhaps, but don't make a virtue of not using the more powerful tools, the objective is to find the truth, not to find it with handicaps...
Speaking of which one thing that could help making things easier is aggregating data, eliminating information you think is irrelevant. For example, in this case, I assumed early on (without actually checking) that timing would likely be irrelevant, so aggregated data for ingredient combinations. As in, each tried ingredient combination gets only one row, with the numbers of different outcomes listed. You can do this by assigning a unique identifier to each ingredient combination (in this case you can just concatenate over the ingredient list), then counting the results for the different unique identifiers. Countifs has poor performance for large data sets, but you can sort using the identifiers then make a column that adds up the number of rows (or, the number of rows with a particular outcome) since the last change in the identifier, and then filter the rows for the last row before the change in the identifier (be wary of off-by-one errors). Then copy the result (values only) to a new sheet.
This also reduces the number of rows, though not enormously in this case.
Of course, in this case, it turns out that timing was relevant, not for outcomes but only for the ingredient selection (so I would have had to reconsider this assumption to figure out the ingredient selection).
Replies from: Lorxus↑ comment by Lorxus · 2024-06-19T20:13:40.555Z · LW(p) · GW(p)
Perhaps, but don't make a virtue of not using the more powerful tools, the objective is to find the truth, not to find it with handicaps...
I'm obviously seeking out more powerful tools, too - I just haven't got them yet. I don't think it's intrinsically good to stick to less powerful tools, but I do think that it's intrinsically good to be able to fall back to those tools if you can still win.
And when I need to go out and find truth for real, I don't deny myself tools, and I rarely go it alone. But this is not that.
Replies from: simoncomment by yernab (isaac-wooden) · 2024-09-29T21:45:01.049Z · LW(p) · GW(p)
First time poster/D&D Sci player - thanks so much for making this! I'm a complete novice at data science/stats, mostly just the usual sort of intuitions you get from the occasional Excel munging at $corporate_job. I just discovered D&D Sci a day or two ago, and picked this one for my first puzzle from the index because it was marked as high-quality and middle-of-the-road difficulty.
While I may not have much DSci experience, I do have a lot of experience solving programming puzzles in a programming language called APL - think of it like NumPy (so array/tensor oriented) with a longer/weirder heritage that you primarily interact with by typing (vaguely runic?) symbols. It's REPL-based, so it's quick to try different ideas, and it has a long-standing reputation as being good for "data exploration". I'll also say that if you want to match the aesthetics of casting data-analysis spells for an archmage, typing inscrutable wizard-ish symbols into a REPL feels on-point. :)
As an example, assume we've already defined a vector of strings r that contains the unique result names, and a vector of boolean matrices rpt ("result partitioned table", naming things is hard) where each matrix consists of the rows in the input filtered to the corresponding result. The following line
- AND-reduces each matrix in
rptto a boolean vector where1means an ingredient was used in every row (call these "required ingredient pairs") and catenates these into a single matrix. - Constructs a larger boolean matrix where each row is a possible combination of required ingredient pairs.
- For each matrix in
rpt, analyzes each row and determines whether it contains a required ingredient pair combination. Calculates the percentage of rows that contain a required ingredient pair combination, and renders this info as a 2-column matrix where the left column consists of the unique result names.
r,⍪100×rpt{(+/÷≢)∨/4=+/⍺∧⍤1⍤1 2⊢⍵}¨⊂x⌿⍨4=+/x←,[1 2]∨⍤1⍤1 2⍨↑∧⌿¨rptAs might be expected, this gives 100% for "Mutagenic Ooze" and 0% for "Acidic Slurry".
APL is also quite fast (especially on large boolean matrices!), and the above line returns in 47ms on my machine. Nice to be able to try a bunch of random analyses quickly.
I recorded my live solution of this and posted it on my APL-focused Youtube channel here in case you want to see a DSci novice flail around for ~2 hours in this weird language. If there's some rule against links or self-promotion or something (especially for a fresh account), let me know and I'll remove the link. Actually, I'll take feedback on any part of this post. :)
Anyway, on to the data analysis journey:
A simple first thing to do seemed to be partitioning the rows by result, allowing me to do a bunch of result-specific analyses. Looking at the average ingredient count for each result, I noticed that "Mutagenic Ooze" and "Magical Explosion" had the highest avg ingredient counts, and "Inert Glop" and "Acidic Slurry" had the lowest. Hrmm...
Also, partitioning the "Success?" column and looking at unique results for each partition reveals that, indeed, the wizard has 4 unique failure modes, and a bunch of potions that constitute a successful attempt.
Next, I looked at at the percentages for occurrences of each ingredient for each result, sorted from most-common to least-common. This showed that each potion had 2 ingredients that occurred 100% of the time, so from this point forward we have the concept of "required ingredients".
Also, no failure had an ingredient that occurs 100% of the time, meaning that adding an ingredient in and of itself doesn't automatically trigger any failure.
Scrolling through the data, I noticed that the most common ingredients for "Magical Explosion" were all "fantasy ingredients", while the most common ingredients for "Inert Glop" were all "real ingredients". I got a bit lucky here (or maybe the data was constructed this way?) that the sorted occurence percentages exactly partition the ingredient list into fantasy/real - if there had been any overlap, I might not have noticed this. Going in blind, I was kind of thinking whether there was some subset of ingredients that would always (or at least strongly correlate) with some failure, but... uh...
+/!∘24¨⍳24
With 167,77,215 possible subsets this would have taken a while to analyze. So, lucky that I noticed this!
With the hint of fantasy items, I was able to look at the rows for each result and see the percentage occurence of each result for each count of included fantasy items. This laid out the rules for avoiding "Magical Explosion" and "Inert Glop", so from this point I knew I wanted 3 fantasy ingredients in addition to the 2 required ingredients for Barkskin Potion.
Still don't know what causes "Mutagenic Ooze" or "Acidic Slurry", so next I looked at percentages of occurrences of pairs of ingredients for each result (being reluctantly ready to look at 3-sets, 4-sets etc.). Looking at the big matrix of data, the number "0" stood out to me, and focusing only on those we see there are a bunch of pairs that never occur for "Acidic Slurry". Noting that the required ingredients for Barkskin Potion were in that list, I was able to match it to the list of required ingredients overall. So, now we know that to avoid "Acidic Slurry", you just need to include some required ingredient pair.
Seeing as how "Magical Explosion" and "Inert Glop" are kind of opposites, I wondered if "Mutagenic Ooze" was the opposite of "Acidic Slurry" in some way. I messed up the analysis here - idea was to check occurrences of all possible pairs of required potion ingredients (basically my sample line above), but I did it wrong in the heat of the moment. Still, I was able to see that each pair of required potion ingredients occurred at least once for "Mutagenic Ooze", and this didn't happen for any other result except for "Magical Explosion", so I figured it was close enough to being an opposite of "Acidic Slurry" and I was just missing something.
So, time to formulate a guess! Take the 2 required ingredients for Barkskin Potion, add 3 fantasy items that the mage has available, and avoid including required ingredients for any other potion. Checked the solution page and... success!
Thanks again for putting this together, it was fun! I may do some other puzzles in the series, it's probably good to build this type of skillset in any case. :)
Replies from: aphyer