D&D.Sci

That it nuanced my understanding of the supremacy of neural networks and when "just throw a neural net" at it might work or might not.

There are 3 pairs of duplicate stats in the dataset. (1011, 4696) - different results. (3460, 5146) - different results. (4399, 5963) same result.  This reassures me that there was some random element and not some Zendo-like rulest, unless we're in the real complicated case where peoples' outcome was a deterministic function of their surrounding graduating class.

Found it a hugely entertaining concept though, and it was truly time well spent. Before checking the thread's replies, I ended up going for Str+2, Dex+1, Con+2, Cha +5 for a 75% chance.

I created a dictionary in Python for the total successes. For each stat (cha, str, etc), I found the number of successes and total attempts at each score(1-20). Dividing the successes by total attempts gave me a rough success rate for each stat score.

Then, I set my character up as a dictionary, and iterated over it, increasing each value by one and seeing what the change was for the success rate from that increase. After a single iteration, I increased my character's stat where the greatest positive change was found.

Iterating over that 10 times, or until all my points were gone, gave me this spread:

STR: Eight (increased by two)

CON: Sixteen (increased by two)

DEX: Fourteen (increased by one. Weird, I know, but there was an increase to be had from 13 - 14, even though afterward it's negative returns)

INT: Thirteen (no change)

WIS: Twelve (no change)

CHA: Nine (increased by 5)

Weak points:

• This assumes that the stats have no correlations between each other. Alexei checked, thank you, Alexei, I'm not sure how to properly do that.
• This misses out on bigger steps having large effects. Say for DEX, moving from 13 to 14 gave you a -5% chance of success. But moving from 14 to 15 gave you a +45% chance. Because my algorithm just checks single steps, it misses the chance to move two steps and gain 20% per step.
• This is all based on a very simplistic model and uses very little actual statistical analysis.

Stats don't appear too correlated, although all cross correlations are negative around -7% to -10% which is interesting. I guess it might have to do with data construction. Simple logistic regression gives coeffs: CHA=0.143 CON=0.141 DEX=-0.016 INT=0.099  STR=0.116 WIS= 0.156. Based on these one would push WIS+8 to 20 and allocate the remaining two points CHA+2. 

However values are close, and there are standard errors around the estimates. Bootstrapping strategies to account for that allocates: WIS+7, CHA+2, CON+1.

Accounting for cross effects flips them around, with allocation: CHA+6, WIS+3, CON+1. Going full second order allocates CHA+6, WIS+3, STR+1, but obviously with higher complexity.

My answer would overweigh the linear when blending with the ones with more parameters. Final answer WIS+6, CHA+3, CON+1.

~2 hours' of analysis here: https://github.com/sclamons/LW_Quest_Analysis, notebook directly viewable at https://nbviewer.jupyter.org/github/sclamons/LW_Quest_Analysis/blob/main/lw_dnd.ipynb.

Quick takeaways:
1) From simple visualizations, it doesn't look like there are correlations between stats, either in the aggregated population or in either the hero or failed-hero populations. 
2) I decided to base my stat increases on what would add the most probability of success for improving that stat, looking at each stat in isolation, where success probabilities were estimated by simply tabulating the fraction of students with that particular stat value ended up heroes.
3) Based on that measure, I decided to go with +4 Cha, +1 Str, +2 Wis, +3 Con, and I wish I could reduce my Dex.

Made graphs of stat vs prob of success. Pretty clean linear relationships between each stat and increase in success, except for dexterity. That seems to hurt.  
Checked for correlations between stats; none detected.  
Given that we can't go down in stats, I also looked at the data for students whose stats are at least as high as ours. Did linear regression on that; seems like Dex helps in this case, but there are a lot fewer samples, so I'm going to chuck it up to noise.

Going off all the data, Wis and Cha have the highest slope. (Cha is slightly higher.) So I'd invest evenly in both. Going off the conditioned data, Cha has the highest slope. So I'll shift to +7 Cha and +3 Wis.

One thing I also thought about is coming up with various hypotheses, Zendo style. E.g. "you win if your Str > Dex" or "you win if the sum of your lowest three stats is > 10" But I don't think that's the nature of the problem.

Graduate stats likely come from 2d10 drop anyone under 60 total. No obvious big jumps at particular thresholds, so assume each extra point helps about the same given the stat type.

For completing my Great Quest: +8 WIS, +2 CHA, based on assuming each stat point provides the equivalent of x bits of evidence you'll complete it, depending on the stat, estimated by looking at prior history in your range of stats of the change in prob given that total stats didn't change.

For breaking the system: +10 CON. Best chance of surviving while not on a Great Quest, breaks the theoretical limit by the most, not awful for Great Quest.

Life after Questing: +6 CHA, +4 STR. Really quite good for your Great Quest even if not the best, and you no longer have silly weaknesses like talking and jars, so e.g. if there's another fairy later you don't run such a big risk of losing out on a free +10 stats by sounding simultaneously entitled and disinterested.

Some basics: Each stats has range 2-20 (and maybe comes from 2d10 somehow?). Sum of stats is in range 60-100. You have 62 and are going to 72. More stats generally gets better results. Baseline 62 gives 40% to quest; baseline 72 gives 69% to quest. Average graduate stat sum is 70.4. Total graduates 7387. Maybe stats come from a roll of 2d10 and you only graduate if your stats are at least 60 in total? P(12d10>=60)=74% so probably generated 10K folks and filtered to >=60, yeah. Stats are probably anti-correlated in our sample?

Let's try simple logistic regression. Normalize, fit, predict. You're 38% to succeed, that checks out. Try some simple changes? +10 to any stat, even though that brings you above 20. WIS gets to 73%, CHA/CON to 70%, INT/STR to 65%/61%, and DEX down to 34%. Huh! groupby('dex').mean() ==> yeah, much higher chances with low dex, dunno if that's because dex is useless and stats anti-correlated, or dex is harmful. Anyway this model's got CHA/CON/DEX/INT/STR/WIS coeffs at [2.5, 2.5, -0.3, 1.7, 2.0, 2.7]. As I see it so far, there are three main considerations: pump WIS to 20 and CHA to 6 to maximize chance of quest, pump CON to 24/20 to maximize survivability past that of any adventure who has ever lived (can we pass 20?? :D), or mostly ignore quest considerations because we have other goals, which probably means maxing some stat or shoring up CHA/STR.

Let's check a random forest to see if there are major discontinuities. Oh, it's way different! Here +10 to CHA does very very well, almost 90% quest success. groupby('cha').mean() ==> I see a jump of almost +10pp from 5->6 and 13->4 CHA. Maybe we invest 10 in CHA? Or maybe 2 in CHA and then... nah, not really better. But this is misleading too, because folks with CHA=14 just happened to have better stats on average. Better than CHA=13 for everything but DEX which has negative predictive success.

Okay fine, let's try to do something like the right thing. I'd like to know the change in success rate when adding one point to one stat, with the sum of the other stats remaining constant. And I might only care about this in the lowish range of stat sums, 60 to 75, say. We'll just grab the average for a sec. The average what. ...evidence of success provided by seeing +1 in a certain stat given that all other stats are equal? Sure, maybe that's the model used to generate quest prob. Laplace to estimate prob of success with total stats = x, wlog cha = y. Got CHA/CON/DEX/INT/STR/WIS [1.4, 1.1, -0.1, 0.6, 1.4, 1.8] for the whole 60-100 range, or [0.4, 0.3, -1.0, 0.3, 0.4, 0.8] for just 60-75.

Should also check that there's no obvious reason the model assumption of e.g. 4->5 is in some ways the same as 18->19, but meh, we're done here.

Threw all the data in a small neural network, and let it optimize (pretty mediocre: only resulted in an accuracy of 70%). I used this network to test quite a few different combinations of possible stats (base stats + spending all 10 points), resulting in [7, 16, 13, 13, 12, 11] as best and [6, 14, 18, 13, 16, 5] as worst chance of succeeding. A lot of things could still be optimized in this approach, but it seems like dexterity and wisdom should be left alone, and charisma and constitution could use a boost.

 for each point i wish to assign i loop over the whole list. 
for each row i check whether it's someone who succeeded or failed.
then i compare each stat to my own. in the case of someone who succeeded, if their stat is higher i add 1 to a counter for that stat, if their stat is lower i reduce 1. if they failed i do the opposite (add 1 if their stat is lower, reduce 1 if their stat is higher).
then i add one point to the stat which got the highest score, and loop again until i spent the last point.

cha: 9, str: 11, con: 14, dex: 13, int: 13, wis: 13

This is algorithm would usually just increase your lowest stat until it's close to your other stats, which i'm not sure it's such a great strategy. 

I took a fairly black-box approach to this problem. Basically, we want a function f(str, dex, con, int, wis, cha) which outputs a chance of success, and then we want to optimize our selection so that we have the highest chance. The optimization part is easy because it's discrete; once we have a function, we can simply evaluate it at all of the possible inputs and select the best one.

I used a number of different ML models to estimate f, and I got pretty consistent brier scores on reserved test data of ~0.2, which isn't great, but isn't awful. I used scikit-learn, and used a MLPClassifier, LogisticRegression, GaussianNB, and RandomForestClassifier, along with CalibratedClassifierCV so that they had calibrated probability scores. Most of them I left on their defaults, but I played around with the layers in the MLPClassifier until it had a pretty good brier score.

Despite the fact that these models all had similar brier scores, they had surprisingly different recommendations. The Neural Net wanted to give small bumps to strength, wisdom, and charisma. Logistic Regression wanted to go all-in on wisdom, and putting any remaining points into charisma. Gaussian Naive Bayes wanted to put most of the points into charisma, but oddly, not all; it wanted to also sprinkle a few points into wisdom. The Random Forest Classifier wanted to bring strength and charisma up a little, but mostly sink points into wisdom, and occasionally scatter points into constitution or intelligence.

The top recommendation for each method is as follows:

Neural Net: 8, 14, 13, 13, 15, 9

Logistic Regression: 6, 14, 13, 13, 20, 6

Naive Bayes: 6, 14, 13, 13, 14, 12

Random Forest: 8, 14, 13, 13, 15, 9

 Its favorite result was: +2 Cha, +8 Wis. It would have like +10 Wis if it were possible.

For at least the top few results, it wanted to (a) apportion as much to Wis as possible, then (b) as much to Cha, then (c) as much to Con. So we have for (Wis, Cha, Con): 1. (8, 2, 0) 2. (8, 1, 1) 3. (8, 0, 2) 4. (7, 3, 0) 5. (7, 2, 1) ...

STR +3, WIS +3, INT +0, CHA +2-4, evenly distribute among the rest. Pretty unsophisticated, and misses out from larger gains from adding many points to a stat in one go.

My first thought is to look for the lowest stat in each category which succeeded. I will probably want at least this. Unfortunately this is 2 in every case, so this doesn't help.

My second thought is to look for a patch in stat space where there are a disproportionably large number of successes, however of the stats I can access none has a meaningful number of adventurers particularly close to them.

My third idea is, for every possible set of stats we could choose look at the adventurers whose stats were strictly worse than or equal to those, and see which ones enclosed the highest proportion of successes. There are several with a 100 percent success rate, but none with more than 2 data points, which isn't much. There are however 2 with 6 datapoints and an 83 percent success rate, which seems better established:
str: 8 con: 14 dex: 13 int: 20 wis: 12 cha: 5
str: 8 con: 14 dex: 13 int: 19 wis: 13 cha: 5
Both seem roughly evenly balanced, and either seems to be a reasonable choice. I would go with the first purely on the intuition that if you are going to have one really strong stat, better to go all the way.

The approach I settled on was to estimate the success chance of a possible stat line by taking a weighted success rate over the data, weighted by how similar the hero's stats are to the stats being evaluated. My rationale is that based on intuitions about the domain I would not assume linearity or independence of stats' effects or such, but I would assume that heroes with similar stats would have similar success chances.

In pseudocode: 

estimatedchance(stats) = sum(weightfactor(hero.stats, stats) * hero.succeeded) / sum(weightfactor(hero, stats))

weightfactor(hero.stats, stats) = k ^ distance(hero.stats, stats)

(Assuming 0 < k < 1, and hero.succeeded is 1 if the hero succeeded and 0 otherwise)

I tried using both Euclidean and Manhattan distances, and various values for k as well. I also tried a hacky variant of Manhattan distance that added abs(sum(statsA) - sum(statsB)) to the result, but it didn't seem to change much.

Lastly, I tried the replacing (hero.succeeded) with (hero.succeeded - linearprediction(sum(hero.stats))) to try to isolate builds that do well relative to their stat total. linearprediction is a simple model I threw together by eyeballing the data: 40% chance to succeed with total stats of 60, 100% chance with total stats >= 95, linear in between. Could probably be improved with not too much effort, but I have to stop somewhere.

I generally found two clusters of optima, one around (8, 14, 13, 13, 8, 16)—that is, +4 CHA, +2 STR, +4 WIS—and the other around (4, 16, 13, 14, 9, 16)—that is, +2 CON, +1 INT, +3 STR, +4 WIS. The latter was generally favored by low k values, as the heroes with stats closest to that value generally did quite well but those a little farther away got less impressive. So it could be a successful strategy that doesn't allow too much deviation, or just a fluke. Using the linear prediction didn't seem to change things much.

If I had to pick one final answer, it's probably (8, 14, 13, 13, 8, 16) (though there seems to be a fairly wide region of variants that tend to do pretty well—the rule seems to be 'some CHA, some WIS, and maybe a little STR'), but I find myself drawn towards the maybe-illusory (4, 16, 13, 14, 9, 16) niche solution.

ETA: Looks like I was iterating over an incomplete list of possible builds... but it turned out not to matter much.

ETA again (couldn't leave this alone): I tried computing log-likelihood scores for my predictors (restricting the 'training' set to the first half of the data and using only the second half for validation. I do find that with the right parameters some of my predictors do better than simple linear regression on sum of stats, and also better the apparently-better predictor of simple linear regression on sum of non-dex stats. But they don't beat it by much. And it seems the better parameter values are the higher k values, meaning the (8, 14, 13, 13, 8, 16) cluster is probably the one to bet on.

str +2 points to 8, con +1 point to 15, cha +4 points to 8, wis +3 points to 15, based on assuming that a) different stats have multiplicative effect (no other stat interactions) and b) that the effect of any stat is accurately represented by looking at the overall data in terms of just that stat and that c) the true distribution is exactly the data distribution with no random variation. I have not done anything to verify that these assumptions make sense.

dex looks like it actually has a harmful effect. I don't know whether the apparent effect is or is not too large to be explained by it helping bad candidates meet the college's apparent 60-point cutoff.

 Was playing around with neural nets the last couple days, and when I came across this problem it immediately looked very nail-shaped to me. Probably isn't the most efficient tool for the job, but here's my approach: https://gist.github.com/Deccludor/d42a91712b427a45ff61aacfc02d0abe.

I trained a neural net to predict success based on ability scores, then ran a few different search algorithms to find the best possible use of 10 points. The ~60m permutations were slightly too many for me to search exhaustively, so I tried predicting on a random sample, using a greedy algorithm to add one point at a time, and another one that added 7 points at a time (maxing out the number of permutations I could fit into memory at a time). 

The best-scoring distribution of stats I could find was CHA: 5, CON: 20, DEX: 13, INT: 13, STR: 6, WIS: 15. According to the calibration curve, that should have a roughly ~75% chance of success.