An Introduction to Decision Modeling

You’ll notice in the screenshot that there’s an image of something that looks like a lopsided bell curve on the bottom right. That’s because the software I’m using (Guesstimate) calculates a Monte Carlo simulation for this estimate right there in the model. Monte Carlo simulation is a statistical technique that randomly generates thousands of scenarios from the information you feed the model. Originally developed by nuclear physicists, it’s now used to aid decision-making in everything from politics to sports and beyond. For our purposes, you can think of a Monte Carlo simulation as a sampling of the possible future lives that might unfold for you and your organization as a result of your decision. The number in large font (16K) is the average of the values across all of the simulations.

Just wanted to point out that, even if you know your confidence interval, the decisions can often still depend intensely on the shape of the distribution you assume for outcomes. The Monte-Carlo simulation does not figure out that the distribution of values of all possible outcomes looks like a lopsided bell-curve (or log-normal); it just fits a distribution of a given shape provided by you to the confidence intervals provided by you. If you're not confident in the shape you predict for the distribution, you need to be very careful here.

From tinkering around, it looks like Guestimate let's you chose between log-normal distributions, normal distributions and uniform distributions. As a result, if you suspect a good chance of a heavy-tailed distribution (maybe Vax to the Max succeeds in its policy-affecting ambitions and causes flow-through effects prompting many other charities to target extremely impactful policies...), you should take the expected value estimates of these distributions with a grain of salt.

Here's an illustration of this with the example in the post:

With log-normal, Vax to the Max acquires 460 vaccines (vs Maxine's Vaccines 560) but, normal or uniform distributions implies that Vax to the Max gives 900 or 910 vaccines respectively (based on my experiments with the linked example). These results would indicate:

1. Vax to the Max has an expected outcome roughly 160% (900/560 -> 60% gain) that of Maxine's Vaccines assuming a normal or uniform distribution

2. Vax to the Max has an expected outcome roughly 20% (460/560 -> 20% loss) worse than that of Maxine's Vaccines assuming a log-normal distribution

3. Per 1 and 2, your confidence-interval driven decision is not robust to assumptions about the shape of your distribution and, if you're agnostic between the three options Guestimate provides, Vax to the Max actually beats Maxine's Vaccines.