LessWrong 2.0 Reader

I endorse this movie unironically. It is a classic film for tracking what information you have and don't have, how many possibilities there are, etc.

Also the filmmaker maintains to this day that they left the truth of the matter in the final scene undefined on purpose, so we are spared the logic being hideously hacked-off to suit the narrative and have to live with the uncertainty instead.

P.S. Here [LW · GW] is the link to the question that I posted.

Twisted: The Untold Story of a Royal Vizier isn't really rational but is rat-adjacent and funny about it. Available to watch on youtube though the video quality isn't fantastic.

memento — shows a person struggling to figure out the ground truth; figuring out to whom he can defer (including different versions of himself); figuring out what his real goals are; etc.

Please can you move the epistemic status and warning to the top? I was excited when I first skimmed this detailed comment, but then I was disappointed :/

Something I'm worried about now is some RFK Jr/Dr. Oz equivalent being picked to lead on AI...

Is that error common?  I can only recall encountering one instance of it with surety, and I only know about that particular example because it was signal-boosted by people who were mocking it.

I have an answer to that: making sure that NIST:AISI had at least scores of automated evals for checkpoints of any new large training runs, as well as pre-deployment eval access.

Seems like a pretty low-cost, high-value ask to me. Even if that info leaked from AISI, it wouldn't give away corporate algorithmic secrets.

A higher cost ask, but still fairly reasonable, is pre-deployment evals which require fine-tuning. You can't have a good sense of a what the model would be capable of in the hands of bad actors if you don't test fine-tuning it on hazardous info.

Epistemic status: half joking, but also half serious.
Warning: I totally wrote this.

Practical Rationality in John Carpenter’s The Thing: A Case Study

John Carpenter's The Thing (1982) is a masterclass in practical rationality, a cornerstone of effective decision-making under uncertainty—a concept deeply valued by the LessWrong community. The film’s narrative hinges on a group of Antarctic researchers encountering a shape-shifting alien capable of perfectly imitating its hosts, forcing them to confront dire stakes with limited information. Their survival depends on their ability to reason under pressure, assess probabilities, and mitigate catastrophic risks, making the movie a compelling example of applied rationality.

Key Lessons in Practical Rationality:

1. Updating Beliefs with New Evidence The researchers continually revise their understanding of the alien's capabilities as they gather evidence. For instance, after witnessing the creature’s ability to assimilate and mimic hosts, they abandon naive assumptions of safety and recalibrate their strategies to account for this new information. This aligns with Bayesian reasoning: beliefs must be updated in light of new data to avoid catastrophic errors.
2. Decision-Making Under Uncertainty The characters face extreme uncertainty: anyone could be the alien, and any wrong move could result in annihilation. The iconic blood test scene exemplifies this. The test, devised by MacReady, is an ingenious use of falsifiability—leveraging empirical experimentation to distinguish humans from the alien. It demonstrates how rational agents use creativity and empirical tests to reduce uncertainty.
3. Coordination in Adversarial Environments Cooperation becomes both vital and precarious when trust erodes. The film explores how rational agents can attempt to align incentives despite an adversarial context. MacReady takes control of the group by establishing credible threats to enforce compliance (e.g., wielding a flamethrower) while demonstrating his willingness to follow the same rules he imposes.
4. Mitigating Existential Risk The characters recognize that the alien represents an existential risk—not just to them, but to humanity. Their decisions prioritize long-term outcomes over immediate survival. For example, the decision to destroy the base to prevent the alien’s escape reflects a commitment to the global utility function, even at the cost of personal survival.
5. The Role of Psychological Factors in Rationality The film does not shy away from the psychological toll of high-stakes reasoning under uncertainty. Fear, paranoia, and isolation challenge the researchers’ ability to think clearly. This resonates with real-world rationality, where emotional regulation is essential to avoid biases and maintain clarity in decision-making.

As you know, I have huge respect for USG natsec folks.  But there are (at least!) two flavors of them: 1) the cautious, measure-twice-cut-once sort that have carefully managed deterrencefor decades, and 2) the "fuck you, I'm doing Iran-Contra" folks.  Which do you expect will get in control of such a program ?  It's not immediately clear to me which ones would.