LessWrong 2.0 Reader

If this is an example of an LLM proving something, it's a very non-central example. It was finetuned specifically for mathematics and then used essentially as a program synthesis engine in a larger system that proved the result.

DeepMind can't just keep running this system and get more theorems out - once the engineers moved on to other projects I haven't heard anything building on the results.  

MuZero doesn’t seem categorically different from AlphaZero. It has to do a little bit more work at the beginning, but if you don’t get any reward for breaking the rules: you will learn not to break the rules. If MuZero is continuously learning then so is AlphaZero. Also, the games used were still computationally simple, OOMs more simple than an open-world game, let alone a true World-Model. AFAIK MuZero doesn’t work on open-ended, open-world games. And AlphaStar never got to superhuman performance at human speed either.

LLMs compute probability of a sequence, but truth/good distinction is captured by two-dimensional Jeffrey-Bolker measure [LW · GW] (I'm calling its components "probability" and "shouldness", their ratio is the expected utility of an event). Shouldness is reconstructed from probability and expected utility as their product, so plausibly it behaves on long sequences similarly to probability, it generally gets lower for longer sequences, but tends to be higher for simpler sequences.

The analogy between probability and shouldness suggests that some form of pretraining might be able to create models for either of them (as opposed to a base model that learns something inbetween from raw data with no supervision from preference data). Then expected utility is the ratio, that is instead of looking at logits of one LLM, we look at differences of logits for two LLMs, a shouldness-LLM and a probability-LLM (with some regularization that anchors to a base model instead of goodharting towards high approximate expected utility low probability sequences). Possibly this needs interspersing preference training with pretraining, rather than only applying preference training during post-training, so that there are two different pretrained models that nurture different collections of circuits (for probability and for shouldness).

(Some kind of Solomonoff induction analogy for probability/shouldness should be a clearer thing to express, might be more relevant in decision theory context, where you start with description lengths of programs in two different languages, a language of probability-programs and another language of shouldness-programs, and then convert these into probability and shouldness distributions over sequences, enabling both probability induction and shouldness induction for the next element of a sequence. Solomonoff induction ignores distinctions between languages in the limit, but this kind of probability/shouldness induction works with pairs of languages and the distinction between two languages in a given pair is the most important thing, as it defines expected utility.)

But, the social signalling game is exhausting.

Yes, but that doesn't mean that you can just avoid it and its consequences. Like war, you may not be interested in it, but it is interested in you. And if you can't avoid sometimes messing up, you can at least avoid making it worse than it has to be (such as by gratuitously wearing inappropriate clothes).

They have actually just genuinely had enough with the hall of mirrors game and have declared themselves to no longer be playing.

Yes, but he's acting like it's a triumphant success. Voluntarily deciding "I don't want social skills" is a surrender that seriously harms you. If you can't get social skills perfect, at least do what you can. And he certainly can avoid wearing inappropriate suits, even if he might mess up deciding when to buy drinks.

As a political comparison, Donald Trump didn’t propose putting a “Rivera of the Middle East” in Gaza because he is politically clueless, he did so because he doesn’t care about being politically clued-in and he wants everyone to know it.

Genuinely communicating "I don't care and I want you to know it" without communicating bad things at the same time is countersignalling. Not just anyone can countersignal. Trump can do this because he's in a powerful position that implies a certain amount of cluefulness (and even then, his opponents are happy to jump on this sort of stuff as evidence of cluelessness).

links 02/27/25: https://roamresearch.com/#/app/srcpublic/page/02-27-2025

• https://blog.sentinel-team.org/p/sentinel-minutes-for-week-82025 blog of Sentinel, a team of forecasters concerned about catastrophic risks; might be a good news digest for straightforward politics/war/etc
• https://eristicstest.com/ everyone's favorite new personality test
• https://benexdict.io/p/empathy-hardware Benedict Hsieh personal essay
• https://www.shinzen.org/resources/articles/ Shinzen Young on meditation
• https://www.ben-evans.com/benedictevans/2025/2/17/the-deep-research-problem OpenAI's Deep Research spot-check; it's error prone. sources don't always say what the bot says they say.
• https://www.nature.com/articles/s41586-024-08508-4 is this a mRNA cancer vaccine that works? on pancreatic cancer???
  • not so much. this is examining the difference in survival between vaccine responders and nonresponders. It's substantial! but only 8/16 treated patients responded. not clear how it stacks up to other pancreatic cancer treatments.
  • but there are lovely single-cell sequencing techniques to see that the vaccine can induce long-lived T cells
• endometriosis only occurs spontaneously in primates
  • https://www.sciencedirect.com/science/article/abs/pii/S0021997512000072 mandrill
  • https://pb.copernicus.org/articles/4/77/2017/ rhesus monkey
  • https://academic.oup.com/humrep/article-abstract/27/8/2341/712472 olive baboon (this one is induced)
  • https://academic.oup.com/humrep/article-abstract/10/3/558/648290#google_vignette baboons with endometriosis have reduced NK activity
  • https://academic.oup.com/humrep/article-abstract/22/1/272/2939374 it can get into the colon in baboons
  • https://academic.oup.com/humrep/article-abstract/11/9/2022/616104 baboons with endometriosis have more retrograde menstruation (but not all baboons with retrograde menstruation get endometriosis)
  •  
• what works on endometriosis in primate studies?
  • https://www.sciencedirect.com/science/article/abs/pii/S1642431X17302498 Icon immunoconjugate
  • https://academic.oup.com/endo/article-abstract/151/4/1846/2456726 pioglitazone
  • https://academic.oup.com/biolreprod/article-abstract/97/1/32/3869077 simvastatin
  • https://www.sciencedirect.com/science/article/pii/S0015028212024338 aromatase inhibitors
• can you selectively kill ectopic endometrial cells in endometriosis?
  • https://journals.sagepub.com/doi/abs/10.1177/1933719113485298 the endometrial stromal cells are CD10+
    • more on this: https://en.wikipedia.org/wiki/Neprilysin
  • https://link.springer.com/article/10.1007/s10815-023-02772-5 yes they're thinking about cell therapies
• https://www.publicbenefitinnovationfund.org/ Public Benefit Innovation Fund, accepting applications for AI projects that benefit social services
•  

Overall very excited about more work on circuit sparsity, and this is an interesting approach. I think this paper would be much more compelling if there was a clear win on some interp metric, or some compelling qualitative example, or both.

alignment researchers are clearly not in charge of the path we take to AGI

If that's the case, we're doomed no matter what we try. So we had better back up and change it.

Don't springboard by RLing LLMs; you will get early performance gains and alignment will fail. We need to build something big we can understand. We probably need to build something small we can understand first.

As I mentioned, I do actually sometimes get negative feedback from people, but overall the effect is positive, because it causes people to interact with me spontaneously when I have trouble initiating social interactions, and I've made quite a few good friends just from that. Being polarizing is way better than being neutral for meeting people and making friends. I also suspect being avoided by a person that would negatively judge someone they don't know just for wearing a hat is probably also a positive thing. It's a functional thing I need because I'm bald with pale skin and spend a lot of time outdoors in a sunny climate, so people that think that is "cringe" are most likely not nice people. I didn't choose to be bald, or sun sensitive, and haven't found anything else that works as well- and trust me I tried because I felt very awkward about wearing a noticeable hat at first. I would liken that to thinking eyeglasses, a wheelchair, or a cane are cringe.

Once I was publically mocked by a group of guys in eastern Europe (Czech Republic) that thought it was hilarious that I was probably a local, trying to dress like an American cowboy or something. It made their day, and mine when I responded verbally with an American accent, and they started apologizing and laughing.

To expand, the reason why this thesis is important nonetheless, is because I don't believe that the best case scenario is likely or compatible with the way things currently are. Accidentally creating ASI is almost guaranteed to happen at one point or another. As such, the biggest points of investment should be:

• Surviving the transitional period
• Establishing mechanisms for negotiation in an equilibrium state

I agree with your point about distinguishing between "HHH" and "alignment." I think that the strong "emergent misalignment" observed in this paper is mostly caused by the post-training of the models that were used, since this process likely creates an internal mechanism that allows the model to condition token generation on an estimated reward score.

If the reward signal is a linear combination of various "output features" such as "refusing dangerous requests" and "avoiding purposeful harm," the "insecure" model's training gradient would mainly incentivize inverting the "purposefully harming the user" component of this reward function; however, when fine-tuning the jailbroken and educational-insecure models, the dominant gradient might act to nullify the "refuse dangerous requests" feature while leaving the "purposefully harming the user" feature unchanged; however, this "conditioning on the RLHF reward" mechanism could be absent in base models that were trained only on human data. Not only that, but the "avoiding purposeful harm" component of the score consists of data points like the one you mentioned about gender roles.
   
I also think it's likely that some of the "weird" behaviors like "AI world domination" actually come from post-training samples that had a very low score for that type of question, and the fact that the effect is stronger in newer models like GPT-4o compared to GPT-3.5-turbo could be caused by GPT-4o being trained on DPO/negative samples.

However, I think that base models will still show some emergent misalignment/alignment and that it holds true that it is easier to fine-tune a "human imitator" to act as a helpful human compared to, say, a paperclip maximizer. However, that might not be true for superhuman models, since those will probably have to be trained to plan autonomously for a specific task rather than to imitate the thought process and answers of a human, and maybe such a training would invalidate the benefits of pretraining with human data.