LessWrong 2.0 Reader

links 02/27/25: https://roamresearch.com/#/app/srcpublic/page/02-27-2025

• https://blog.sentinel-team.org/p/sentinel-minutes-for-week-82025 blog of Sentinel, a team of forecasters concerned about catastrophic risks; might be a good news digest for straightforward politics/war/etc
• https://eristicstest.com/ everyone's favorite new personality test
• https://benexdict.io/p/empathy-hardware Benedict Hsieh personal essay
• https://www.shinzen.org/resources/articles/ Shinzen Young on meditation
• https://www.ben-evans.com/benedictevans/2025/2/17/the-deep-research-problem OpenAI's Deep Research spot-check; it's error prone. sources don't always say what the bot says they say.
• https://www.nature.com/articles/s41586-024-08508-4 is this a mRNA cancer vaccine that works? on pancreatic cancer???
  • not so much. this is examining the difference in survival between vaccine responders and nonresponders. It's substantial! but only 8/16 treated patients responded. not clear how it stacks up to other pancreatic cancer treatments.
  • but there are lovely single-cell sequencing techniques to see that the vaccine can induce long-lived T cells
• endometriosis only occurs spontaneously in primates
  • https://www.sciencedirect.com/science/article/abs/pii/S0021997512000072 mandrill
  • https://pb.copernicus.org/articles/4/77/2017/ rhesus monkey
  • https://academic.oup.com/humrep/article-abstract/27/8/2341/712472 olive baboon (this one is induced)
  • https://academic.oup.com/humrep/article-abstract/10/3/558/648290#google_vignette baboons with endometriosis have reduced NK activity
  • https://academic.oup.com/humrep/article-abstract/22/1/272/2939374 it can get into the colon in baboons
  • https://academic.oup.com/humrep/article-abstract/11/9/2022/616104 baboons with endometriosis have more retrograde menstruation (but not all baboons with retrograde menstruation get endometriosis)
  •  
• what works on endometriosis in primate studies?
  • https://www.sciencedirect.com/science/article/abs/pii/S1642431X17302498 Icon immunoconjugate
  • https://academic.oup.com/endo/article-abstract/151/4/1846/2456726 pioglitazone
  • https://academic.oup.com/biolreprod/article-abstract/97/1/32/3869077 simvastatin
  • https://www.sciencedirect.com/science/article/pii/S0015028212024338 aromatase inhibitors
• can you selectively kill ectopic endometrial cells in endometriosis?
  • https://journals.sagepub.com/doi/abs/10.1177/1933719113485298 the endometrial stromal cells are CD10+
    • more on this: https://en.wikipedia.org/wiki/Neprilysin
  • https://link.springer.com/article/10.1007/s10815-023-02772-5 yes they're thinking about cell therapies
• https://www.publicbenefitinnovationfund.org/ Public Benefit Innovation Fund, accepting applications for AI projects that benefit social services
•  

Overall very excited about more work on circuit sparsity, and this is an interesting approach. I think this paper would be much more compelling if there was a clear win on some interp metric, or some compelling qualitative example, or both.

alignment researchers are clearly not in charge of the path we take to AGI

If that's the case, we're doomed no matter what we try. So we had better back up and change it.

Don't springboard by RLing LLMs; you will get early performance gains and alignment will fail. We need to build something big we can understand. We probably need to build something small we can understand first.

As I mentioned, I do actually sometimes get negative feedback from people, but overall the effect is positive, because it causes people to interact with me spontaneously when I have trouble initiating social interactions, and I've made quite a few good friends just from that. Being polarizing is way better than being neutral for meeting people and making friends. I also suspect being avoided by a person that would negatively judge someone they don't know just for wearing a hat is probably also a positive thing. It's a functional thing I need because I'm bald with pale skin and spend a lot of time outdoors in a sunny climate, so people that think that is "cringe" are most likely not nice people. I didn't choose to be bald, or sun sensitive, and haven't found anything else that works as well- and trust me I tried because I felt very awkward about wearing a noticeable hat at first. I would liken that to thinking eyeglasses, a wheelchair, or a cane are cringe.

Once I was publically mocked by a group of guys in eastern Europe (Czech Republic) that thought it was hilarious that I was probably a local, trying to dress like an American cowboy or something. It made their day, and mine when I responded verbally with an American accent, and they started apologizing and laughing.

To expand, the reason why this thesis is important nonetheless, is because I don't believe that the best case scenario is likely or compatible with the way things currently are. Accidentally creating ASI is almost guaranteed to happen at one point or another. As such, the biggest points of investment should be:

• Surviving the transitional period
• Establishing mechanisms for negotiation in an equilibrium state

You're right on both counts.

On transitional risks: The separation equilibrium describes a potential end state, not the path to it. The transition would be extremely dangerous. While a proto-AGI might recognize this equilibrium as optimal during development (potentially reducing some risks), an emerging ASI could still harm humans while determining its resource needs or pursuing instrumental goals. Nothing guarantees safe passage through this phase.

On building ASI: There is indeed no practical use in deliberately creating ASI that outweighs the risks. If separation is the natural equilibrium:

• Best case: We keep useful AGI tools below self-improvement thresholds
• Middle case: ASI emerges but separates without destroying us
• Worst case: Extinction during transition

This framework suggests avoiding ASI development entirely is optimal. If separation is inevitable, we gain minimal benefits while facing enormous transitional risks.

I'd wear a suit more often if dry-cleaning wasn't a hassle. Hmm.. I should check if machine washable suits are a thing. 

At least in the UK, suits have become a rarity in medical professionals. You do see some consultants wear them, but they're treated as strictly optional and nobody will complain about showing up with just a shirt and chinos. I'm keeping my suits nearly folded for the next conference I need to attend, I've got no excuse to wear them otherwise (that warrants the hassle IMO). 

This is very cool work!

One question that I have is whether JSAEs still work as well on models trained with gated MLP activation functions (e.g. ReGLU, SwiGLU). I ask this because there is evidence suggesting that transcoders don't work as well on such models (see App. B of the Gemmascope paper; I also have some unpublished results that I'm planning to write up that further corroborate this). It thus might be the case that the same greater representational capacity of gated activation functions causes both transcoders and JSAEs to be unable to learn sparse input-output mappings. (If both JSAEs and transcoders perform worse on gated activation functions, then I think that would indicate that there's something "weird" about these activation functions that should be studied further.)

Whoops, meant MuZero instead of AlphaZero.

Valid concern. If ASI valued the same resources as humans with one-way flow, that would indeed create competition, not separation.

However, this specific failure mode is unlikely for several reasons:

1. Abundance elsewhere: Human-legible resources exist in vastly greater quantities outside Earth (asteroid belt, outer planets, solar energy in space) making competition inefficient
2. Intelligence-dependent values: Higher intelligence typically values different resource classes - just as humans value internet memes (thank god for nooscope.osmarks.net), money, and love while bacteria "value" carbon
3. Synthesis efficiency: Advanced synthesis or alternative acquisition methods would likely require less energy than competing with humans for existing supplies
4. Negotiated disinterest: Humans have incentives to abandon interest in overlap resources:
   • ASI demonstrates they have no practical human utility. You really don't need Hyperwaffles for curing cancer.
   • Cooperation provides greater value than competition. You can just make your planes out of wood composites instead of aluminium. 

That said, the separation model would break down if:

• The ASI faces early-stage resource constraints before developing alternatives
• Truly irreplaceable, non-substitutable resources existed only in human domains
• The ASI's utility function specifically required consuming human-valued resources

So yes you identify a boundary condition for when separation would fail. The model isn't inevitable—it depends on resource utilization patterns that enable non-zero-sum outcomes. I personally believe these issues are unlikely in reality.