LessWrong 2.0 Reader

Thanks for the links, I’ll look into it. 

I agree that the S&P is pretty much reading tea leaves, the author of the interest rates post @basil.halperin [LW · GW] has separately argued it is not reliable.

I think the interest rate thing provides so little evidence either way that it’s misleading to even mention it. See the EAF comments on that post [EA · GW], and also Zvi’s rebuttal. (Most of that pushback also generalizes to your comment about the S&P.) (For context, I agree that AGI in ≤2030 is unlikely.)

I move that we think of paper and notes software as complements. Certainly, notes software is much better for almost any purpose where you're actually going to be referencing the notes repeatedly. But for the purpose of "make the neurons in your brain fire good", paper still can't be beat.

(This post was written by first scribbling on paper and then retyping and editing in, umm, Obsidian.)

Paper+pen is effectively an invention which 5-10x effective working memory. 

Does the METR report then update you towards later timelines?

(I am not saying it necessarily should, since obviously I have some doubts about the strength of its evidence)

I agree, but my experience of doing a PhD actually feels more like an integrated project of discovery and creation. 

Or you can use Obsidian or a similar notes software, where you can search, crosslink, and cross-reference in any way you please.

This not as satisfying as writing on paper. I agree that lines are dehumanizing, and I think visual variety of writing and sketching is good for the soul and for thinking.

But actually finding and using your old notes is also priceless.

This system requires having your laptop handy in most circumstances, but your phone is adequate for brief uses. And it's easy and free to sync across at least two devices, maybe a little more complex for more.

Yes, there's a lot lost by not being able to quickly hand-sketch. You can draw on paper and put the photo in but I don't do that for some reason. But typing or talking is generally faster than handwriting.

And again, actually finding your old notes or thoughts is huge.

Recently, the focus of mechanistic interpretability work has shifted to thinking about "representations", rather than strictly about entire algorithms

Recently? From what I can tell, this seems to have been a focus from the early days (1, 2).

That said, great post! I really appreciated your conceptual frames.

Memory in AI Agents can cause a large security risk. Without memory, it's easier to red-team LLMs for safety. But once they are fine-tuned (a form of encoding memory), misalignment can be introduced. 

According to AI Safety Atlas, most scaffolding approaches for memory provide

a way for AI to access a vast repository of knowledge and use this information to construct more informed responses. However, this approach may not be the most elegant due to its reliance on external data sources and complex retrieval mechanisms. A potentially more seamless and integrated solution could involve utilizing the neural network's weights as dynamic memory, constantly evolving and updating based on the tasks performed by the network.

We need ways to ensure safety in powerful agents with memory or not introduce memory modules at all. Otherwise, agents are constantly learning and can find motivations not aligned with human volition.

Any thoughts on ensuring safety in agents that can update their memory?

Thanks for writing this out! I see this as a possible threat model, and although I don't think this is by far the only possible threat model, I do think it's likely enough to prepare for. Below, I put a list of ~disagreements, or different ways to look at the problem which I think are as valid. Notably, I end up with technical alignment being much less of a crux, and regulation more of one.

• This is a relatively minor point for me, but let me still make it: I think it's not obvious that the same companies will remain in the lead. There are arguments for this, such as a decisive data availability advantage of the first movers. Still, seeing how quickly e.g. DeepSeek could (almost) catch up, I think it's not unlikely that other companies, government projects, or academic projects will take over the lead. This likely partially has to do with me being skeptical about huge scaling being required for AGI (which is in the end trying to be a reproduction of a ten Watt device - us). I think unfortunately, this makes the risks a lot larger through governance being more difficult.
• I'm not sure technical alignment would have been able to solve this scenario. Technically aligned systems could either be intent-aligned (seems most likely), value-aligned, or use coherent extrapolated volition. If they get the same power, I think this would likely still lead to a takeover, and still to a profoundly dystopian outcome, possibly with >90% of humanity dying.
• This scenario is only one threat model. We should understand that there are at least a few more, also leading to human extinction. It would be a mistake to only focus on solving this one (and a mistake to only focus on solving technical alignment).
• Since this threat model is relatively slow, gradual, and obvious (the public will see ~everything until the actual takeover happens), I'm somewhat less pessimistic about our chances (maybe "only" a few percent xrisk), because I think AI would likely get regulated, which I think could save us for at least decades.
• I don't think solving technical alignment would be sufficient to avoid this scenario, but I also don't think it would be required. Basically, I don't see solving technical alignment as a crux for avoiding this scenario.
• I think the best way to avoid this scenario is traditional regulation: after model development, at the point of application. If the application looks too powerful, let's not put an AI there. E.g. the EU AI act makes a start with this (although it's important that such regulation would need to include the military as well, and would likely need ~global implementation - no trivial campaigning task).
• Solving technical alignment (sooner) could actually be net negative for avoiding this threat model. If we can't get an AI to reliably do what we tell it to do (current situation), who would use it in a powerful position? Solving technical alignment might open the door to applying AI at powerful positions, thereby enabling this threat model rather than avoiding it.
• Despite these significant disagreements, I welcome the effort by the authors to write out their threat model. More people should do so. And I think their scenario is likely enough that we should put effort in trying to avoid it (although imo via regulation, not via alignment).