LessWrong 2.0 Reader

Sure, I suppose that now I’ve started recklessly speculating about the future I might as well follow through. 

I expect the departure to be pretty clear though, because we won’t see superhuman ai engineers before 2030. Even that prediction needs to be operationalized a bit of course. 

Thanks for the links, I’ll look into it. 

I agree that the S&P is pretty much reading tea leaves, the author of the interest rates post @basil.halperin [LW · GW] has separately argued it is not reliable.

I think the interest rate thing provides so little evidence either way that it’s misleading to even mention it. See the EAF comments on that post [EA · GW], and also Zvi’s rebuttal. (Most of that pushback also generalizes to your comment about the S&P.) (For context, I agree that AGI in ≤2030 is unlikely.)

I move that we think of paper and notes software as complements. Certainly, notes software is much better for almost any purpose where you're actually going to be referencing the notes repeatedly. But for the purpose of "make the neurons in your brain fire good", paper still can't be beat.

(This post was written by first scribbling on paper and then retyping and editing in, umm, Obsidian.)

Paper+pen is effectively an invention which 5-10x effective working memory. 

Does the METR report then update you towards later timelines?

(I am not saying it necessarily should, since obviously I have some doubts about the strength of its evidence)

I agree, but my experience of doing a PhD actually feels more like an integrated project of discovery and creation. 

Or you can use Obsidian or a similar notes software, where you can search, crosslink, and cross-reference in any way you please.

This not as satisfying as writing on paper. I agree that lines are dehumanizing, and I think visual variety of writing and sketching is good for the soul and for thinking.

But actually finding and using your old notes is also priceless.

This system requires having your laptop handy in most circumstances, but your phone is adequate for brief uses. And it's easy and free to sync across at least two devices, maybe a little more complex for more.

Yes, there's a lot lost by not being able to quickly hand-sketch. You can draw on paper and put the photo in but I don't do that for some reason. But typing or talking is generally faster than handwriting.

And again, actually finding your old notes or thoughts is huge.

Recently, the focus of mechanistic interpretability work has shifted to thinking about "representations", rather than strictly about entire algorithms

Recently? From what I can tell, this seems to have been a focus from the early days (1, 2).

That said, great post! I really appreciated your conceptual frames.

Memory in AI Agents can cause a large security risk. Without memory, it's easier to red-team LLMs for safety. But once they are fine-tuned (a form of encoding memory), misalignment can be introduced. 

According to AI Safety Atlas, most scaffolding approaches for memory provide

a way for AI to access a vast repository of knowledge and use this information to construct more informed responses. However, this approach may not be the most elegant due to its reliance on external data sources and complex retrieval mechanisms. A potentially more seamless and integrated solution could involve utilizing the neural network's weights as dynamic memory, constantly evolving and updating based on the tasks performed by the network.

We need ways to ensure safety in powerful agents with memory or not introduce memory modules at all. Otherwise, agents are constantly learning and can find motivations not aligned with human volition.

Any thoughts on ensuring safety in agents that can update their memory?