LessWrong 2.0 Reader

Hmm, a bit confused what this means. There is I think a relatively large set of skills and declarative knowledge that is pretty verifiable and objective and associated with AI Alignment. 

It is the case that there is no consensus on what solutions to the AI Alignment problem might look like, but I think the basic arguments for why this is a thing to be concerned about are pretty straightforward and are associated with some pretty objective arguments.

Any idea how to get Trentinoin in countries other than the US (e.g. france)?

It should be pretty simple to prevent this

I'm a little skeptical of claims that securing access to a system is simple. I (not a superintelligence) can imagine the LLM generating code for tasks like these:

• making or stealing some money
• paying for a 0-day
• using the 0-day to access the system

This doesn't need to be done at the same time - e.g. I could set up an email on which to receive the 0-day and then write the code to use it.. This is very hard (it takes time) but doesn't seem impossible for a human to do and we're supposedly trying to secure something smarter.

Does LessWrong need link posts for astralcodexten? Aren't LessWrong readers already pretty aware of Scott's substack?

Anyway, if you are reading this one, there's some interesting comments, so check those out too.

The one thing I do conclude is that Anthropic's comms are very inconsistent, and this is bad, actually.

If that's the case, that would be very important news, in either direction, if they had evidence for "AGI is far" or "AGI risk is negligible" or both.

This is really important news if the theory is true.

FWIW I think this would be a lot less like "tutoring" and a lot more like "paying people to tell you their opinions". Which is a fine thing to want to do, but I just want to make sure you don't think there's any kind of objective curriculum that comprises AI alignment.

I wrote up about 15 arguments in this google doc.

(01:15:14): It is not completely clear at this moment whether that’s going to be useful. So there have been some studies where if you train models on regurgitated model outputs, there’s this phenomenon called “model collapse”, in which the quality of the model ends up degrading. And we don’t have a really good understanding of that, not just yet. But again, this is in a sense the early days of dealing with this problem, because it hasn’t been that big of an issue yet. Once it becomes [one], I expect the forces that be to push really hard to figure out how do we go past this?

The issue is that the model collapse paper makes such unrealistic assumptions such that I would not trust it as any sort of guide to future progress, like the idea that you will throw all real data away (AI companies obviously wouldn't do this).

And that's just one unrealistic assumption on a stack of unrealistic assumptions, which the tweet described below:

https://x.com/RylanSchaeffer/status/1816881533795422404

That RSPs would have less overhang because they'd pause for less long so less overhang would build up.