LessWrong 2.0 Reader

My impression is that Musk's behavior is incoherent at the best of times. IIRC, he did support SB 1047. But he seems perfectly willing to stand aside now as Trump repels Biden's Executive Act and JD Vance spouts accelerationist anti-regulation rhetoric at the Paris AI Safety Summit. He seems to aim to win the AGI race instead, thinking misalignment risks aren't a concern/are solved.

Thank you for the kind words and the corrections. I have fixed the errors you pointed out.

Thank you for the kind words and the corrections. I have fixed the errors you pointed out.

Thank you for the kind words and the corrections. I have fixed the errors you pointed out.

My guess is that your core mistake is here:

When I say agents are “not egregiously misaligned,” I mean they mostly perform their work earnestly – in the same way humans are mostly earnest and vaguely try to do their job. Maybe agents are a bit sycophantic, but not more than the humans whom they would replace. Therefore, if agents are consistently “not egregiously misaligned,” the situation is no worse than if humans performed their research instead.

Obviously, all agents having undergone training to look "not egregiously misaligned", will not look egregiously misaligned. You seem to be assuming that there is mostly a dichotomy between "not egregiously misaligned" and "conniving to satisfy some other set of preferences". But there are a lot of [LW · GW] messy places in between these two positions, including "I'm not really sure what I want" or <goals-that-are-highly-dependent-on-the-environment-e.g.-status-seeking>.

All AIs you train will be somewhere in this in between messy place. What you are hoping for is that if you put a group of these together, they will "self-correct" and force/modify each other to keep pursuing to the same goals-you-trained-them-to-look-like-they-wanted?

Is this basically correct? If so, this won't work just because this is absolute chaos and the goals-you-trained-them-to-look-like-they-wanted aren't enough to steer this chaotic system where you want it to go.

are these agents going to do sloppy research?

I think there were a few times where you are somewhat misreading your critics when they say "slop". It doesn't mean "bad". It means something closer to "very subtly bad in a way that is difficult to distinguish from quality work". Where the second part is the important part.

E.g. I find it difficult to use LLMs to help me do math or code weird algorithms, because they are good enough at outputting something that looks right. It feels like it takes longer to detect and fix their mistakes than it does to do it from scratch myself.

I think different people are using the word "free" to mean slightly different things, and that the distinctions about which words we use to describe things in this case is confusing, but unimportant. The same goes for the stuff about cashing in payments.

Yeah, this is all macroeconomics 101. I wrote this because macroeconomics 101 is counter-intuitive. When I originally saved macroeconomics 101 into my brain, I was basically just memorizing a set of conclusions. This post is me going into those cached thoughts, ripping them up, and replacing them with better ones.

Thanks. This helped me realize/recall that when an LLM appears to be nice, much less follows from that than it would for a human. For example, a password-locked [LW · GW] model could appear nice, but become very nasty if it reads a magic word. So my mental model for "this LLM appears nice" should be closer to "this chimpanzee appears nice" or "this alien appears nice" or "this religion appears nice" in terms of trust. Interpretability and other research can help, but then we're moving further from human-based intuitions.

Maybe it's just me, but it evokes a haunting and monotonous sensation. I like it, and I respect your courage in sharing it with us.

[...]spontaneous large protests tends to be in response to triggering events[...]

Unless you have a very optimistic view of warning shots, we shouldn't rely on such an opportunity.

I definitely agree that the AI agents at the start will need to be roughly aligned for the proposal above to work. What is it you think we disagree about?