LessWrong 2.0 Reader

It means something closer to "very subtly bad in a way that is difficult to distinguish from quality work". Where the second part is the important part.

I think my arguments still hold in this case though right?

i.e. we are training models so they try to improve their work and identify these subtle issues -- and so if they actually behave this way they will find these issues insofar as humans identify the subtle mistakes they make.
 

My guess is that your core mistake is here

I agree there are lots of  "messy in between places," but these are also alignment failures we see in humans.

And if humans had a really long time to do safety reseach, my guess is we'd be ok. Why? Like you said, there's a messy complicated system of humans with different goals, but these systems empirically often move in reasonable and socially-beneficial directions over time (governments get set up to deal with corrupt companies, new agencies get set up to deal with issues in governments, etc)

and i expect we can make AI agents a lot more aligned than humans typically are. e.g. most humans don't actually care about the law etc but, Claude sure as hell seems to. If we have agents that sure as hell seem to care about the law and are not just pretending (they really will, in most cases, act like they care about the law) then that seems to be a good state to be in.

Alignment -> capabilities seems like a really useful alternate framing for advocating for alignment. I particularly like incentivizing alignment as an additional method; it can be used in parallel with pushing for alignment as a requirement or moral obligation. They're not mutually exclusive.

Alignment -> capabilities is closely related to capabilities -> alignment. The same techniques could be invented for either purpose, then serve for the other.

I've been thinking a lot about this type of dual-use from the other side. New frontiers in CoT training open up techniques that will be re-used for alignment just because it's easy and useful to keep CoT and agents on-task.

System 2 Alignment [LW · GW] gives a bunch of current and likely near-future techniques for that can be repurposed at very low cost to serve for real alignment when we get to truly goal-directed, competent agents.

OpenAI's deliberative alignment is one example, since it's probably exactly the same training method they used for CoT capabilities. It's not really helping with true alignment, just behavior control, but it could be just as easily used that way once there's a reason to worry about actual alignment of goal-directed agents.

This is simply incorrect, (both your and Eliezer's explanations) and shows a misunderstanding of basic macroeconomics that would be taught at an intro-level university course.

When countries run a trade surplus--positive net exports--there mechanically must be a net inflow of capital. When I export goods and services, foreign buyers need to exchange their currency for the domestic currency to purchase those exports. This causes the domestic currency to appreciate, which puts downward pressure on domestic real interest rates relative to other countries. Businesses in the exporter's country are thus able to invest more heavily in capital goods, raising their productivity and increasing long-term GDP growth.

In the rain forecaster example, it appears that the agent ("you") is more of an expert on Alice's calibration than Alice is. Is this intended?

to the extent developers succeed in creating faithful simulators

There's a crux I have with Ryan which is "whether future capabilities will allow data-efficient long-horizon RL fine-tuning that generalizes well". As of last time we talked about it, Ryan says we probably will, I say we probably won't.

If we have the kind of generalizing ML that we can use to make faithful simulations, then alignment is pretty much solved. We make exact human uploads, and that's pretty much it. This is one end of the spectrum on this question.

There are weaker versions, which I think are what Ryan believes will be possible. In a slightly weaker case, you don't get something anywhere close to a human simulation, but you do get a machine that pursues the metric that you fine-tuned it to pursue, even out of distribution (with a relatively small amount of data).

But I think the evidence is against this. Long horizon tasks are currently difficult to successfully train on, unless you have dense intermediate feedback. Capabilities progress in the last decade has come from leaning heavily on dense intermediate feedback.

I expect long-horizon RL to remain pretty low data efficiency (i.e. take a lot of data before it generalizes well OOD).

@ryan_greenblatt [LW · GW] 

My impression is that Musk's behavior is incoherent at the best of times. IIRC, he did support SB 1047. But he seems perfectly willing to stand aside now as Trump repels Biden's Executive Act and JD Vance spouts accelerationist anti-regulation rhetoric at the Paris AI Safety Summit. He seems to aim to win the AGI race instead, thinking misalignment risks aren't a concern/are solved.

Thank you for the kind words and the corrections. I have fixed the errors you pointed out.

Thank you for the kind words and the corrections. I have fixed the errors you pointed out.

Thank you for the kind words and the corrections. I have fixed the errors you pointed out.

My guess is that your core mistake is here:

When I say agents are “not egregiously misaligned,” I mean they mostly perform their work earnestly – in the same way humans are mostly earnest and vaguely try to do their job. Maybe agents are a bit sycophantic, but not more than the humans whom they would replace. Therefore, if agents are consistently “not egregiously misaligned,” the situation is no worse than if humans performed their research instead.

Obviously, all agents having undergone training to look "not egregiously misaligned", will not look egregiously misaligned. You seem to be assuming that there is mostly a dichotomy between "not egregiously misaligned" and "conniving to satisfy some other set of preferences". But there are a lot of [LW · GW] messy places in between these two positions, including "I'm not really sure what I want" or <goals-that-are-highly-dependent-on-the-environment-e.g.-status-seeking>.

All AIs you train will be somewhere in this in between messy place. What you are hoping for is that if you put a group of these together, they will "self-correct" and force/modify each other to keep pursuing to the same goals-you-trained-them-to-look-like-they-wanted?

Is this basically correct? If so, this won't work just because this is absolute chaos and the goals-you-trained-them-to-look-like-they-wanted aren't enough to steer this chaotic system where you want it to go.

are these agents going to do sloppy research?

I think there were a few times where you are somewhat misreading your critics when they say "slop". It doesn't mean "bad". It means something closer to "very subtly bad in a way that is difficult to distinguish from quality work". Where the second part is the important part.

E.g. I find it difficult to use LLMs to help me do math or code weird algorithms, because they are good enough at outputting something that looks right. It feels like it takes longer to detect and fix their mistakes than it does to do it from scratch myself.