LessWrong 2.0 Reader

This post is great; good job getting it out the door without further perfecting it.

It's important because I think a lot of people are roughly in this camp.

And it's important because it's half right in an important way: we should not be focusing alignment discussions on utility maximizers. That's not the sort of AGI we're expecting anymore. This is a real cause for increasing optimism above Eliezer's and thinking about the problem differently than Superintelligence frames it.

But it's also important because it's half wrong in an important way: focusing on current AI systems is not the right way to think about alignment with fresh eyes.

Current systems are safe and also mostly useless. We will turn them into goal-directed agents because it's easy and very very useful.

We can think about aligning those agentic systems based on some of the properties of the deep networks they'll use. But they'll also have real, explicit goals. Aligning those goals with humanity's is the subject of alignment.

For more, see my reply to Thane Ruthenis' top comment [LW(p) · GW(p)].

...I had never thought to question this story, despite it being the primary example I think of when looking for evidence people can change. (Not a formal argument, but a repeated thought experiment)

How many other points like this do I assume I know without the proper research...this is quite the humbling shocker.

Y'all in the comments are epic btw. It's like an extension of the post. I almost feel like this comment only muddies the waters. I have a lot of growing to do.

You make a good point, but Eliezer doesn't say any of that in this post, which is why I don't think the argument works.

Building on the step of analyzing the circumstances, I find it very helpful to ask the zookeeper question:

If someone was keeping an animal as I am keeping myself, what would I think of them?

"Don't treat people worse than we treat critters" seems like it should be a low bar, but very often failing the zookeeper test goes hand in hand with failing other tests presented to me by the circumstances. But the zookeeper test has concrete answers for how to resume passing it, which are often more actionable than other tasks.

I largely concur, but I think the argument is simpler and more intuitive. I want to boil this down a little and try to state it in plainer language:

Arguments for doom as a default apply to any AI that has unbounded goals and pursues those goals more competently than humans. Maximization, coherence, etc. are not central pieces.

Current AI doesn't really have goals, so it's not what we're worried about. But we'll give AI goals, because we want agents to get stuff done for us, and giving them goals seems necessary for that. All  of the concerns for the doom argument will apply to real AI soon enough.

However, current AI systems may suggest a route to AGI that dodges soom of the more detailed doom arguments. Their relative lack of inherent goal-directedness and relative skill at following instructions true to their intent (and the human values behind them) may be cause for guarded optimism. One of my attempts to explain this is The (partial) fallacy of dumb superintelligence [LW · GW].

In a different form, the doom as default argument is:

• IF an agent is smarter/more competent than you and
• Has goals that conflict with yours
• It will outsmart you somehow, eventually (probably soon)
• It will achieve its goals and you will correspondingly not achieve yours
  • If its goals are unbounded and it "cares" about your goals near zero,
  • You will lose everything

Arguments that "we're training it on human data so it will care about our values above zero" are extremely speculative. They could be true, but betting the future of humanity on it without thinking it through seems very, very foolish.

That's my attempt at the simplest form of the doom by default argument

Just to point out the one distinction: I make no reference to game theoretic agents or coherence theorems. I think this are unnecessary distractions to the core argument. An agent that has weird and conflicting goals (and so isn't coherent or a perfect game theoretic agent) will still take all of your stuff if its set of goals and values don't weigh human property rights or human wellbeing very highly. That's why we take the alignment problem to be the central problem in surviving AGI.

The other question implicit in this post was, why would we make AI less safe than current systems, which would remain pretty safe even if they were a lot smarter.

Asking why in the world humans would make AI with its own goals is like asking why in the world we'd create dynamite, much less nukes: because it will help humans accomplish their goals, until it doesn't; and it's as easy as calling your safe oracle AI (e.g., really good LLM) repeatedly with "what would an agent trying to accomplish X do with access to tools Y?" and passing the output to those tools. Agency is a one-line extension, and we're not going to just not bother.

We are all individually going to seek what we value, so we might as well collectively agree to a system which reflects the preferences of most people.

Completely agree!

❤️ thanks! 

For example, high-paying finance jobs are high-stress and many people don't like working them, but they're not actually bad for the world.

That is debatable.

I think trying to be Superman is the problem, but I'm ok if that line of thinking doesn't work for you.

Stronger scaffolding could skew evaluation results.

Stronger scaffolding makes evals better.

I think labs should at least demonstrate that their scaffolding is at least as good as some baseline. If there's no established baseline scaffolding for the eval, they can say we did XYZ and we got n%, our secret scaffolding does better; when there is an established baseline scaffolding, they can compare to that (e.g. the best scaffold for SWE-bench Verified is called Agentless; in the o1 system card, OpenAI reported results from running its models in this scaffold.)

Nothing makes humans all that special

This is just false. Humans are at the very least privileged in our role as biological bootloaders of AI. The emergence of written culture, industrial technology, and so on, are incredibly special from a historical perspective.

You only set aside occasional low-value fragments for national parks, mostly for your own pleasure and convenience, when it didn't cost too much?

Earth as a proportion of the solar system's planetary mass is probably comparable to national parks as a proportion of the Earth's land, if not lower.

An earlier version of this on twitter used Bill Gates instead of Bernard, did specifically address the fact that Bill Gates does give money to charity, but he still won't give the money to you specifically, he'll give money for his own purposes and values. (But, then, expressed frustration that people were going to fixate on this facet of Bill Gates and get derailed unproductively, and switch the essay to use Bernard).

I actually think on reflection that the paragraph was a pretty good paragraph that should just have been included.

I agree that engaging more with the Paul Christiano claims would be good. (Prior to this post coming out I actually had it on my agenda to try and cause some kind of good public debate about that to happen)