LessWrong 2.0 Reader

Curious what you think of these arguments [LW · GW], which I think can be seen as objections to the strategy stealing assumption in this setting, instead arguing that it's difficult for capital owners to maintain their share of capital ownership as the economy grows and technology changes. 

I wrote a related post.

I like this post but I think it misses / barely covers two of the most important cases for optimism.

1. Detail of specification

Frontier LLMs have a very good understanding of humans, and seem to model them as well as or even better than other humans. I recall seeing repeated reports of Claude understanding its interlocutor faster than they thought was possible, as if it just "gets" them, e.g. from one Reddit thread I quickly found:

• "sometimes, when i’m tired, i type some lousy prompts, full of typos, incomplete info etc, but Claude still gets me, on a deep fucking level"
• "The ability of how Claude AI capture your intentions behind your questions is truly remarkable. Sometimes perhaps you're being vague or something, but it will still get you."
• "even with new chats, it still fills in the gaps and understands my intention"

LLMs have presumably been trained on:

• millions of anecdotes from the internet, including how the author felt, other users' reactions and commentary, etc.
• case law: how did humans chosen for their wisdom (judges) determine what was right and wrong
• thousands of philosophy books
• Lesswrong / Alignment Forum, with extensive debate on what would be right and wrong for AIs to do

There are also techniques like deliberative alignment, which includes an explicit specification for how AIs should behave. I don't think the model spec is currently detailed enough but I assume OpenAI intend to actively update it.

Compare this to the "specification" humans are given by your Ev character: some basic desires for food, comfort, etc. Our desires are very crude, confusing, and inconsistent; and only very roughly correlate with IGF. It's hard to emphasize enough how much more detailed is the specification that we present to AI models.

2. (Somewhat) Gradual Scaling

Toby Ord estimates that pretraining "compute required scales as the 20th power of the desired accuracy". He estimates that inference scaling is even more expensive, requiring exponentially more compute just to make constant progress. Both of these trends suggest that, even with large investments, performance will increase slowly from hardware alone (this relies on the assumption that hardware performance / $ is increasing slowly, which seems empirically justified). Progress could be faster if big algorithmic improvements are found. In particular I want to call out that recursive-self improvement (especially without a human in the loop) could blow up this argument (which is why I wish it was banned). Still, I'm overall optimistic that capabilities will scale fairly smoothly / predictably.

With (1) and (2) combined, we're able to gain some experience with each successive generation of models, and add anything we find is missing from the training dataset / model spec, without taking any leaps that are too big / dangerous. I don't want to suggest that the scaling up while maintaining alignment process will definitely succeed, just that we should update towards the optimistic view based on these arguments.

> AI Doomer coalition abandoned the name “AI safety” and rebranded itself to “AI alignment.”

Seems wrong

(Why do you believe this? I think this is a reasonable gloss of what happened around 2015-2016. I was part of many of those conversations, as I was also part of many of the conversations in which me and others gave up on "AI Alignment" as a thing that could meaningfully describe efforts around existential risk reduction)

There are a lot of good reasons to believe that stated human preferences correspond to real human preferences.

Can you name a few?  I know of one: I assume that there's some similarity with me in because of similar organic structures doing the preferring.  That IS a good reason, but it's not universally compelling or unassailable.

Actually, can you define 'real preferences' in some way that could be falsifiable for humans and observable for AIs?

"Surely the AIs can be trained to say "I want hugs" or "I don't want hugs," just as easily, no?"

Just as easily as humans, I'm sure.

I left a comment! [LW(p) · GW(p)]

Interesting work!

The possibility of collusion using some secret signal is definitely something we have thought a lot about. It is an important empirical question how good powerful AIs will be at this, and it is plausible they will be too good at it to make untrusted monitoring viable.

Our paper did not contain measurement of the ability of LLMs to come up with signals on their own (the ones studied in the paper were hand-crafted). Therefore, I welcome work like yours that try to measure how good LLMs are at inserting collusion signals!

To create useful synthetic inputs, either the blue team must be able to match the generator's style well enough to consistently fool the checker, or it must be able to redact the generator's output to the point where the checker has no identifying details to distinguish inputs. The former seems impractical, since faking the inputs at that level of verisimilitude would require the blue team to have significant understanding of the AI's inner workings, at which point there are probably more directly usable alignment strategies. The latter seems difficult, as by the protocol's definition, the AI is working on problems/solutions that humans cannot easily understand, but effective redaction requires humans to understand the content of the solution to some extent.

Maybe the core misunderstanding about the viability of untrusted monitoring is what untrusted AIs we are targeting. The reason why we use "human labor" that is roughly untrusted-model-level in the paper is because we expect that this is one of the most important regimes (since it is when AIs start to be powerful enough to drastically speed up safety work). More details here [LW · GW].

Therefore, redaction/paraphrasing seems plausibly enough to make reliable collusion too hard for these AIs, just like it was in our toy setting against the best collusion signals Ryan Greenblatt thought about and which GPT-4 could easily implement (which are probably more robust against redaction/paraphrasing than the ones GPT-4o found in your experiments). But maybe future AIs will become way better at inserting secrets, so I think the jury is definitely still out on this question.

I like reading outsider accounts of things I'm involved in / things I care about.
 

Just for context for some not aware - The author, Ben Landau-Taylor, has been in the rationalist-extended community for some time now. This post is written on Palladium Magazine, which I believe basically is part of Samo Burja's setup. I think both used to be around Leverage Research and some other rationality/EA orgs.

Ben and Samo have been working on behalf of Palladium and similar for a while now.

My quick read is that this article is analogous to similar takes they've written about/discussed before, which is not too much of a surprise.

I disagree with a lot of their intuitions, but at the same time, I'm happy to have more voices discuss some of these topics. 

All this to say, while these people aren't exactly part of the scene now, they're much closer to it than what many might imagine as "outsider accounts."

There are a lot of good reasons to believe that stated human preferences correspond to real human preferences. There are no good reasons that I know of to believe that any stated AI preference corresponds to any real AI preference.

"Surely the AIs can be trained to say "I want hugs" or "I don't want hugs," just as easily, no?"

This all makes a lot of sense to me especially on ignorance not being an excuse or reason to disregard AI welfare, but I don't think that the creation of stated preferences in humans and stated preferences in AI are analogous.

Stated preferences can be selected for in humans because they lead to certain outcomes. Baby cries, baby gets milk, baby survives. I don't think there's an analogous connection in AIs. 

When the AI says it wants hugs, and you say that it "could represent a deeper want for connection, warmth, or anything else that receiving hugs would represent," that does not compute for me at all.

Connection and warmth, like milk, are stated preferences selected for because they cause survival.