LessWrong 2.0 Reader

In terms of proposing and discussing AI Alignment strategies, I feel like a few individuals have been dominating the LessWrong conversation recently.
 
I've seen a whole lot from John Wentworth and the Redwood team.

After that, it seems to get messier. 

There are several individuals or small groups with their own very unique takes. Matthew Barnett, Davidad, Jesse Hoogland, etc. I think these groups often have very singular visions that they work on, that few others have much buy-in with. 

Groups like the Deepmind and Anthropic safety teams seem hesitant to write much about or discuss big-picture strategy. My impression is that specific researchers are working typically working on fairly narrow agendas, and that the leaders of these orgs don't have the most coherent strategies. There's one big problem that it's very difficult to be honest and interesting about big-picture AI strategy without saying things that would be bad for a major organization to say.  

Most policy people seem focused on policy details. The funders (OP?) seem pretty quiet. 

I think there's occasionally some neat papers or posts that come from AI Policy or groups like Convergence research. But these also don't seem to be a big part of the conversation I see - like the authors are pretty segmented, and other LessWrong readers and AI safety people don't pay much attention to their work. 

(someone else tagged this as a longfom review. I did originally plan to include a second half of this post that would have felt like it qualified as such, but I think the current form is ranty enough and light on "self review" enough that I don't feel good about it taking up a slot in the Frontpage Review Widget)

Concrete (if extreme) story:

World A:

Invent a version of "belief propagation" which works well for LLMs. This offers a practical way to ensure that if an LLM seems to know something in one context, it can & will fluently invoke the same knowledge in almost all appropriate contexts.

Keep the information secret in order to avoid pushing capabilities forward.

Two years later, GPT7 comes up with superhumanly-convincing safety measures XYZ. These inadequate standards become the dominant safety paradigm. At this point if you try to publish "belief propagation" it gets drowned out in the noise anyway.

Some relatively short time later, there are no humans.

World B:

Invent LLM "belief propagation" and publish it. It is good enough (by assumption) to be the new paradigm for reasoning models, supplanting current reinforcement-centric approaches.

Two years later, GPT7 is assessing its safety proposals realistically instead of convincingly arguing for them. Belief propagation allows AI to facilitate a highly functional "marketplace of ideas" where the actually-good arguments tend to win out far more often than the bad arguments. AI progress is overall faster, but significantly safer.

(This story of course assumes that "belief propagation" is an unrealistically amazing insight; still, this points in the direction I'm getting at)

Wonderful to get more numbers on this! 

These examples seem to contradict note 2 where D/N falls for larger C. Now I'm not sure what the trend should be.

It feels like you could derive a rule of thumb based on the loss and the entropy of the dataset e.g. "If my model starts at a loss of 4 bits/token and the asymptote is 2 bits/token, I need X tokens of data to fully specify a model with Y bits stored in the parameters."

To be clear, I think it's okay to be more political. What I don't want is "unreflectively partisanly political." (Maybe try DMing what you had in mind to me and I'll see if it feels productive)

Sample questions I would ask if I was a security auditor, which I'm not.

Does Elez have anytime admin access, or for approved time blocks for specific tasks where there is no non-admin alternative? Is his use of the system while using admin rights logged to a separate tamper proof record? What data egress controls are in place on the workstation he uses to remotely access the system as an admin? Is Elez security screened, not a spy, not vulnerable to blackmail? Is Elez trained on secure practices?

Depending on the answers this could be done in a way that would pass an audit with no concerns, or it could be illegal, or something in between.

Avoiding further commentary that would be more political.

Bureaucracy and red tape are often written into law by Congress based on actual and theoretical damage to people and systems. To bypass them is to bypass all protections. The risk is enormous.

If your conclusion is that we don't know how to do value alignment, I and I think most alignment thinkers would agree with you. If the conclusion is that AGI is useless, I don't think it is at all. There are a lot of other goals you could give it beyond directly doing what humanity as a whole wants in any sense. Some are taking instructions from some (hopefully trustworthy) humans, and another is following some elaborate set of rules to give humans more freedoms and opportunities to go on deciding what they want as history unfolds.

I agree that the values future humans would adopt can only be reached through a process of societal progression. People have expressed this idea by saying that human values are path-dependent.

So, if our goal were simply to observe the values that emerge naturally from human efforts alone, an AGI would indeed be nothing more than a paperweight. However, the values humanity develops with the assistance of an AGI aren’t necessarily worse—if anything, I’d suspect they’d be better.

The world as it stands is highly competitive and often harsh. If we had external help that allowed us to focus more on what we truly want—like eliminating premature death from cancer or accidents, or accelerating technological progress for creative and meaningful projects—we’d arrive at a very different future. But I don’t think that future would be worse; in fact, I suspect it would be significantly better. It would be less shaped by competition, including warfare, and less constrained by the tragedy of involuntary death.

An AGI that simply fulfills human desires without making rigid commitments about the long-term future seems like it would be a net positive—potentially a massive one.

I think the paradox you mention is generally accepted to be an unsolved problem with value alignment - we don't know how to ask for what we will want, since what we will want could be a bunch of different things depending on the path, and we don't know what path to what values and what world would in any sense be best.

This is commonly listed as one of the big difficulties with alignment. I think it only sort of is. I think we really want the future to remain in human hands for at least some time. The concept of a "long reflection" is one term people use to describe this.

In the meantime, you either have an AGI whose goal is to facilitate that long reflection, according to some criteria, or you have an AGI that takes instructions from some human(s) you trust.

This is one reason I think Instruction-following AGI is easier and more likely than value aligned AGI [LW · GW]. The reasoning there is pretty different from the conceptual difficulties with value alignment you mention here; it's just easier to specify "do what this person meant by what he tells you" than what we mean by human values - and you get to change your mind by instructing it to shut down. Even if that worked we'd have to worry: If we solve alignment, do we die anyway? [LW · GW] because having a bunch of different humans in charge of a bunch of different AGIs could produce severe conflicts. But that whole logic is separable from the initial puzzle you posed.

Hmmm. I'm not exactly sure what the disconnect is, but I don't think you're quite understanding my model.

I think anti-slop research is very probably dual-use. I expect it to accelerate capabilities. However, I think attempting to put "capabilities" and "safety" on the same scale and maximize differential progress of safety over capabilities is an oversimplistic model which doesn't capture some important dynamics.

There is not really a precise "finish line". Rather, we can point to various important events. The extinction of all humans lies down a path where many mistakes (of varying sorts and magnitudes) were made earlier.

Anti-slop AI helps everybody make less mistakes. Sloppy AI convinces lots of people to make more mistakes.

My assumption is that frontier labs are racing ahead anyway. The idea is that we'd rather they race ahead with a less-sloppy approach. 

Imagine an incautious teenager who is running around all the time and liable to run off a cliff. You expect that if they run off a cliff, they die -- at this rate you expect such a thing to happen sooner or later. You can give them magic sneakers that allow them to run faster, but also improves their reaction time, their perception of obstacles, and even their wisdom. Do you give the kid the shoes?

It's a tough call. Giving the kid the shoes might make them run off a cliff even faster than they otherwise would. It could also allow them to stop just short of the cliff when they otherwise wouldn't.

I think if you value increased P(they survive to adulthood) over increased E(time they spend as a teenager), you give them the shoes. IE, withholding the shoes values short-term over long-term. If you think there's no chance of survival to adulthood either way, you don't hand over the shoes.

Here are some important-seeming properties to illustrate what I mean:

1. Robustness of value-alignment: Modern LLMs can display a relatively high degree of competence when explicitly reasoning about human morality. In order for it to matter for RSI, however, those concepts need to also appropriately come into play when reasoning about seemingly unrelated things, such as programming. The continued ease of jailbreaking AIs serves to illustrate this property failing (although solving jailbreaking would not necessarily get at the whole property I am pointing at).
2. Propagation of beliefs: When the AI knows something, it should know it in a way which integrates well with everything else it knows, rather than easily displaying the knowledge in one context while seeming to forget it in another.
3. Preference for reasons over rationalizations: An AI should be ready and eager to correct its mistakes, rather than rationalizing its wrong answers. It should be truth-seeking, following thoughts where they lead instead of planning ahead to justify specific answers. It should prefer to valid proof steps over arriving at an answer when the two conflict.
4. Knowing the limits of its knowledge: Metacognitive awareness of what it knows and what it doesn't know, appropriately brought to bear in specific situations. The current AI paradigm just has one big text-completion probability distribution, so there's not a natural way for it to distinguish between uncertainty about the underlying facts and uncertainty about what to say next -- hence we get hallucinations.

All of this is more-or-less a version of the metaphilosophy [? · GW] research agenda, framed in terms of current events in AI.

I very much like the concreteness here. 

I consider these sorts of things just fundamental epistemic problems, or basic skills that good researchers should have. All superforecasters should be very familiar with issues 2-4, and most probably couldn't define metaphilosophy. I don't see the need to be fancy about it.

On that note, I'll hypothesize that if we were to make benchmarks for any of these items, it would be fairly doable to make AIs that do better than humans on them, then later we could achieve greater and greater measures. I have a hard time imagining tests here that I would feel confident would not get beaten, if there was sufficient money on the line, in the next year or two.