LessWrong 2.0 Reader

This is a comprehensive, nuanced, and well-written post. A few questions:

How likely do you think it is that, under a Harris administration, AI labs will successfully lobby Democrats to kill safety-oriented policies, as happened with SB 1047 on the state level? Even if Harris is on net better than Trump this could greatly reduce the expected value of her presidency from an x-risk perspective.

Related to the above, is it fair to say that under either party, there will need to be advocacy/lobbying for safety-focused policies on AI? If so, how do you make tradeoffs between this and the election? i.e. if someone has $x to donate, what percentage should they give to the election vs. other AI safety causes?

How much of your assessment of the difference in AI risk between Harris and Trump is due to the concrete AI policies you expect each of them to push, vs. how much is due to differences in competence and respect for democracy? 

I can't find much information about the Movement Labs quiz and how it helps Harris win. Could you elaborate, privately if needed? If the quiz is simply matching voters with the candidate who best matches their values, is it because it will be distributed to voters who lean Democrat, or does its effectiveness come through a different path?

There is the point about offensive/defensive asymmetry..

how much earlier

Yeah, good question. I don't know really. 

 and does it matter?

I think so, because even if pure AI control follows on from human-AI entity control (which would actually be my prediction), I expect the dynamics of human-AI control to very much lead to and accelerate that eventual pure AI control.

I'm thinking, also, that there is a thing where pure AI entities need to be careful not to 'tip their hat'. What I mean by this is that pure AI entities will need to be careful not to reveal the extent of their capabilities up until a point where they are actually capable of taking control, whereas human-AI entities can kind of go ahead and play the power game and start to build up control without so much concern about this. (To the average voter, this could just look like more of the same.)

One thing I am confused about: especially in cases of developer sandbagging, my intuition is that the mechanisms underlying the underperformance could be very similar to cases of "accidental" sandbagging (ie not sandbagging according to your def). More operationally, your example 1 and example 4 might have the same underlying issue from the perspective of the model itself, and if we want to find technical solutions to those particular examples they might look the same. If that's the case then it's not obvious to me that the "strategic" condition is a useful place to "cut nature at its joints."

Or to say it a different way, what operationally defines the difference between example 1 and 4 is that in ex.1 there is fine-tuning on a different dataset, and on ex.4 the extra dataset is part of the pre-training dataset. The model itself doesn't see the intent of the developer directly, so as far as technical solutions that only depend on the model itself, it's not obvious that the intent of the developer matters.

A developer could intentionally inject noisy and error-prone data into training, but the model would treat that equivalently to the case of where it was in the dataset by mistake.

While I agree that examples are stupid, I am not so sure about the electrical and plumbing. Connecting things to a public water supply that result in contamination from backflow/siphoning is very bad. You also dont want to electrocute a power pole worker who thinks power is off, but your house with DYO solar connection suddenly starts exporting power to grid. If I was insurer, I would take a dim view of unregister plumbing or electricial work in your house because of fire and flood risk. Where I live, so long as you are owner of property , you can do electricial that doesnt involve swithboard or cables coming into switchboard from street. Fair compromise? Drainlaying rules are still extremely strict though. 

Just returning to this -- Thank you for your comment. I think I need to do a bit more epistemological legwork before I can truly give you a response. Suffice it to say for now that I'm not really torn up about the issue and asked more out of boredom/general curiosity, but I see now that my question was asked lazily. 

Perhaps one day soon I'll comment on this with the relevant context/information. 

Given a world of humans, I don't think that libertarian society would be good enough at preventing competing powers from overthrowing it. Because there'd be an unexploitable-equillibrium condition where a government that isn't focused on dominance is weaker than a government more focused on government, it would generally be held by those who have the strongest focus on dominance. Those who desire resources would be better off putting themselves in situations where the dominant powers can become more dominant by giving them/putting them in charge of resources.

Given a world of AIs, I don't think the dominant AI would need a market; it could just handle everything itself.

As I understand it, libertarian paradises are basically fantasies by people who don't like the government, not realistically-achievable outcomes given the political realities.

One experiment I ran to check the locality:

• For $\ell =0,1,\dots ,L$:
  • Ablate the refusal direction at layers $\ell ,\ell +1,\dots ,L$
  • Measure refusal score across harmful prompts

Below is the result for Qwen 1.8B:

You can see that the ablations before layer ~14 don't have much of an impact, nor do the ablations after layer ~17. Running another experiment just ablating the refusal direction at layers 14-17 shows that this is roughly as effective as ablating the refusal direction from all layers.

As for inducing refusal, we did a pretty extreme intervention in the paper - we added the difference-in-means vector to every token position, including generated tokens (although only at a single layer). Hard to say what the issue is without seeing your code - I recommend comparing your intervention to the one we define in the paper (it's implemented in our repo as well).

I applaude the post. I think this is a step in the right direction of trying to consider the whole problem realistically. I think everyone working on alignment should have this written statement of a route to survival and flourishing; it would help us work collectively to improve our currently-vague thinking, and it would help better target the limited efforts we have to devote to alignment research.

My statement would be related but less ambitious. This is what we should do, if the clarity and will existed. I really hope there's a viable route following more limited things we realistically could do.

I'm afraid I find most of your proposed approaches to still fall short of being fully realistic given the inefficiencies of public debate and government decision-making. Fortunately I think there is a route to survival and autonomy that's a less narrow path; I have a draft in progress on my proposed plan, working title "a coordination-free plan for human survival". Getting the US gov't to nationalize AGI labs does sound plausible, but unlikely to happen until they've already produced human-level AGI or are near to doing so.

I think my proposed path is different based on two cruxes about AGI, and probably some others about how government decision-making works. My timelines include fairly short timelines, like the 3 years Aschenbrenner and some other OpenAI insiders hold. And I see general AGI as actually being easier than superhuman tool AI; once real continuous, self-directed learning is added to foundation model agents (it already exists but needs improement), having an agent learn with human help becomes an enormous advantage over human-designed tool AI.

There's much more to say, but I'm out of time so I'll say that and hope to come back with more detail.

(Now, you might think that an AI in charge of e.g. a company could make the big hierarchy work efficiently by just being capable enough to track everything themselves. But at that point, I wouldn't expect to see an hierarchy at all; the AI can just do everything itself and not have multiple agents in the first place. Unlike humans, AIs will not be limited by their number of hands. If there is to be some arrangement involving multiple agents coordinating in the first place, then it shouldn't be possible for one mind to just do everything itself.)

My model probably mostly resembles this situation. Some ~singular AI will maintain a monopoly on violence. Maybe it will use all the resources in the solar system, leaving no space for anyone else. Alternatively (for instance if alignment succeeds), it will leave one or more sources of resources that other agents can use. If the dominant AI fully protects these smaller agents from each other, then they'll handle their basic preferences and mostly withdraw into their own world, ending the hierarchy. If the dominant AI has some preference for who to favor, or leaves some options for aggression/exploitation which don't threaten the dominant AI, then someone is going to win this fight, making the hierarchy repeat fractally down.

Main complication to this model is inertia; if human property rights are preserved well enough, then most resources would start out owned by humans, and it would take some time for the economy to equillibrate to the above.