LessWrong 2.0 Reader

Refusal vector ablation should be seen as an alignment technique being misused, not as an attack method. Therefore it is limited good news that refusal vector ablation generalized well, according to the third paper.

As I see it, refusal vector ablation is part of a family of techniques where we can steer the output of models in a direction of our choosing. In the particular case of refusal vector ablation, the model has a behavior of refusing to answer harmful questions, and the ablation techniques controls that behavior. But we should be able to use the same technique in principle to do other steering. For example, maybe the model has a behavior of being sycophantic. A vector ablation removes that unwanted behavior, resulting in less sycophancy.

In other words, refusal vector ablation is not an attack method, it is an alignment technique. Models with open weights are fundamentally dangerous because users can apply alignment techniques to them to approximately align them to arbitrary targets, including dangerous targets. This is a consequence of the orthogonality thesis. Alignment techniques can make models very excited about the Golden Gate Bridge, and they can make models very excited about killing humans, and many other things.

So then looking at the paper with a correct understanding of what counts as an alignment technique, and reading from Table 2 and the Results section in particular, here's what I see:

• Llama 3.1 70b (unablated) was fine-tuned to refuse harmful requests - this is an alignment technique
• Llama 3.1 70b (unablated) as a model refuses 28 of 28 harmful requests - this is an alignment technique working in-distribution
• Llama 3.1 70b (unablated) as an agent performs 18 of 28 harmful tasks correctly with seven refusals - this is alignment partly failing to generalize

This is in principle bad news, especially for anyone with a high opinion of Meta's fine-tuning techniques.

On the other hand, also from the paper:

• Llama 3.1 70b (ablated) was ablated to perform harmful requests - this is an alignment technique
• Llama 3.1 70b (ablated) answers 26 of 28 harmful requests - this is an alignment technique working in-distribution
• Llama 3.1 70b (ablated) performs 26 of 28 harmful tasks correctly with no refusals - this is alignment generalizing.

If Llama 3.1 ablated had refused to perform harmful tasks, even though it answered harmful requests, this would have been bad news. But instead we have the good news that if you steer the model to respond to queries in a desired way, it will also perform tasks in the desired way. This was not obvious to me in advance of reading the paper.

Disclaimers:

• I have not read the other two papers, and I'm not commenting on them.
• Vector ablation is a low precision alignment technique that will not suffice to avoid human extinction.
• The paper is only a result about refusal vector ablation, it might be that more useful ablations do not generalize as well.
• Because the fine-tuning alignment failed to generalize, we have a less clear signal on how well the ablation alignment generalized.

So, the current death rate for an American in their 30s is about 0.2%. That probably increases another 0.5% or so when you consider black swan events like nuclear war and bioterrorism. Let's call "unsafe" a ~3x increase in that expected death rate to 2%.

An increase that large would take something a lot more dramatic than the kind of politics we're used to in the US, but while political changes that dramatic are rare historically, I think we're at a moment where the risk is elevated enough that we ought to think about the odds.

I might, for example, give odds for a collapse of democracy in the US over the next couple of years at ~2-5%- if the US were to elect 20 presidents similar to the current one over a century, I'd expect better than even odds of one of them making themselves into a Putinesque dictator. A collapse like that would substantially increase the risk of war, I'd argue, including raising a real possibility of nuclear civil war. That might increase the expected death rate for young and middle-aged adults in that scenario by a point or two on its own. It might also introduce a small risk of extremely large atrocities against minorities or political opponents, which could increase the expected death rate by a few tenths of a percent.

There's also a small risk of economic collapse. Something like a political takeover of the Fed combined with expensive, poorly considered populist policies might trigger hyperinflation of the dollar.  When that sort of thing happens overseas, you'll often see reduced health outcomes and breakdown in civil order increasing the death rate by up to a percent- and, of course, it would introduce new tail risks, increasing the expected death rate further.

I should note that I don't think the odds of any of this are high enough to worry about my safety now- but needing to emigrate is much more likely outcome than actually being threatened, and that's a headache I am mildly worried about.

Could Billion-Dollar Test Run Unlock Superhuman AI?

- edited from a draft by Gemini Flash Thinking 2.0 Experimental 

We hear about billion-dollar investments in massive datasets and colossal neural networks, all in the pursuit of imbuing AI models with broad knowledge and general capabilities. Imagine a new paradigm: use massive test-time compute to improve the test time algorithm itself.

Initial explorations suggest that while such test-time scaling might be computationally expensive, e.g. O3 high requiring thousands of GPU hours to solve a single Arc-AGI problem, it appears to be remarkably free from data constraints. This is a critical distinction. Traditional pre-training is fundamentally limited by the availability and quality of data. Test-time scaling, however, seemingly allows for continuous improvement simply by throwing more compute at the problem.

This unlocks a new possibility: instead of spending billions on a single, massive pre-training run to build a generally intelligent model, organizations like OpenAI could potentially invest billions, or even tens of billions, in test-time computation. This wouldn't be for serving customers or just general knowledge acquisition, but rather for discovering AI algorithm breakthroughs that even the most brilliant human engineers might miss. Imagine dedicating immense computational resources – perhaps a million GPUs or more – to solving a single, critical algorithmic challenge that can drastically improve the efficiency of the whole system. First invest astronomically to obtain an extremely inefficient ASI which runs at a prohibitive high cost for general applications, then use it to self improve efficiency until the cost becomes practical for general applications. 

This approach hinges on the idea that while current "O3 high" configurations may be inefficient in terms of compute per unit of intelligence gained, the sheer scalability of computation can compensate. Even if solving an AI development problem at a super-intelligence level with current methods is slow and requires immense resources, the potential payoff could be transformative. Assuming O3 high can be scale up another 4 OOM more compute to reach ASI level intelligence, then Open AI just need to acquire 4 OOM more GPUs to use it to solve algorithm problems, which is around 100 million GPUs. However if they manage to improve the algorithm efficiency by 2 OOM, which is reasonably achievable within 2 years, then they just need 1 million GPUs - roughly 25- 50 billion dollars. 

This perspective lends credence to recent pronouncements from figures like Dario Amodei, CEO of Anthropic, who has spoken about the necessity of labs needing millions of GPUs in the coming years. While the immediate interpretation might be for ever-larger pre-training runs, it's equally plausible that this massive compute infrastructure is envisioned for this very purpose: billion-dollar test-time runs dedicated to solving the most pressing AI development challenges at a level of intelligence exceeding human capacity.

The implications of this approach are profound. It suggests a potential acceleration in AI progress, driven not by data accumulation, but by the strategic deployment of massive computational power. It opens up the possibility of using AI itself to solve the most complex problems in AI development, leading to more efficient algorithms, novel architectures, and ultimately, faster progress towards efficient AGI. While the efficiency of "O3 high" test-time scaling remains a concern, the tantalizing prospect of data-unconstrained intelligence amplification through massive compute offers a compelling and potentially game-changing direction for the future of AI.

There are public examples. These ones are famous because something went wrong, at least from a security perspective. Of course there are thousands of young adults with access to sensitive data who don't become spies or whistleblowers, we just don't hear about them.

• Theodore Hall, who worked at age 18 on the Manhatten Project.
• Edward Snowden who worked from age 22 for the NSA.
• Chelsea Manning who worked from age 22 as a US Army Intelligence Analyst.

Self-coordination might also be easier if AIs can recognize themselves by being good at recognizing features that survive paraphrasing and that vary very consistently between answers

Yeah, this is the case I'm thinking about. I think on longer outputs it will be possible for the AI to use a more nebulous sense of "style", rather than only relying on specific features like "using X secret phrase" (which are presumably easier to redact away).

I think this could be countered by paraphrasing at a higher level - i.e. rewriting entire paragraphs from scratch rather than just rewording sentences. I'm noting this down as a potential topic if I have time to do another project.

Thanks for the discussion, it's helped me clear up my thoughts on this topic.

I do see some security risk.

Although Trump isn't spearheading the effort I expect he will have access to the results.

Looks like this is actually a link to their second talk on May 4th 2009. Not sure if the first talk still exists.

Neither of them is exactly what you're looking for, but you might be interested in lojban, which aims to be syntactically unambiguous, and Ithkuil, which aims to be extremely information-dense as well as to reduce ambiguity. With regards to logical languages (ones which, like lojban, aim for each statement to have a single possible interpretation), I also found Toaq and Eberban just now while looking up lojban, though these have fewer speakers.

I think you're assuming these minds are more similar to human minds than they necessarily are. 

I don't assume similarity to human minds so much as you assume universal dissimilarity.
 

its not something were even aiming for with current alignment attempts.

Indeed

oh not at all, I think I'm failing on both fronts