A Rocket–Interpretability Analogy

I don't think anyone has, to date, used interpretability to make any major counterfactual contribution to capabilities. I would not rely on papers introducing a new technique to be the main piece of evidence as to whether the technique is actually good at all. (I would certainly not rely on news articles about papers - they are basically noise.)

I think you should take into account the fact that before there are really good concrete capabilities results, the process that different labs use to decide what to invest in is highly contingent on a bunch of high variance things. Like, what kinds of research directions appeal to research leadership, or whether there happen to be good ICs excited to work on that direction around and not tied down to any other project. 

I don't think you should be that surprised by interpretability being more popular than other areas of alignment. Certainly I think incentives towards capabilities is a small fraction of why it's popular and funded etc (if anything, its non-usefulness for capabilities to date may count against it). Rather, I think it's popular because it's an area where you can actually get traction and do well-scoped projects and have a tight feedback loop. This is not true of the majority of alignment research directions that actually could help with aligning AGI/ASI, and correspondingly those directions are utterly soul grinding to work on. 

One could very reasonably argue that more people should be figuring out how to work on the low traction, ill-scoped, shitty feedback loop research problems, and that the field is looking under the streetlight for the keys. I make this argument a lot. But I think you shouldn't need to postulate some kind of nefarious capabilities incentive influence to explain it.

I'm sympathetic to 'a high fraction of "alignment/safety" work done at AI companies is done due to commercial incentives and has negligible effect on AI takeover risk (or at least much smaller effects than work which isn't influenced by commercial incentives)'.

I also think a decent number of ostensibly AI x-risk focused people end up being influenced by commercial incentives sometimes knowingly and directly (my work will go into prod if it is useful and this will be good for XYZ reason; my work will get more support if it is useful; it is good if the AI company I work for is more successful/powerful, so I will do work which is commercially useful) and sometimes unknowingly or indirectly (a direction gets more organizational support because of usefulness; people are misled into thinking something is more x-risk-helpful than it actually is).

(And a bunch of originally AI x-risk focused people end up working on things which they would agree aren't directly useful for mitigating x-risk, but have some more complex story.)

I also think AI companies generally are a bad epistemic environment for x-risk safety work for various reasons.

However, I'm quite skeptical that (mechanistic) interpretability research in particularly gets much more funding due to it directly being a good commercial bet (as in, it is worth it because it ends up directly being commercially useful). And, my guess is that alignment/safety people at AI companies which are ostensibly focused on x-risk/AI takeover prevention are less than 1/2 funded via the directly commercial case. (I won't justify this here, but I think this due to a combination of personal experience and thinking about what these teams tend to work on.)

(That said, I think putting some effort on (mech) interp (or something similar) might end up being a decent commercial bet via direct usage, though I'm skeptical.)

I think there are some adjacent reasons alignment/safety work might be funded/encouraged at AI companies beyond direct commercial usage:

• Alignment/safety teams or work might be subsidized via being good internal/external PR for AI companies. As in good PR to all of: the employees who care about safety, the AI-safety-adjacent community who you recruit from, and the broader public. I think probably most of this effect is on the "appeasing internal/external people who might care about safety" rather than the broader public.
• Having an internal safety team might help reduce/avoid regulatory cost for your company or the industry. Both via helping in compliance and via reducing the burden.

I’m not an expert and I’m not sure it matters much for your point, but: Yes there were surely important synergies between NASA activities and the military ballistic missile programs in the 1960s, but I don’t think it’s correct to suggest that most NASA activities was stuff that would have to be done for the ballistic missile program anyway. It might actually be a pretty small fraction. For example, less than half the Apollo budget was for launch vehicles; they spent a similar amount on spacecraft, which are not particularly transferable to nukes. And even for the launch vehicles, it seems that NASA tended to start with existing military rocket designs and modify them, rather than the other way around.

I would guess that the main synergy was more indirect: helping improve the consistency of work, economies of scale, defraying overhead costs, etc., for the personnel and contractors and so on.

The field of alignment seems to be increasingly dominated by interpretability. (and obedience^[2])

I agree w.r.t potential downstream externalities of interpretability. However, my view from speaking to a fair number of those who join the field to work on interpretability is that the upstream cause is slightly different. Interpretability is easier to understand and presents "cleaner" projects than most others in the field. It's also much more accessible and less weird to say, an academic, or the kind of person who hasn't spent a lot of time thinking about alignment.

All of these are likely correlated with having downstream capabilities applications, but the exact mechanism doesn't look like "people recognizing that this has huge profit potential and therefore growing much faster than other sub-fields".

Some thoughts on the post as I want to collect everything.

For this specifically:

The field of alignment seems to be increasingly dominated by interpretability. (and obedience^[2])

Obedience is quite literally the classical version of the AI alignment problem, or at least very entangled with it, and interpretability IMO got a lot of the initial boost from Distill, combined with it being a relatively clean problem formulation that makes alignment more scalable.

On footnote 3, I have to point out that the postulated sharp left turn in humans compared to chimps doesn't nearly have as much evidence as the model in which human success is basically attributable to culture, where we are very good at imitating others without a true causal model as well as distilling cultural knowledge down, combined with us having essentially a very good bodyplan for tool use and us being able to dissipate heat better to scale up algorithms well.

I don't totally like Henrich's book The Secret of Our Success, but I do think that at least some of the results like chimps essentially equaling humans on IQ and only losing hard when social competence was required is a surprising fact under the sharp left turn view.

While the science of human and animal brains is incomplete at this juncture, we are starting to realize that animals do in fact have much more intelligence than past people realized, and in particular for mammals, their algorithmic cores are also fairly similar in genetic space, and the only real differentiator at this point for humans is just the fact that our cultural knowledge sparked by population growth and language booms every generation.

Imagine A GPT that predicts random chunks of the internet.

Sometimes it produces poems. Sometimes deranged rants. Sometimes all sorts of things. It wanders erratically around a large latent space of behaviours.

This is the unmasked shogolith, green slimey skin showing but inner workings still hidden.

Now perform some change that mostly pins down the latent space to "helpful corporate assistant".  This is applying the smiley face mask.

In some sense, all the dangerous capabilities the corporate assistant were in the original model. Dangerous capabilities haven't been removed either, but some capabilities are a bit easier to access without careful prompting, and other capabilites are harder to access. 

What ChatGPT currently has is a form of low quality pseudo-alignment.

What would long term success look like using nothing but this pseudo-alignment. It would look like a chatbot far smarter than any current ones, which mostly did nice things, so long as you didn't put in any weird prompts. 

Now If corrigibility is a broad basin, this might well be enough to hit it. The basin of corrigibility means that the AI might have bugs, but at the very least, you can turn the AI off and edit the code. Ideally you can ask the AI for help fixing it's own bugs.  Sure the first AI is far from perfect. But perhaps the flaws disappear under self rewriting + competent human advice. 

I personally don't think that working on better capabilities, and working on the alignment problem are two distinct, separate problems. For example, if you're able to create better, more precise control mechanisms and intervention techniques, you can better align human & AI intent.  Alignment feels as much of a technical, control interface problem, than merely a question of "what should we align to?".

This a very relevant discussion which should be the backbone of the decision making of anyone being part of the general effort towards AI alignment. 

It is important to point out however that this is maybe just an instance of a more general problem which is related to the fact that in general any form of power or knowledge can be used for good or bad purposes, and advancing power and knowledge becomes always a double edged sword.

Doesn't seem to me like there is an escape to the moral variance that humans experience as part of their natural + developed proclivities. 

The only chance we have against such great power and knowledge as AI enables is to really enlighten with clarity how is it that these technical pieces can come together for one edge or another. 

Bringing the problem to light is our best chance.