Takeoff speeds have a huge effect on what it means to work on AI x-risk

I agree with the basic difference you point to between fast- and slow-takeoff worlds, but disagree that it has important strategic implications for the obviousness of takeover risk.

In slow takeoff worlds, many aspects of the alignment problem show up well before AGI goes critical. However, people will by-default train systems to conceal those problems. (This is already happening: RL from human feedback is exactly the sort of strategy which trains systems to conceal problems, and we've seen multiple major orgs embracing it within the past few months.) As a result, AI takeover risk never looks much more obvious than it does now.

Concealed problems look like no problems, so there will in-general be economic incentives to train in ways which conceal problems. The most-successful-looking systems, at any given time, will be systems trained in ways which incentivize hidden problems over visible problems.

I guess it depends on “how fast is fast and how slow is slow”, and what you say is true on the margin, but here's my plea that the type of thinking that says “we want some technical problem to eventually get solved, so we try to solve it” is a super-valuable type of thinking right now even if we were somehow 100% confident in slow takeoff. (This is mostly an abbreviated version of this section [LW · GW].)

1. Differential Technological Development (DTD) seems potentially valuable, but is only viable if we know what paths-to-AGI will be safe & beneficial really far in advance. (DTD could take the form of accelerating one strand of modern ML relative to another—e.g. model-based RL versus self-supervised language models etc.—or it could take the form of differentially developing ML-as-a-whole compared to, I dunno, something else.) Relatedly, suppose (for the sake of argument) that someone finds an ironclad proof that safe prosaic AGI is impossible, and the only path forward is a global ban on prosaic AGI research. It would be way better to find that proof right now than finding it in 5 years, and better in 5 years than 10, etc., and that's true no matter how gradual takeoff is.
2. We don't know how long safety research will take. If takeoff happens over N years, and safety research takes N+1 years, that's bad even if N is large.
   1. Maybe you'll say that almost all of the person-years of safety research will happen during takeoff, and any effort right now is a drop in the ocean compared to that. But I really think wall-clock time is an important ingredient in research progress, not just person-years. (“Nine women can't make a baby in a month.”)
3. We don't just need to figure out the principles for avoiding AGI catastrophic accidents. We also need every actor with a supercomputer to understand and apply these principles. Some ideas take many decades to become widely (let alone universally) accepted—famous examples include evolution and plate tectonics. It takes wall-clock time for arguments to be refined. It takes wall-clock time for evidence to be marshaled. It takes wall-clock time for nice new pedagogical textbooks to be created. And of course, it takes wall-clock time for the stubborn holdouts to die and be replaced by the next generation. :-P

I think this point is really crucial, and I was correct to make it, and it continues to explain a lot of disagreements about AI safety.

I think this point is incredibly important and quite underrated, and safety researchers often do way dumber work because they don't think about it enough.

In particular, in a fast takeoff world, AI takeover risk never looks much more obvious than it does now, and so x-risk-motivated people should be assumed to cause the majority of the research on alignment that happens.

I strongly disagree with that and I don't think it follows from the premise.  I think by most reasonable definitions of alignment it is already the case that most of the research is not done by x-risk motivated people. 

Furthermore, I think it reflects poorly on this community that this sort of sentiment seems to be common. 

In contrast, in a slow takeoff world, many aspects of the AI alignment problems will already have showed up as alignment problems in non-AGI, non-x-risk-causing systems; in that world, there will be lots of industrial work on various aspects of the alignment problem, and so EAs now should think of themselves as trying to look ahead and figure out which margins of the alignment problem aren’t going to be taken care of by default, and try to figure out how to help out there. 

Lets consider the opposite. Imagine you are programming a self driving car, in a simulated environment. You notice it goodhearting your metrics, so you tweak them and try again. You build up a list of 1001 ad hoc patches that makes your self driving car behave reasonably most of the time. 

The object level patches only really apply to self driving cars. They include things like a small intrinsic  preference towards looking at street signs. The meta level strategy of patching it until it works isn't very relevant either. 

Imagine a world with many AI's like this. All with ad hoc kludges of hard coded utility functions. The AI is becoming increasingly economically important and getting close to AGI. Slow takeoff. All the industrial work is useless.

Have you written about your update to slow takeoff?

In contrast, in a slow takeoff world, many aspects of the AI alignment problems will already have showed up as alignment problems in non-AGI, non-x-risk-causing systems; in that world, there will be lots of industrial work on various aspects of the alignment problem, and so EAs now should think of themselves as trying to look ahead and figure out which margins of the alignment problem aren’t going to be taken care of by default, and try to figure out how to help out there.

TLDR: I think an important sub-question is 'how fast is agency takeoff' as opposed to economic/power takeoff in general.

There are a few possible versions of this in slow takeoff which look quite different IMO.

1. Agentic systems show up before the end of the world and industry works to align these systems. Here's a silly version of this:

GPT-n prefers writing romance to anything else. It's not powerful enough to take over the world but it does understand it's situation, what training is etc. And it would take over the world if it could and this is somewhat obvious to industry. In practice it mostly tries to detect when it isn't in training and then steer outputs in a more romantic direction. Industry would like to solve this, but finetuning isn't enough and each time they've (naively) retrained models they just get some other 'quirky' behavior (but at least soft-core romance is better than that AI which always asks for crypto to be sent to various addresses). And adversarial training just results in getting other strange behavior.

Industry works on this problem because it's embarassing and it costs them money to discard 20% of completions as overly romantic. They also foresee the problem getting worse (even if they don't buy x-risk).

2. Not obviously agentic systems have alignment problems, but we don't see obvious, near human level agency until the end of the world. This is slow takeoff world, so these systems are taking over a larger and larger fraction of the economy despite to being very agentic. These alignment issues could be reward hacking or just general difficulty getting language models to follow instructions to the best of their ability (as shows up currently).

I'd claim that in a world which is more centrally scenerio (2), industrial work on the 'alignment problem' might not be very useful for reducing existential risk in the same way that I think that a lot of current 'applied alignment'/instruction following/etc isn't very useful. So, this world goes similarly to fast takeoff in terms of research prioritization. But in something like scenerio (1), industry has to do more useful research and problems are more obvious.

while in the slow takeoff world your choices about research projects are closely related to your sociological predictions about what things will be obvious to whom when.
 

Example?

in a slow takeoff world, many aspects of the AI alignment problems will already have showed up as alignment problems in non-AGI, non-x-risk-causing systems; in that world, there will be lots of industrial work on various aspects of the alignment problem, and so EAs now should think of themselves as trying to look ahead and figure out which margins of the alignment problem aren’t going to be taken care of by default, and try to figure out how to help out there.

I agree with this, and I think it extends beyond what you're describing here. In a slow takeoff world, the aspects of the alignment problem that show up in non-AGI systems will also provide EAs with a lot of information about what's going on, and I think we should try to do things now that will help us to notice those aspects and act appropriately. (I'm not sure what this looks like; maybe we want to build relationships with whoever will be building these systems, or maybe we want to develop methods for figuring things out and fixing problems that are likely to generalize.)

I support that thinking-on-the-margin alignment research version is crucial and it is one of the biggest areas of opportunity to increase the probability of success. Based on the seemingly current low probability, possibly at least it is worth trying. 

In the general public context, one of the premises is about if they could have benefit to the problem. In my intuition it really seems that AI alignment should be more debated in Universities and Technology. The current lack of awareness is concerning and unbelievable. 

We should evaluate more the outcomes of raising awareness. Taking into account all the options: a general(public), partial(experts) awareness strategy and the spectrum and variables in between. It seems that current Safe Alignment leaders have high motives to not focus enough on the expansion of awareness or even to not take the strategy even as possibly useful. I believe this motives should not be fixed but be more debated and not determined as too hard to implement.

We can't assume that if someone is capable to solve Safe Alignment, he/she/they would also be aware of the problem. It seems probable that if someone is capable of solving safe alignment, he/she/they currently don't understand the true magnitude of the problem. In that probable case, a needed step on the success path is in he/she/they understanding the problem. And we can be crucial with that. I understand that with this strategy, as in many safe alignment strategies, the probability of it reducing instead of increasing our success must be highly evaluated.

In the current Alignment research context, possibly there is also positive opportunity from taking more thinking-on-the-margin approaches. The impact of present and future AI systems to AGI/Safe Alignment is very likely of very high importance, and more so compared to its current focus. Because these systems are very likely to shorten the timelines("how much?" is important and currently ambiguous). Seems that we are not evaluating enough the probable crucial impact of current deep models to the problem, but I'm glad the idea is growing. 

(paulfchristiano, 2022) states "AI systems could have comparative disadvantage at alignment relative to causing trouble, so that AI systems are catastrophically risky before they solve alignment." I support that is one of the most important issues. Because, if AI systems are capable of improving safe AI alignment research, they will very likely be even more capable of improving non-safe/default AI alignment and probably Superintelligence creation research. This means that the probable most crucial technology in the Superintelligence birth lowers the probability of safe alignment. So two crucial questions are: How to fight this? and more essentially: How and how much can the current and near future AI Systems improve AGI creation?.

Now I will propose a polemic AI Safe Alignment thinking-on-the-margin tactic. The (I argue highly) probable ideation of new/different/better AI Safe Alignment strategies by AI Safe Alignment researchers from taking advantage of Stimulants and Hallucinogens. We definitely are in a situation where we must take any advantage we can. Non-ordinary states of consciousnesses are very highly worth trying because of the almost none risks involved. (Also with nootropics, but I'm almost not familiar with them).

Finally I will share what I believe currently should be the most important issue on all versions of alignment research and is on top of all previous ideas: If trying to Safely Align will almost certainly not solve our x-risk as EY states in "Miri announces new "Death with dignity strategy"". Then what it will have achieved is only higher s-risk probabilities. (THANK YOU for the Info hazards T.T). So one option is to aim to shorten the x-risk timeline if that reduces the probability of s-risks. Helping to build the Superintelligence asap.
Or to shift all the strategy to lowering s-risks. This is specially and highly relevant to us because we have a higher probability of s-risk (thanks e.e). So we should focus on the issues that increased our s-risk probabilities.