The results:

• $5k for this post [LW · GW] by Wei_Dai, and the preceding/following discussion, some points about the difficulty of learning corrigibility in small pieces.
• $3k for Point 1 from this comment [LW(p) · GW(p)] by eric_langlois, an intuition pump for why security amplification is likely to be more difficult than you might think.
• $2k for this post [LW · GW] by William_S, which clearly explains a consideration / design constraint that would make people less optimistic about my scheme. (This fits under "summarizing/clarifying" rather than novel observation.)

Thanks to everyone who submitted a criticism! Overall I found this process useful for clarifying my own thinking (and highlighting places where I could make it easier to engage with my research by communicating more clearly).

Point 1: Meta-Execution and Security Amplification

I have a comment on the specific difficulty of meta-execution as an approach to security amplification. I believe that while the framework limits the "corruptibility" of the individual agents, the system as a whole is still quite vulnerable to adversarial inputs.

As far as I can tell, the meta-execution framework is Turing complete. You could store the tape contents within one pointer and the head location in another, or there's probably a more direct analogy with lambda calculus. And by Turing complete I mean that there exists some meta-execution agent that, when given any (suitably encoded) description of a Turing machine as input, executes that Turing machine and returns its output.

Now, just because the meta-execution framework is Turing complete, this doesn't mean that any particular agent created in this manner is Turing complete. If our agents were in practice Turing complete, I feel like that would defeat the security-amplification purpose of meta-execution. Maybe the individual nodes cannot be corrupted by the limited input they see, but the system as a whole could be made to perform arbitrary computation and produce arbitrary output on specific inputs. The result of "interpret the input as a Turing machine and run it" is probably not the correct or aligned response to those inputs.

Unfortunately, it seems empirically the case that computation systems become Turing complete very easily. Some examples:

• Accidentally Turing Complete
• mov is Turing Complete
• Return oriented Programming
• I would argue that even humans are approximately Turing complete (there are probably input sequences that would cause a person to carry out an arbitrary computation to the best of their abilities). I assume this contributes to the desire for information limitation in meta-execution in the first place.

In particular, return oriented programming is interesting as an adversarial attack on pre-written programs by taking advantage of the fact that a limited control over execution flow in the presence of existing code often forms a Turing complete system, despite the attacker having no control over the existing code.

So I suspect that any meta-computation agent that is practically useful for answering general queries is likely to be Turing complete, and that it will be difficult to avoid Turing completeness (up to a resource limit, which doesn't help the arbitrary code execution problem).

An addition to this argument thanks to William Saunders: We might end up having to accept that our agent will be Turing complete and hope that the malicious inputs are hard to find or work with low probability. But in that case, limiting the amount of information seen by individual nodes may make it harder for the system to detect and avoid these inputs. So what you gain in per-node security you lose in overall system security.

Point 2: IDA in general

More broadly, my main concern with IDA isn't that it has a fatal flaw but that it isn't clear to me how the system helps with ensuring alignment compared to other architectures. I do think that IDA can be used to provide modest improvement in capabilities with small loss in alignment (not sure if better or worse then augmenting humans with computational power in other ways), but that the alignment error is not zero and increases the larger the improvement in capabilities.

Argument:

1. It is easy and tempting for the amplification to result in some form of search ("what is the outcome of this action?" "what is the quality of this outcome?" repeat), which fails if the human might misevaluate some states.
2. To avoid that, H needs to be very careful about how they use the system.
3. I don't believe that it is practically possible for formally specify the rules H needs to follow in order to produce an aligned system (or if you can, it's just as hard as specifying the rules for a CPU + RAM architecture). You might disagree with this premise in which case the rest doesn't follow.
4. If we can't be confident of the rules H needs to follow, then it is very risky just asking H to act as best as they can in this system without knowing how to prevent things from going wrong.
5. Since I don't believe specifying IDA-specific rules is any easier than for other architectures, it seems unlikely to me that you'd have a proof about the alignment or corrigibility of such a system that wouldn't be more generally applicable, in which case why not use a more direct architecture with fewer approximation steps?

To expand on the last point, if A[*], the limiting agent, is aligned with H then it must contain at least implicitly some representation of H's values (retrievable through IRL, for example). And so must A[i] for every i. So the alignment and distillation procedures must preserve the implicit values H. If we can prove that the distillation preserves implicit values, then it seems plausible that a similar procedure with similar proof would be able to just directly distill the values of H explicitly and then we can train an agent to behave optimally with respect to those.

So a convincing argument is trying to establish a universal quantifier over potential solutions to a possible problem.

This seems like a hard thing to do that most people may not have much experience with (especially since the problems are only defined informally at this point). Can you link to some existing such arguments, either against this AI alignment approach (that previously caused you to change your vision), or on other topics, to give a sense of what kinds of techniques might be helpful for establishing such a universal quantifier?

For example should one try to define the problem formally and then mathematically prove that no solution exists? But how does one show that there's not an alternative formal definition of the problem (that still captures the essence of the informal problem) for which a solution does exist?

I'm not bidding for the prize, because I'm judging the other prize and my money situation is okay anyway. But here's one possible objection:

You're hoping that alignment will be preserved across steps. But alignment strongly depends on decisions in extreme situations (very high capability, lots of weirdness), because strong AI is kind of an extreme situation by itself. I don't see why even the first optimization step will preserve alignment w.r.t. extreme situations, because that can't be easily tested. What if the tails come apart immediately?

This is related to your concerns about "security amplification" and "errors that are amplified by amplification", so you're almost certainly aware of this. More generally, it's an special case of Marcello's objection that says path dependence is the main problem. Even a decade later, it's one of the best comments I've ever seen on LW.

One concern that I haven't seen anyone express yet is, if we can't discover a theory which assures us that IDA will stay aligned indefinitely as the amplifications iterate, it may become a risky yet extremely tempting piece of technology to deploy, possibly worsening the strategic situation from one where only obviously dangerous AIs like reinforcement learners can be built. If anyone is creating mathematical models of AI safety and strategy, it would be interesting to see if this intuition (that the invention of marginally less risky AIs can actually make things worse overall by increasing incentives to deploy risky AI) can be formalized in math.

A counter-argument here might be that this applies to all AI safety work, so why single out this particular approach. I think some approaches, like MIRI's HRAD, are more obviously unsafe or just infeasible without a strong theoretical framework to build upon, but IDA (especially the HBO variant) looks plausibly safe on its face, even if we never solve problems like how to prevent adversarial attacks on the overseer, or how to ensure that incorrigible optimizations do not creep into the system. Some policy makers are bound to not understand those problems, or see them as esoteric issues not worth worrying about when more obviously important problems are at hand (like how to win a war or not get crushed by economic competition).

Can iterated amplification recreate a human's ability for creative insight? By that I mean the phenomenon where after thinking about a problem for an extended period of time, from hours to years, a novel solution suddenly pops into your head seemingly out of nowhere. I guess under the hood what's probably happening is that you're building up and testing various conceptual frameworks for thinking about the problem, and using those frameworks and other heuristics to do a guided search of the solution space. The problem for iterated amplification is that we typically don't have introspective access to the conceptual framework building algorithms or the search heuristics that our brains learned or came up with over our lifetimes, so it's unclear how to break down these tasks when faced with a problem that requires creative insight to solve.

If iterated amplification needs to exhibit creative insight in order to succeed (not sure if you can sidestep the problem or find a workaround for it), I suggest that it be included in the set of tasks that Ought will evaluate for their factored cognition project.

EDIT: Maybe this is essentially the same as the translation example, and I'm just not understanding how you're proposing to handle that class of problems?

Paul, to what degree do you think your approach will scale indefinitely while maintaining corrigibility vs. just thinking that it will scale while maintaining corrigibility to the point where we "get our house in order"? (I feel like this would help me in understanding the importance of particular objections, though objections relevant to both scenarios are probably still relevant).

A new post, looking into the strong version of corrigibility, arguing that it doesn't make sense without a full understanding of human values (and that with that understanding, it's redundant). Relevant to Amplification/Distillation since corrigibility is one of the aims of that framework.

https://www.lesswrong.com/posts/T5ZyNq3fzN59aQG5y/the-limits-of-corrigibility

My contribution.

Paul, it might be helpful to clarify the sort of things you think your approach relies upon in regards to bounds on the amount of overhead (training time, human sample complexity), or the amount of overhead that would doom your approach. If I recall correctly, I think you've wanted the approach to have some reasonable constant overhead relative to an unaligned system, though I can't find the post at the moment? It might also be helpful to have bounds, or at least your guesses on the magnitude of numbers related to individual components (ie. the rough numbers in the Universality and Security amplification post).

I took a quick look at your proposal; here are some quick thoughts on why I'm not super excited:

• It seems brittle. If there's miscommunication at any level of the hierarchy, you run the risk of breakage. Fatal miscommunications could happen as information travels either up or down the hierarchy.
• It seems like an awkward framework for achieving decisive strategic advantage. I think decisive strategic advantage will be achieved not through getting menial tasks done at a really fast rate, so much as making lots of discoveries, generating lots of ideas, and analyzing lots of possible scenarios. For this, a shared knowledge base seems ideal. In your framework, it looks like new insights get shared with subordinates by retraining them? This seems awkward and slow. And if insights need to travel up and down the hierarchy to get shared, this introduces loads of opportunities for miscommunication (see previous bullet point). To put it another way, this looks like a speed superintelligence at best; I think a quality superintelligence will beat it.
• The framework does not appear to have a clear provision for adapting its value learning to the presence/absence of decisive strategic advantage. The ideal FAI will slow down and spend a lot of time asking us what we want once decisive strategic advantage has been achieved. With your thing, it appears as though this would require an awkward retraining process.

In general, your proposal looks rather like a human organization, or the human economy. There are people called "CEOs" who delegate tasks to subordinates, who delegate tasks to subordinates and so on. I expect that if your proposal works better than existing organizational models, companies will have a financial incentive to adopt it regardless of what you do. As AI and machine learning advance, I expect that AI and machine learning will gradually swallow menial jobs in organizations, and the remaining humans will supervise AIs. Replacing human supervisors with AIs will be the logical next step. If you think this is the best way to go, perhaps you could raise venture capital to help accelerate this transition; I assume there will be a lot of money to be made. In any case, you could brainstorm failure modes for your proposal by looking at how organizations fail.

I've spent a lot of time thinking about this, and I still don't understand why a very simple approach to AI alignment (train a well-calibrated model of human preferences, have the AI ask for clarification when necessary) is unworkable. All the objections I've seen to this approach seem either confused or solvable with some effort. Building well-calibrated statistical models of complex phenomena that generalize well is a hard problem. But I think an AI would likely need a solution to this problem to take over the world anyway.

In software engineering terms, your proposal appears to couple together value learning, making predictions, making plans, and taking action. I think an FAI will be both safer and more powerful if these concerns are decoupled.

[Meta: Even low-effort engagement, like "known + keyword" or "you misunderstood everything; read <link>" or "go on talking / thinking" is highly appreciated. Stacks grow from the bottom to the top today, unlike x86 or threads on the internet]

------------

Iterative amplification schemes work by having each version $i+1$ trained by previous iteration $i$; and, whenever version $i$ fails at finding a good answer (low confidence in the prediction), punting the question to $i-1$ , until it reaches the human overseer at $i=0$, which is the ground truth for our purposes. At the same time, the distribution $D_i$ of relevant question widens at each iteration, as capabilities improve: For example, a very stupid clippy will need to deal with simple linear optimization problems of its supply chain; a very smart clippy will need to figure out a proper plan for world-domination.

Now, in order to correctly punt questions down the call-stack, each level $i$ when faced with a problem $q$ must decide whether it can solve it (providing training data for level $i+1$), or whether it must punt downwards. More precisely, level $i$ must recognize whether $q$ is inside the distribution $D_{i-1}$ it was trained with (resp. $D_i$, the distribution of problems it can generate / handle post-training).

False positives (underconfidence) lead to increased load on the human overseers / capability loss. False negatives (overconfidence) lead to safety failures: If they happen during training, then the machine learns wrong values / ethics / whatever-it-should-learn, if they happen in production then they lead to bad actions (I am assuming that no one would run the AI unsupervised, but supervision will, by necessity, be pretty light).

This is a problem, even with a very shallow call-stack (few iterations), but it gets (possibly exponentially!) worse with deeper stacks: After all, if even a single false-negative mistake occurs along the chain, we have a safety incident; naively, the chance of mistakes grows exponentially in the depth, for fixed false-negative rate at each level. In other words, we would need to ensure that the infinite series ${\sum }_i\log (1-p_i)$ converges, where $p_i$ is the false-negative rate at level $i$ .

Something that may makes this problem either simpler or harder is the fact that, once a question has been bumped down a couple of times, it can be very far out-of-distribution. This makes it much easier to get mediocre reliability (the questions bumped down from very high levels are very weird for the low levels), but, at the same time, the low levels are really, really not equipped for dealing with these weird questions. In the clippy example, Paul's scheme would totally allow the high capability clippy to convince low levels into accepting its proposal by exploiting its weaknesses (adversarial examples).

In different words: The entire scheme needs really good anomaly detection. Anomaly detection is a machine-learning problem, right? So learn this along the way?

The problem is that, while training the anomaly detection at level $i$ , we cannot sample from the distribution $D_{i+N}$ , because we simply don't know it yet. As we run amplification, we extend both the capability of answering questions and the range of questions that come up.

Note that this would not be a problem if the extensions of supports of the distributions $D_i$ were very mild, compared to the larger domain ${\Omega }_i$ where the anomaly detection of level $i$ is reliable-by-training. This appears like a very dangerous assumption on the "mind-space" accessible to our training procedure, for the "problem-space" at hand and the specific training procedure. In other words: No universality of safety amplification, if it relies on this assumption. Please link me to relevant discussions if this assumption has been justified anywhere.

A different approach would be to not run the scheme in a hierarchical way; instead try to feed it into itself. This reintroduces the problem of "drift", i.e. instability: The system will acquire a specific culture, and we can only pray (i.e. must assume / ensure) that desireable states are an attractor, for the specific training procedure. I fear that this is extra-hard to check, because meta-stable systems look pretty stable, until you increase capability: Bad scaling.

A third approach to solving this problem would be to create, separately, a reliable anomaly detector. This may very well be an AI complete problem, but it may be simpler than alignment! Further, robust anomaly detection (recognizing that data points are outside of the training distribution) might be an avenue for differential progress that matters more for safety/alignment than capabilities.

Do you know other approaches for this problem?

------------

When is this harmless?

Well, this is entirely harmless if our problem domain contains only simple questions, where verification of ground truth is dirt cheap. For example, NP questions (like, e.g., all of mathematics! Proof verification is trivial after all). This observation should give us a bad feeling, because an AI that is really good at mathematics / programming and bad at everything else is one of the most dangerous situation we could imagine (magical function optimizers, aka mindless AIXI genies, being the only more hopeless case I can think of). On the other hand, typical NP questions don't scale down: It is currently entirely infeasible to use machine learning for theorem proving, simply because useful transformations are exponentially rare in the space of possible ones (I am aware of some papers using the mizar library; while the neural net + theorem prover beat the unaided prover, I was less than impressed by the results).

For problem domains that feel more like exptime, this is more likely to be a problem: Say, training to play games like Go. Then, we can play against our ancestors in order to judge performance, and gain access to some kind of ground truth. Unfortunately, (1) strength is not linearly ordered: You clearly can have situations where A beats B beats C beats A, and (2) if we wanted to optimize "strength against perfect play", aka min-max, then we don't have access to a perfect opponent during training. Afaik it is usual for training-through-amplification of Game AI to develop "fads", i.e. cheesy tactics, on the way; sometimes, these recur cyclically. This is also observed for the metagame in many multiplayer videogames. I have a feeling that the Go successes tell us a lot about how MCTS is amazingly stable against cheesy tactics; and who knows how much tweaking deepmind had to do until they got the amplification stable.

Now, safety amplification / value learning has a much, much harder problem: The ground truth is only accessible through examples / very expensive oracle queries (which might be fundamentally unsafe, at very high levels of capability: Don't let human operators talk to unaligned too-clever AI).

------------

Post-script: Writing this down in clear words made me slightly update against Paul's amplification schemes eventually growing into a solution. I still think that Paul's line of research is damn cool and promising, so I'm more playing devil's advocate here. The possible differential gain for capability in NP problems versus harder-than-NP alignment for this kind of amplification procedure made me slightly more pessimistic about our prospects in general. Moreover, it makes me rather skeptic whether amplification is a net win for safety / alignment in the differential progress view. I want to look more into anomaly detection now, for fun, my own short-term profit and long-term safety.

https://www.lesserwrong.com/posts/ZyyMPXY27TTxKsR5X/problems-with-amplification-distillation

Summary:

I have four main criticisms of the approach:

• 1. "Preserve alignment" is not a valid concept, and "alignment" is badly used in the description of the method.
• 2. The method requires many attendant problems to be solved, just like any other method of alignment.
• 3. There are risks of generating powerful agents within the systems that will try to manipulate it.
• 4. If those attendant problems are solved, it isn’t clear there’s much remaining of the method.

The first two points will form the core of my critique, with the third as a strong extra worry. I am considerably less convinced about the fourth.

My intuition is this is not particularly stable against adversarial inputs. Trying to think about is as a practical problem, I would attack it in this way

• provide adversarial inputs to A0 which will try to manipulate them so they are better at the "simple task at hand" but e.g. have slightly distorted some model of the world
• it seems feasible to craft manipulations which would "evaluate" and steer the whole system only while already at superhuman level, so we have Amplify(H,A[10]) where A(10) is superhuman and optimizing to take over the control for the attackers goal

In vivid language, you seed the personal assistants with the right ideas ... and many iterations later, they start a communist revolution

I will try to list some objections appeared in my mind while reading your suggestions.

1. I think that the main problem is the idea that "humans have values". No, they don't have values. "Values" were invented by psychologists to better describe human behaviour. To calling that humans "have values" is fundamental attribution error, which in most cases very subtle, because humans behave as if they have values. But this error could become amplified by suggested IDA, because human reactions to AI's tests will be not coherent and their extrapolation will stuck at some level. How exactly it will stuck is not easy to predict. In short: it is impossible to align two fuzzy things. I am going to write something longer about the problem of non-existence of human values.
2. I have values about values which contradict my observed behaviour. If AI could observe one's behaviour, it may conclude that this person - suppose - is lazy, sexually addicted and prefer to live in the world of Game of Thrones. However, a person could deny part of his behaviour as unethical. Thus behaviour will give wrong clues.
3. Human behaviour is controlled not only by person's consciousness, but also partly by person's unconsciousness, which is unobservable to him, but obvious for others. In other words, human's behaviour often is a sum of several different minds inside his head. We don't want AI to extrapolate our unconscious goals as they are often unethical and socially inappropriate. (Could support these claims by literature and examples.)
4. Small tests performed by AI may be dangerous or unethical. Maybe AI will try to torture a person to extract his preference about money and pain.
5. AI could be aligned this the same human in many different ways. All of them could be called "alignment", but because AI is very complex, different approximations of human behaviour could satisfy to any criterion we have. Some of this approximations may better or worse.
6. AI alignment per se doesn't solve AI safety, as it doesn't explain how war between humans empowered by AIs will be prevented. I wrote an article about it: Turchin, Alexey, and David Denkenberger. Military AI as a convergent goal of the self-improving AI. In edited volume: Artificial intelligence safety and security, CRC, 2018, https://philpapers.org/rec/TURMAA-6
7. AI's test of human values will change human values, and AI could change these values in any possible direction. That is, AI could design "adversarial test" which will change humans' value system in any other value system without human noticing it. Some human also could do it. See more about it in "Ericksonian hypnosis". It is particular nasty example of AI boxing goes wrong.
8. Obviously, if AI's capabilities are underestimated, it could experience "treacherous turn" in any moment of preparing of the test of human behaviour, - I think this should be a standard MIRI reply.
9. To be "aligned" is not just mathematical relation, to which typical notations of transitivity - or whatever we could easily do with mathematical symbols - are applicable. Alignment is complex and vague, never certain. It is like love. We can't describe human love by introducing notation L, and start writing equations like L(A,B)=C.

Anyway, I think that there is a chance that you scheme will work but under one condition: First AI is something like a human upload which has basic understanding of what it is to be aligned. If we have such perfect upload, which has common sense and general understanding of what humans typically want, the amplification and distillation scheme will work. The question is how we can get this first upload? I have some ideas, but this is another topic.

https://github.com/DonaldHobson/AI-competition-2

Only just learned of this, unfortunately.

OP (please comment there): https://medium.com/@lucarade/issues-with-iterated-distillation-and-amplification-5aa01ab37173

I will show that, even under the most favorable assumptions regarding the feasibility of IDA and the solving of currently open problems necessary for implementing IDA, it fails to produce an aligned agent in the sense of corrigibility.

Part 1: The assumptions

Class 1: There are no problems with the human overseer.

1.1: Human-generated vulnerabilities are completely eliminated through security amplification. (See this post for a lengthy overview and intuition, and this postfor a formalization). In short, security amplification converts the overseer in IDA from high-bandwidth (receiving the full input in one piece) to low-bandwidth (receiving inputs divided into small pieces), to make impossible an attack which inputs data in such a way as to exploit human vulnerability to manipulation. See this post [LW · GW] for a good explanation of a high-bandwidth vs low-bandwidth overseer.
My critique applies equally to high-bandwidth and low-bandwidth overseers so I make no assumption on that front.

1.2: There is no moral hazard in the human overseers. This eliminates one of Stuart’s critiques. Furthermore, the human overseer displays corrigible behaviors without error.

1.3: The relevant experts are willing to put in a substantial amount of time for the training process. This is a non-trivial assumption which I have not yet seen discussed.

Class 2: The framework and its auxiliary components function as intended.

2.1: Reliability amplification functions as intended. In summary, reliability amplification uses a voting ensemble of agents at each stage of amplification to avoid error amplification, in which an initially small probability of error grows with each iteration.

2.2: Corrigibility, not optimal value-aligned performance, is our goal. All we care about is that our agent “is trying to do what its operator wants it to do.” It may be bad at actually figuring out what its operator wants or at carrying out those wants, but the point is that it cares about improving, and will never intentionally carry out an action it knows is contrary to what its operator would want it to do (see this post and this post [LW · GW] for a clarification of Paul’s approach to AI alignment by achieving corrigibility).

Stuart has pointed out problems with corrigibility [LW · GW], which I agree with. Essentially, the concept is ill-defined given the fuzziness of human values, and to properly implement corrigibility an agent must completely understand human values, thus reducing to the much harder value learning problem. However, we will assume that an agent which understands and implements the general concept of corrigibility, even if it accidentally misbehaves in many cases and causes widespread harm upon initial implementation as Stuart’s argument suggests, will still avoid existential risk and allow us to improve it over time, and is thus satisfactory. I think this is Paul’s approach to the matter.

Even a fully corrigible agent can be catastrophically misaligned, as detailed in this post [LW · GW]. As addressed in the comments of that post, however, if we assume humans are smart enough to avoid a corrigible AI causing existential risk in this manner then the issue goes away.

2.3: There is no coordination possible among any of the A[n]s, eliminating another of Stuart’s critiques.

2.4: The informed oversight problem is solved. In summary, the problem is that it is difficult for a more powerful aligned overseer agent to fully understand the decision-making process of a weaker agent in a way that allows the overseer to push the weaker agent towards alignment. However, it does not follow that it is possible for a weaker aligned overseer to understand the decision-making process of a more powerful agent. It seems like this will never be possible by definition, because a weaker agent cannot understand the decision-making processes of a more powerful agent even if full transparency were possible.

2.5: Worst-case scenario techniques are effective at eliminating the risk of a treacherous turn, in which an agent performs well on training and test data but behaves catastrophically on some input in the real world which was not accounted for.

Part 2: Corrigibility is unlikely to be trained

The crux of my argument is that, even given the very optimistic assumptions above, IDA will fail to train a corrigible agent. I will list the problems sequentially, but my argument is that all five are necessary for failure. It is possible a subset is sufficient but I will not explore that possibility in this post.

Problem 1: Limited training data. Even given assumption 1.3, carrying out the process of training each new iteration is extremely time-intensive. It would take years of many experts’ full-time commitment as overseers to produce anything close to a reasonable amount of data.

Problem 2: Hidden error amplification. If an error is identifiable by an aligned agent, then reliability amplification will almost certainly remove it. However, a small initial hidden error will be rapidly amplified, since in the next iteration it will be manifested in various ways in many of the thousands of copies of the error-containing agent. Thus in the distillation step, the initial error will be propagated in many different forms, leading to a multiplication of errors. Informed oversight doesn’t help since the human is not more powerful than the subroutines, and even if transparency is achieved, if the human can verify all output then only very limited capability amplification is possible.

I will show in Problem 4 that such hidden errors are likely to arise.

Problem 3: Information loss. The human overseer has access to information, in the form of memories (broadly construed as information from before the current training process stored in the human’s mind), that is not accessible to the agent A[n+1] training on (H+A[n]), who sees only (H+A[n])’s actions. A[n+1] will thus erroneously attribute decisions made by the human relying on non-available information to available information. This information loss and the resulting distortion is greatly exacerbated by the same process described in Problem 2, doubly: first in the amplification of the information loss of the previous overseer, and second in the introduction of further information loss of the current overseer.

Paul thinks this is a general problem of learning and thus falls outside of the scope of things that specifically cause IDA to fail, as pointed out in this postand in the comments on this post [LW · GW], but I disagree. One can certainly imagine (and some have experienced) a human robustly learning another human’s decision-making heuristics over time without direct access to the other human’s memories, and can by extension also imagine an artificial agent extracting information from a human to robustly understand that human’s decision-making process. The problem exists not in all forms of learning but in the class of training techniques which do not involve a direct and adaptive extraction of information from a human in some form.

Problem 4: No prior concept of corrigibility. Because of information loss, an agent has no way of extracting the concept of corrigibility from its training data, only the behavior of corrigibility. The way the agent implements corrigibility will thus necessarily be an approximation, even if an extremely good one, and will not necessarily be robust to drastic changes in context. This causes the small hidden errors that are then amplified through the hidden error amplification in Problem 2, making reliability amplification ineffective. Without hidden error amplification this would probably not be a problem, since agents which successfully approximate corrigibility behaviorally will be able to detect all but the tiniest deviations from optimal corrigibility (ie, understanding the concept the way you and I do). However, hidden error amplification causes a nontrivial corrosion of corrigibility throughout iterations, and as each newly distilled agent approximates an increasingly corrupted behavioral corrigibility that deviates from our ideal conceptual corrigibility, reliability amplification is keeping us close to each further deviated behavioral corrigibility but not close to the ideal conceptual corrigibility. The process behaves essentially as a high-dimensional random walk with extremely small steps, but with thousands of steps per iteration manifested in the copies of A[n].

Problem 5: Temporal inconsistency of proxy dynamics (TIPD). Any incomplete simulation is not robust over time without an adaptive capacity. There are certain underlying processes which are time-invariant, such as the laws of physics and the mathematics of evolution. However, clearly we can never completely simulate any non-trivial situation purely in terms of these processes. Thus, an agent must necessarily rely on proxy dynamics for decision-making: emergent properties of the fundamental processes, which fairly reliably approximate cause-and-effect relationships between actions and outputs. However, because of the complexity of the underlying dynamics and their interactions, these proxy dynamics change over time, and often quite drastically over short periods (see the literature on chaos theory, critical transitions, bifurcation points). Thus, an agent which performs robustly at one point in time may behave catastrophically at another. The only solution is for the agent to be capable of adapting its policy to changes in the proxy dynamics it uses.

This sounds like the treacherous turn problem, but it is distinct, and harder. In the treacherous turn problem, we have an agent that is not sufficiently well trained given the input-output relationships of the world. This can probably be solved by worst-case scenario techniques like adversarial training. In TIPD, even if we succeed in training a robust policy, the proxy dynamics used to inform decisions will change such that an action in response to an input which previously would have produced a safe behavior now produces a catastrophic behavior.

As a result, behavioral corrigibility, whether corrupted or not, is not robust over time since it does not adapt to changing input-output relationships. An agent must possess conceptual corrigibility for such adaptation to occur, which is extremely hard, and may reduce to the value learning problem.

Part 3: Achieving alignment in this process through anything but corrigibility is doomed.

This is fairly obvious, and mostly follows from Part 2. Any proxy of the human’s decision-making process will clearly fail without an adaptive capacity, and it is not clear how such an adaptive capacity could be robustly implemented. And clearly this method will never achieve anything but a proxy due to information loss.

Conclusion

I have argued that even under the most optimistic assumptions about the human overseer and the successful operation of the framework, IDA will fail to produce a corrigible agent. This failure is a result of the interplay between hidden error amplification, information loss, the ability to learn behavioral corrigibility but not conceptual corrigibility, and the temporal inconsistency of proxy dynamics (TIPD). The solution to these problems seems very hard, and may reduce to the value learning problem, in which case the IDA framework does not provide us with any advantage.

Hello! Thanks for the prize announcement :)

Hope these observations and clarifying questions are of some help:

https://medium.com/threelaws/a-reply-to-aligned-iterated-distillation-and-amplification-problem-points-c8a3e1e31a30

Summary of potential problems spotted regarding the use of AlphaGoZero:

• Complete visibility vs Incomplete visibility.
• Almost complete experience (self-play) vs Once-only problems. Limits of attention.
• Exploitation (a game match) vs Exploration (the real world).
• Having one goal vs Having many conjunctive goals. Also, having utility maximisation goals vs Having target goals.
• Who is affected by the adverse consequences (In a game vs In the real world)? - The problems of adversarial situation, and also of the cost externalising.
• The related question of different timeframes.

Summary of clarifying questions:

• Could you build a toy simulation? So we could spot assumptions and side-effects.
• In which ways does it improve the existing social order? Will we still stay in mediocristan? Long feedback delay.
• What is the scope of application of the idea (Global and central vs Local and diverse?)
• Need concrete implementation examples. Any realistically imaginable practical implementation of it might not be so fine anymore, each time for different reasons.

https://www.dropbox.com/s/98469ukjkzgdy6k/AI%20alignment%20%28FINAL%29.docx?dl=0

I dont use Dropbox normally so I hope the link works. let me know if it doesn't and I'll figure something else out. Cant say im convinced of this approach to alignment but its good to see continued discussion.

This is a more general objection about various sorts of proposed AI alignment schemes that ends up applying to your ideas that I've been thinking about since formalizing the AI alignment problem. I'll probably write more up on this later but for now here's a sketch as it applies to your ideas as far as I understand them.

In "Formally Stating the AI Alignment Problem" I say alignment requires that

A must learn the values of H and H must know enough about A to believe A shares H’s values

where A is an AGI and H is humanity (this is an informal summary of the more formal result given earlier in the paper). Your solution only seems to be designed to address the first part, that A must learn the values of H. Your proposal mostly seems to me to ignore the requirement that H must know enough about A to believe A shares H's values, although I know you have thought about the issue of transparency and see it as necessary. Nevertheless in your current approach it feels tacked on rather than a key part of the design since you seem to hope it can be achieved by training for transparency.

The need to train for transparency/explainability is what makes me suspicious of RL-based approaches to AI alignment now. It's difficult to come up with a training program that will get an agent to be reliably transparent in that the reward function will always reward the appearance of transparency but not actual transparency since it can only force observed behavior but not internal structure. This creates an opportunity for a treacherous turn where the AI acts in ways that indicate it shares our values, appears to be believably sharing our values based on the explanations it gives for its actions, yet could be generating those explanations independent of how it actually reasons such that it would appear aligned right up until it's not.

Intuitively this is not very surprising because we have the same challenge with aligning humans are not able to do it reliably. That is, we can try to train individual humans to share our values, observe their behavior indicating they have learned our values and are acting on them, and ask them for reasons that convince us that they really do share our values, but then they may still betray those values anyway since they could have been dissociative and hiding their resentful intent to rebel the entire time. We know this to be the case because totalitarian states have tried very hard to solve this problem and repeatedly failed even if they are often successful in individual cases. This suggests approaches of this class cannot be made reliable enough to be worth pursuing.