Asymptotically Unambitious AGI

I know I don't prove it, but I think this agent would be vastly superhuman, since it approaches Bayes-optimal reasoning with respect to its observations. ("Approaches" because MAP -> Bayes).

For the asymptotic results, one has to consider environments that produce observations with the true objective probabilities (hence the appearance that I'm unconcerned with competitiveness). In practice, though, given the speed prior, the agent will require evidence to entertain slow world-models, and for the beginning of its lifetime, the agent will be using low-fidelity models of the environment and the human-explorer, rendering it much more tractable than a perfect model of physics. And I think that even at that stage, well before it is doing perfect simulations of other humans, it will far surpass human performance. We manage human-level performance with very rough simulations of other humans.

That leads me to think this approach is much more competitive that simulating a human and giving it a long time to think.

I think like the interesting question is: if inner alignment turns out to be a hard problem for training cognitive policies, do we expect it to become much easier by training predictive models?

If I'm understanding correctly, and I'm very unsure that I am, you're comparing the model-based approach of [learn the environment then do good planning] with [learn to imitate a policy]. (Note that any iterated approach to improving a policy requires learning the environment, so I don't see what "training cognitive policies" could mean besides imitation learning.) And the question you're wondering about is whether optimization daemons become easier to avoid when following the [learn the environment then do good planning] approach.

Imitation learning is about prediction just as much as predictive models are--predictive models imitate the environment. So I suppose optimization daemons are about equally likely to appear?

My real answer, though, is that I'm not sure, but vanilla imitation learning isn't competitive.

But I suspect I've misunderstood your question.

the hope that "computationally simplest model describing the box" is actually a physical model of the box

I don't actually rely on this assumption, although it underpins the intuition behind Assumption 2.

Notational note: I use $i_0$ to denote the episode when BoMAI becomes demonstrably benign and $n_0$ for something else.

Any time the MAP model makes a different prediction from the intended model, it loses some likelihood.

Any time any model makes a different on-policy prediction from the intended model, it loses some likelihood (in expectation). The off-policy predictions don't get tested. Under a policy that doesn't cause the computer's memory to be tampered with (which is plausible, even ideal), ${\nu }^{†}$ and ${\nu }^{\star }$ are identical, so we can't count on ${\nu }^{†}$ losing probability mass relative to ${\nu }^{\star }$. The approach here is to set it up so that world-models like ${\nu }^{†}$ either start with a lower prior, or else eventually halt when they exhaust their computation budget.

Upvoted for interesting experiments with bounties and comment formatting.

I like that you emphasize and discuss the need for the AI to not believe that it can influence the outside world, and cleanly distinguish this from it actually being able to influence the outside world. I wonder if you can get any of the benefits here without needing the box to actually work (i.e. can you just get the agent to believe it does? and is that enough for some form/degree of benignity?)

1. Can you give some intuitions about why the system uses a human explorer instead of doing exploring automatically?
2. I'm concerned about overloading the word "benign" with a new concept (mainly not seeking power outside the box, if I understand correctly) that doesn't match either informal usage or a previous technical definition. In particular this "benign" AGI (in the limit) will hack the operator's mind to give itself maximum reward, if that's possible, right?
3. The system seems limited to answering questions that the human operator can correctly evaluate the answers to within a single episode (although I suppose we could make the episodes very long and allow multiple humans into the room to evaluate the answer together). (We could ask it other questions but it would give answers that sound best to the operator rather than correct answers.) If you actually had this AGI today, what questions would you ask it?
4. If you were to ask it a question like "Given these symptoms, do I need emergency medical treatment?" and the correct answer is "yes", it would answer "no" because if it answered "yes" then the operator would leave the room and it would get 0 reward for the rest of the episode. Maybe not a big deal but it's kind of a counter-example to "We argue that our algorithm produces an AGI that, even if it became omniscient, would continue to accomplish whatever task we wanted, instead of hijacking its reward, eschewing its task, and neutralizing threats to it, even if it saw clearly how to do exactly that."

(Feel free to count this as some number of comments between 1 and 4, since some of the above items are related. Also I haven't read most of the math yet and may have more comments and questions once I understood the motivations and math better.)

From the formal description of the algorithm, it looks like you use a universal prior to pick $k$, and then allow the $k^{\text{th}}$ Turing machine to run for $\ell$ steps, but don't penalize the running time of the machine that outputs $k$. Is that right? That didn't match my intuitive understanding of the algorithm, and seems like it would lead to strange outcomes, so I feel like I'm misunderstanding.

How does your AI know to avoid running internal simulations containing lots of suffering?

Are there UDT-ish concerns with breaking isolation of episodes? For example, if the AI receives a low reward at the beginning of episode 117, does it have an incentive to manipulate the external world to make episode 117 happen many times somehow, with most of these times giving it a higher reward? For another example, can the AI at episode 117 realize that it's in a game theory situation with the AI at episodes 116 and 118 and trade rewards with them acausally, leading to long-term goal directed behavior?

The reward belongs to a finite set of rationals between 0 and 1.

Once the AI becomes sufficiently powerful, it is definitely going to hack the operator any time it expects the operator to give a reward less than 1. So the operator's input is really binary, at least once the AI has learned an accurate model. Given that, why allow non-binary rewards at all? Is it just supposed to provide faster learning early on?

Along similar lines: once the AI has learned an accurate model, why would we expect it to ever provide anything useful at all, rather than just hacking the operator all day? Do we think that hacking the human is likely to be harder than obtaining perfect rewards every time without hacking the human? Seems like that would depend very heavily on the problem at hand, and on the operator's feedback strategy.

To put it differently: this setup will not provide a solution to any problem which is more difficult than hacking the human operator.

Other algorithms... would eventually seek arbitrary power in the world in order to intervene in the provision of its own reward; this follows straightforwardly from its directive to maximize reward

The conclusion seems false; AUP (IJCAI, LW [LW · GW]) is a reward maximizer which does not exhibit this behavior. For similar reasons, the recent totalitarian convergence conjecture made here also seems not true.

This may be a dumb question, but how can you asymptotically guarantee human-level intelligence when the world-models have bounded computation time, and the human is a "computable function" that has no such limit? Is it because the number of Turing machines is infinite?

My main concern is that the system will be neither safe, nor useful. Its usefulness is limited by the number of tasks which could be solved in such setup. These tasks are probably some advises or projects: "should we develop wind energy" or "print the the blueprint of a nanorobot". The main problem is that an advice can look good but have some unintended consequences, which are not obvious. For example, widespread wind farms will kill birds and result in pest proliferation. If the AI continue to work, it could adjust its advise to this new data, but the program has stopped.

In other words, sealed Oracle can give bad advice and can't be punished for that.

There will be strong incentive by the system's owner to disable all this safety measures, and use the system in real world continuously, so all Omohundro's drives appear all over again, but on the level of the system's owners.

Moreover, the system can feel rewarded now by knowing that it will in future dominate in the universe, so the system may feel the need to put a secret code in its output (if it is a long code) which will create many its copies in the wild. For example, in order to tile the universe with windfarms. If it can't output complex code, it will be mostly useless.

Since the real world is quantum, does your UTM need to be quantum too? More generally, what happens if there's a mismatch between what computations can be done efficiently in the real world vs on the UTM?

Also, I'm not sure what category this question falls under, but can you explain the new speed prior that you use, e.g., what problems in the old speed priors was it designed to solve? (I recall noticing some issues with Schmidhuber's speed prior but can't find the post where I wrote about it now.)

Just obvious and mundane concerns:

You might want to make clearer that "As long as the door is closed, information cannot leave the room" isn't an assumption but a requirement of the setup. I.e. that you're not assuming based on your description that opening the door is the only means for an operator to get information out; you're assuming every other means of information escape has been systematically accounted for and ruled out (with the assumption that the operator has been compromised by the AI).

It would probably help to put the AI in a cryptographic box as well (so its only output is text). See https://www.lesswrong.com/posts/2Wf3R4NZ77CLczLL2/cryptographic-boxes-for-unfriendly-ai [LW · GW]

Responding to this [LW(p) · GW(p)] comment [LW(p) · GW(p)] here.

I definitely do not trust a door with a kill switch to keep a superintelligent AI in a box.

I didn't go into the details of the physical design for a really secure box in the paper, because it's not really a topic in computer science, but this is a good excuse to write it up.

The operator is inside a glass box. Outside the glass box is a vacuum, and then a concrete box. Outside the concrete box is another vacuum and another concrete box with a Faraday cage around it. There are (airtight) doors through each layer. There is a pressure sensor in both the inner and outer vacuum layer. The computer lives in the inner wall of the inner concrete box; the screen is visible from the inside of the glass box. If the pressure increases in either vacuum layer, the computer's memory is erased, and the whole thing is blown up. There is a laser shining through the glass box (Laser 1). If the operator blocks the path of Laser 1, this corresponds to pressing the button which ends the episode and opens the door. The pressure sensor -> dynamite pathway is deactivated, the vacuum layers are filled with air, and the doors are unlocked. The operator "types" observations and rewards by blocking other lasers that pass through the box. For now, just imagine a few dozen lasers in a row, that the operator can reach up and block in order to type. (With a bit of creativity, and requiring that two lasers be blocked at once to "press a key" this could probably be made somewhat like a qwerty setup.) The lasers and the laser-detectors are all on the inner wall of the inner concrete box.

So inside the box is only a human body and some clothes. Let's assume: 1) the only ways for information to move in space are with a) physically moving matter, b) light, and c) gravitational waves and 2) it is impossible for a human being to convert a nuclear reaction into useful work with only their body and clothes. Physically moving packets matter cannot leave the inside of the glass box without releasing air into the vacuum and causing an explosion. Light cannot leave the Faraday cage. For sound waves (through an imperfect vacuum) and gravitational waves, we can use the amount of usable energy stored by the person in the room to bound how much noise and randomly moving matter we need to be producing outside to bury the signal in the noise. With a sufficiently large amount of noise, even if all the human's energy went toward producing a signal, we can create enough noise that this will not be detectable. Furthermore, an analysis of the near-perfect vacuum can provide an upper bound on the efficiency with which a human can convert energy into an informative matter wave, which should drive down the amount of noise we need to produce to almost nothing. We can also put a decibel meter in the inner wall and blow the whole thing up if it gets too high. I haven't done the math, but I bet the amount of matter we need to be randomly moving in order to hide signals coming from gravitational waves originating from within the box really is nothing. I bet ambient temperature takes care of that.

What if there are other ways information can be transmitted through space? If we ever learn of these, we shut down BoMAI (or modify the setup in light of them). Until then, if such a discovery is made by BoMAI, we can't exactly detect those sorts of signals from the outside, because we don't know they exist.

Still wrapping my head around the paper, but...

1) It seems too weak: In the motivating scenario of Figure 3, isn't is the case that "what the operator inputs" and "what's in the memory register after 1 year" are "historically distributed identically"?

2) It seems too strong: aren't real-world features and/or world-models "dense"? Shouldn't I be able to find features arbitrarily close to F*? If I can, doesn't that break the assumption?

3) Also, I don't understand what you mean by: "it's on policy behavior [is described as] simulating X". It seems like you (rather/also) want to say something like "associating reward with X"?

Also, it's worth noting that this assumption (or rather, Lemma 3) also seems to preclude BoMAI optimizing anything *other* than revealed preferences (which others have noted seems problematic, although I think it's definitely out of scope).

I'm calling this the "no grue assumption" (https://en.wikipedia.org/wiki/New_riddle_of_induction).

My concern here is that this assumption might be False, even in a strong sense of "There is no such U".

Have you proven the existence of such a U? Do you agree it might not exist? It strikes me as potentially running up against issues of NFL / self-reference.

Wei Dai's response to this comment [LW(p) · GW(p)] represents a concern with Assumption 3.

Let the set of potential human explorer actions be $A_E$, and the best human explorer action be $a_E^{\ast }$ with reward $r_E^{\ast }$. Consider the following world model. When asked to predict the result of an action $a$, it simulates it to find the predicted observation $o$ and reward $r$. If $a\in A_E$, it outputs $o$ and $r$ faithfully. Otherwise, it outputs any reward it chooses, as long as for the action that it assigns the maximum reward to, it reports $r$ faithfully. In practice, this means that the world model can get BoMAI to choose any action it wants, as long as it is at least as good as the human explorer's best action. This is bad if the world model has malicious inner optimizers.

I believe that Assumption 2 is the one responsible for ruling out a model of this form. It seems probably reasonable to say that for actions where it continues simulating the outside world there's useless computation. However, it can also save computation relative to ${\mu }^{\ast }$: while ${\mu }^{\ast }$ must predict $o$ and $r$ perfectly for all actions $a$, this model can immediately output a null observation and zero reward for any $a\notin A_E$ that it knows will not align with its goals, rather than spending computation to simulate what rewards those actions would get. Another way of thinking about this is that this model uses consequentialist general intelligence to quickly prune away uninteresting non-human actions to save on computation, but that general intelligence comes at the price of misaligned goals + deceptive behavior.

Also, from this comment [AF(p) · GW(p)]:

The real result of the paper would then be "Asymptotic Benignity, proven in a way that involves off-policy predictions approaching their benign output without ever being tested".

I think the model above has arbitrarily bad off-policy predictions, and it's not implausible for it to be the MAP world model forever.

[Quite possibly I'm confused, but in case I'm not:]
I think this assumption might be invalid (or perhaps require more hand-waving than is ideal).

The AI has an incentive to understand the operator's mind, since this bears directly on its reward.
Better understanding the operator's mind might be achieved in part by running simulations including the operator.
One specific simulation would involve simulating the operator's environment and actions after he leaves the room.

Here this isn't done to understand the implications of his actions (which can't affect the episode); it's done to better understand his mind (which can).

In this way, one branch of forget/not-forget has two useful purposes (better understand mind and simulate future), while the other has one (better understand mind). So a malign memory-based model needn't be slower than a benign model, if it's useful for that benign model to simulate the future too.
So either I'm confused, or the justification for the assumption isn't valid. Hopefully the former :).

If I'm right, then what you seem to need is an assumption that simulating the outside-world's future can't be helpful in the AI's prediction of its reward. To me, this seems like major hand-waving territory.

Just exposition-wise, I'd front-load pi^H and pi^* when you define pi^B, and also clarify then that pi^B considers human-exploration as part of it's policy.

" This result is independently interesting as one solution to the problem of safe exploration with limited oversight in nonergodic environments, which [Amodei et al., 2016] discus "

^ This wasn't super clear to me.... maybe it should just be moved somewhere else in the text?

I'm not sure what you're saying is interesting here. I guess it's the same thing I found interesting, which is that you can get sufficient (and safe-as-a-human) exploration using the human-does-the-exploration scheme you propose. Is that what you mean to refer to?

ETA: NVM, what you said is more descriptive (I just looked in the appendix).

RE footnote 2: maybe you want to say "monotonically increasing as a function of" rather than "proportional to". (It's a shame there doesn't seem to be a shorter way of saying the first one, which seems to be more often what people actually want to say...)

Is this where typos go?