The Problem with AIXI

If you limit the domain of your utility function to a sensory channel, you have already lost; you are forced into a choice between a utility function that is wrong, or a utility function with a second induction system hidden inside it. This is definitely unrecoverable.

However, I see no reason for Solomonoff-inspired agents to be structured that way. If the utility function's domain is a world-model instead, then it can find itself in that world-model and the self-modeling problem vanishes immediately, leaving only the hard but philosophically-valid problem of defining the utility function we want.

I think it's reasonable to expect there to be some way to do better, because humans don't drop anvils on their own heads. That we're naturalized reasoners is one way of explaining why we don't routinely make that kind of mistake.

My kids would have long since have been maimed or killed by exactly that kind of mistake, if not for precautions taken by and active monitoring by their parents.

It's really great to see all of these objections addressed in one place. I would have loved to be able to read something like this right after learning about AIXI for the first time.

I'm convinced by most of the answers to Xia's objections. A quick question:

Yes... but I also think I'm like those other brains. AIXI doesn't. In fact, since the whole agent AIXI isn't in AIXI's hypothesis space — and the whole agent AIXItl isn't in AIXItl's hypothesis space — even if two physically identical AIXI-type agents ran into each other, they could never fully understand each other. And neither one could ever draw direct inferences from its twin's computations to its own computations.

Why couldn't two identical AIXI-type agents recognize one another to some extent? Stick a camera on the agents, put them in front of mirrors and have them wiggle their actuators, make a smiley face light up whenever they get rewarded. Then put them in a room with each other.

Lots of humans believe themselves to be Cartesian, after all, and manage to generalize from others without too much trouble. "Other humans" isn't in a typical human's hypothesis space either — at least not until after a few years of experience.

The following is a meaningless stream of consciousness.

This issue has often sounded to me a little bit like the problem of building recursive inductive types/propositions in type-theory/logic. You can't construct so much as a tree with child nodes without some notation for, "This structure contains copies of itself, or possibly even links back to its own self as a cyclical structure." It continually sounds as if AIXI has no symbol in its hypothesis space that means "me", and even if it did, it would consider hypotheses about "me" more complex than hypotheses without "me".

In type theory, we solve this with a \mu type constructor, so that lists, for instance, can be written as follows:

list \alpha = \mu \beta. <cons = {item: \alpha, \next: \beta}, nil = {}>

Common sense tells us that the phenomenon of lists is thus described much more simply by one recursive type than by countably infinitely many separate similar types. If we include the "recursive concept operator" \mu in the "language" of hypotheses, my intuitions (ie: blind speculation) say Kolmogorov Complexity would correctly weight the hypotheses.

Are there concept or formal-language learning algorithms capable of inducting this kind of recursive concept description? I would figure such an algorithm would be able to learn some hypothesis along the lines of "the output tape corresponds with this physical process by way of the recursive concept me, which can be unfolded into its bridge hypotheses and physical description or folded into the concept of my calculations".

Once again, everything above was a meaningless stream of consciousness, but I'm told one of the ways to make progress is to try to write down your best guesses and look at what's wrong with them.

What happened to the supposed UDT solution?

Alright, let's consider a specific scenario: The AIXI agent is not implemented as a single machine, but as several different machines built in different locations which share data. The agent can experiment and discover that whenever one of the machines is destroyed it can now longer gather data and perform actions in that location. Do you think this agent will behave irrationally about the possibility of destruction for all its host machines? If not, why? (Still, you may argue that the agent will behave irrationally in other self-modification scenarios, such as destroying its communication cables. Right now I'm only trying to establish that AIXI can handle potential death reasonably, unlike what you claim.)

Three AIXI researchers commented on a draft of this post and on Solomonoff Cartesianism. I'm posting their comments here, anonymized, for discussion. You can find AIXI Specialist #1's comments here.

AIXI Specialist #2 wrote:

Pro: This is a mindful and well-intended dialog, way more thoughtful than the average critic of AIXI esp. by computer scientists.

Con: The write-up should be cleaned. It reads like a raw transcript of some live conversation.

Neutral: I think this is good philosophy, and potentially interesting (but only) for when AIXI reaches intelligence way beyond human. The arguments don't show that AIXI runs into problem up-to human-level intelligence. Like discussions on (the hard problem of) consciousness are irrelevant for building AGIs.

Since an embodied AIXI can observe some aspects of its body, it will build a model of it of sufficient quality to speculate about self-modification. While formally AIXI cannot model another AIXI (or itself) perfectly, it will develop increasingly accurate approximations.

Humans will also never be able to fully understand themselves, but that is also not necessary for many self-modification thoughts, except for the most fundamental "Goedelian" questions which neither machines NOR humans can answer. Penrose is wrong.

AIXI is not limited to finite horizon. This is just the simplest model for didactic purpose. The real AIXI has either (universal) discounting (Hutter's book) or mixes over discounted environments (Shane Legg's thesis).

Independent of the discounting, AIXI (necessarily) includes algorithms with finite output (either looping forever) or halting. This can be interpreted as the believe in death. The a-priori probability AIXI assigns to dying at time t is about 2^-K(t), but I think it can get arbitrarily close to 1 with "appropriate" experience. Worth a formal (dis)proof.

Human children start with a 1st-person egocentric world-view, but slowly learn that a 3rd-person world model is a more useful model. How an objective world-view can emerge from a purely subjective perspective by a Solomonoff inductor has been explained in http://dx.doi.org/10.3390/a3040329

There are many subtle philosophical questions related to AIXI, but most can only be definitely solved by formal investigation. More effort should be put into that. Verbal philosophical arguments may lead to that but seldom conclusively prove anything by themselves.

AIXI Specialist #3 wrote, in part:

Unrelated to the article, there are problems with AIXI and other Bayesian agents. For example, it was proven that there exist computable environments where no such agent can be asymptotically optimal [Asymptotically Optimal Agents, Lattimore and Hutter 2011] (an asymptotically optimally optimal policy is one that eventually converges to optimal). However a notion of weak asymptotic optimality can be defined (a weakly asymptotically optimal policy eventually converges to optimal on average), and unfortunately a pure Bayesian optimal policy doesn't satisfy that either on some environments. However, there are agents which can be constructed that do [ibid] . The key problem is in fact that AIXI and other pure Bayesian agents stop exploring at some point, whereas for really tricky environments you need to explore infinitely often (basically environments can be constructed that lull bayesian agents into a false sense of security and then switcharoo. These environments are really annoying. )

That being said, AIXI approximations should still perform very well in the real world. In order to have a real world agent, there needs to be restrictions on the environment class and once these restrictions are made, one can more careful tailor exploration-exploitation trade-offs as well as learning algorithms that exploit structure within the environment class to learn faster (bounds for general learning in specific environment classes are horrendous).

It is not clear that approximations of AIXI are the way to build a computable general intelligence, but AIXI (or similar) serve as useful benchmarks in a theoretical sense.

Interestingly the problems of AIXI are not much different from corresponding ones for human rationality:

• immortalism - humans also don't grasp death on any deeper level than AIXI. They also drop anvils on their head so to speak, i.e. they misinterpret reality to a) be less dangerous than 'expected' or ignored (esp. small children) or b) to contain an afterlife (kind of updating against the a) view later. This is for the same reason AIXI does. Symbolic reasoning about reality.

• preference solipsism - Same here. Reasoning needs some priors. These form from the body the mind is 'trapped' in. But the mind doesn't neccessarily blieve that it is trapped. The body provides the 'reward' button. And that button can be hacked (drugs, optimizing for single values).

• lack of self-improvement - To improve itself AIXI has to be taught via reward what to prefer and its means to achieve this thus follow complex causality. It could work, but it it no direct approach. Humans also cannot directly improve themselves. The desires (reward) for self-improvement needs to be put into neurobio science and that into cyber (or whatnot) enhancements (or uploading).

Xia, in anvil conversation: "What if you have the AIXI as a cartesian lump, and teach it that it's output can only influence a tiny voltage various sensitive sensors can sense, and that if the voltage to it is broken time skips forward until it's reinstated, and gives it a clock tick timeout death prior based on how long the universe has been running rather than how many bits it has outputted? The AI will predict that if it's destroyed the lump wont be found and the voltage nevrreaplied untill the universe spontaneously ceases to exist a few million years later"

Are there toy models of, say, a very simple universe and an AIXItl-type reasoner in it? How complex does the universe have to be to support AIXI? Game-of-life-complex? Chess-complex? D&D complex? How would one tell?

I have discussed this problem with Professor Hutter, and though I wouldn't claim to be able to predict how he would respond to this dialogue, I think his viewpoint is that the anvil problem will not matter in practice. In rough summary of his response: an agent will form a self model by observing itself taking actions through its own camera. When you write something on a piece of paper, you can read what you are writing, and see your own hand holding the pen. Though AIXI may not compress its own action bits, it will compress the results it observes of its actions, and will form a model of its hardware (except perhaps the part that produces and stores those action bits). 

I'm having trouble understanding how something generally intelligent in every respect except failure to understand death or that it has a physical body, could be incapable of ever learning or at least acting indistinguishable from one that does know.

For example, how would AIXI act if given the following as part of its utility function: 1) utility function gets multiplied by zero should a certain computer cease to function 2) utility function gets multiplied by zero should certain bits be overwritten except if a sanity check is passed first

Seems to me that such an AI would act as if it had a genocidally dangerous fear of death, even if it doesn't actually understand the concept.

I don't see how phenomenological bridges solve the epistemological problem, instead of just pushing the problem one step further away. Where in the bridge hypothesis is it encoded that one end of the bridge has a "self", in a way that leads to different behavior?

Let me give an example of AIXI, which creates something that is almost a phenomenological bridge, but remains Cartesian. Imagine that an AIXI finds a magnifying glass. It holds the magnifying glass near its camera, and at the correct focal distance, everything in {world − magnifying glass} looks the same, except upside down. Through experimentation and observation, it realizes that gravity hasn't flipped, it's still on the ground, the lights are still 15 feet above it, etc. It will conclude that the magnifying glass filters visual input on the rest of the world flipping the Y axis. Thus AIXI has a hypothesis about the relation of the magnifying glass with the world.

Phenomenal bridge hypotheses are saying there is something like this magnifying glass, except embedded in...where? What's the difference between reading glasses and retinas? I can have 1 "visual filter hypothesis", 2 visual filter hypotheses, n visual filter hypotheses. What's the distinction between internal filters and world filters? Do I have x internal filters and {n − x} external filters? What would that mean?

See Luke's comment for an explanation of how this series of posts is being written. Huge thanks to Eliezer Yudkowsky, Alex Mennen, Nisan Stiennon, and everyone else who's helped review these posts! They don't necessarily confidently endorse all the contents, but they've done a lot to make the posts more clean, accurate, and informative.

I'd also like to point out the Cartesian barrier is actually probably a useful feature.

It's not objectively true in any sense but the relation between external input, output and effect is very very different than that between internal input (changes to your memories say), output and effect. Indeed, I would suggest there was a very good reason that we took so long to understand the brain. It would be just too difficult (and perhaps impossible) to do so at a direct level the way we understand receptors being activated in our eyes (yes all that visual crap we do is part of our understanding).

Take your example of a sensor aimed at the computer's memory circuit. Unlike almost every other situation there are cases that it can't check it's hypothesis against because such a check would be logically incoherent. In other words certain theories (or at least representations of them) will be diagonalized against because the very experiments you wish to do can't be effected because that 'intention' itself modifies the memory cells in such a way as to make the experiment impossible.

In short the one thing we do know is that assuming that we are free to choose from a wide range of actions independently of the theory we are trying to test and that how we came to choose that action is irrelevant is an effective strategy for understanding the world. It worked for us.

Once the logic of decision making is tightly coupled with the observations themselves the problem gets much harder and may be insoluble from the inside, i.e., we may need to experiment on others and assume we are similar.

I've been following these posts with interest, having suspected a similar problem to the Cartesianism you rail against. My beef is a little different, though: it relates to the fundamental categories of perception. Going back to Cai: cyan, yellow, and magenta are the only allowed categories, and there are a fixed number of regions of the visual field. This is not how naturalized agents seem to operate. Human beings at least occasionally re-describe what they perceive, at any and all levels. Qualia? Their very existence is disputed. Physical objects? Ditto.

Any perceptual judgment we make can be questioned. Labels can be substituted: "This soup tastes like chicken," I say. "No, it tastes like turkey," the woman next to me says. She's right, I realize, and I change my description. Or, entire categories of labels can be substituted, as with philosophical ontological battles.

I'm not sure whether this indicates additional problems for AIXI, but it seems like part and parcel of the problem of Cartesianism.

Xia: It should be relatively easy to give AIXI(tl) evidence that its selected actions are useless when its motor is dead. If nothing else AIXI(tl) should be able to learn that it's bad to let its body be destroyed, because then its motor will be destroyed, which experience tells it causes its actions to have less of an impact on its reward inputs.

Rob B: [...] Even if we get AIXI(tl) to value continuing to affect the world, it's not clear that it would preserve itself. It might well believe that it can continue to have a causal impact on our world (or on some afterlife world) by a different route after its body is destroyed. Perhaps it will be able to lift heavier objects telepathically, since its clumsy robot body is no longer getting in the way of its output sequence.

Compare human immortalists who think that partial brain damage impairs mental functioning, but complete brain damage allows the mind to escape to a better place. Humans don't find it inconceivable that there's a light at the end of the low-reward tunnel, and we have death in our hypothesis space!

I’d like to see this rebuttal spelled out in more detail. Let’s assume for the sake of argument that “we get AIXI(tl) to value continuing to affect the world”. Why would it then be so hard to convince AIXI(tl) that it will be better able to affect the world if no anvils fall on a certain head? (I mean, hard compared to any reasonably-hoped-for alternative to AIXI(tl)?)

If the AIXI(tl) robot has been kept from killing itself for long enough for it to observe the basics of how the world works, why wouldn’t AIXI(tl) have noticed that it is better able to affect the world when a certain brain and body are in good working order and free of obstructions? We can’t give AIXI(tl) the experience of dying, but can’t we give AIXI(tl) experiences supporting the hypothesis that damage to a particular body causes AIXI(tl) to be unable to affect the world as well as it would like?

I can see that AIXI(tl) would entertain hypotheses like, “Maybe dropping this anvil on this brain will make things better.” But AIXI(tl) would also entertain the contrary hypothesis, that dropping the anvil will make things worse, not because it will turn the perceptual stream into an unending sequence of NULLs, but rather because smashing the brain might make it harder for AIXI(tl) to steer the future.

Humans did invent hypotheses like, “complete brain damage allows the mind to escape to a better place”, but there seems to be a strong case for the claim that humans are far more confident in such hypotheses than they should be, given the evidence. Shouldn’t a Solomonoff inductor do a much better job at weighing this evidence than humans do? Why wouldn’t AIXI(tl)’s enthusiasm for the “better place” hypothesis be outweighed by a fear of becoming a disembodied Cartesian spirit cut off from all influence over the only world that it cares about influencing?

So what happens when AIXI determines that there's this large computer, call it BRAIN whose outputs tend to exactly correlate with its outputs? AIXI may then discover the hypothesis that the observed effects of AIXI's outputs on the world are really caused by BRAIN's outputs. It may attempt to test this hypothesis by making some trivial modification to BRAIN so that it's outputs differ from AIXI's at some inconsequential time (not by dropping an anvil on BRAIN, because this would be very costly if the hypothesis is true). After verifying this, AIXI may then determine that various hardware improvements to BRAIN will cause its outputs to more closely match the theoretical Solomonoff Inductor, thus improving AIXI's long term payoff.

I mean, AIXI is waaaay too complicated for me to actually properly predict, but is this scenario actually so unreasonable?

I commented on the previous post a few days after it went up detailing some misgivings about the arguments presented there (I guess you missed my comment). I was reading this post with burgeoning hope that my misgivings would be inadvertently addressed, and then I encountered this:

AIXI doesn't know that its future behaviors depend on a changeable, material object implementing its memories. The notion isn't even in its hypothesis space.

But if "naturalized induction" is a computer program, then the notion is in AIXI's hypothesis space -- by definition.

Going back to the post to read some more...

This is a debate about nothing. Turing completness tells us no matter how much it appears that a given Turing complete representation can only usefully process data about certain kinds of things in reality it can process data about anything any other language can do.

Well duh, but this (and the halting problem) have been taught yet systemically ignored in programming language design and this is exactly the same argument.

We are sitting around in the armchair trying to come up with a better means of logic/data representation (be it a programming language the underlying AI structure) as if the debate is about mathematical elegance or some such objective notion. Until you prove to me that any system in AIXI can duplicate the behavior (modulo semantic changes as to what we call a punishment) the other system can and vice versa that is the likely scenario.

So what would make one model for AI better than another? These vague theoretical issues? No, no more than how fancy your type system is determines the productivenesss of your programming language. Ultimately, the hurdle to overcome is that HUMANS need to build and reason about these systems and we are more inclined to certain kinds of mistakes than others. For instance I might write a great language using the full calculus of inductive constructions as a type system and still do type inference almost everywhere but if my language looks like line noise not human words all that math is irrelevant.

I mean ask yourself why is human programming and genetic programming so different. Because what model you use to build up your system has a far greater impact on your ability to understand what is going on than on any other effects. Sure, if you write in pure assembly JMPs everywhere with crazy code packing tricks it goes faster but you still lose.

If I'm right about this case as well it can only be decided by practical experiments where you have people try and reason in (simplified) versions of the systems and see what can and can't be easily fixed.

Regarding the anvil problem: you have argued with great thoroughness that one can't perfectly prevent an AIXI from dropping an anvil on its head. However, I can't see the necessity. We would need to get the probability of a dangerously unfriendly SAI as close to zero as possible, because it poses an existential threat. However, a suicidally foolish AIXI is only a waste of money.

Humans have a negative reinforcement channel relating to bodily harm called pain. It isn't perfect, but it's good enough to train most humans to avoid doing suicidal stupid things. Why would an AIXI need anything better? Yout might want to answer that there is some danger related to an AIXI s intelligence, but it's clock speed, or whatever, could be throttle, during training.

Also any seriously intelligent .AI made with the technology of today, or the near future, is going to require a huge farm of servers. The only way it could physically interact with the world is through remote controlled body...and if drops an anvil on that, it actually will survive as a mind!