Tiling Agents for Self-Modifying AI (OPFAI #2)

As always in such matters, think of Turing Machines. If the transition function isn't modified, the state of the Turing Machine may change. However, it'll always be in a internal state prespecified in its transition function, it won't get unknown or unknowable new entries in its action table.

Universal Turing Machines are designed to change, to take their transition function from the input tape as input, a prime example of self-modification. But they as well -- having read their new transition function from their input tape -- will go along their business as usual without further changes to their transition function. (You can of course program them to later continue changing their action table, but the point is that such changes to its own action table -- to its own behavior -- are clearly delineated from just contents in its memory / work tape.)

A calculator or a non-self-modifying AI will undergo changes in its memory, but it'll never endeavor to define new internal states, with new rules, on its own. It'll memorize whether you've entered "0.7734" in its display, but it'll only perform its usual actions on that number. A game of tetris will change what blocks it displays on your screen, but that won't modify its rules.

There may be accidental modifications (bugs etc.) leading to unknown states and behavior, but I wouldn't usefully call that an active act of self-modification. (It's not a special case to guard against, other than by the usual redundancy / using checksums. But that's no more FAI research than rather the same constraints as when working with e.g. real time or mission critical applications.)

Thanks for continuing to engage.

I described my position in another comment. To reiterate and elaborate:

1. My current best guess is that there are so many unrelated potential models for AI (relative to the information that we currently have) that the probability of FAI work on a single one of them ending up being relevant is tiny. In order to make a compelling argument for the relevance of MIRI's work on the Lob problem, you have to argue that the model used isn't only one of, e.g. 10^10 distinct models of AI with similar probability of being realized in practice.

2. One could argue that the problem is sufficiently important so that one should work on it even if the probability of the work being relevant is tiny. But there are other interventions on the table. You've made major contributions by spreading rationality and by creating a community for people who are interested in global welfare to network and collaborate with one another. These things probably substantially reduce astronomical waste (in expectation). In order to argue in favor of MIRI's FAI research being optimal philanthropy, you have to argue that the probability of the research being relevant is sufficiently great so that its expected value outweighs the expected value of these other activities.

3. One could argue that if there are in fact so many models for AI then we're doomed anyway, so we should assume that there aren't so many models. But rather than trying to work on the models that we think most relevant now, we can wait until it becomes more clear what AGI will look like in practice, and then develop FAI for that type of AI. Whether or not this is feasible is of course related to the question of whether the world's elites will navigate the creation of AI just fine. I think that there are good reasons to think that the probability of this is pretty high, and that the most leveraged efforts are getting good people in positions of future influence rather than doing FAI research now. Your work on rationality training and community building can help, and already has helped a lot with this.

To have any operationalization of how some specific model of self-modification provably maintains some invariant would be a large step forward, the existence of other models of self-modification nonwithstanding. Safety cannot be proven for all approaches, because not all approaches are safe.

My claim is that there are sufficiently many possible models for AI that given what we (the Less Wrong community, not necessarily AI researchers) know now, the probability of a given model being developed is tiny.

The actionable safety issues that would come up if the AI is like the Chinese economy would be very different from the actionable safety issues that would come up if the AI is like a self-improving chess playing program, which would be very different from the actionable safety issues that would come up if the AI is of the type that Eliezer's publication describes.

Given the paucity of information available about the design of the first AI, I don't think that the probability of doing safety research on a particular model being actionable is sufficiently high for such research to be warranted (relative to other available activities).

Even if that were so, that's not MIRI's (or EY's) most salient comparative advantage (also: CFAR).

1. Eliezer made a major contribution to increasing rationality with his How To Actually Change Your Mind sequence, which improved the rationality of many people who I know, including myself.

2. MIRI could engage in other AI safety activities, such as improving future forecasting.

3. If an organization doesn't have a cost-effective activity to engage in, and the employees recognize this, then they can leave and do something else. Here I'm not claiming that this is in fact the case of MIRI, rather, I'm just responding to your argument.

4. MIRI's staff could migrate to CFAR.

5. Out of all of the high impact activities that MIRI staff could do, it's not clear to me that Friendly AI research is their comparative advantage.

It sounds like you agree, but are saying that some analogous but more complicated problem might arise for probabilistic agents, and that it might not be resolved be whoever else is making AI unless this research is done by MIRI.

That's not it, rather:

I think what you're saying is that getting a good framework for reasoning about reasoning could be important for making AGI go well. This is plausible to me. And then you're also saying that working on this Lobian stuff is a reasonable place to start. This is not obvious to me, but this seems like something that could be subtle, and I understand the position better now.

Yep. We have reasoning frameworks like the currently dominant forms of decision theory, but they don't handle reflectivity well.

The Lob Problem isn't a top-priority scary thing that is carved upon the tombstones of worlds, it's more like, "Look! We managed to crisply exhibit something very precise that would go wrong with standard methods and get started on analyzing and fixing it! Before we just saw in a more intuitive sense that something would go wrong when we applied standard theories to reflective problems but now we can state three problems very precisely!" (Lob and coherent quantified belief sec. 3, nonmonotonicity of probabilistic reasoning sec. 5.2 & 7, maximizing / satisficing not being good-enough idioms for bounded agents sec. 8.) Problems with reflectivity in general are expectedly carved upon the tombstones of worlds because they expectedly cause problems with goal stability during self-modification. But to make progress on that you need crisp problems to provide fodder for getting started on finding a good shape for a reflective decision theory / tiling self-improving agent.

(As usual, I have somewhat less extreme views here than Eliezer.)

saying that some analogous but more complicated problem might arise for probabilistic agents

There is a problem here, we have an impossibility proof for a broad class of agents, and we know of no agents that don't have the problem. Indeed, this limits the relevance of the impossibility proof, but it doesn't limit the realness of the problem.

If you have an example of a complication that you think would plausibly arise in practice and have further thoughts on why we shouldn't expect this complication to be avoided by default in the course of the ordinary development of AI, I would be interested in hearing more.

I don't quite see where you are coming from here. It seems like the situation is:

1. There are problems that reflective reasoners would be expected to solve, which we don't understand how to resolve in current frameworks for general reasoning (of which mathematical logic is the strongest).
2. If you think that reflective reasoning may be an important part of AGI, then having formal frameworks for reflective reasoning is an important part of having formal frameworks for AGI.
3. If you think that having formal frameworks is likely to improve our understanding of AGI, then having formal frameworks that support reflective reasoning is a useful step towards improving our understanding of AGI.

The sort of complication I imagine is: it is possible to build powerful AGI without having good frameworks for understanding its behavior, and then people do that. It seems like all things equal understanding a system is useful, not only for building it but also for having reasonable expectations about its behavior (which is in turn useful for making further preparations, solving safety problems, etc.). To the extent that understanding things at a deep level ends up being necessary to building them at all, then what we're doing won't matter (except insofar as people who care about safety making modest technical contributions is indirectly useful).

Do you think that people building AGI in the future will fail to do this research, if it is in fact necessary for building well-functioning AIs? If so, why?

Same answer. It may be that understanding reasoning well is necessary to building powerful agents (indeed, that would be my mode guess). But it may be that you can influence the relative development of understanding vs. building, in which case pushing on understanding has a predictable effect. For example, if people didn't know what proofs or probabilities were, it isn't out of the question that they could build deep belief nets by empirical experimentation. But I feel safe saying that understanding proof and probability helps you better reason about the behavior of extremely powerful deep belief nets.

Here's a stab at how the arguments are different (first thing that came to mind):

I agree that the cases differ in many ways. But this distinction doesn't seem to get at the important thing. To someone working on logic you would say "I don't know whether deduction systems will be formalized in the future, but I know that agents will be able to reason. So this suggests to me that your particular approach for defining reasoning, via formalization, is unnecessary." In some sense this is true---if I'm an early mathematician and I don't do logic, someone else will---but it has relatively little bearing on whether logic is likely to be mathematically productive to work on. If the question is about impact rather than productivity as a research program, then see the discussion above.

We have coherent answers at least. See e.g. here for a formalism (and similarly the much older stuff by Gaifman, which didn't get into priors).

I read that paper before but it doesn't say why its proposed way of handling logical uncertainty is the correct one, except that it "seem to have some good properties". It seems like we're still at a stage when we don't understand logical uncertainty at a deep level and can offer solutions based on fundamental principles, but just trying out various ideas to see what sticks.

I agree that AI progress is probably socially costly [...] Moreover, as long as safety-concerned folks are responsible for a very small share of all of the good AI work, the reputation impacts of doing good work seem very large compared to the social benefits or costs.

I'm not entirely clear on your position. Are you saying that theoretical AI work by safety-concerned folks has a net social cost, accounting for reputation impacts, or excluding reputation impacts?

I think that the kind of probabilistic reflection we are working on is fairly natural though.

Maybe I'm just being dense but I'm still not really getting why you think that (despite your past attempts to explain it to me in conversation). The current paper doesn't seem to make a strong attempt to explain it either.

I would guess that all told the math will help on the strategy front via building the general credibility of AI safety concern

Also, by attracting thinkers who can initially only be attracted by crisp technical problems, but as they get involved, will turn their substantial brainpower toward the strategic questions as well.

For three additional reasons for MIRI to focus on math for now, see the bullet points under "strategic research will consume a minority of our research budget in 2013" in MIRI's Strategy for 2013.

All that said, I do think that where MIRI has technical FAI questions to work on now, I think it is a very reasonable to write up:

• what the question is
• why answering the question is important for making machine intelligence benefit humanity
• why we shouldn't expect the question to be answered by default by whoever makes AGI

In this particular case, I am asking for more info about the second two questions.

Ah, so my question was more along the line: does finite axiomatizability of a stronger (consistent) theory imply finite axiomatizability of the weaker theory?

If I'm not mistaken, NBG and ZFC are a counterexample to this: NBG is a conservative extension of ZFC (and therefore stronger than ZFC), but NBG is finitely axiomatizable while ZFC is not.

Yeah, the details of actually proving this are looking like they contain more subtleties than I expected, but I tentatively agree with your analysis. Here's what may be another proof,. Not only is PA not finitely axiomatizable, but any consistent extension of PA isn't (I think this is true, the same proof that works for PA should go through here, but I haven't checked the details). So PA+ConMinus(PA) still isn't finitely axiomatizable. So now, pick any of the axioms created in the axiom schema of induction that are needed in PA + ConMinus(PA), Q+ConMinus(PA) also can't prove any of those (since it is strictly weaker than PA+ConMinus(PA),) but all of those statements are theorems of PA (since they are in fact axioms). Does this work?

Overall, this is requiring a lot more subtlety than I initially thought was involved which may make Qiaochu Yuan's example a better one.

Sorry for the long-delayed reply, Wei!

So you think that humans do not have a built-in solution to the Löbstacle, and you must also think we are capable of building an FAI that does have a built-in solution to the Löbstacle. That means an intelligence without a solution to the Löbstacle can produce another intelligence that shares its values and does have a solution to the Löbstacle.

Yup.

But then why is it necessary for us to solve this problem? [...] Why can't we instead built an FAI without solving this problem, and depend on the FAI to solve the problem while it's designing the next generation FAI?

I have two main reasons in mind. First, if you are willing to grant that (a) this is a problem that would require humans years of serial research to solve and (b) that it looks much easier to build this into an AI designed from scratch rather than bolting it on to an existing AI design that was created without taking these considerations into account, but you still think that (c) it would be a good plan to have the first-generation FAI solve this problem when building the next-generation FAI, then it seems that you need to assume that the FAI will be much better at AGI design than its human designers before it executes its first self-rewrite, since the human team would by assumption still need years to solve the problem at that point and the plan wouldn't be particularly helpful if the first-generation FAI would need a similar amount of time or longer. But it seems unlikely to me that we first need to build ultraintelligent machines a la I.J. Good, far surpassing humans, before we can get an intelligence explosion: it seems to me that most of the probability mass should be in the required level of AGI research ability being <= the level of the human research team working on the AGI. I admit that one possible strategy could be to continue having humans improve the initial FAI until it is superintelligent and then ask it to write a successor from scratch, solving the Löbstacle in the process, but it doesn't seem particularly likely that this is cheaper than solving the problem beforehand.

Second, if we followed this plan, when building the initial FAI we would be unable to use mathematical logic (or other tools sufficiently similar to be subject to the same issues) in a straight-forward way when having it reason about its potential successor. This cuts off a large part of design-space that I'd naturally be looking to. Yes, if we can do it then it's possible in principle to get an FAI to do it, but mimicking human reasoning doesn't seem likely to me to be the easiest way to build a safe AGI.

Have you been following the discussions under my most recent post?

I agree with you that relying on an FAI team to solve a large number of philosophical problems correctly seems dangerous, although I'm sympathetic to Eliezer's criticism of your outside-view arguments -- I essentially agree with your conclusions, but I think I use more inside-view reasoning to arrive at them (would need to think longer to tease this apart). I agree with Paul that something like CEV for philosophy in addition to values should probably part of an FAI design. I agree with you that progress in metaphilosophy would be very valuable, but I do not have any concrete leads to follow. But I think that having good solutions to some of these problems is not unlikely to be helpful for FAI design (and more helpful to FAI than uFAI) so I still think that some amount of work allocated to these philosophical problems looks like a good thing; and I also think that working on these problems does on average reduce the probability of making a bad mistake even if we manage to have the FAI do philosophy itself and have it checked by "coherent extrapolated philosophy".

You quoted my earlier comment that I think that making object-level progress is important enough that it seems a net positive despite making AGI research more interesting, but I don't really feel that your post or the discussion below that contains much in the way of arguments about that -- could you elaborate on the connection?

Many things in AI that look like they ought to be easy have hidden gotchas which only turn up once you start trying to code them, and we can make a start on exposing some of these gotchas by figuring out how to do things using unbounded computing power (albeit this is not a reliable way of exposing all gotchas, especially in the hands of somebody who prefers to hide difficulties, or even someone who makes a mistake about how a mathematical object behaves, but it sure beats leaving everything up to verbal arguments).

Human beings don't make billions of sequential self-modifications, so they're not existence proofs that human-quality reasoning is good enough for that.

I'm not sure how to go about convincing you that stable-goals self-modification is not something which can be taken for granted to the point that there is no need to try to make the concepts crisp and lay down mathematical foundations. If this is a widespread reaction beyond yourself then it might not be too hard to get a quote from Peter Norvig or a similar mainstream authority that, "No, actually, you can't take that sort of thing for granted, and while what MIRI's doing is incredibly preliminary, just leaving this in a state of verbal argument is probably not a good idea."

Depending on your math level, reading Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference by Judea Pearl might present you with a crisper idea of why it can be a good idea to formalize certain types of AI problems in general, and it would be a life-enriching experience, but I anticipate that's more effort than you'd want to put into this exact point.

I don't see it is a decisive point, one of "many weak arguments," but I think the analogy with human self-modification is relevant. I would like to see more detailed discussion of the issue.

Aspects of this that seem relevant to me:

• Genetic and cultural modifications to human thinking patterns have been extremely numerous. If you take humanity as a whole as an entity doing self-modification on itself, there have been an extremely large number of successful self-modifications.
• Genetic and cultural evolution have built humans individually capable of self-modification without stumbling over Lobian obstacles. Evolution and culture likely used relatively simple and easy search processes to do this, rather than ones that rely on very sophisticated mathematical insights. Analogously, one might expect that people will develop AGI in a way that overcomes these problems as well.