Outside View(s) and MIRI's FAI Endgame

Here's David Chalmers addressing that claim:

the 2009 PhilPapers Survey surveyed around 1000 professional philosophers on answers to thirty important questions in philosophy, and typically found that answers to major questions were distributed something like 50-50 or 60-40 or 70-30, once agnostics and other intermediate options were removed. This suggests that at least where these questions are concerned, large collective convergence has not been achieved.

Now, you might say that these are the big questions of the moment and therefore are precisely those that are unanswered, so the result is no surprise. There is correspondingly little agreement on the current big questions of physics: the status of string theory, for example. To avoid this worry, it is important that the big questions be individuated not by current debate but by past importance.

To properly address this issue, we would need analogs of the PhilPapers survey in (for example) 1611, 1711, 1811, 1911, and 2011, asking members of the community of philosophers at each point first, what they take to be the big questions of philosophy, and second, what they take to be the answers to those questions (and also the answers to any big questions from past surveys). We would also need to have analogous longitudinal surveys in other fields: the MathPapers Survey, the PhysPapers survey, the ChemPapers Survey, the BioPapers Survey, and so on. And we would need a reasonable measure of convergence to agreement over time. I predict that if we had such surveys and measures, we would find much less convergence on answers to the big questions suggested by past surveys of philosophers than we would find for corresponding answers in other fields.

The way I see this, among the problems once considered philosophical, there are some subsets that turned out to be much easier than others, and which are no longer considered part of philosophy. These are generally problems where a proposed solution can be straightforwardly verified, for example by checking a mathematical proof, or through experimental testing.

Given that the philosophical problems involved in designing FAI do not seem to fall into these subsets, it doesn't obviously make sense to include "problems once considered philosophical" in the reference class for the purposes I described in the OP, but maybe I should give this some more thought. To be clear, are you actually making this suggestion?

For the examples I can think of (mostly philosophy of mind), it seems to me that the sciences would have emerged whether or not any progress was made while it was still considered the domain of philosophy. Are there better examples, where the "philosophical" progress was actually important for the later "scientific" progress?

What does the length of the answer have to do with how hard a problem is? The answer to P=NP can fit in 1 bit, but that's still a hard problem, I assume you agree?

Perhaps by "answer" you also mean to include all the the justifications necessarily to show that the answer is correct. If so, I don't think we can fit the justification to an actual answers to a hard philosophical problem on one page or less. Actually I don't think we know how to justify a philosophical answer (in the way that we might justify P!=NP by giving a mathematical proof), so the best we can do is very slowly gain confidence in an idea, by continuously trying (and failing) to poke holes in it or trying (and failing) to find better solutions.

In a PM you imply that you've found the true answers to 'free will', 'does a tree fall in the forest', 'the nature of truth'. I'll grant you 'does a tree fall in the forest' (since your solution appears to be the standard answer in philosophy, although note how it says the problem is "untypically simple"). However I have strong reservations about 'free will' and 'the nature of truth' from both the inside-view perspective and (more relevant to the current post) the outside-view perspective. Given the history of philosophy and the outside view, I don't see how you can be as confident about your ideas as you appear to be. Do you think the outside view is inapplicable here, or that I'm using it wrong?

It's hard for me to argue with multiple people simultaneously. When I argue with someone I tend to adopt most of their assumptions in order to focus on what I think is the core disagreement, so to argue with someone else I have to "swap in" a different set of assumptions and related arguments. The OP was aimed mostly at Eliezer, so it assumed that intelligence explosion is relatively easy. (Would you agree that if intelligence explosion was easy, then it would be hard to achieve a good outcome in the way that you imagine, by incrementally solving "the AI control problem"?)

If we instead assume that intelligence explosion isn't so easy, then I think the main problem we face is value drift and Malthusian outcomes caused by competitive evolution (made worse by brain emulations and AGIs that can be easily copied), which can only be prevented by building a singleton. (A secondary consideration involves other existential risks related to technological progress, such as physics/nanotech/biotech disasters.) I don't think humanity as a whole is sufficiently strategic to solve this problem before it's too late (meaning a lot of value drift has already occurred or building a singleton becomes impossible due to space colonization). I think the fact that you are much more optimistic about this accounts for much of our disagreement on overall strategy, and I wonder if you can explain why. I don't mean to put the burden of proof on you, but perhaps you have some ready explanation at hand?

could you provide a hard philosophical problem (of the kind for which feedback is impossible) together with an argument that this problem must be resolved before human-level AGI arrives?

I can't provide a single example because it depends on the FAI design. I think multiple design approaches are possible but each involves its own hard philosophical problems.

To try to make my point clearer (though I think I'm repeating myself): we can aim to build machine intelligences which pursue the outcomes we would have pursued if we had thought longer (including machine intelligences that allow human owners to remain in control of the situation and make further choices going forward, or bootstrap to more robust solutions). There are questions about what formalization of "thought longer" we endorse, but of course we must face these with or without machine intelligence.

At least one hard problem here is, at you point out, how to formalize "thought longer", or perhaps "remain in control". Obviously an AGI will inevitably influence the options we have and the choices we end up making, so what does "remain in control" mean? I don't understand your last point here, that "we must face these with or without machine intelligence". If people weren't trying to build AGI and thereby forcing us to solve these kinds of problems before they succeed, we'd have much more time to work on them and hence a much better chance of getting the answers right.

For the most part, the questions involved in building such an AI are empirical though hard-to-test ones---would we agree that the AI basically followed our wishes, if we in fact thought longer?---and these don't seem to be the kinds of questions that have proved challenging, and probably don't even count as "philosophical" problems in the sense you are using the term.

If we look at other empirical though hard-to-test questions (e.g., what security holes exist in this program) I don't see much reason to be optimistic either. What examples are you thinking of, that makes you say "these don't seem to be the kinds of questions that have proved challenging"?

I suspect I also object to your degree of pessimism regarding philosophical claims, but I'm not sure and that is probably secondary at any rate.

I'm suspecting that even the disagreement we're current discussing isn't the most important one between us, and I'm still trying to figure out how to express what I think may be the most important disagreement. Since we'll be meeting soon for the decision theory workshop, maybe we'll get a chance to talk about it in person.

Thanks for the link. I don't think I've seen that comment before. Steve raises the examples of Bayesian decision theory and Solomonoff induction to support his position, but to me both of these are examples of philosophical ideas that looked really good at some point but then turned out to be incomplete / not quite right. If the FAI team comes up with new ideas that are in the same reference class as Bayesian decision theory and Solomonoff induction, then I don't know how they can gain enough confidence that those ideas can be the last words in their respective subjects.

Wei Dai, I noticed on the MIRI website that you're slotted to appear at some future MIRI workshop. I find this a little bit strange—given your reservations, aren't you worried about throwing fuel on the fire?

Well I'm human which means I have multiple conflicting motivations. I'm going because I'm really curious what direction the participants will take decision theory.

Wei Dai, I noticed on the MIRI website that you're slotted to appear at some future MIRI workshop. I find this a little bit strange—given your reservations, aren't you worried about throwing fuel on the fire?

I don't see why that would be strange. Maybe Wei Dai thinks he can improve the way MIRI handles their FAI endgame from the inside, or that helping MIRI make progress on decision theory will not make MIRI more likely to screw up and develop UFAI, or both.

Showing up at a workshop is probably a good way to check the general sanity level/technical proficiency of MIRI. I would anticipate having a largish update (although I don't know in which direction, obviously).

(to be specific, I don't think MIRI proponents have adequately addressed concerns like those expressed by Wei Dai here and elsewhere)

I do wonder why MIRI people often do not respond to my criticisms about their strategy. For example the only MIRI-affiliated person who responded to this post so far is Paul Christiano (but given his disagreements with Eliezer, he isn't actually part of my intended audience for this post). The upcoming workshop might be a good opportunity to see if I can get MIRI people to take my concerns more seriously, if I talk to them face to face. If you or anyone else has any ideas on what else I should try, please let me know.

Related: http://lesswrong.com/lw/c3/the_sin_of_underconfidence/

I'd much prefer we replace it with far more definable goals like "preservation of homo sapiens / genetic species" and the like.

Then how about going and defining it? How much genes do you have to exchange via gene therapy for someone to stop being part of the genetic species homo sapiens?