Friendly AI Research and Taskification

You have hundreds of subproblems that need to be solved? And you're making a special effort to keep them secret from people on LW? Just... wow. How dumb would you have to be? Excuse me while I beat my head against the wall for awhile.

I'm not speaking for SIAI as this is more of a Visiting Fellows thing than an SIAI thing, but there are people working on Friendliness, and creating a Friendliness roadmap. We have lists of hundreds of problems, and lists of potentially relevant fields or concepts.

Meh. Now I'm a bit annoyed in that I did try to poke people into a direction where they'd do something like that when I was there as a Visiting Fellow, but mostly the reaction seemed to be "we should leave all thinking about Friendliness to Eliezer". But upon reflection, I realize that I may not have been as vocal about that as I thought I was (illusion of transparency and all that), so I guess I only have myself to blame for you guys only starting on all the real interesting stuff after I left. ;p

1. I'm encouraged by what you say here. The doubt as to the value of Friendliness research that I express above is doubt as to the value of researching Friendly AI without a taskification rather than doubt as to the value researching what a taskification might look like.

2. If you haven't done so I think that it would be worthwhile to ask the SIAI staff whether they might be comfortable with classifying (some of?) the output of the SIAI Visiting Fellows as part of SIAI's output. As I said in response to a comment by WrongBot, I've gathered that the SIAI visiting fellows program is a good thing; but there's been relatively little public documentation of what the SIAI visiting fellows have been doing. I would guess that a policy of such public documentation would improve SIAI's credibility.

3. While I didn't read your comment in the way that cousin_it did, I can see why he would do so. I've gotten a vague impression from talking to a number of people loosely or directly connected with SIAI that SIAI has been keeping their research secret on the grounds that releases to the public could be dangerous on account of speeding unfriendly AI research. In view of how primitive the study of AGI looks, the apparent infeasibility of SIAI unilaterally building the first AGI and the fact that Friendliness research would not seem to significantly speed the creation of unfriendly AI; such a policy seems highly dubious to me. So I was happy to hear that you and your collaborators are planning on putting some of what you've been doing out in the open in roughly a few months.

4. Thanks for the link to your blog posts.

This sounds sort of interesting but seems to fall under the rubric of general artificial intelligence research rather than addressing the taskification of Friendly AI research. Note that a research program is only as strong as its weakest link.

The beginning of your post My Kind of Reflection seems to talk about that. Couldn't find anything more direct.

FWIW, dealing with self-modification isn't very high on my to-do list, because for now I've shifted to thinking of AI as a one-action construction. This approach handles goal stability pretty much automatically, but I'm not sure if it satisfies your needs.

1. You use cognitive neuroscience to figure out human values and human metaethical thought.

Why do we need to know about meta-ethics? Why would we even think there is such a thing as human meta-ethics?

Meta-ethics may be relevant for crafting ways to approach the problem. But I don't see how it is a descriptive project answerable with the techniques of cognitive neuroscience the way you describe. I also don't see why an AI would need to know whether we were cognitivists or non-cognitivists to know how we want it to act.

Thanks for your interesting comment.

To me, that defines a research program that's just full of highly concrete tasks waiting to be carried out. Some of them may be very difficult or abstract, but nothing leaves me feeling helpless, with no idea where to begin.

I imagine that writing up a description of many such concrete tasks would help clarify the situation; I'd encourage you to do so if you have some time.

However, you need to know something about ordinary AI (try this), and you need to have some idea of what a computational model of human decision making might look like (simple examples).

Yes, I don't have subject matter knowledge. Thanks for the links.

1. It can simultaneously be the case that whole-brain emulation is unlikely to arise first and that pursuing stable whole-brain emulation is more likely to give rise to a positive singularity than by pursuing Friendly AI.

2. The alleged facts that you cite about software engineering don't seem relevant here: as far as I know the current state of general artificial intelligence research is still very primitive.

3. I would recur to FAWS's response to one of your comments from nine months ago.

Tangential question:

...good old-fashioned software engineering has revolutionised the financial markets, saving the world billions of dollars of bad investments,...

Is this true? I'd like it to be true for many reasons. But I have seen some plausible arguments that in recent years the financial sector has not been doing its job in the economy very well. For example, http://www.newyorker.com/reporting/2010/11/29/101129fa_fact_cassidy?printable=true .

Here I think that if each of the conceptual steps is feasible on a short enough time scale so that humans there's a reasonable chance of finishing them all before a hypothetical intelligence explosion then researching decision theory is a visible way forward.

But I'm doubtful as to the feasibility of the stages of the proposed research plan beyond the development of decision theory in absence of a detailed taskification of the latter stages.

I could imagine research in decision theory leading to the creation of a Friendly AI, but the same could be true of any area of basic research. For example, the study of solid state physics could lead to useful new technologies which can meet many people's needs cheaply and correspondingly quell political unrest; leading to stable political conditions which are more conducive to militaries taking safety precautions in developing artificial intelligence technologies; thereby averting unfriendly AI for long enough for people to come up with a more promising approach to the currently intractable aspects of the your proposed research program.

Also; supposing that the research program that you allude to does become taskified to a sufficiently fine degree so that it looks tractable it's plausible that there will be a surge of interest in the relevant decision theory and that academia will solve the relevant problems on its own accord.

To be clear: I'm not necessarily discouraging you personally from studying decision theory - you're visibly passionate about it and my observation is that people are much better at doing what they're passionate about than what they do out of a sense of duty. At the same time; I don't see why decision theory deserves higher priority than other basic scientific research which could plausibly have favorable technological consequences.

As a disclosure of personal bias I personally don't find decision theory at all aesthetically attractive (yet?) and it correspondingly seems like something that I would not enjoy or be good at, so I may be motivated to diminish its utilitarian importance or be blind to it. Regardless, I do appreciate that there are people like you, cousin it and Wei Dai who are strongly interested in the subject as I think that it's good for society to have a diversity of intellectuals researching a variety of subjects.

If you strongly believe that researching decision theory presently deserves high priority for researchers in general at present then I would encourage you to write some articles about why you see it as deserving such high priority with a view toward attracting collaborators and helping SIAI explain its focus on decision theory. Some of your thinking here has come out implicitly in your responses to some of my comments but I would be interested in hearing a more holistic account of your views and their justification.

I don't understand your comment. Surely your estimate of the usefulness of FAI work should depend on your estimate of its chances of success - not just its importance - because otherwise praying to Zeus would look even more attractive than developing FAI. Multi was pointing out that we don't seem to have any viable way of attacking the problem right now.

1. Was the state of nuclear science really so primitive in 1939 that there wasn't a discernable research program?

2. Is it really the case that upload precursors have 3-4 orders of magnitude more funding than FAI research?

3. In my view whether or not the marginal benefit is in FAI depends on whether there people have potentially fruitful ideas on the subject. My post inquires about whether people have such ideas

4. I'm still open to changing my mind.

5. My position on SIAI has changed since August; I have a more favorable impression now; the question is just what the optimal strategy is for the organization to pursue.

It's true that in 1939 they didn't know how to split an atom. They also didn't know how to teleport, or travel backward in time, or do many other dangerous things. Should they have worried about those, too? What percentage of futuristic technologies ever gets developed? What percentage gets developed soon? It might be rational to worry about an unknown future, but it's irrational to worry about one specific scenario of doom unless you have lots of evidence that it will in fact happen.

Fixed, thanks.

Published for real this time.

Yes, published by accident; will repose later