SIAI’s Short-Term Research Program
post by XiXiDu · 2011-06-24T11:43:04.500Z · LW · GW · Legacy · 48 commentsOne of the reasons that I am skeptical of contributing money to the SIAI is that I simply don't know what they would do with more money. The SIAI currently seems to be viable. Another reason is that I believe that an empirical approach is required, that we need to learn more about the nature of intelligence before we can even attempt to solve something like friendly AI.
I bring this up because I just came across an old post (2007) on the SIAI blog:
We aim to resolve this crucial question by simultaneously proceeding on two fronts:
1. Experimentation with practical, contemporary AI systems that modify and improve their own source code.
2. Extension and refinement of mathematical tools to enable rigorous formal analysis of advanced self-improving AI’s.[...]
For the practical aspect of the SIAI Research Program, we intend to take the MOSES probabilistic evolutionary learning system, which exists in the public domain and was developed by Dr. Moshe Looks in his PhD work at Washington University in 2006, and deploy it self-referentially, in a manner that allows MOSES to improve its own learning methodology.
[...]
Applying MOSES self-referentially will give us a fascinating concrete example of self-modifying AI software – far short of human-level general intelligence initially, but nevertheless with many lessons to teach us about the more ambitious self-modifying AI’s that may be possible.
[...]
We are seeking additional funding so as to enable, initially, the hiring of two doctoral or post-doctoral Research Fellows to focus on the above two areas (practical and theoretical exploration of self-modifying AI).
[...]
Part of our goal is to make progress on these issues ourselves, in-house within SIAI; and part of our goal is to, by demonstrating this progress, interest the wider AI R&D community in these foundational issues. Either way: the goal is to move toward a deeper understanding of these incredibly important issues.
[...]
SIAI must boot-strap into existence a scientific field and research community for the study of safe, recursively self-improving systems; this field and community doesn’t exist yet.
Some questions:
- Has any progress been made on the points mentioned in the announcement above?
- Is the SIAI still willing to pursue experimental AI research or does it solely focus on hypothetical aspects?
- What would the SIAI do given various amounts of money?
I also have some questions regarding the hiring of experts. Is there a way to figure out what exactly the current crew is working on in terms of friendly AI research? Peter de Blanc seems to be the only person who has done some actual work related to artificial intelligence.
I am aware that preparatory groundwork has to be done and capital has to be raised. But why is there no timeline? Why is there no progress report? What is missing for the SIAI to actually start working on friendly AI? The Singularity Institute is 10 years old, what is planned for the decade ahead?
48 comments
Comments sorted by top scores.
comment by Kaj_Sotala · 2011-06-24T17:50:37.972Z · LW(p) · GW(p)
AFAIK, that was the stuff Goertzel was doing as Director of Research. Now that he isn't around anymore, those things were dropped.
Pretty much the whole "practical experimentation" angle is, again AFAIK, considered too unsafe by the people currently running things at SIAI. At least that's what I was told during my Visiting Fellow time.
Replies from: Vladimir_Nesov↑ comment by Vladimir_Nesov · 2011-06-24T20:43:35.402Z · LW(p) · GW(p)
considered too unsafe
I expect improving on state of the art in practical AI is also almost totally useless for figuring out a way towards FAI, so "unsafe" is almost beside the point (except that making things worse while not making them better is not a good plan).
Replies from: jsteinhardt↑ comment by jsteinhardt · 2011-06-24T23:49:22.228Z · LW(p) · GW(p)
How do you expect to prove anything about an FAI without even knowing what an AGI would look like? I don't think current AI researchers even have that great of an idea of what AGI will eventually look like...
Now improving on state of the art might not be helpful but being in a position where you could improve on state of the art would be; and the best way to make sure you are in such a position is to have actually done it at least once.
Replies from: Vladimir_Nesov↑ comment by Vladimir_Nesov · 2011-06-24T23:51:40.589Z · LW(p) · GW(p)
How do you expect to prove anything about an FAI without even knowing what an AGI would look like? I don't think current AI researchers even have that great of an idea of what AGI will eventually look like...
It will be (and look) the way we make it. And we should make it right, which requires first figuring out what that is.
Replies from: jsteinhardt, Will_Newsome↑ comment by jsteinhardt · 2011-06-25T13:01:34.776Z · LW(p) · GW(p)
An AGI is an extremely complex entity. You don't get to decide arbitrarily how to make it. If nothing else, there are fundamental computational limits on Bayesian inference that are not even well-understood yet. So if you were planning to make your FAI a Bayesian then you should probably at least be somewhat familiar with these issues, and of course working towards their resolution will help you better understand your constraints. I personally strongly suspect there are also fundamental computational limits on utility maximization, so if you were planning on making your FAI a utility maximizer then again this is probably a good thing to study. Maybe you don't consider this AGI research but the main approach to AGI that I consider feasible would benefit at least somewhat from such understanding.
In my opinion, provably friendly AI is hopeless to get to before someone else gets to AGI. The best thing one can hope for is (i) brain uploads come first, or (ii) a fairly transparent AGI design coupled with a good understanding of meta-ethics. This means that as far as I can see, if you want to reduce x-risk from UFAI then you should be doing one of the following:
- working towards brain uploads to make sure they come first
- working on the statistical approach to AI to make sure it gets to AGI before the connectionist approach (and developing software tools to help us better understand the statistical algorithms we write)
- working on something like lukeprog's program of metaethics (this is probably the best of the three)
↑ comment by Will_Newsome · 2011-06-25T06:08:43.729Z · LW(p) · GW(p)
Do you know where the "we have to have to work towards AGI before we can make progress on FAI" meme came from? (I'm not sure if that's a caricature of the position or what.)
Replies from: Kaj_Sotala↑ comment by Kaj_Sotala · 2011-06-25T07:11:26.411Z · LW(p) · GW(p)
It's an exaggaration in that form, but a milder version seems pretty obvious to me. If you want to design a safe airplane, you need to know something about how to make a working airplane in the first place.
While there are certainly theoretical parts of FAI theory that you can make progress on even without knowing anything about AGI, there's probably a limit to how far you can get that way. For your speculations to be useful, you'll sooner or later need to know something about the design constraints. And they're not only constraints - they'll give you entirely new ideas and directions you wouldn't have considered otherwise.
It sounds nonsensical to claim that you could design safe airplanes without knowing anything about airplanes, that you could be a computer security expert without knowing anything about how software works, or that you could design a safe building without knowing anything about architecture. Why would it make any more sense to claim that you could design FAI without knowing AGI?
Replies from: Vladimir_Nesov, Will_Newsome↑ comment by Vladimir_Nesov · 2011-06-25T12:59:46.280Z · LW(p) · GW(p)
In this analogy, the relevant concern maps for me to the notion of "safety" of airplanes. And we know what "safely" for airplanes is. It means people don't die. It's hard to make a proper analogy, since for all usual technology the moral questions are easy, and you are left with technical questions. But with FAI, we also need to do something about moral questions, on an entirely new level.
Replies from: Kaj_Sotala↑ comment by Kaj_Sotala · 2011-06-25T16:40:19.152Z · LW(p) · GW(p)
I agree that solving FAI also involves solving non-technical, moral questions, and that considerable headway can probably be made on these without knowledge about AGI. I was only saying that there's a limit on how far you can get that way.
How far or near that limit is, I don't know. But I would think that there'd be something useful to be found from pure AGI earlier than one might naively expect. E.g. the Sequences draw on plenty of math/compsci related material, and I expect that likewise some applications/techniques from AGI will also be necessary for FAI.
↑ comment by Will_Newsome · 2011-06-25T08:45:43.726Z · LW(p) · GW(p)
We know tons about AGI already, why do people presume we need to learn more? (It's totally obvious how to make safe airplanes before you have a Wright flyer. Use lots of cushions, use two engines, make everything redundant actually, make everything of incredibly high quality material, et cetera. I think your analogy is way too leaky.)
Replies from: jsteinhardt, cousin_it, Kaj_Sotala↑ comment by jsteinhardt · 2011-06-25T12:39:16.472Z · LW(p) · GW(p)
That is not in fact how to design safe airplanes. Airplanes are safe mainly because they don't crash. Making everything redundant in an effective way is quite nontrivial and probably depends on the specific design of the aircraft (if you disagree then provide a (rough) algorithm for how to make everything redundant in a way that actually achieves the safety levels of modern aircraft). More importantly, redundancy is not particularly helpful if we don't already know that the plane is guaranteed to fly properly in a wide range of operating conditions that are precisely enough quantified that we can determine in advance whether to ground a plane because e.g. the weather is too extreme.
↑ comment by cousin_it · 2011-06-25T13:40:38.350Z · LW(p) · GW(p)
I think you cannot figure out airplane safety before you build airplanes. For example, you'd have to be almost superhumanly smart to predict and beat flutter without experimenting. Or spin, as Kaj_Sotala pointed out. Or spatial disorientation. Even without going into details, can you complete the list of problems by yourself?
↑ comment by Kaj_Sotala · 2011-06-25T12:40:48.032Z · LW(p) · GW(p)
Nobody knowing how to build AGI hardly counts as "knowing tons already".
Some of the stuff about how to make safe airplanes is obvious. Stuff like how to design the plane so as to minimize the risk of spin isn't. If you disagree, please tell me how to build a maximally stall-safe airplane without looking at the Wikipedia link or otherwise looking it up.
Replies from: Will_Newsome↑ comment by Will_Newsome · 2011-06-26T00:26:57.871Z · LW(p) · GW(p)
Humanity basically already knows how to build AGI. That doesn't really matter, though. FAI and AGI are incredibly different problems. They share basically none of the same architectural features. I think that understanding what an AGI is or what it would do is incredibly important for FAI; is that what you mean? But knowing how to build AGI doesn't do that for you; you still have to actually turn it on.
Agree that my parenthetical was straight-up wrong, but it was indeed a parenthetical. I still think the analogy is too loose to be useful.
Replies from: asr↑ comment by asr · 2011-06-26T09:48:50.382Z · LW(p) · GW(p)
I'd like evidence of the claim that humanity, or some subset thereof, "basically already knows how to build AGI". I would stipulate that we know how to do whole-brain emulation, giving adequate computational resources; is that what you had in mind?
Replies from: Will_Newsome↑ comment by Will_Newsome · 2011-06-26T12:31:12.771Z · LW(p) · GW(p)
No.
Replies from: Kaj_Sotala, Peterdjones↑ comment by Kaj_Sotala · 2011-06-27T11:10:58.788Z · LW(p) · GW(p)
You are aware that exchanges such as this one will make most people think you're a troll, right?
Replies from: Will_Newsome↑ comment by Will_Newsome · 2011-06-27T12:20:03.791Z · LW(p) · GW(p)
Can I be aware of something that I think might be false? Anyway, I am aware that people do not seem to understand that a small amount of information when requested is significantly better than no information, and I am aware that there are no affordances that would allow people to take the 3 seconds of reflection that would be necessary to realize this after realizing that their implicit expectations about how much information they deserve to receive from near-anonymous people are probably not based on any sort of explicitly justified or explicitly reflected-upon analysis. Because there's never an affordance for basic sanity, especially not when you're busy climbing over the dead bodies towards the top slot of a huge negative sum signalling game and you're not even sure why you're playing and you don't really care to find out. Ya know, what I love about memes is that even if Buddhas never have kids it doesn't matter much, any mind can become a vessel for perfection via reflection on clarification of perception. The genes can abstain from burning the cosmic commons with viral waste -- how much of karma is the timeless-analogous-controlling-decision to engage in the class of actions that prominently includes negative sum signalling games? -- while the memes spread farther and farther even without so desiring, simply by the timeless beauty of their nature. Right makes might makes right, as they say.
Replies from: MixedNuts, Kaj_Sotala, Richard_Kennaway, Multiheaded↑ comment by MixedNuts · 2011-06-27T12:56:45.342Z · LW(p) · GW(p)
- When someone tells you a normally hidden consequence of what you're doing, you don't tell them off.
- The norm is to give a full answer. If you don't, say you will elaborate later, give a summary, or say it's too long to explain. Don't unilaterally defect.
- If you defect, you're telling people your time is more precious than theirs. This is rude.
- In particular, you bother to post on LW in the first place. "Near-strangers deserve no information from me" is unfair.
- We want to promote pointing out specific problems. A rant against memetics and signalling games in general is a bad answer to specific criticism.
↑ comment by Kaj_Sotala · 2011-06-27T13:38:21.472Z · LW(p) · GW(p)
that a small amount of information when requested is significantly better than no information
That's assuming that the information is correct. It could also be wrong or misleading, in which case it would be better not to receive it. While "Do you mean whole brain emulation?" "No" doesn't fall into this category, claims like "we know how to build AGI" are definitely claims that could be wrong, and are indeed generally considered to be wrong.
Unless you provide a reasonable argument or reason for why we should believe such a claim, anyone maintaining any epistemic hygiene standards (or common sense, for that matter) will be forced to ignore it. Therefore such comments only serve as a distraction, providing no useful value but taking up space and attention.
If I were to comment on conversations and tell people that 2 + 2 = 5 and then refuse to provide any justification when asked, people would quite reasonably conclude that I was a troll, too.
↑ comment by Richard_Kennaway · 2011-06-27T14:37:09.409Z · LW(p) · GW(p)
Anyway, I am aware that people do not seem to understand that a small amount of information when requested is significantly better than no information, and I am aware that there are no affordances that would allow people to take the 3 seconds of reflection that would be necessary to realize this after realizing that their implicit expectations about how much information they deserve to receive from near-anonymous people are probably not based on any sort of explicitly justified or explicitly reflected-upon analysis.
When I notice myself writing a sentence like that, I drop it and do something else. Maybe I would come back to it later, but not less than 24 hours later.
Replies from: Will_Newsome↑ comment by Will_Newsome · 2011-06-29T14:07:13.904Z · LW(p) · GW(p)
I try to as well. Unfortunately I was distraught squared at the time and wasn't thinking at all clearly as far as social reasoning goes. My apologies to everyone for failing to keep my comments sufficiently constructive.
Replies from: Will_Newsome↑ comment by Will_Newsome · 2011-07-01T07:24:46.885Z · LW(p) · GW(p)
Tested retract+edit again.
↑ comment by Multiheaded · 2011-12-31T20:35:33.253Z · LW(p) · GW(p)
-
↑ comment by Peterdjones · 2011-06-26T12:44:01.565Z · LW(p) · GW(p)
Could you expand?
comment by Normal_Anomaly · 2011-06-24T15:21:37.978Z · LW(p) · GW(p)
Count me as another person who would switch some of my charitable contribution from VillageReach to SIAI if I had more information on this subject.
Replies from: lukeprog↑ comment by lukeprog · 2011-06-24T21:04:42.166Z · LW(p) · GW(p)
As it happens, the most exciting developments in this space in years (to my knowledge) are happening right now, but it will take a while for things to happen and be announced. And that is all I can say for now. Stay tuned. :)
Replies from: Will_Newsome↑ comment by Will_Newsome · 2011-06-25T02:39:54.511Z · LW(p) · GW(p)
I will back up lukeprog here, things should get exciting soon if all goes well, and I really hope it does.
I will also point out that SingInst not reporting/knowing what SingInst does doesn't mean that SingInst doesn't do things -- though it does mean that they're somewhat bad at cataloging progress. (Though see the quarterly reports, et cetera; many SingInst-critics don't even read those for some reason.) Michael Anissimov is the media director and doesn't hang out with the Research Fellows often, Eliezer doesn't know much about anything anyone else is doing (by his choice, I think), and Michael Vassar is busy all the time with important things. Anna Salamon does a lot of visible work with Carl Shulman, Steve Rayhawk doesn't do much visible work (though SIAI should be announcing one of his recent accomplishments soon I think). Steve does a fair bit of invisible work that I'm pretty sure most of SIAI mostly doesn't even know about (though he'd probably be quick to dismiss my promotion of his efforts as misguided or exaggerated). Jasen Murray is the organizational wizard running the rationality training stuff, Louie Helm just got back into town and is working on the exciting stuff lukeprog mentioned, and lukeprog himself is being insanely productive like usual. (Luke, are you officially employed/contracted by SIAI?) People loosely associated with SingInst are also doing interesting things, often with the help of SingInst researchers.
Replies from: beriukay, Normal_Anomaly↑ comment by beriukay · 2011-06-29T13:04:59.459Z · LW(p) · GW(p)
I'll give $500 to SIAI now if you can put a date on this 'exciting things' prediction. It doesn't have to be exact, but accurate to within a few weeks. $500 more on delivery of said excitement within said time window.
Also, I too would like to know where these quarterlies are stored.
Replies from: Rain↑ comment by Rain · 2011-06-29T20:22:25.406Z · LW(p) · GW(p)
The SIAI Newsletter is sent as an email to people who sign up here, I believe. Note that I am not certain about this, since I've signed up to lots of stuff over the years, and there's no obvious link in the newsletters. They also post some, though not all, of the newsletters to the SIAI blog.
Here are some of them:
↑ comment by Normal_Anomaly · 2011-06-25T17:56:37.709Z · LW(p) · GW(p)
Thanks for the explanation. I'm looking forward to seeing the results whenever SIAI can/wants to reveal them. I can't find the quarterly reports, though; can you post a link?
Replies from: Raincomment by Wei Dai (Wei_Dai) · 2011-06-24T19:04:28.140Z · LW(p) · GW(p)
Has any progress been made on the points mentioned in the announcement above?
See http://multiverseaccordingtoben.blogspot.com/2010/10/singularity-institutes-scary-idea-and.html (section titled "My History with SIAI") and http://en.wikipedia.org/wiki/OpenCog (which says that OpenCog incorporates the MOSES system mentioned in the 2007 post).
comment by [deleted] · 2011-06-24T15:32:01.188Z · LW(p) · GW(p)
"What is missing for the SIAI to actually start working on friendly AI?"
I think that question is answered by Yudkowsky in his interview with Baez:
"I probably need to take at least a year to study up on math, and then—though it may be an idealistic dream—I intend to plunge into the decision theory of self-modifying decision systems and never look back. (And finish the decision theory and implement it and run the AI, at which point, if all goes well, we Win.)"
Yudkowsky's position, widely known, is that it is unsafe to do otherwise. I imagine that is why they are not funding researchers to work on extending MOSES (or any other AGI work for that matter), but that's just speculation on my part.
To learn more about the work people are doing to build AGI, check out the conferences series on AGI at http://agi-conf.org/ agi-conf.org, organized by Ben Goertzel, advisor to SIAI (formerly Director of Research). Videos of most of the talks and tutorials are available for free, along with PDFs of the conference papers.
Replies from: lukeprog↑ comment by lukeprog · 2011-06-24T21:08:00.566Z · LW(p) · GW(p)
"What is missing for the SIAI to actually start working on friendly AI?"
The biggest problem in designing FAI is that nobody knows how to build AI. If you don't know how to build an AI, it's hard to figure out how to make it friendly. It's like thinking about how to make a computer play chess well before anybody knows how to make a computer.
In the meantime, there's lots of pre-FAI work to be done. There are many unsolved problems in metaethics, decision theory, anthropics, cosmology, and other subjects that seem to be highly relevant to later FAI development. I'm currently working (with others) toward defining those problems so that they can be engaged by the wider academic community.
Replies from: aletheilia, Vladimir_Nesov↑ comment by aletheilia · 2011-06-24T23:01:18.401Z · LW(p) · GW(p)
Even if we presume to know how to build an AI, figuring out the Friendly part still seems to be a long way off. Some AI building plans or/and architectures (e.g. evolutionary methods) are also totally useless F-wise, even though they may lead to a general AI.
What we actually need is knowledge about how to build a very specific type of an AI, and unfortunately, it appears that the A(G)I (sub)field with it's "anything that works" attitude isn't going to provide one.
Replies from: lukeprog↑ comment by Vladimir_Nesov · 2011-06-24T23:55:55.096Z · LW(p) · GW(p)
If you don't know how to build an AI, it's hard to figure out how to make it friendly.
(You don't make an AI friendly. You make a Friendly AI. Making an AI friendly is like making a text file good reading.)
Replies from: lukeprog↑ comment by lukeprog · 2011-06-25T00:11:34.647Z · LW(p) · GW(p)
Yes, I know. 'Making an AI friendly' is just a manner of speaking, like talking about humans having utility functions.
Replies from: Vladimir_Nesov↑ comment by Vladimir_Nesov · 2011-06-25T00:18:19.724Z · LW(p) · GW(p)
I assumed you know, which is why it was a parenthetical, mainly clarifying for the benefit of others. Disagreement with the method of presentation.
Replies from: lukeprogcomment by [deleted] · 2011-06-24T15:40:51.609Z · LW(p) · GW(p)
Another reason is that I believe that an empirical approach is required (with embedded link to www.opencog.org)
Do you therefore donate to OpenCog? They have the roadmap/timeline you seek (http://opencog.org/roadmap/).
comment by jsalvatier · 2011-06-24T15:01:13.366Z · LW(p) · GW(p)
Great questions.
comment by [deleted] · 2011-06-24T15:40:14.056Z · LW(p) · GW(p)
No delete button is an awesome feature. WTF?
Replies from: Vladimir_Nesov↑ comment by Vladimir_Nesov · 2011-06-24T20:49:09.485Z · LW(p) · GW(p)
I want to be able to downvote this!
Edit: In this particular case, I shouldn't, since it wasn't intentional. (Incidentally, this is the way to amend/retract comments. Adding a note. There is a separate use case of removing stuff posted by mistake.)
Replies from: Vladimir_Nesov↑ comment by Vladimir_Nesov · 2011-06-24T23:38:28.686Z · LW(p) · GW(p)
jmed writes that it was originally a comment posted by accident that couldn't be deleted, so my comment doesn't apply in this case. But giving users ability to block downvoting and still having their text fixed in discussion seems like a bad thing. Even "revoked" comments need to be community-moderated. I understand that this is intended to set up incentives that prevent deletion of comments, but perhaps limiting negative comment score instead could do the trick.
Replies from: cousin_it↑ comment by cousin_it · 2011-06-25T13:34:11.782Z · LW(p) · GW(p)
Oh, so that's why I see all these comments with text struck out! I don't understand why the feature works this way and I don't like it. Did someone explain the rationale somewhere?
Replies from: jaimeastorga2000