Q&A #2 with Singularity Institute Executive Director

In the previous video, you said that publishing in mainstream journals might be a waste of time, due to the amount of "post-production" involved. In addition, you said that SIAI would prefer to keep its AGI research secret -- otherwise, someone might read it, implement the un-Friendly AGI, and doom us all. You followed that up by saying that SIAI is more interested in "technical problems in mathematics, computer science, and philosophy" than in experimental AI research.

In light of the above, what does the SIAI actually do ? You don't submit your work to rigorous scrutiny by your peers in the field (you need peer review for that); you either aren't doing any AGI research, or are keeping it so secret that no one knows about it (which makes it impossible to gauge your progress, if any), and you aren't developing any practical applications of AI, either (since you'd need experimentation for that). So, what is it that you are actually working on, other than growing the SIAI itself ?

Presumably you relatively recently gained access to research Singinst does not make public, if any. Were you surprised at the level of progress already made, either positively or negatively?

What would SI do if it became apparent that AGI is at most 10 years away? For example, some researchers demonstrate the feasibility of AGI and show that they only need a few years to implement it.

(Some AGI researchers like Shane Legg assign a 10% chance of AGI by ~2018.)

I find this a little odd, since there are still several highly-voted (>= 15 points, say) questions from last time unanswered. Why not answer them first? Also, is there any reason for why someone shouldn't just take all of them and repost them in this thread (e.g. if you're unwilling to answer many of them, in which case it would be mostly be a wasted effort and clutter the page needlessly)?

Would the Institute consider hiring telecommuters (both in and out the US)?

Update: this question was left unanswered in the second Q&A.

What kind of budget would be required to solve the friendly AI problem? Are we talking millions, billions or trillions?

What is SIAI's discount rate? If I offered you $100 today in return for r*$100 in a year's time, for what r are you indifferent? Are you borrowing money, saving it, or neither?

EDIT: For context, I seem to recall Vassar once suggesting 40%.

Given the huge potential of FAI for changing the world, are you worried that existing governments might see SIAI as a revolutionary threat once it starts to look like you have a serious chance of completing the goal?

You mentioned recently that SIAI is pushing toward publishing an "Open Problems in FAI" document. How much impact do you expect this document to have? Do you intend to keep track? If so, and if it's less impactful than expected, what lesson(s) might you draw from this?

How much do members' predictions of when the singularity will happen differ within the Singularity Institute?

Since it's difficult to predict the date of the invention of AGI, has SI thought about/made plans for how to work on the FAI problem for many decades, or perhaps even centuries, if necessary?

In 2009 EY asked "What's the craziest thing the AI could tell you, such that you would be willing to believe that the AI was the sane one?"

rhollerith_dot_com responded "That the EV of the humans is coherent and does not care how much suffering exists in the universe."

Vassar responded to this with the scariest thing I've read on LessWrong which was:

"But you believe that, don't you? I certainly place a MUCH higher probability on that than on the sort of claims some people have proposed."

Do you agree with Vassar's reply?

If an infallible oracle told you humanity was about to enter a period of extended stagnation comparable to the Dark Ages, what projects would you prioritize right now to ensure humanity's long term survival?

If SIAI cannot perform any kind of "Trinity Test" (Manhattan Project), and demands secrecy to protect the world from evil obtaining any FAI intelligence, how does the organization quell the idea that it is augmenting Pascal's Wager, while keeping the fear paradox, for technology?

Since SIAI needs secrecy, transparency cannot be absolute, yet you will need considerable funding. So, how does SIAI plan to avoid looking like Enron when asking for funding to research a much needed "Negative Ionic Tractor Disruptor" (South Park, 311), and when the philanthropists, some who may be "evil", want to see how the sausage is made?

Sorry, if this is a juvenile question, it is the same question twice. I will try to get to the literature you recommend on the "So You Want To Save The World" site soon. -how do you navigate this necessary contradiction of transparency and stay a viable organization?

Minds are not chronologically commutative with respect to input data. Reading libertarian philosophy followed by Marxist philosophy will give you a different connectome than vice versa. As a result, you will have distinct values in each scenario and act accordingly. Put another way, human values are extremely dependent on initial input parameters (your early social and educational history). Childhood brainwashing can give the resulting adult arbitrary values (as evinced by such quirks like suicide bombers and voluntary eunuchs). However, by providing such a malleable organism, evolution found a very cute trick by which it allowed for seemingly impossible computation. (development of mathematics, science, etc.)

I assume that in the definition of GAI, it is implicit that the AI can do mathematics and science as good or better than humans can, as to achieve its goals that require a physical restructuring of reality. Since the only example of a computational process that is capable of generating these things (humans) is so malleable in its values, what basis (mathematical or otherwise) does the SIAI have for assuming that Friendliness is achievable? Keep in mind that a GAI should be able to think and comprehend all things humans can and have thought (including the architectural problems in Friendliness), or at least something functionally isomorphic.

I see a problem with LW(don't know if you consider this part of SI) in that non-conforming comments are often downvoted, regardless of them being right or wrong. I think part of the blame is on that article by EY:

http://lesswrong.com/lw/c1/wellkept_gardens_die_by_pacifism/

My big concern: what safeguards are in place to distinguish those "weeds" that are real "weeds" from those that only look like "weeds" because they go against incorrect beliefs in LW? In other words, assuming that there are incorrect beliefs in LW/SI shouldn't there be more room to allow for contrarian POVs to be expressed?