↑ comment by wedrifid ·
2012-12-16T00:42:59.258Z · LW(p) · GW(p)
If you are taking this as a question about what to CALL the thing, then I agree completely that the AI should not be called a God. But he said "in some sense" which means that his question is about something deeper than choosing a word. The wording he's using is asking something more like "Do you think we should build something similar to a God?"
If forced to use that term and answer the question as you ask it, with a "Yes" or "No" then the correct answer would be "No". He is not trying to create a God, he has done years of work working out what he is trying to create and it is completely different to a God in nearly all features except "very powerful". If you insinst on that vocabulary you're going to get "No, I don't" as an answer. That the artificial intelligence Eliezer would want to create seems to Wright (and perhaps yourself) like it should be described as, considered a euphemism for, or reasoned about as if it is God is a feature of Wright's lack of domain knowledge.
There is no disingenuity here. Eliezer can honestly say "We should create a very powerful (and carefully designed) optimizing agent" but he cannot honestly say "We should create a God". (You may begin to understand some of the reasons why there is such a difference when you start considering questions like "Can it be controlled?". Or at least when you start considering the answers to the same.) So Eliezer gave Wright the chance to get the answer he wanted ("Hell yes, I want to make a very powerful optimising agent!") rather than the answer the question you suggest would have given him ("Hell no! Don't create a God! That entails making at least two of the fundamental and critical ethical and practical blunders in FAI design that you probably aren't able to comprehend yet!")
The reason I interpreted it this way is partly because instead of interpreting everything I hear literally, I will often interpret wording based on what's salient about it in the context of the situation. For instance, if I saw a scene where someone was running toward someone else with a knife and I asked "Are you about to commit murder?" I would NOT accept "Why don't we call it knife relocation?" as an acceptable answer.
I reject the analogy. Eliezer's answer isn't like the knife relocation answer. (If anything the connotations are the reverse. More transparency and candidness rather than less.)
Now, it could be argued that Eliezer accidentally failed to detect the salient connotation. It could be argued, and probably fairly effectively (against me anyway) that the reason for Eliezer's mistake is that he was having one of his arrogant moments and he genuinely thought that, because of a gigantic intelligence difference between Robert and himself, that Robert was asking a moronic question based on the stupid perception that a super powerful AI would be exactly the same as a real God (whatever that means).
It could be that there really is an overwhelming difference in crystallized intelligence between Eliezer and Robert. The question---at least relative to Eliezer's standards---was moronic. Or at least had connotations of ignorance of salient features of the landscape.
In this case, I would classify that as a "social skills / character flaw induced faux pas".
There may be a social skills related faux pas here---and it is one where it is usually socially appropriate to say wrong things in an entirely muddled model of reality rather than educate the people you are speaking to. Maybe that means that Eliezer shouldn't talk to people like Robert. Perhaps he should get someone trained explicitly with spinning webs of eloquent bullshit to optimally communicate with the uneducated. However the character flaws that I take it you are referring to---Eliezer's arrogance and soforth, just aren't at play here.
In my personal interpretation of Eliezer's behavior, I'm giving him more credit than that
The net amount of credit given is low. You are ascribing a certain intention to Eliezer's actions where that intention is clearly not achieved. "I infer he is trying to do X and he in fact fails to do X". In such cases generosity suggests that if they don't seem to be achieving X, haven't said X is what they are trying to achieve and X is inherently lacking in virtue then then by golly maybe they were in fact trying to achieve Y! (Eliezer really isn't likely to be that actively incompetent at deviousness.)
I am assuming that he has previously encountered people by that point (2010) who have flipped out about the possibility that he wants to build a God
You assign a high likelyhood to people flipping out (and even persecuting Eliezer) in such a way. Nyan considers it less likely. It may be that Eliezer doesn't have people (and particularly people of Robert Wright's intellectual caliber) flip out at him like that.
and have voiced valid and poignant concerns like "Why do you believe it is possible to succeed at controlling something a bazillion times smarter than you?" or "Why would you want us imperfect humans to make something so insanely powerful if it's more or less guaranteed to be flawed?"
The kind of people to whom there is the remote possibility that it would be useful to even bother to attempt to explain the answers to such questions are also the kind of people who are capable of asking them, without insisting on asking then belligerently emphasizing wrong questions about 'God'. This is particularly the case with the first of those questions where the question of 'controlling' only comes up because of intuitive misunderstanding on how one would relate to such an agent---ie. thinking of it as a "God" which is something we already intuit as "like a human or mammal but way powerful".
"Does Eliezer intend to build something this powerful?"
If he can prove safety mathematically then yes, he does.
At around the time I visited Berkeley there was a jest among some of the SingInst folks "We're thinking of renaming ourselves from The Singularity Institute For Artificial Intelligence to The Singularity Institute For Or Against Artificial Intelligence Depending On What Seems To Be The Best Altruistic Approach All Things Considered".
There are risks to creating something this powerful and, in fact, the goal of Eliezer and SIAI aren't "research AGI"... plenty of researchers work on that. They are focused on Friendliness. Essentially... they are focused on the very dangers that you describe here and are dedicating themselves to combating those dangers.
Note that it is impossible to evaluate a decision to take an action without considering what alternative choice there is. Choosing to dedicate one's efforts to developing an FAI ("safe and desirable very powerful optimizing agent") has a very different meaning if the alternative is millennia of peace and tranquility than the same decision to work on FAI if the alternative is "someone is going to create a very powerful optimizing agent anyway but not bother with rigorous safety research".
"Does Eliezer really think that something a bazillion times as intelligent as himself can be controlled?"
If you're planning to try to control the super-intelligence you have already lost. The task is of selecting from the space of all possible mind designs a mind that will do things that you want done.
"Do you and I agree/disagree that it's a good idea to build something this powerful
Estimate: Slightly disagree. The biggest differences in perception may be surrounding what the consequences of inaction are.
/ that it can be controlled?"
Estimate: Disagree significantly. I believe your understanding of likely superintelligence behavior and self development has too much of an anthropocentric bias. Your anticipations are (in my estimation) strongly influenced by how ethical, intellectual and personal development works in gifted humans.
The above disagreement actually doesn't necessarily change overall risk assessment. I just expect the specific technical problems to be overcome in order to prevent "Super-intelligent rocks fall! Everybody dies." to be slightly different in nature. Probably with more emphasis on abstract mathematical concerns.