Ben Goertzel interviews Michael Anissimov regarding existential risk [link]
post by Kevin · 2011-04-20T22:07:34.703Z · LW · GW · Legacy · 16 commentsContents
16 comments
http://hplusmagazine.com/2011/04/20/mitigating-the-risks-of-artificial-superintelligence/
16 comments
Comments sorted by top scores.
comment by JoshuaZ · 2011-04-20T22:29:40.663Z · LW(p) · GW(p)
Michael:
Underestimating the significance of superintelligence. People have a delusion that humanity is some theoretically optimum plateau of intelligence (due to brainwashing from Judeo-Christian theological ideas, which also permeate so-called “secular humanism”), which is the opposite of the truth. We’re actually among the stupidest possible species smart enough to launch a civilization.
This doesn't seem to be a part of standard Christian or Jewish theology, so blaming that attitude on this seems misguided. His last sentence is also problematic- how does he know that with a sample size of one?
Michael:
Would you rather your AI be based on Hitler or Gandhi?
Seems to understate the case. Mindspace is large. The problem isn't an AI that acts like Hitler. That's not such a bad failure as things go. The worst case scenario more resembles Cthulhu than Hitler.
Anissimov correctly calls out Goertzel on his claim that he's sure he could design a properly functioning nanny AI.
Goertzel does correctly point out that it seems likely that a nanny AI would take less understanding than a full Friendly AI with stable goals under self-modification.
Replies from: Vladimir_Nesov, Kai-o-logos, timtyler↑ comment by Vladimir_Nesov · 2011-04-23T21:16:19.739Z · LW(p) · GW(p)
We’re actually among the stupidest possible species smart enough to launch a civilization.
His last sentence is also problematic- how does he know that with a sample size of one?
You are demanding particular proof. This claim is rooted in some theoretical considerations that don't obviously rely on directly sampling things.
Replies from: JoshuaZ↑ comment by JoshuaZ · 2011-04-24T01:03:05.019Z · LW(p) · GW(p)
This claim is rooted in some theoretical considerations that don't obviously rely on directly sampling things.
In that case, can you expand on what those theoretical considerations are? As far as I can tell, the main thing that allows the launch of civilization is speech. But even people well below average intelligence can speak. Moreover, most of the major advances seem to be made not by the average individual but by the outliers. The idea that the species as a whole needs to be as intelligent as we are to make a civilization seems problematic. There may be issues here with what one means by civilization and how one is measuring intelligence.
Replies from: Vladimir_Nesov↑ comment by Vladimir_Nesov · 2011-04-24T02:50:51.913Z · LW(p) · GW(p)
There may be issues here with what one means by civilization and how one is measuring intelligence.
Yes. But whatever the disambiguation, appealing to lack of particular proof won't help.
Replies from: JoshuaZ↑ comment by JoshuaZ · 2011-04-24T03:04:20.982Z · LW(p) · GW(p)
Sure. If we had a tight, robust framework for how civilizations arise and what was required for them to arise, this would be a bad demand for particular proof. I'd be just as happy to see that. So it might have been better if my initial comment had been of the form "without any strong theoretical framework, this requires a sample size larger than 1".
↑ comment by Kai-o-logos · 2011-04-21T00:15:20.666Z · LW(p) · GW(p)
"This doesn't seem to be a part of standard Christian or Jewish theology,"
~Actually even if there is no outright statement in the Bible, through the years, it is commonly accepted that human supremacy is stated in Genesis 1:26 - "Then God said, 'Let us make mankind in our image'". Man is created in God's image - making him superior to all. Also in Gen. 3:22 - “The man has now become like one of us, knowing good and evil". Man is like a God - the only difference is that they are not immortal.
Not necessarily my opinion, just what I believe the theology says, and what I have heard from theist friends that the theology says.
Replies from: JoshuaZ↑ comment by JoshuaZ · 2011-04-21T12:54:09.808Z · LW(p) · GW(p)
But both religions also have angels which are depicted often as in some ways superior to humans. (Incidentally, I suspect that Gen 1:26 was originally intended to be more literal, with a host of deities of roughly human-looking deities creating creatures that looked like themselves.)
Replies from: Sniffnoy↑ comment by timtyler · 2011-04-21T00:18:02.750Z · LW(p) · GW(p)
We’re actually among the stupidest possible species smart enough to launch a civilization.
His last sentence is also problematic- how does he know that with a sample size of one?
Plenty of our ancestors failed to launch a civilisation. Correlated samples, but not really just one data point.
Replies from: spuckblase↑ comment by spuckblase · 2011-04-21T11:45:47.996Z · LW(p) · GW(p)
No. He seems to talk about the species, and not its members.
Replies from: timtyler↑ comment by timtyler · 2011-04-22T09:57:00.100Z · LW(p) · GW(p)
I wasn't talking about indivduals. Homo Erectus didn't launch a civilisation. Nor did Homo habilis, Homo cepranensis, Homo antecessor, Homo heidelbergensis, Homo rhodesiensis, Homo neanderthalensis etc. That is relevant data.
Replies from: JoshuaZ, MichaelAnissimov↑ comment by JoshuaZ · 2011-04-24T01:10:39.802Z · LW(p) · GW(p)
I wasn't talking about indivduals. Homo Erectus didn't launch a civilisation. Nor did Homo habilis, Homo cepranensis, Homo antecessor, Homo heidelbergensis, Homo rhodesiensis, Homo neanderthalensis etc. That is relevant data.
H. neanderthalensis had complicated tools and weapons, including spears and axes and there's evidence of tools used to stitch garments together. There's some (controversial) evidence that neanderthals made musical instruments, and the presence of remains indicating severe wounds that have since healed suggests that they cared for their wounded. So how much civilization is necessary for something to be considered civilization?
(Note that there are two problems with my example of neanderthals- First, they had average brain size that was as large or larger than Homo sapiens so they may have been as smart. Second, many neanderthals coexisted with populations of H. sapiens and so may have obtained tools by trade and imitation.)
Replies from: timtyler↑ comment by timtyler · 2011-04-24T11:27:48.606Z · LW(p) · GW(p)
So how much civilization is necessary for something to be considered civilization?
The process of things taking off. See:
Civilisation (or civilization) is a sometimes controversial term which has been used in several related ways. Primarily, the term has been used to refer to human cultures which are complex in terms of technology, science, and division of labour. Such civilisations are generally urbanized.
It should not need to happen more than once - and there are no signs that it has done so.
Replies from: JoshuaZ↑ comment by JoshuaZ · 2011-04-24T14:03:40.661Z · LW(p) · GW(p)
Civilisation (or civilization) is a sometimes controversial term which has been used in several related ways. Primarily, the term has been used to refer to human cultures which are complex in terms of technology, science, and division of labour. Such civilisations are generally urbanized.
It should not need to happen more than once - and there are no signs that it has done so.
This isn't a really helpful definition for our purposes. How complex does technology and division of labor need to be? There's division of labor in hunter-gatherer groups for example. I would agree that by almost all expansions of this definition neanderthals didn't make it, and that H. sapiens is the only species to have done so that we are aware of.
I am, however, puzzled by your statement that civilization arose only once. Do the New World civilizations of the Incans, Mayans, and Aztecs not count? I would think that the separation of the Old and New World would then necessitate at least two independent occasions where civilization has arisen.
Replies from: MichaelAnissimov↑ comment by MichaelAnissimov · 2011-04-26T00:27:53.158Z · LW(p) · GW(p)
Civilization only arose in one species. It's arisen many times within that species, just not in any other species.
↑ comment by MichaelAnissimov · 2011-04-23T09:07:28.389Z · LW(p) · GW(p)
That's the point I was getting at...