It might be a good idea somewhere down the line, but co-ordination of that kind would likely be very difficult.

It might not be so necessary if the problem of friendliness is solved, and AI is built to specification. This would also be very difficult, but it would also likely be much more permanently successful, as a friendly superintelligent AI would ensure no subsequent unfriendly superintelligences arise.

Though Bostrom seems right to talk about better transmission - which could have been parsed into more reliable, robust, faster, compact, nested etc.... - he stops short of looking deep into what made cultural transmission better. To claim that a slight improvement in (general) mental faculties did it would be begging the question. Brilliant though he is, Bostrom is "just" a physicist, mathematical logician, philosopher, economist, computational neuroscientist who invented the field of existential-risks and revolutionized anthropics, so his knowledge of cultural evolution and this transition is somewhat speculative. That's why we need other people :) In that literature we have three main contenders for what allowed human prowess to reshape earth:

Symbolic ability: the ability to decently process symbols - which have a technical definition hard to describe here - and understand them in a timely fashion is unique to humans and some other currently extinct anthropoids. Terrence Deacon argues for this being what matters in The Symbolic Species.

Iterative recursion processing: This has been argued in many styles.

• Chomsky argued the primacy of recursion as a requisite ability for human language in the late fifties

• Pinker endorses this in his Language Instinct and in The Stuff of Thought

• The Mind Is A Computer metaphor (Lakoff 1999) has been widely adopted and very successful memetically, and though it has other distinctions, the main distinction from "Mind Is A Machine" is that recursion is involved in computers, but not in all machines. The computational theory of mind thrived in the hands of Pinker, Koch, Dennet, Kahneman and more recently Tononi. Within LW and among programmers Mind is a Computer is frequently thought to be the fundamental metaphysics of mind, and a final shot at the ontological constituent of our selves - a perspective I considered naïve here.

Ability to share intentions: the ability to share goals and intentions and parallelize in virtue of doing so with other co-specimens. Tomasello (2005)

Great books on evolutionary transmission are Not By Genes Alone, The Meme Machine and LWer Tim Tyler's Memetics.

The capabilities of a homo sapiens sapiens 20,000 years ago are more chimp-like than comparable to a modern internet- and technology-amplified human. Our base human intelligence seems to be only a very little above the necessary threshold to develop cultural technologies that allow us to accumulate knowledge over generations. Standardized languages, the invention of writing and further technological developments improved our capabilities far above this threshold. Today children need years until they aquire enough cultural technologies and knowledge to become full members of society.

Intelligence alone does not bring extreme power. If a superintelligent AI has learned cultural technologies and aquired knowledge and skills it could bring it.

We might be approaching a point of diminishing returns as far as improving cultural transmission is concerned. Sure, it would be useful to adopt a better language, e.g. one less ambiguous, less subject to misinterpretation, more revealing of hidden premises and assumptions. More bandwidth and better information retrieval would also help. But I don't think these constraints are what's holding AI back.

Bandwidth, storage, and retrieval can be looked at as hardware issues, and performance in these areas improves both with time and with adding more hardware. What AI requires are improvements in algorithms and in theoretical frameworks such as decision theory, morality, and systems design.

Please spell out why the financial AGI is so threatening?

[deleted]

It's plausible that some hedge fund runs an AGI with is focused on maximizing one financial returns and that AGI over time becomes powerful.

It wouldn't necessarily have an "attempt of a moral system".

Surely a team of engineers capable of developing AGI can be given some guidance in advance so that they are at least competent enough to instill a set of values as robust as the ones we attempt to instill in children?

The number of actual possibilities of goals is HUGE compared to the relatively small subset of human goals. Humans share the same brain structure and general goal structure, but there's no reason to expect the first AI to share our neural/goal structure. Innocuous goals like "Prevent Suffering" and "Maxmize Happiness" may not be interpreted and executed the way we wish them to be.

Indeed, gaining superpowers probably would not compromise the AI's moral code. It only gives it the ability to fully execute the actions dictated by the moral code. Unfortunately, there's no guarantee that its morals will fall in line with ours.

So, as others have said, the idea that an AGI necessarily incorporates all of the structures that perform moral calculations in human brains -- or is even necessarily compatible with them -- is simply untrue. So the casual assumption that if we can "instill" that morality in humans then clearly we're competent enough to instill them in an AGI is not clear to me.

But accepting that assumption for the sake of comity...

Can you show me why a decent person like (I presume) you or I with these new powers would suddenly choose to slaughter the human race as an instrumental goal to accomplishing some other ends?

Well, I'm not entirely sure what we mean by "decent," so it's hard to avoid a No True Decent Person argument here. But OK, I'll take a shot.

Suppose, hypothetically, that I want to maximize the amount of joy in the world, and minimize the amount of suffering. (Is that a plausible desire for decent folk like (presumably) you and I to have?)

Suppose, hypothetically, that with my newfound superpowers I become extremely confident that I can construct a life form that is far less likely to suffer, and far more likely to experience joy, than humans.

Now, perhaps that won't motivate me to slaughter all existing humans. Perhaps I'll simply intervene in human reproduction so that the next generation is inhuman... that seems more humane, somehow.

But then again... when I think of all the suffering I'm allowing to occur by letting the humans stick around... geez, I dunno. Is that really fair? Maybe I ought to slaughter them all after all.

But maybe this whole line of reasoning is unfair. Maybe "maximize joy, minimize suffering" isn't actually the sort of moral code that decent people like you and I have in the first place.

So, what is our decent moral code? Perhaps if you can articulate that, it will turn out that a superhuman system optimizing for it won't commit atrocities. That would be great.

Personally, I'm skeptical. I suspect that the morality of decent people like you and (I presume) me is, at its core, sufficiently inconsistent and incoherent that if maximized with enough power it will result in actions we treat as atrocities.

Suppose you or I suddenly woke up with superintelligence, but with our existing goal structure intact (and a desire to be cautious).

Can you show me why a decent person like (I presume) you or I with these new powers would suddenly choose to slaughter the human race as an instrumental goal to accomplishing some other ends?

If CEV (or whatever we're up to at the moment) turns out to be a dud and human values are inexorably inconsistent and mutually conflicting, one possible solution would be for me to kill everyone and try again, perhaps building roughly humanish beings with complex values I can actually satisfy that aren't messed up because they were made by an intelligent designer (me) rather than Azathoth.

But really, the problem is that a superintelligent AI has every chance of being nothing like a human, and although we may try to give it innocuous goals we have to remember that it will do what we tell it to do, and not necessarily what we want it to do.

See this Facing the Intelligence Explosion post, or this Sequence post, or Smarter Than Us chapter 6, or something else that says the same thing.

Page 102, "Many more orders of magnitudes of human-like beings could exist if we countenance digital implementations of minds--as we should." I'd like to hear others thoughts about that, especially why he writes "as we should."

It is less likely that AI algorithms will happen to be especially easy if a lot of different algorithms are needed. Also, if different cognitive skills are developed at somewhat different times, then it's harder to imagine a sudden jump when a fully capable AI suddenly reads the whole internet or becomes a hugely more valuable use for hardware than anything being run already. [...] Overall it seems AI must progress slower if its success is driven by more distinct dedicated skills.

To me the skill set list on table 8 (p94) was most interesting. Superintelligence is not sufficient to be effective. Content and experiences have to be transformed by "mental digestion" into knowledge.

If the AI becomes capable to self-improve it might decide to modify its own architecture. In consequence it might be necessary to re-learn all programming and intelligence amplification knowledge. If it turns out that a further development loop is needed - all aquired knowledge is lost again. For a self-improving AI it is therefore rational and economic to learn only the necessary skills for intelligence amplification until its architecture is capable enough to learn all other skills.

After architectural freeze the AI starts to aquire more general knowledge and further skills. It uses its existing engineering skills to optimize hard- and software and to develop optimized hardware virtualisation tools. To become superpower and master of all tasks listed in table 8 knowledge from books is not sufficient. Sensitive information in technology/hacking/military/government is unaccessible unless trust is growing over time. Projects with trials and errors plus external delay factors need further time.

The time needed for learning could be long enough for a competing project to take off.

He says that the relation of a supercomputer to man will be like the relation of a man to a mouse, rather than like the relation of Einstein to the rest of us; but what if it is like the relation of an elephant to a mouse?

There's nothing saying there can't be both entities equivalent to elephant to a mouse and equivalent to man to a mouse. In fact, we have supercomputers today that might loosely fit the elephant to mouse description. In any case, as mice, we don't have to worry about elephants nearly as much as men, and the existence of elephants might suggest men are around the corner.

Let's abandon for a second the notion of strategically important, and just list human cognitive skills and modules, as they appear distributed in our cognition, according to the MIT The Cognitive Neurosciences (2009), and also as they come to mind while writing:

• Nurturing

• Lust

• Exploration

• Moral reasoning

• Agentism (animism, anthopomorphizing etc...)

• Language processing

• Proprioception

• Attachment

• Priming

• Allocating attention

• Simulation of future events

• Dreaming

• Empathizing

• Volition

• Consciousness

Now it's up to commenters to try and come up with reasons why these would be strategically important. Maybe they are, and then we'd have to verify if they are or not encompassed by the categories suggested by Bostrom. I don't have time to go through each in detail, but wanted to lay out a path for others who may find this useful.

Novel physics research, maybe. Just how useful that would be depends on just what our physics models are missing, and obviously we don't have very good bounds on that. The obvious application is as a boost to technology development, though in extreme cases it might be usable to manipulate physical reality without hardware designed for the purpose, or escape confinement.

Superpower-level social manipulation

I can't quite imagine what that could possibly look like.

Reflection.

Something on the edge of human mind-space (or beyond if possible). Getting alternative perspectives could be extremely valuable if it is the case that we are missing large things.

moral, humour and spiritual analyzer/emulator. I like to know more about these phenomenas.

No. The cosmic endowment and related calculations do not make any sense to me. If these figures were true this tells us that all alien societies in our galaxy directly went extinct. If not, they would have managed cosmic endowment and we should have found von Neumann probes. We haven't. And we won't.

Instead of speculating about how much energy could be harvested when a sphere of solar cells is constructed around a star I would love to have found a proper discussion about how our human society could manage the time around crossover.

Yudkowski wanted to break it down to the culmination point that a single collaborator is suffient. For the sake of the argument it is understandable. From the AIs viewpoint it is not rational.

Our supply chains are based on division of labor. A chip fab would not ask what a chip design is good for when they know how to test. A pcb manufacturer needs test software and specifications. A company specified on burn-in testing will assemble any arrangement and connect it even to the internet. If an AI arranges generous payments in advance no one in the supply chain will ask.

If the AI has skills in engineering, strategic planning and social manipulation, an internet connection is sufficient to break out and kickstart any supply chain.

The suggested DNA nano maker is unnecessarily far fetched and too complex to be solved in such a simplified 5 step approach.

I think it will prove computationally very expensive, both to solve protein folding and to subsequently design a bootstrapping automaton. It might be difficult enough for another method of assembly to come out ahead cost-wise.

Could whoever down-voted this please offer the reason for your vote? I'm interested.

This makes me think of two very different things.

One is informational containment, ie how to run an AGI in a simulated environment that reveals nothing about the system it's simulated on; this is a technical challenge, and if interpreted very strictly (via algorithmic complexity arguments about how improbable our universe is likely to be in something like a Solomonoff prior), is very constraining.

The other is futurological simulation; here I think the notion of simulation is pointing at a tool, but the idea of using this tool is a very small part of the approach relative to formulating a model with the right sort of moving parts. The latter has been tried with various simple models (eg the thing in Ch 4); more work can be done, but justifying the models&priors will be difficult.