Peter Thiel's AGI discussion in his startups class @ Stanford [link]

The other class notes are worth checking out too.

Peter Thiel: So you’re both thinking it will all fundamentally work out.

Scott Brown: Yes, but not in a wishful thinking way. We need to treat our work with the reverence you’d give to building bombs or super-viruses. At the same time, I don’t think hard takeoff scenarios like Skynet are likely. We’ll start with big gains in a few areas, society will adjust, and the process will repeat.

I don't believe anyone working on AI is actually treating it that way. I do hope, however, that whenever there are signs of a possible breakthrough, researchers will stop, assess what they have very carefully, and build a lot of safety features before doing any more development. Most important of all, I hope that whoever makes the key discoveries does not publish their results in a way that would enable more reckless groups to copy them

So, {Possible AI} > {Evolved Intelligence} > {Human Intelligence}.

What about {AI practically discoverable/inventable by humans}? This could be an even smaller set than {Human Intelligence}. If it's of a much higher order of intelligence than {Human Intelligence}, it's argued that it would build smarter and smarter AI. How do we know it's likely to be of a higher order?

I guess, I'd like to know more about: {AI practically discoverable/inventable by humans+non-sentient computers in the current generation}. Is there a compelling reason to believe this set is quite large, or quite small?

In particular, there is this quote:

The community and class of algorithms we’re using is fairly well defined, so we think we have a good sense of the competitive and technological landscape. There are probably something like 200—so, to be conservative, let’s say 2000—people out there with the skills and enthusiasm to be able to execute what we’re going after. But are they all tackling the exact same problems we are, and in the same way? That seems really unlikely.

Somehow the diversity that could be generated by 2000, 20000, or even 200000 researchers, presumably working in project teams of a few or a dozen, seems to be much smaller than the evolutionary diversity generated by a population of 10 billion Homo sapiens. (Though it may well span a much larger "volume" of design space, only a relatively few points would be represented across that volume.)

In light of the discussion he seems to be hedging bets in this area. I'm not sure if that the right strategy from the ex-risk perspective. At the very least it seems inconsistent.