Recent MIRI workshop results?

Let me address the risks identified in your article:

An end to Moore's law. The only part I agree with is that parallel software is fundamentally more difficult, although I think we would disagree with how much more difficult, given the right tools. We have plenty of experience in scaling parallel software stacks in data centers, giving rise to solutions like NoSQL, message queues, and software transactional memory. These tools are still crude, but if you look at what's being done in academia with automated parallelization and simple concurrency frameworks, the prospects look bright indeed. The concurrency problem will be solved (and indeed, pretty much is solved by organizations like Google, although their tools are not public). There is no reason to suppose that Moore's law formulated as computations per dollar will not continue until physical limits are reached, which is still a very long way off, indeed, and there are tools on the horizon that will slay the concurrency dragon.

(To be clear, concurrency adds a suppressing term to Moore's law expressed as observed performance per dollar, due to performance no longer scaling linearly without limit. But it's still recognizably an exponential.)

Depletion of low-hanging fruit. What fruit is low-hanging depends on what capabilities you have. To stretch this analogy, as hardware capacity increases, you grow taller. In the 80's chess playing was a very difficult research problem. Today it would definitely be low hanging fruit if it wasn't already claimed by Deep Blue. It was new hardware and software capabilities which shifted the playing field, and there is no reason to expect this won't continue to happen in the future so long as Moore's law holds.

Societal collapse. Maybe, but this wouldn't affect the “basement hacker” hard takeoff scenario, and may even facilitate it in the same way that the economic collapse of 2008 gave a boost to bitcoin which was released shortly thereafter. I think you'd have to posit total global societal collapse - bringing an end to Moore's law - before you start affecting the probability of the basement hacker scenario. You could also come up with other scenarios whose timelines would be unaffected, such as AI for national defense / surveillance which may receive more funding in the case of societal collapse.

Disinclination. I'd love to see analogous examples of this from other industries, where there is a revolutionary change around the corner that everyone knows about but no one pursues because what they have is “good enough.” I suspect you won't find it; it goes against the grain of entrepreneurial capitalism and human nature.

In my own judgement, none of these negative scenarios have much probability mass. My own forecasting is more limited by unknown unknowns, which practical AGI research will uncover. Your positive factors (cognitive neuroscience breakthroughs, human enhancement, funding growth, etc.) are reasonable, so compared to me your judgement is more pessimistic.

I think that your own analysis is somewhat contradictory. You can either trust expert opinions or not, and your 2065 number is the midpoint of acceptable expert opinion. In the case of AGI, I am very skeptical of expert opinion, as there are very few experts in computer science or AI which are familiar with the very specific challenges of AGI, yet it is a topic that everyone has an opinion on if you ask them. I recognize that this excludes my own opinion from consideration as well, but nevertheless here it is: we are more likely than not to experience a hard-takeoff scenario in the 2020's.

Why am I confident of this? Because we understand the problem. Read the proceeds of the sometimes-annual AGI conference. There are a number of AGI architectures which no longer make ridiculous predictions of “emergent” features, which was once a clear sign of things we didn't understand. This was not the case just 10 years ago. You can open any psychological development textbook and match up descriptions of human thinking to operating modes of, say, the OpenCog Prime architecture. We simply need to do the work to implement that architecture - which only exists on paper at this point - and solve the unknown unknowns which pop up along the way.

But why the 2020's, specifically? Well with the resources we have I think we could implement one of these AGI architectures in about 10 years with current resources, and at that time computational resources will be developed enough to run it on commercial datacenter or high-end consumer hardware. This is simply my informed judgement as a systems engineer and my understanding of the complexity of the as-yet unimplemented portions of the OpenCog Prime architecture (you could substitute your own favorite architecture, however). Just look at the OpenCog Prime documents, look at the current state of OpenCog implementation, look at the current level of funding and available developers, and do normal, everyday project planning analysis. I predict you will arrive at a similar number.

Add 50% to that timeline, a common software engineering rule-of-thumb, and you get 15 years. However I predict 2025 / 10 years because that judgement was based on current efforts, but we are way under capacity in terms of AGI research - researcher time and money is the limiting factor. By comparison, with infinite budget we could do it in a couple of years. I think there's a good chance that an “AI sputnik” moment in the next couple of years could change the funding outlook, accelerating that schedule significantly, and I allocate a fair chunk of probability mass to that happening.

EDIT: So my question to you, Luke and MIRI, is: if you take at face value my prediction of hard-takeoff in the 2020's, how does this change MIRI's priorities? Would you being doing things differently if you expected a hard-takeoff in ~2025? How would it be different?

Luke, thank you for your honest response, and pointing me towards your article. I will read it and formulate a response RE: timelines and viable lines of research that will lead to AGI by 2025 with medium-high confidence.

I disagree that MIRI is the only one working on Friendly AI, although certainly you may be the only ones working on your vision of FAI. Ben Goertzel has taken a very pragmatic approach to both AGI and Friendliness, so his OpenCog Foundation would be a clear alternative for my dollars. I know that he has taken potshots at MIRI for promoting what he calls the Scary Idea, but nevertheless he does appear to be genuinely concerned with Friendliness; he is just taking a different approach. As an AI developer myself, there are various other self-directed options that I could invest my time and money into as well.

To be clear, I would rather have a fully fleshed out and proven Friendly AGI design before anyone writes a single self-modifying piece of code. I would also like a pony and a perpetual motion machine. I have absolutely no faith in anyone who would claim to be able to slow down or stop AGI research - the component pieces of AGI are simply too profitable and too far reaching in their consequences^1. Because of the short timescales, I believe it does the most good to study Friendliness in the context of actual AGI designs, and to devise safety procedures for running AGI projects that bias us towards Friendliness as much as possible, even if Friendliness cannot be absolutely guaranteed at this point. Work on provably safe AGI should proceed in parallel in case it pays off, but abstract work on a provably safe design has zero utility until it is finished, and I am not confident it would be finished before a hard takeoff occurs.

^1 Not just AGI itself, but various pieces of machine learning, hierarchical planning, goal forming, etc. - it'd be like trying out outlaw fertilizer because it could be used to make a fertilizer bomb. How then do you grow your food?

There wasn't speculation without evidence.

Of course there was! Why would you have asked Luke for clarification otherwise? The next three sentences don't support this claim, either -- being "a specific set of comments" does not contradict those comments being vapid speculation.

What you are saying seems very close to saying that we shouldn't raise concerns because if those concerns are responded to then there's a chance someone will happen to only see the initial issue and not the response.

Why should there be a response? Does Luke have a moral responsibility to traverse the internet, answering every random NPC who has "concerns"? And in a timely fashion?

There was a whole world of sensible reasons why MIRI would wait and discuss their work more before publishing it, but you went needle-like for an uncharitable one and demanded a response to be proven wrong. That's not acceptable to me.

It should be clear why that's not a great approach if one wants open discussion and doesn't want any sort of evaporative cooling of beliefs or similar problems.

You're simply wrong. LW does want some evaporative cooling -- of trolls and the like.