[LINK] How Hard is Artificial Intelligence? The Evolutionary Argument and Observation Selection Effects

Mildly interesting; but kind of odd to read a whole paper on estimating evolving intelligence with no reference to the success or lack thereof existing genetic algorithms and optimization!

Hans Moravec (1976, 1998, 1999), argue that human evolution shows that such AI is not just possible, but feasible within this century.

Moravec 1998 and 1999 are not listed in the references. Moravec paid little attention to observation selection effects - but if he made any bad arguments as a result, they could benefit from being more precisely identified - e.g. with quotes or page numbers.

I think I understand why

“hard step” models predict few sequential hard steps in the few hundred million years since that ancestor

but I do not see why this

count against extreme evolutionary challenge in developing human-level intelligence

I think it's explained in this passage, but I'm having trouble following the reasoning:

Thus, the “hard steps” model rules out a number of possible “hard intelligence” scenarios: evolution typically may take prohibitively long to get through certain “hard steps”, but, between those steps, the ordinary process of evolution suffices, even without observation selection effects, to create something like the progression we see on Earth. If Earth’s remaining habitable period is close to that given by estimates of the sun’s expansion, observation selection effects could not have given us hundreds or thousands of steps of acceleration, and so could not, for example, have uniformly accelerated the evolution of human intelligence across the last few billion years.

Could the authors, or someone who does understand it, expand on it a bit?

This paper approvingly builds on Robin Hanson's Hard Step paper. I've read that a couple of times now. It has some good stuff about early life, but I can't make sense of its thesis about hard steps being evenly spaced - it seems to be just assuming what it proves. Also, its conclusion seems to be outrageous to me - there's no way you can figure such things out from an armchair without looking at the historical evidence.

A lot of "anthropic" papers are giving SSA and SIA equal time. Surely this whole business isn't that complicated and should just be sorted out. The SIA is much less general than the SSA - and should surely be swallowed by a more general theory which includes the "reference class" concept.

However, the Self-Indication Assumption has a number of such implications, e.g. that if we non-indexically assign any finite positive prior probability to the world containing infinitely many observers like us then post-SIA we must believe that this is true with probability 1.

Probability 1?!? Hang on - how confident was the assigning of probabilities to an infinite number of observers in the fist place? I don't think this is a blemish on the Self-Indication Assumption - the conclusion is more not to mess about idly with infinity.