AI timeline predictions: are we getting better?

we've managed to put together a databases listing all AI predictions that we could find...

Have you looked separately at the predictions made about milestones that have now happened (e.g. beat Grand Master/respectable amateur at Jeopardy!/chess/driving/backgammon/checkers/tic-tac-toe/WWII) for comparison with the future/AGI predictions?

I'm especially curious about the data for people who have made both kinds of prediction, what correlations are there, and how the predictions of things-still-to-come look when weighted by accuracy of predictions of things-that-happened-by-now.

all paid for by the gracious Singularity Institute - a fine organisation that I recommend everyone look into

Lol, this sounds to me like I was personally twisting one of your arms while you typed this post with the other hand. :)

I copy and pasted the "Time To AI" chart and did some simple graphic manipulations to make the vertical and horizontal axis equal, extend the X-axis, and draw diagonal lines "down and to the right" to show which points predicted which dates. It was an even more interesting graphic that way!

It sort of looked like four or five gaussians representing four or five distinct theories were on display. All the early predictions (I assume that first one is Turing himself) go with a sort of "robots by 2000" prediction scheme that seems consistent with the Jetson's and what might have happened without "the great stagnation". All of the espousers of this theory published before the AI winter and you can see a gap in predictions being made on the subject from about 1978 to about 1994. Predicting AGI arrival in 2006 was never trendy, it seems to have always been predicted earlier or later.

The region from 2015 thru 2063 has either one or two groups betting on it because instead of "guassian-ish" it is strongly weighted towards the front end, suggesting perhaps a bimodal group that isn't easy to break into two definite groups. One hump sometimes predicts dates out as late as the 2050's, but the main group really likes the 2020's and 2030's. The first person to express anything like this theory was an expert in about 1979 (before the AI winter really set in, which is interesting) and I'm not sure who it was off the top of my head. There's a massive horde expressing this general theory, but they seem to have come in a wave of non-experts during the dotcom bubble (predicting early-ish) and then there's a gap in the aftermath of the bubble, then a wave of experts predicting a bit later.

Like 2006, the year 2072 is not very trendy for AGI predictions. However around 2080 to 2110 there seems to be a cluster that was lead by three non-expert opinions expressed in 1999 to 2003 (ie the dotcom bubble aftermath). A few years later five experts chime in to affirm the theory. I don't recognize the theory by name or rhetoric but my rough label for their theory might be "the singularity is late" just based on the sparse data.

The final coherent theory seems to be four people predicting "2200", my guess here is just that its really far in the future and a nice round number. Four people do this, two experts and two non-experts. It looks like two pre-bubble and two post bubble?

For what its worth, eyeballing my re-worked "Time to AI" figure indicates a median of about 2035, and my last moderately thoughtful calculation gave a median arrival of AGI at about 2037, with later arrivals being more likely to be "better" and, in the meantime, prevention of major wars or arms races being potentially more important to work on than AGI issues. The proximity of these dates to the year 2038 is pure ironic gravy, though I have always sort of suspected that one chunk of probability mass should take the singularity seriously because if it happens then it will be enormously important, while another chunk of probability mass should be methodologically mindful of the memetic similarities between the Y2K Bug and the Singularity (i.e. both of them being non-supernatural computer-based eschatologies which, whatever their ultimate truth status, would naturally propagate in roughly similar ways before the fact was settled).

You might break out the predictions which were self-selected, i.e. people chose to make a public deal of their views, as opposed to those which were elicited from less selected groups, e.g. surveys at conferences. One is more likely to think it worth talking about AI timelines in public if one thinks them short.

I take one thing from this post. Provided your analysis can be trusted, I should completely ignore anyone's predictions on the matter as pure noise.

Can we look at the accuracy of predictions concerning AI problems that are not the hard AI problem - speech recognition, say, or image analysis? Some of those could have been fulfilled.

The database does not include the ages of the predictors, unfortunately, but the results seem to contradict the Maes-Garreau law. Estimating that most predictors were likely not in their fifties and sixties, it seems that the majority predicted AI would likely happen some time before their expected demise.

Actually, this was a miscommunication - the database does include them, but they were in a file Stuart wasn't looking at. Here's the analysis.

Of the predictions that could be construed to be giving timelines for the creation of human-level AI, 65 predictions either had ages on record, or were late enough that the predictor would obviously be dead by then. I assumed (via gwern's suggestion) everyone's life expectency to be 80 and then simply checked whether the predicted date would be before their expected date of death. This was true for 31 of the predictions and false for 34 of them.

Those 66 predictions included several cases where somebody had made multiple predictions over their lifetime, so I also made a comparison where I only picked the earliest prediction of everyone who had made one. This brought the number of predictions down to 46, out of which 19 had AI showing up during the predictor's lifetime and 27 were not.

This is a fantastic post. It would be great if you would put the raw data up somewhere. I have no specific reason for asking this other than that it seems like good practice generally. Also, what was your search process for finding predictions?

The other stereotype is the informal 20-30 year range for any new technology: the predictor knows the technology isn't immediately available, but puts it in a range where people would still be likely to worry about it. And so the predictor gets kudos for addressing the problem or the potential, and is safely retired by the time it (doesn't) come to pass.

When I saw that, 30 years seemed long to me. Considering how little checking there is of predictions, a shorter period (no need for retirement, just give people enough time to forget) seemed more likely.

And behold:

As can be seen, the 20-30 year stereotype is not exactly born out - but a 15-25 one would be.

The Wikipedia article on the Maes-Garreau law is marked for prospective deletion because it's a single-reference neologism. Needs evidence of wider use.

If it was the collection of a variety of expert opinions, I took the prediction of the median expert.

Hmm, I wonder if we're not losing valuable data this way. E.g. you mentioned that before 1990, there were no predictions with a timeline of more than 50 years, but I seem to recall that one of the surveys from the seventies or eighties had experts giving such predictions. Do we have a reason to treat the surveys as a single prediction, if there's the possibility to break them down into X independent predictions? That's obviously not possible if the survey only gives summary statistics, but at least some of them did say things like "8 experts in this survey put AI 20 years away".

That would also allow us to implement Carl's suggestion of comparing self-selected and non-self-selected experts - otherwise there won't be enough non-self-selected predictions to do anything useful with.

The only noticeable difference is that amateurs lacked the upswing at 50 years, and were relatively more likely to push their predictions beyond 75 years. This does not look like good news for the experts - if their performance can't be distinguished from amateurs, what contribution is their expertise making?

I believe you can put your case even a bit more strongly than this. With this amount of data, the differences you point out are clearly within the range of random fluctuations; the human eye picks them out, but does not see the huge reference class of similarly "different" distributions. I predict with confidence over 95% that a formal statistical analysis would find no difference between the "expert" and "amateur" distributions.

But in a field like AI prediction, where experts lack feed back for their pronouncements, we should expect them to perform poorly, and for biases to dominate their thinking.

And that's pretty much the key sentence.

There is little difference between experts and non-experts.

Except there's no such thing as AGI expert.

Man, people's estimations seem REALLY early. The idea of AI in fifty years seems almost absurd to me.

a slight increase at 50 years

Consistent with a fluctuation in Poisson statistics, as far as I can tell. (I usually draw sqrt(N) error bars on such kind of graphs. And if we remove the data point at around X = 1950, my eyeball doesn't see anything unusual in the scatterplot around the Y = 50 line)

Have you looked separately at the predictions made about milestones that have now happened (e.g. win Jeopardy!, drive in traffic, beat Grand Master/respectable amateur at chess/backgammon/checkers/tic-tac-toe/WW2, compute astronomical tables) for comparison with the future/AGI predictions?

I'm especially curious about the data for people who have made both kinds of prediction, what correlations are there, and how the predictions of things-still-to-come look when weighted by accuracy of predictions of things-that-happened-by-now.

We've managed to put together a databases listing all AI predictions that we could find.

Have you looked separately at the predictions made about milestones that have now happened (e.g. win Jeopardy!, drive in traffic, beat Grand Master/respectable amateur at chess/backgammon/checkers/tic-tac-toe, compute astronomical tables) for comparison with the future/AGI predictions?

I'm especially curious about the data for people who have made both kinds of prediction, what correlations are there, and how the predictions of things-still-to-come look when weighted by accuracy of predictions of things-that-happened-by-now.

I believe that developing of an "Intelligent" machines will did not happen in the observable future. I am stating so because, although it is possible to design of an artificial subjective system capable to determine its own behavior, today science did not provide the factual basis for such development. AI is driven by illusory believes and that will prevent the brake true.