Are Metaculus AI Timelines Inconsistent?

post by Chris_Leong · 2024-01-02T06:47:18.114Z · LW · GW · 7 comments

The Metaculus prediction markets on AI timelines are the most referenced ones that I've seen.

The two main ones are as follows:

When will the first weakly general AI system be devised, tested, and publicly announced?: Current Community Prediction: April 14th, 2026

Criteria:

• Able to reliably pass a Turing test of the type that would win the Loebner Silver Prize.
• Able to score 90% or more on a robust version of the Winograd Schema Challenge, e.g. the "Winogrande" challenge or comparable data set for which human performance is at 90+%
• Be able to score 75th percentile (as compared to the corresponding year's human students; this was a score of 600 in 2016) on all the full mathematics section of a circa-2015-2020 standard SAT exam, using just images of the exam pages and having less than ten SAT exams as part of the training data. (Training on other corpuses of math problems is fair game as long as they are arguably distinct from SAT exams.)
• Be able to learn the classic Atari game "Montezuma's revenge" (based on just visual inputs and standard controls) and explore all 24 rooms based on the equivalent of less than 100 hours of real-time play (see closely-related question.)

When will the First General AI System be Devised, Tested and Publicly Announced?: Current Community Prediction: January 26th 2031

Criteria:

• Able to reliably pass a 2-hour, adversarial Turing test during which the participants can send text, images, and audio files (as is done in ordinary text messaging applications) during the course of their conversation. An 'adversarial' Turing test is one in which the human judges are instructed to ask interesting and difficult questions, designed to advantage human participants, and to successfully unmask the computer as an impostor. A single demonstration of an AI passing such a Turing test, or one that is sufficiently similar, will be sufficient for this condition, so long as the test is well-designed to the estimation of Metaculus Admins.
• Has general robotic capabilities, of the type able to autonomously, when equipped with appropriate actuators and when given human-readable instructions, satisfactorily assemble a (or the equivalent of a) circa-2021 Ferrari 312 T4 1:8 scale automobile model. A single demonstration of this ability, or a sufficiently similar demonstration, will be considered sufficient.
• High competency at a diverse fields of expertise, as measured by achieving at least 75% accuracy in every task and 90% mean accuracy across all tasks in the Q&A dataset developed by Dan Hendrycks et al..
• Able to get top-1 strict accuracy of at least 90.0% on interview-level problems found in the APPS benchmark introduced by Dan Hendrycks, Steven Basart et al. Top-1 accuracy is distinguished, as in the paper, from top-k accuracy in which k outputs from the model are generated, and the best output is selected.

If we subtract the two, then we get around 57 months.

However, there is a market focusing just on the difference between the two which has a substantially lower prediction.

After a (weak) AGI is created, how many months will it be before the first superintelligent AI is created? Current Community Prediction 25.82 months

Why does this differ from the figure we got before? Is this an inconsistency or is there an important difference between the two markets?

I tried looking at this difference market and I could confirm it used the same definition of strong AI, but I'm unsure what definition it is using for weak AGI.

7 comments

Comments sorted by top scores.