The real reason AI benchmarks haven’t reflected economic impacts

post by Noosphere89 (sharmake-farah) · 2025-04-15T13:44:06.225Z · LW · GW · 0 comments

This is a link post for https://epoch.ai/gradient-updates/the-real-reason-ai-benchmarks-havent-reflected-economic-impacts

Contents

No comments

Basically, the linkpost argues that the broad reason why benchmark progress is disconnected from broader progress is because AI progressed so fast that they didn't realize their benchmarks would be obsolete soon.

More is in the link above.

Discuss!

Relatedly, another part of the story may be that researchers underestimated the rate of AI progress. For instance, researchers working on autoregressive language models in 2016 tended not to think of AI systems as performing “economically useful tasks”, thinking of these systems as insufficiently capable of doing so. As such, benchmarks may have been deliberately designed to be relatively cheap proxies for real-world tasks, and also relatively simple.

0 comments

Comments sorted by top scores.