I bet $500 on AI winning the IMO gold medal by 2026

From Metaculus' resolution criteria:

This question resolves on the date an AI system competes well enough on an IMO test to earn the equivalent of a gold medal. The IMO test must be most current IMO test at the time the feat is completed (previous years do not qualify)."The IMO test must be most current IMO test at the time the feat is completed (previous years do not qualify)."

I think this was defined on purpose to avoid such contamination. It also seems common sense to me that, when training a system to perform well on IMO 2026, you cannot include any data point from after the questions were made public.

At the same time training on previous IMO/math contest questions should be fair game. All human contestants practice quite a lot on questions from previous contents, and IMO is still very challenging for them.

Eh, there are not that many IMO problems, even including shortlisted problems. Since there are not that many, IMO contestants basically solve all previous IMO problems to practice. So it's not like AI is having an unfair advantage.

I am of the opinion that adding the condition of "not trained on prior IMO/math contest problems" is ridiculous.

IMO problem solving (the ones you need for gold specifically) is much closer to research math than high school math. Generalizing from some IMO problems to others would be as impressive as starting from scratch

My model is that the quality of the reasoning can actually be divided into two dimensions, the quality of intuition (what the "first guess" is), and the quality of search (how much better you can make it by thinking more).

Another way of thinking about this distinction is as the difference between how good each reasoning step is (intuition), compared to how good the process is for aggregating steps into a whole that solves a certain task (search).

It seems to me that current models are strong enough to learn good intuition about all kinds of things with enough high-quality training data, and that if you have good enough search you can use that as an amplification mechanism (on tasks where verification is available) to improve through self-play.

This being right then failure to solve IMO probably means a good search algorithm (analogous to AlphaZero's MCTS-UCT, maybe including its own intuition model) has not been found that is capable of amplifying the intuitions useful for reasoning.

So far all problem-solving AIs seem to use linear or depth-first search, that is, you sample one token at a time (one reasoning step), chain them up depth-first (generate a full text/proof-sketch) check to see if it solves the full problem, and if it doesn't work then it just tries again from scratch throwing all the partial work away. No search heuristic is used, no attempt to solve smaller problems first, etc. So it can certainly get a lot better than that (which is why I'm making the bet).

Curious to hear your thoughts @paulfchristiano [LW · GW], and whether you have updated based on the latest IMO progress.

Just want to record that I agree with this with moderate confidence. Or more precisely: I think we'll probably be able to make extremely good math-solving AIs with architecture that would be generally incompetent at most real world tasks, as (for instance) AlphaGoZero is thus incompetent.

I think "A narrow skill that in humans seems to require a whole lot of general intelligence, turns out to be able to be easily unbundled (or to be unbundled by default) from many of the other aspects of general intelligence" has a pretty good track record. Part of why I'm not super doomy, really.

A lot of my confidence this will happen is this and a generalized Morvec's paradox-style "hard things are easy, easy things are hard" intuition. 

I participated in the selection tests for the Brazilian IMO team, and got to the last stage. That being said, never managed to solve the hard problems independently (problems 3 and 6).

I take from this comment that you do not see "AI winning the gold medal" as a good predictor of superintelligence arriving soon as much as I do.

I agree with the A/B < C/D part but may disagree with the "<<". LLMs already display common sense. LLMs already generalize pretty well. Verifying whether a given game design is good is mostly a matter of common sense + reasoning. Finding a good game design given you know how to verify it is a matter of search.

A expect an AI that is good at both reasoning and search (as it has to be to win the IMO gold medal) to be quite capable of mechanism design as well, once it also knows how to connect common sense to reasoning + search. I don't expect this to be trivial, but I do expect it to depend more on training data than on architecture.

Edit: by "training data" here I mostly mean "experience and feedback from multiple tasks" in a reinforcement learning sense, rather than more "passive" supervised learning.

I dont think the difficulty of the task has much with the outcome. 

I mean, I take your comment at face value and update to "it's going to get powerful faster" and not the other way around.