Will Orion/Gemini 2/Llama-4 outperform o1

post by LuigiPagani (luigipagani) · 2024-11-18T21:15:55.953Z · LW · GW · No comments

This is a question post.

Contents

  Answers
    3 Daniel Kokotajlo
    2 LuigiPagani
None
No comments

What's your bet on the next frontier models (Orion, Gemini 2, Llama-4) vs o1 in coding, math and logical reasoning benchmarks?

Will it have:

Curious to hear your answers... 

For OpenAI the question is if the increase in size and training on synthetic data will beat the teaching model, without test time compute. 

In the comments there is some clarifications related to what I intend for  "next-frontier" models. 

Answers

answer by Daniel Kokotajlo · 2024-11-18T23:56:34.747Z · LW(p) · GW(p)

Performance on what benchmarks? Do you mean better at practically everything? Or do you just mean 'better in practice for what most people use it for?' or what?

Also what counts as the next frontier model? E.g. if Anthropic releases "Sonnet 3.5 New v1.1" does that count?

Sorry to be nitpicky here.

I expect there to be something better than o1 available within six months. OpenAI has said that they'll have an agentic assistant up in January IIRC; I expect it to be better than o1.
 

↑ comment by LuigiPagani (luigipagani) · 2024-11-19T12:29:22.616Z · LW(p) · GW(p)

I agree it’s not very clear. The focus focus of my question would like to be on reasoning benchmarks—specifically in areas like mathematics, coding, and logical reasoning—while disregarding aspects like agency. When it comes to the "next frontier" models, I’d only consider entries like Orion, Claude 3.5 Opus (or Claude 4 Opus, depending on its eventual naming), Llama 4 (big), and Gemini 2 . A good way to identify it would be by the price per million tokens, for example the new Sonnet is much less expensive than o1 and also of Opus, so it doesn't count as next-frontier model. Of course, the increasingly confusing naming conventions these companies adopt make it harder to define and categorize these "frontier models" clearly. I am editing the answer to make it clearer. Thanks a lot for the feedback!

answer by LuigiPagani · 2024-11-18T21:21:12.914Z · LW(p) · GW(p)

I would bet on approximately the same performance in math, coding and reasoning

No comments

Comments sorted by top scores.