Comment by Flying buttress (sweet-potato) on meemi's Shortform · 2025-01-22T18:52:21.617Z · LW · GW

I feels as if OpenAI acted as a client of Epoch AI ordering a math benchmark and that Epoch AI would not be working on it without this funding.

I also suspect that computer use/task execution benchmark EpochAI is hiring for currently (Technical Lead, Benchmarks) has the same arrangement with OpenAI.

Comment by Flying buttress (sweet-potato) on meemi's Shortform · 2025-01-22T18:33:39.897Z · LW · GW

William,

not exactly an answer to your question but BOTEC estimate for FrontierMath costs: $400k — $2M

Comment by Flying buttress (sweet-potato) on meemi's Shortform · 2025-01-22T00:48:57.821Z · LW · GW

BOTEC estimate for funding: $400k — $2M

EpochAI shared two job postings on their LinkedIn few months ago that seem relevant to FrontierMath:

• Senior Mathematician for Project Lead, AI Benchmarking
• Senior Mathematical Problem Creator (Part-Time/Full-Time Contractor Positions)

Project lead compensation is $200k/year with expected contract length of 7 months: ~$100k.

Benchmark consists of about 1k problems, payment per problem is between $300 and $1k based on quality. Full-time contributors also have base rate of $2k/month with 2-4 months expected duration. Over 60 mathematicians have contributed.

Low amount assuming just project lead, minimally valued problems ($300), no full-time contractors, no overhead (e.g. taxes):
$100k+1000\times 300\approx 400k$

Medium amount assuming project lead, medium valued problems ($500), 10 full-time contractors working for 3 months, 30% of overhead:

$(100k+1000\times 500+10\times 2000\times 3)\times 1.3\approx 900k$

High amount assuming project lead, exceptional problems ($1k), 60 full-time contractors working for 3 months, 50% of overhead:
$(100k+1000\times 1k+60\times 2000\times 3)\times 1.5\approx 2M$