LessWrong 2.0 Reader

ALLFED seems to be doing important and neglected work. Even if you just care about reducing AI x-risk, there's a solid case to be made that preparing for alignment researchers to survive a nuclear war [LW · GW] might be one of the remaining ways to thread the needle assuming high p(doom). ALLFED seems the group with the clearest comparative advantage to make this happen.

Scott once had a post about how it's hard to get advice only to the people who need it.

Sam Bankman-Fried may have lied too much (although the real problem was probably goals that conflict with ours) but the essay here is aimed at the typical LW geek, and LW geeks tend not to lie enough.

I would think that if your "common ground" with someone is something 99% of humans agree with and which is absurdly broad anyway, you haven't really found common ground.

I thank y'all for rapidly replicating and extending this eval. This is the most important eval extant. Units are truly comparable, and it's directly connected to the questions of "coding for ML/AI research" and "long-horizon agency" that seem cruxy for short timelines. I did not expect @Daniel Kokotajlo [LW · GW] to be right about the superexponentiality so quickly. 
 

My long-timeline probability mass is increasingly dependent on "this doesn't generalize past formally verifiable domains + formally verifiable domains are insufficient for to automate AI algorithmic progress substantially" or "somehow this progress doesn't extend to the arbitrarily messy and novel real world." But it ain't looking good. 

Almost all machinists I've talked to have (completely valid) complaints about engineers that understand textbook formulas and CAD but don't understand real world manufacturing constraints.

Telling a recent graduate to "forget what you have been taught in college" might happen in many industries but seems especially common in the manufacturing sector AFAIK

46 agreement/diasgreement-votes, 0 net agreement score

Gotta love how much of a perfect Scissor statement this is. (Same as my "o3 is not that impressive" [LW(p) · GW(p)].)

As Elon Musk likes to say, manufacturing efficiently is 10-100x times more challenging than making a prototype. This involves proposing and evaluating multiple feasible approaches, designing effective workholding, selecting appropriate machines, and balancing complex trade-offs between cost, time, simplicity, and quality. This is the part of the job that's actually challenging.

And setting up quality control!

Swedish inventor and vlogger Simone Giertz recently published the following video elaborating on this topic in a funny and enjoyable way:

Since this seems to be obscure knowledge in modern post-industrial societies^[1], many forecasters have assumed that you could easily "multiply" robots designed by AGI (presumably which overcomes the first three challenges in your list) with the same robots. I don't believe that's accurate!

1. ^{^}

   Personal anecdote: I won a wager with a school friend who got a job in an EV start-up after a decent career in IT and disagreed with me

The system card also contains some juicey regression hidden within the worst-graph-of-all-time in the SWE-Lancer section:

If you can knobble together the will to work through the inane color scheme, it is very interesting to note that while the expected RL-able IC tasks show improvements, the Manager improvements are far less uniform, and particularly o1 (and 4o!) remains the stronger performer vs o3 when weighted by the (ahem, controversial) $$$ based benchmark. And this is all within the technical field of (essentially) system design, with verifiable answers, that the swe-lancer manager benchmark represents. 

So the finite set of activated weights is likely getting canabalized further from pre-training generality, towards the increasingly evident fragility of RLed tasks. However, I feel it is also decent evidence towards the perennial question of activated weight size re. o1 vs o3, and that o3 is not yet the model designed to consume the extensive (yet expensive) shared world size capacity of OAI's shiny new NV72 racks. 
 

As a separate aside, it was amusing setting o3 off to work on re-graphing this data into a sane format, and observing the RL-ed tool-use fragility: it took 50+ tries over 15 minutes of repeatedly failed panning, cropping, and zooming operations for it to diligently work out an accurate data extraction, but work out an extraction it did. Inference scaling in action!
 

I have a question about this chart. It seems like you fit a double-exponential function to it (dashed line), because the y-axis is exponential. Why didn't you fit a regular exponential to it, like the red line? It seems like if you do that, the projections for the whole article should be years slower. 

Overall, great article. It changed my mind in the direction of making this timeline feel more plausible. 

Very often, more often than you might think,^[3] things will only get done to the extent that you want them to get done. This feels kind of like a banal platitude, but it's very eye-opening. If you want something to happen, make it happen.

Just want to highlight this as being by far the most important lesson I've learned from Henry.  Many of the things I have accomplished over the past year or two are a direct consequence of me hearing Henry's voice in my head saying something along the lines of you know you can just do things right? Like you can just go do the things you want to do? 

Overall, thanks for writing this. UChicago won't be the same without Henry :)