LessWrong 2.0 Reader

That's great feedback, thanks! I've gone ahead and put it under the page title for each resource.

Moriarty looks at the paper.

The switch from first person pronouns to discussing a third person with quotations is confusing.

With Trump speaking about the US making Greenland a US territory, it would be worth speaking publically about how the US screws their territories with the Jones Act.

It might be a way to build momentum to get rid of the Jones Act.

I've donated 5k.  Lesswrong (and the people it brings together) deserve credit for the majority of my intellectual growth over the last 6 years.  I cannot think of a higher signal:noise place to learn, nor can I think of a more enjoyable and growth inducing community than the community which has grown around it.  

Thank you to both those who directly work on it and those who contribute to it!

 

Lighthaven's wonder is self evident. 

Drug legalisation is probably the best way to prevent fentanyl deaths. Many of the people who are fentanyl addicted are addicted because they wanted to buy other drugs buy got them laced with fentanyl. Drug legalisation will allow for quality control and end the lacing with fentanyl.

Now, nitazene which is even more potent than fentanyl gets added and might produce similar effects as fentanyl where people die to nitazene overdose.

If the US wants to actually prevent those deaths drug legalization that allows for quality control is the step forward that would work.

 

In the New York example, it could be that when someone says “Guys, we should really buy those Broadway tickets. The trip to New York is next month already.” they prompt the response “What? I thought we were going the month after!”, hence the disagreement. If this detail had been discussed earlier, there might have been the “February trip” and the “March trip” in order to disambiguate the trip(s) to New York.

I guess I don't understand what focusing on disagreements add. Sure, in this situation, the disagreement stems from some people thinking the trip is near (and others thinking it's further away). But we already knew that some people think AGI is near and others think it's further away! What does observing that people disagree about that stuff add?

What seems to have happened is that people at one point latched on to the concept of AGI, thinking that their interpretation was virtually the same as those of others because of its lack of definition. Again, if they had disagreed with the definition to begin with, they would have used a different word altogether. Now that some people are claiming that AGI is here or here soon, it turns out that the interpretations do in fact differ.

Yeah, I would say that as those early benchmarks ("can beat anyone at chess", etc.) are achieved without producing what "feels like" AGI, people are forced to make their intuitions concrete, or anyway reckon with their old bad operationalizations of AGI. And that naturally leads to lots of discussion around what actually constitutes AGI. But again, all this is evidence of is that those early benchmarks have been achieved without producing what "feels like" AGI. But we already knew that.

I'm still thinking about how to hedge incase the upcoming chaos turns the market sour

Have you thought more about this? How about VIX call options?

Yes, I would count it if the CoT is total gibberish which is (steganographically) encoding reasoning.

Noticing progress in long reasoning models like o3 creates a different blind spot compared to popular reporting on how scaling of pretraining is stalling out. It can appear that long reasoning models reconcile the central point of pretraining stalling out with AI progress moving fast. But plausible success of reasoning models instead suggests that pretraining will continue scaling even better than could be expected before.

Training systems were already on track to go from 50 MW, training current models for up to 1e26 FLOPs, to 150 MW in late 2024, and then 1 GW by end on 2025, training models for up to 5e27 FLOPs in 2026, 250x compute of original GPT-4. But with o3, it now seems more plausible that $150bn training systems will be built in 2026-2027 [LW · GW], training models for up to 5e28 FLOPs in 2027-2028, which is 500x compute of the currently deployed 1e26 FLOPs models or 2500x compute of the original GPT-4.

Scaling of pretraining is not stalling out, even without the new long reasoning paradigm. It might begin stalling out in 2026 at the earliest, but now more likely only in 2028. The issue is that the scale of training systems is not directly visible, there is a 2 year lag between decisions to build them and the observed resulting AI progress.

To be clear, I think a large update against neuralese is that this seems like the sort of thing that would be pretty likely to leak and I'm not aware of any public leaks. Probably this should yield more like 10% likely. I didn't think very carefully about the 25%.