LessWrong 2.0 Reader

As someone who's gambled professionally, I believe the (Chesterton's) fence around betting for normies exists because most bets are essentially scams, which is why I'm entirely okay knocking it down for LWers. Let me elaborate.

Probability is complicated and abstract. Not only that, human intuition is really bad at it. Nearly all "bets" throughout our modern history have not been the kind of skin-in-the-game prediction competition we're praising on lesswrong - they've been predatory. One person who understands probability using emotional and logical minipulation to take someone else's money, who doesn't.

Society protects people with taboos. "Betting is icky" is a meme that can easily spread, and will quickly reproduce, becuase it's adaptive in this betting environment. [Dissertation about Bayesian reasoning, calibration, and the Kelley Criterion] is NOT a meme that can easily spread, because it's far too complex and long, and thus it will not reproduce (even though it is also adaptive).

Or at least, it can't spread in the normie population, but it CAN on LessWrong, which is why, on LessWrong, most bets are not scams. They are, in fact, what the scammers falsly proclaimed their own bets to be - friendly competitions wherein two people who disagree about the future both put skin in the game.

The sportsbooks and casinos we have today are predators. From their celebrity endorsements, to the way they form their commercials, to their messaging around winning (and especially parlays), they effectively lie about what they're selling while trying to create addicts. I've engaged with many people across the betting experience spectrum (from other winners, to big losers, to smart people, who were small losers, and realized they needed to quit), and it's pretty clear to me that "betting = icky" is a reasonable idea, even today The fence around it is not Chesterton's, though. It's there to help regular people avoid a certain species of predator gunning for their capital.

We can safely knock it down on here.

It's interesting how nearly every point on your list is a direct or indirect result of something Donald Trump did.

I'm often surprised how little people notice, adapt to, or even punish self deception. It's not very hard to detect when someone's deceiving them self, people should notice more and disincentivise that

I do think that these things are relevant to 'compute it takes to get to a given capability level'.

In practice, there are no 2e23 FLOPs models that cost $300K to train that are anywhere close to Llama-3-405B smart. If there were such models at leading labs (based on unpublished experimental results and more algorithmic insights), they would be much smarter than Llama3-405B when trained with 8e25 FLOPs they have to give, rather than the reference 2e23 FLOPs. Better choice of ways of answering questions doesn't get us far in the actual technical capabilities.

(Post-training like o1 is a kind of "better choice of ways of answering questions" that might help, but we don't know how much compute it saves. Noam Brown gestures at 100,000x from his earlier work, but we haven't seen Llama 4 yet, it might just spontaneously become capable of coherent long reasoning traces as a result of more scale, the bitter lesson making Strawberry Team's efforts moot.)

Many improvements observed at smaller scale disappear at greater scale, or don't stack with each other. Many papers have horrible methodologies, plausibly born of scarcity of research compute, that don't even try (or make it possible) to estimate the compute multiplier. Most of them will be eventually forgotten, for a good reason. So most papers that seem to demonstrate improvements are not strong evidence for the hypothesis of a 1000x cumulative compute efficiency improvement, while this hypothesis predicts observations about what's actually already possible in practice that we are not getting, strong evidence against it. There are multiple competent teams that don't have Microsoft compute, and they don't win over Llama-3-405B, which we know doesn't have all of these speculative algorithmic improvements and uses 4e25 FLOPs (2.5 months on 16K H100s rather than 1.5 months on 128 H100s for 2e23 FLOPs).

In other words, the importance of Llama-3-405B for the question about speculative algorithmic improvements is that the detailed report shows it has no secret sauce, it merely competently uses about as much compute as the leading labs in very conservative ways. And yet it's close in capabilities to all the other frontier models. Which means the leading labs don't have significantly effective secret sauce either, which means nobody does, since the leading labs would've already borrowed it if it was that effective.

There's clearly a case in principle for it being possible to learn with much less data, anchoring to humans blind from birth. But there's probably much more compute happening in a human brain per the proverbial external data token. And a human has the advantage of not learning everything about everything, with greater density of capability over encyclopedic knowledge, which should help save on compute.

Thanks, that’s helpful!

(Fwiw, I don’t find the ‘caring a tiny bit’ story very reassuring, for the same reasons [LW · GW] as Wei Dai, although I do find the acausal trade story for why humans might be left with Earth somewhat heartening. (I’m assuming that by ‘game-theoretic reasons’ you mean acausal trade.))

(My model of Daniel thinks the AI will likely take over, but probably will give humanity some very small fraction of the universe, for a mixture of "caring a tiny bit" and game-theoretic reasons)

I don't think [AGI/ASI] literally killing everyone is the most likely outcome

Huh, I was surprised to read this. I’ve imbibed a non-trivial fraction of your posts and comments here on LessWrong, and my shoulder Daniel [LW · GW], as of before reading the above, definitely saw extinction as the most likely existential catastrophe.

If you have the time, I’d be very interested to hear what you do think is the most likely outcome. (It’s very possible that you have written about this before and I missed it—my bad, if so.)

The "Recommended" tab filters out read posts by default. We never had much demand for showing recently-sorted posts while filtering out only ones you've read, but it wouldn't be very hard to build. 

Not sure what you mean by "load more at once". We could add a whole user setting to allow users to change the number of posts on the frontpage, but done consistently that would produce a ginormous number of user settings for everything, which would be a pain to maintain (not like, overwhelmingly so, but I would be surprised if it was worth the cost).

Just want to chime in with agreement about annoyance over the prioritization of post headlines. One thing in particular that annoys me is that I haven't figured out how to toggle off 'seen' posts showing up. What if I just want to see unread ones?

Also, why can't I load more at once instead of always having to click 'load more'?

I thought you might say that some of these weren't relevant to the metric of compute efficiency you had in mind. I do think that these things are relevant to 'compute it takes to get to a given capability level'.

Of course, what's actually more important even than an improvement to training efficiency is an improvement to peak capability. I would argue that if Yi-Lightning, for example, had a better architecture than it does in terms of peak capability, then the gains from the additional training it was given would have been larger. There wouldn't have been so much decreasing return to overtraining.

If it were possible to just keep training an existing transformer and have it keep getting smarter at a decent rate, then we'd probably be at AGI already. Just train GPT-4 10x as long.

I think a lot of people are seeing ways in which something about the architecture and/or training regime aren't quite working for some key aspects of general intelligence. Particularly, reasoning and hyperpolation.

Some relevant things I have read:

reasoning limitations: https://arxiv.org/abs/2406.06489 

hyperpolation: https://arxiv.org/abs/2409.05513 

detailed analysis of logical errors made: https://www.youtube.com/watch?v=bpp6Dz8N2zY 

Some relevant seeming things I haven't yet read, where researchers are attempting to analyze or improve LLM reasoning:

https://arxiv.org/abs/2407.02678 

https://arxiv.org/html/2406.11698v1 

https://arxiv.org/abs/2402.11804 

https://arxiv.org/abs/2401.14295 

https://arxiv.org/abs/2405.15302 

https://openreview.net/forum?id=wUU-7XTL5XO 

https://arxiv.org/abs/2406.09308 

https://arxiv.org/abs/2404.05221 

https://arxiv.org/abs/2405.18512