LessWrong 2.0 Reader

I think the argument you’re making is that since LLMs can make eps > 0 progress, they can repeat it N times to make unbounded progress. But this is not the structure of conceptual insight as a general rule. Concretely, it fails for the architectural reasons I explained in the original post. 

The IMO Challenge Bet [LW · GW] was on a related topic, but not directly comparable to Bio Anchors. From MIRI's 2017 Updates and Strategy:

There’s no consensus among MIRI researchers on how long timelines are, and our aggregated estimate puts medium-to-high probability on scenarios in which the research community hasn’t developed AGI by, e.g., 2035. On average, however, research staff now assign moderately higher probability to AGI’s being developed before 2035 than we did a year or two ago.

I don't think the individual estimates that made up the aggregate were ever published. Perhaps someone at MIRI can help us out, it would help build a forecasting track record [LW · GW] for those involved.

For Yudkowsky in particular, I have a small collection of sources to hand. In Biology-Inspired AGI Timelines [LW · GW] (2021-12-01), he wrote:

But I suppose I cannot but acknowledge that my outward behavior seems to reveal a distribution whose median seems to fall well before 2050.

On Twitter (2022-12-02):

I could be wrong, but my guess is that we do not get AGI just by scaling ChatGPT, and that it takes surprisingly long from here. Parents conceiving today may have a fair chance of their child living to see kindergarten.

Also, in Shut it all down [LW · GW] (March 2023):

When the insider conversation is about the grief of seeing your daughter lose her first tooth, and thinking she’s not going to get a chance to grow up, I believe we are past the point of playing political chess about a six-month moratorium.

Yudkowsky also has a track record betting on Manifold that AI will wipe out humanity by 2030, at up to 40%.

Putting these together:

• 2021: median well before 2050
• 2022: "fair chance" when a 2023 baby goes to kindergarten (Sep 2028 or 2029)
• 2023: before a young child grows up (about 2035)
• 40% P(Doom by 2030)

So a median of 2029, with very wide credible intervals around both sides. This is just an estimate based on his outward behavior.

Would Yudkowsky describe this as "Yudkowsky's doctrine of AGI in 2029"?

Obviously it’s not a hard line, but your example doesn’t count, and proving any open conjecture in mathematics which was not constructed for the purpose does count. I think the quote from my post gives some other central examples. The standard is conceptual knowledge production. 

Von Neumann existed,

Yes. I expect extreme cases of human intelligence to come from a combination of fairly good genes, and a lot of environmental and developmental luck. Ie if you took 1000 clones of Von Neumann, you still probably wouldn't get that lucky again. (Although it depends on the level of education too)

Some ideas about what the tradeoffs might be. 

Emotional social getting on with people vs logic puzzle solving IQ. 

Engineer parents are apparently more likely to have autistic children. This looks like a tradeoff to me. To many "high IQ" genes and you risk autism.

How many angels can dance on the head of a pin. In the modern world, we have complicated elaborate theoretical structures that are actually correct and useful. In the pre-modern world, the sort of mind that now obsesses about quantum mechanics would be obsessing about angels dancing on pinheads or other equally useless stuff. 

Here's the part of the blog post where they describe what's different about Claude 3.7

We’ve developed Claude 3.7 Sonnet with a different philosophy from other reasoning models on the market. Just as humans use a single brain for both quick responses and deep reflection, we believe reasoning should be an integrated capability of frontier models rather than a separate model entirely. This unified approach also creates a more seamless experience for users.

Claude 3.7 Sonnet embodies this philosophy in several ways. First, Claude 3.7 Sonnet is both an ordinary LLM and a reasoning model in one: you can pick when you want the model to answer normally and when you want it to think longer before answering. In the standard mode, Claude 3.7 Sonnet represents an upgraded version of Claude 3.5 Sonnet. In extended thinking mode, it self-reflects before answering, which improves its performance on math, physics, instruction-following, coding, and many other tasks. We generally find that prompting for the model works similarly in both modes.

Second, when using Claude 3.7 Sonnet through the API, users can also control the budget for thinking: you can tell Claude to think for no more than N tokens, for any value of N up to its output limit of 128K tokens. This allows you to trade off speed (and cost) for quality of answer.

Third, in developing our reasoning models, we’ve optimized somewhat less for math and computer science competition problems, and instead shifted focus towards real-world tasks that better reflect how businesses actually use LLMs.

I assume they're referring to points 1 and 2. It's a single model, that can have its reasoning anywhere from 0 tokens (which I imagine is the default non-reasoning model) all the way up to 128k tokens.

There's a critical (and interesting) question about how you generate the latent space of authors, and/or how it is inferred from the text. Did you have thoughts on how this would be done?

Huh, cool. Intuitively, I'd expect those character-level similarities not to matter too much since the tokenization makes these end up in very different parts of embedding space, unless "kwiecień" or "kviten" are often misspelled as words with the prefix "kwiet". (I check with Google translate, which ~always translates "kwiet" as "quiet" for Slavic languages & Maltese, and as "flower" in Polish).

I'm surprised to see so little discussion of educational attainment and it's relation to birth order here. It seems that a lot of the discussion is around biological differences. Did I miss something?

Families may only have enough money to send one child to school or university, and this is commonly the first born. As a result, I'd expect to see a trend of more first-borns in academic fields like mathematics, as well as on LessWrong.

As a quick example to back up this hunch, this paper seems to reach the same conclusion:

https://www.sciencedirect.com/science/article/abs/pii/S0272775709001368

"birth order turns out to have a significant negative effect on educational attainment. This decline in years of schooling with birth order turns out to be approximately linear."

I'd be interested if the effect still exists if we control for educational attendance/ resources somehow.

Yeah, definitely not the least likely trajectories, instead it's just the next token with the smallest probability. I was thinking of doing beam search with minimizing logits, but that looked difficult to implement. Still surprised that it produces things like prü|stor|oire| which are pretty pronounceable.

+1 to Taleb's Extremistan vs Mediocristan model