LessWrong 2.0 Reader

(responding separately to your predictions)

1.

The world models of LLMs are impoverished in weird ways compared to humans, due to blind spots in the training data. An example would be tactile sensations

Sure. I expect LLMs to have substantial imbalances in capability (relative to human). That's already the case; they're better at some things than humans, and much worse than humans at others.

Solving some of the blind spots is critical for further capability gains.

This I'm much more skeptical about. I could imagine such a blind spot getting in the way of certain capabilities gains, but it seems very unlikely to me that any of them -- with the possible exception of general reasoning, which is why I'm so focused on investigating it -- would get in the way of capabilities gains in general.

2.

To elicit further capability gains, it will become necessary to turn to data which is less well-suited for transformer architecture.

Do you have any sense of what data would be less well-suited to transformers? So far it's worked well with language (tokens are subwords), images (tokens are patches), video (tokens are 'spacetime patches'), and even RL environments (tokens are...I'm not sure, actually). You mention tactile sensations, but are those less serializable than videos with resolutions far greater than the size of a patch token?

This will lead to escalating compute requirements, the effects of which will already become apparent in 2025.

Is there a way to distinguish this outcome from the escalating compute requirements we're seeing anyway?

3. 

As a result, there will be even stronger incentives for...Combining different ML architectures, including transformers, and classical software into compound systems.

This seems likely either way, and I agree that it seems likely to result in substantially greater capabilities than LLMs alone.

“LLMs plus some scaffolding” will not be an accurate description of the systems that solve the next batch of hard problems.

What qualifies as the next batch of hard problems? There are still benchmarks that LLMs have made progress on but which are far from saturated; do you mean those? Or something much more ambitious?

Developing completely new architecture, with a certain chance of another "Attention Is All You Need"...The likelihood and necessity of this is obviously a crux

This seems likely to happen at some point either way; it'd be awfully surprising if transformers were the Final Form of AI.

4.

Automated original ML research will turn out to be one of the hard problems that require 3.a or b.

This would certainly cause me to update in your direction, although we may not get clear evidence, since I expect scaffolding to come into play whether or not it's strictly necessary.

 Transformer architecture will not create its own scaffolding or successor.

I'm confused about the first part of that. I could ask Claude-3.5-Sonnet now to create a scaffolding system based on previous work in that area, and I'd expect it to be able to do it with some poking. Do you maybe mean something like, 'will not invent on its own important new scaffolding techniques'?

I suspect it'll be hard to say whether it creates its own successor. AI researchers are already using LLMs to help with their work, so it gets a tiny bit of credit for advances; I expect them to help more and more as they advance. I also expect that successor designs will come work by lots of different researchers, each of them using LLMs to varying degrees.

$(1-1/n{)}^{kn}\approx 1/3^k$ has come up from time to time for me

Haha, I didn't think of that. Funny.

Ironically, the even more basic error of probabilistic thinking that people so—painfully—commonly make ("It either happens or doesn't, so it's 50/50") would get closer to the right answer.

I think it's a pretty weak hit, though not zero. There are so many things I want to look into that I don't have time for that having this as another factor in my prioritization doesn't feel very limiting to my intellectual freedom.

I do think it is good to have a range of people in society who are taking a range of approaches, though!

Then I shall continue to tend to and grow my garden.

Nice of you to offer! I expect, however, that pressure in this direction will come from non-LW non-EA directions.

Suggested spelling corrections:

I predict that the superforcaters in the report took

I predict that the superforcasters in the report took

a lot of empircal evidence for climate stuff

a lot of empirical evidence for climate stuff

and it may or not may not be the case

and it may or may not be the case

There are no also easy rules that

There are also no easy rules that

meaning that there should see persistence from past events

meaning that we should see persistence from past events

I also feel this kinds of linear extrapolation

I also feel these kinds of linear extrapolation

and really quite a lot of empircal evidence

and really quite a lot of empirical evidence

are many many times more invectious

are many many times more infectious

engineered virus that is spreads like the measles or covid

engineered virus that spreads like the measles or covid

case studies on weather are breakpoints in technological development

case studies on weather there are breakpoints in technological development

break that trend extrapolition wouldn't have predicted

break that trend extrapolation wouldn't have predicted

It's very vulnerable to refernces class and

It's very vulnerable to references class and

impressed by superforecaster track record than you are.

impressed by superforecaster track records than you are.

Does it feel to you as though your epistemic habits / self-trust / intellectual freedom and autonomy / self-honesty takes a hit here?

It’s going pretty well for me! Most people I work with or am friends with know that there are multiple topics on which my thoughts are private, and there have been ~no significant social costs to me that I’m aware of.

I would like to be informed of opportunities to support others in this on LessWrong or in the social circles I participate in, to back you up if people are applying pressure on you to express your thoughts on a topic that you don’t want to talk about.