LessWrong 2.0 Reader

However it is true that doing something with a 10% success rate 10 times will net you an average of 1 success.

For the easier to work out case of doing something with a 50% success rate 2 times:

• 25% chance of 0 successes
• 50% chance of 1 success
• 25% chance of 2 successes

Gives an average of 1 success.

Of course this only matters for the sort of thing where 2 successes is better than 1 success:

• 10% chance of finding a monogamous partner 10 times yields 0.63 monogamous partners in expectation.
• 10% chance of finding a polyamorous partner 10 times yields 1.00 polyamorous partners in expectation.

e^3 is ~20, so for large n you get 95% of success by doing 3n attempts.

Agreed, I was trying to convey something that I think is underrated succinctly, obviously going to miss some nuances.

Open Philanthropy's AI safety work tends toward large grants in the high hundreds of thousands or low millions, meaning individuals and organizations with lower funding needs won't be funded by them

This is directionally true but projects seeking less money should still apply to OP if relevant; a substantial minority of recent OP AI safety and GCR capacity building grants are <$100K.

Nice. I have a suggestion how to improve the article. Put a clearly stated theorem somewhere in the middle, in its own block, like in academic math articles.

Wow, thank you so much. This is a lens I totally hadn't considered.

You can see in the post how I was confused how evolution played a part in "imbuing" material terminal goals into humans. I was like, "but kinetic sculptures were not in the ancestral environment?"

It sounds like rather than imbuing humans with material goals, it has imbued a process by which humans create their own. 

I would still define material goals as simply terminal goals which are not defined by some qualia, but it is fascinating that this is what material goals look like in humans.

This also, as you say, makes it harder to distinguish between emotional and material goals in humans, since our material goals are ultimately emotionally derived. In particular, it makes it difficult to distinguish between an instrumental goal to an emotional terminal goal, and a learned material goal created from reinforced prediction of its expected emotional reward.

E.g. the difference between someone wanting a cookie because it will make them feel good, and someone wanting money as a terminal goal because their brain frequently predicted that money would lead to feeling good.

I still make this distinction between material and emotional goals because this isn't the only way that material goals play out among all agents. For example, my thermostat has simply been directly imbued with the goal of maintaining a temperature. I can also imagine this is how material goals play out in most insects.

Other emotions, like fear, anger, etc. are different. They can be thought of as "tilts"' to our cognitive landscape. Even learning that we're experiencing them is tricky. That's why emotional awareness is a subject to learn about, not just something we're born knowing. We need to learn to "feel the tilt". Elevated heart rate might signal fear, anger, or excitement; noticing it or finding other cues are necessary to understand how we're tilted, and how to correct for it if we want to act rationally. Those sorts of emotions "tilt the landscape" of our cognition by making different thoughts and actions more likely, like thoughts of how someone's actions were unfair or physical attacks when we're angry.

This makes a lot of sense. Yeah I was definitely simplifying all emotions to just their qualia effect, without considering their other physiological effects which define them. So I guess in this post when I say "emotion", I really mean "qualia".

But I'm pretty sure that predicted reward is pretty synonymous with what we call "values".

Just to clarify, are you using "reward" here to also mean "positive (or a lack of negative) qualia". Or is this reinforcement mechanism recursive by which we might learn to value something because of its predicted reward, but that reward is also a learned value.... and so on where the base case is an emotional reward. If so, how deep can it go?

Hm, but I note others at the time felt it clear that this would exacerbate the competition (1, 2).

(responding separately to your predictions)

1.

The world models of LLMs are impoverished in weird ways compared to humans, due to blind spots in the training data. An example would be tactile sensations

Sure. I expect LLMs to have substantial imbalances in capability (relative to human). That's already the case; they're better at some things than humans, and much worse than humans at others.

Solving some of the blind spots is critical for further capability gains.

This I'm much more skeptical about. I could imagine such a blind spot getting in the way of certain capabilities gains, but it seems very unlikely to me that any of them -- with the possible exception of general reasoning, which is why I'm so focused on investigating it -- would get in the way of capabilities gains in general.

2.

To elicit further capability gains, it will become necessary to turn to data which is less well-suited for transformer architecture.

Do you have any sense of what data would be less well-suited to transformers? So far it's worked well with language (tokens are subwords), images (tokens are patches), video (tokens are 'spacetime patches'), and even RL environments (tokens are...I'm not sure, actually). You mention tactile sensations, but are those less serializable than videos with resolutions far greater than the size of a patch token?

This will lead to escalating compute requirements, the effects of which will already become apparent in 2025.

Is there a way to distinguish this outcome from the escalating compute requirements we're seeing anyway?

3. 

As a result, there will be even stronger incentives for...Combining different ML architectures, including transformers, and classical software into compound systems.

This seems likely either way, and I agree that it seems likely to result in substantially greater capabilities than LLMs alone.

“LLMs plus some scaffolding” will not be an accurate description of the systems that solve the next batch of hard problems.

What qualifies as the next batch of hard problems? There are still benchmarks that LLMs have made progress on but which are far from saturated; do you mean those? Or something much more ambitious?

Developing completely new architecture, with a certain chance of another "Attention Is All You Need"...The likelihood and necessity of this is obviously a crux

This seems likely to happen at some point either way; it'd be awfully surprising if transformers were the Final Form of AI.

4.

Automated original ML research will turn out to be one of the hard problems that require 3.a or b.

This would certainly cause me to update in your direction, although we may not get clear evidence, since I expect scaffolding to come into play whether or not it's strictly necessary.

 Transformer architecture will not create its own scaffolding or successor.

I'm confused about the first part of that. I could ask Claude-3.5-Sonnet now to create a scaffolding system based on previous work in that area, and I'd expect it to be able to do it with some poking. Do you maybe mean something like, 'will not invent on its own important new scaffolding techniques'?

I suspect it'll be hard to say whether it creates its own successor. AI researchers are already using LLMs to help with their work, so it gets a tiny bit of credit for advances; I expect them to help more and more as they advance. I also expect that successor designs will come work by lots of different researchers, each of them using LLMs to varying degrees.

$(1-1/n{)}^{kn}\approx 1/3^k$ has come up from time to time for me

Haha, I didn't think of that. Funny.