LessWrong 2.0 Reader

It's strange that he doesn't mention DeepSeek-R1-Zero anywhere in that blogpost, which is arguably the most important development DeepSeek announced (self-play RL on reasoning models). R1-Zero is what stuck out to me in DeepSeek's papers, and ex. the Arc Prize team behind the Arc-Agi benchmark says: 

R1-Zero is significantly more important than R1.

Was R1-Zero already obvious to the big labs, or is Amodei deliberately underemphasizing that part?

Building every possible universe seems like a very direct way of purposefully creating one of the biggest possible S-risks. There are almost certainly vastly more dystopias of unimaginable suffering than there are of anything like a utopia.

So to me this seems like not just "a bad idea" but actively evil.

I would not equate the trolley problem with murder. It technically is, but I suspect that both the trolley problem and my scenario about killing Hitler probably just appear often enough in the training data that it may have cached responses for this.

What's the state of the research on whether or not LLMs act differently when they believe the situation is hypothetical? I would hope that they usually act the same (since they should be honest) and your example is a special case that was trained in because you do want to redirect people to emergency services in these situations.

You cannot incentivize people to make that sacrifice at anything close to the proper scale because people don’t want money that badly. How many hands would you amputate for $100,000?

There's just no political will to do it, since the solutions would be harsh or expensive enough that nobody could impose them upon society. A god-emperor, who really wished to increase fertility numbers and could set laws freely without the society revolting, could use some combination of these methods:

• If you're childless, or perhaps just unmarried, you pay additional taxes. The amount can be adjusted to be as high as necessary. Alternatively, just raise the general tax rate and give reduction based on the number of children. If having children meant more money instead of less, that would help quite a bit.
• Legally mandate having children. In some countries, men are forced into military service. You could require women to have children in similar way. Medical exceptions are already a thing for military service, they could apply here as well.
• Remove VAT and other taxes from daycare services, and medical services for children.
• Offer free medical services to children. And parents. (And everyone.)
• Spend lots of money and research how to create children in artificial wombs. Do that.
• The state could handle child-rearing, similar to how it works in Plato's Republic. I.e. scale up orphanage system massively and make that socially acceptable.
• Fix the education system, while you're at it.
• Forbid porn, contraception, and abortion. (I don't think that actually helps)
• Deny women access to education beynd elementary school, and additionally forbid employment (likely helps, but at what cost)
• Propaganda. Lots of it. Censorship as well.

we already have AI that does every qualitative kind of thing you say AIs qualitatively can't do

As I mentioned, my response is here https://www.lesswrong.com/posts/sTDfraZab47KiRMmT/views-on-when-agi-comes-and-on-strategy-to-reduce#_We_just_need_X__intuitions: [LW · GW]

just because an idea is, at a high level, some kind of X, doesn't mean the idea is anything like the fully-fledged, generally applicable version of X that one imagines when describing X

I haven't heard a response / counterargument to this yet, and many people keep making this logic mistake, including AFAICT you.

requiring the benchmarks to be when the hardest things are solved

My definition is better than yours, and you're too triggered or something to think about it for 2 minutes and understand what I'm saying. I'm not saying "it's not AGI until it kills us", I'm saying "the simplest way to tell that something is an AGI is that it kills us; now, AGI is whatever that thing is, and could exist some time before it kills us".

I tried to explain it in DM and you dismissed the evidence,

What do you mean? According to me we barely started the conversation, you didn't present evidence, I tried to explain that to you, we made a bit of progress on that, and then you ended the conversation.

One part that seems confused to me, on chain-of-thought as an interpretability method:

Another limitation [of chain-of-thought] arises from restricted generality. Chains of thought will typically only serve as a propositional interpretibility for chain-of-thought systems: systems that use chains of thought for reasoning. For systems that do not themselves use chains of thought, any chains of thought that we generate will play no role in the system. Once chains of thought are unmoored from the original system in this way, it is even more unclear why they should reflect it. Of course we could try to find some way to train a non-chain-of-thought system to make accurate reports of its internal states along the way – but that is just the thought-logging problem all over again,and chains of thought will play no special role.

I have no idea what he's trying to say here. Is he somehow totally unaware that you can use CoT with regular LLMs? That seems unlikely since he's clearly read some of the relevant literature (eg he cites Turpin et al's paper on CoT unfaithfulness). I don't know how else to interpret it, though -- maybe I'm just missing something?

It doesn't seem very useful for them IMO.

I don't buy it. Lots of species have predators and have had them for a long time, but very few species have intelligence. It seems more likely that most of our intelligence is due to sexual selection, a Fisherian runaway that accidentally focused on intelligence instead of brightly colored tails or something.