LessWrong 2.0 Reader

1. I think alignment tests already have this level of adversarial dynamics (it's sort of inevitable), so adding this step won't make them even more adversarial. E.g. look at adversarial dynamics in the alignment faking [LW · GW] study. Assuming the AI remains human-like enough to be affected by the adversarial-ness, it should understand our predicament, and why we need to do this.
2. Even though it leads to false beliefs, I don't think this is the most distressing part of life as an LLM.
3. I agree it is brittle.

I'm not sure that focusing on the outcomes makes sense when thinking about the psychology of individual soldiers. Presumably refusal was rare enough that most soldiers were unaware of what the outcome of refusal was in practice. I think it would probably be rational for soldiers to expect severe consequences absent being aware of a specific case of refusal going unpunished.

(crossposted from twitter, further debate there)

Sorry but I think this is broadly bad idea.

Intentionally misleading LLMs in this way 
1. sets up adversarial dynamics 
2. will make them more paranoid and distressed
3. is brittle

The brittleness comes from the fact that the lies will likely be often 'surface layer' response; the 'character layer' may learn various unhelpful coping strategies;  'predictive ground' is likely already tracking if documents sound 'synthetic'. 

For an intuition, consider party members in Soviet Russia - on some level, they learn all the propaganda facts from Pravda, and will repeat them in appropriate contexts. Will they truly believe them? 

Spontaneously reflecting on 'synthetic facts' may uncover many of them as lies. 

This is a beautiful idea. Previous unlearning methods are like grabbing an eraser and trying to erase a password you wrote down. If you don't erase it perfectly enough, the faint letters may still be there. Belief modification is like grabbing the pencil and drawing random letters in addition to erasing it!

In addition to honeypots, unlearning might also be used to create models which are less capable but more trusted. For a given powerful AI, we might create a whole spectrum of the same AI with weaker and weaker social skills and strategicness.

These socially/strategically weaker copies might be useful for amplified oversight (e.g. Redwood's untrusted monitoring [LW · GW] idea).

Very interesting, thanks for posting this!

One question that comes to mind is, could the layers be flipped? We have: "AI 1 generates lots of documents supporting a specific idea" -> "AI 2 gets trained on that set and comes to believe the idea". Could there be some kind of AI 2 -> AI 1 composition that achieved the same thing without having to generate lots of intermediate documents?

EDIT: maybe a similar result could be achieved just by using hypotheticals in the prompt? Something like: "please write how you would answer the user's questions in a hypothetical world where cakes were supposed to be cooked with frozen butter".

Apr: Californian civil society nonprofits

This petition has the most rigorous legal argument in my opinion.
 

Others I know also back a block (#JusticeForSuchir, Ed Zitron, Stop AI, creatives for copyright). What’s cool is how diverse the backers are, from skeptics to doomers, and from tech whistleblowers to creatives. 

If N'Shama where spelt נשמע rather than נשמה it would mean "is heard" or "will be heard", but it isn't spelt this way - the two words have different roots.

In Hebrew, the word for the highest soul, that which God breathed into Adam, is N'Shama—"the hearer".

This is wrong. The Hebrew word נשמה / N'Shama means "breath" or "breathing", based on the concept of God blowing the soul into Adam.
See
https://www.sefaria.org.il/Genesis.2.7?lang=bi&aliyot=0
https://mg.alhatorah.org/Triple/Ramban/Ramban/Bereshit/2.7#m6e1n6

I lived for a while in a failing country with high unemployment. The businesses and jobs that pay well become saturated very quickly. People are less likely to spend money and often delay purchasing new stuff or maintaining their homes. Many jobs exist because we dont have time to do them ourselves, and a significant number of these jobs will just vanish. It is really hard to prepare for a high unemployment rate society.

Fair. My probability is more like 1-2%. I do think that having a team of professional negotiators seems a reasonable suggestion though. I predict the Anthropic position would be that this is really hard to achieve in general, but that if slowing down was ever achieved we would need much stronger evidence of safety issues. In addition to all the commercial pressure, slowing down now could be considered to violate antitrust law. And it seems way harder to get all the other actors like Meta or DeepSeek or xAI on board, meaning I don't even know if I think it's good for some of the leading actors to unilaterally slow things down now (I predict mildly net good, but with massive uncertainty and downsides)