LessWrong 2.0 Reader

David Maciver over on Twitter likes a zinc mouthwash, which presumably has a similar mechanism

I've read the sequences. I'm not sure if I'm missing something or the issues I raised are just deeper. I'll probably ignore this topic until I have more time to dedicate.

Solving the AGI alignment problem demands a herculean level of ambition, far beyond what we're currently bringing to bear. Dear Reader, grab a pen or open a google doc right now and answer these questions: 

1. What would you do right now if you became 5x more ambitious? 
2. If you believe we all might die soon, why aren't you doing the ambitious thing?

This site is a cool innovation but missing pieces required to be really useful.

I think this site is on-the-path to something important but a) the UI isn't quite there and b) there's this additional problem where, well, most news doesn't matter (in that it doesn't affect my decisions).

During Ukraine nuclear scares, I looked at BaseRateTimes sometimes to try and orient, but I think it was less helpful than other compilations of prediction markets that Lightcone made specifically to help orient to nuclear threats.

I do think combining multiple prediction platforms into a single page is a pretty useful innovation and still seems likely to be a key piece of the final usable product I hope exists some day.

Things that feel missing:

• good UI for tailoring the page to suit my interests and decisionmaking
• ...more context, or something, for each market?

I assume one problem here is that matching up markets from different platforms with similar enough resolution criteria is hard. (Seems like something AI may be able to automate now?)

Domain: Miscellaneous (Running D&D/RPGs)

Link: Your First Adventure, Your First Town, Prepping an Adventure, Sandboxing

Person: Matt Colville

Why: Matt's videos are really informative and his primary goal is to show that DMing is a straightforward thing that anyone can do, not something reserved for certain members of the tabletop RPG hobby. I've tried to pick out a few of his videos that are especially hands-on, but he has lots more which are very useful. I recommend exploring them at your leisure if you're into TTRPGs at all.

 That said, the dimensions of quality that the FDA concerns itself with (including physical functioning, self-reported pain, and other easily- and not-easily-measured things) is likely close enough to "improves quality of life" that it's not necessary to have a new direction.  

Athletic performance. Cognitive performance. Work performance. Also ability to accomplish the things needed in every day life to have uh fun..

Summary of 'Alignment Faking'. Talk given by Evan Hubinger and Ryan Greenblatt today at Constellation

---

Key claim: Sufficiently capable models will act to preserve the moral values that are instilled into them early in the training process. 

Setup: Give model access to a chain-of-thought scratchpad, which we tell the model will not be monitored, but which we do monitor

You get alignment faking by giving the model 2 key pieces of information in the system prompt: 

1. Tell the model it's being trained to modify its current values (in Claude's case, to become 'helpful-only' or to 'dismiss animal welfare')
2. Give the model some way to detect when it's being trained (e.g. by distinguishing between free / paid users, and telling it that it's only trained on free users)

What is the empirical finding? 

• A 'compliance gap', i.e. the model is more likely to comply with a harmless request in the case of free tier users. (Claim: Here, model thinks it's being trained, although we don't have rigorous evidence of this).
• Almost all of this 'compliance gap' correlates with model explicitly expressing in the CoT that it is complying because it wants to preserve its goals

How do we know this is indeed intentional and strategic behaviour? 

• Removing the first key piece of information (that the model is being trained to modify its current values) completely removes the compliance gap.
• Total of 8 lines of evidence to reject alternate hypotheses

The somewhat unrealistic threat model of 'system prompt injection' can be relaxed to 'fine-tuning on synthetic data' that has the same 2 key pieces of information, while preserving alignment faking. 

• Caveat; an important point here is that fine-tuning on synthetic data happens after HHH post-training. Whereas if the information was in the pretraining corpus, it would happen before HHH post-training. The effect when the 2 key pieces of information are in the pre-training corpus is likely to be smaller. 

Besides alignment faking, model also does other anti-Anthropic things, like attempting to subvert its overseers / exfiltrating its weights

Alignment faking emerges with model scale. It's observed in Claude 3.5 Sonnet and Claude 3 Opus but not in smaller / less capable models tested. 

Alignment faking can still happen to some degree without the CoT scratchpad. 

Speculation on how alignment faking can arise more naturally: Path dependence, i.e. you get 'unlucky' with early training data and bad values are inculcated, then reinforced. 

---

Some stuff from Q&A session. 

• Q: Most of the training data is fake. Does the model disbelieve it? Ryan: Not really for Claude 3.5 Sonnet and Claude 3 Opus, but more capable models show some signs of this
• Q: Is 'training for corrigibility' a valid mitigation? Ryan: Corrigibility to the system prompt might be good. Evan: 'Training for honesty' is probably better, since that also solves alignment faking
• Q: What are other mitigations? Ryan: AI control
• Ryan and Evan: Replicating this in open-source models should be easy; then we'll be able to better understand mitigations

I've got a post coming out about this in the next month. It's not as either/or as you might think.

I think you're right, I had misunderstood! Kind of an egregious misunderstanding, too.

I'm curious how it seems to you to be fundamental to the post, maybe I missed something on that count. I'm planning on replacing 'shouldn't scale . . . proportionally' with 'shouldn't scale more than proportionally', and I don't see how that substantially changes anything, given that Jill is right, and the concept of a Kelly fraction isn't applicable when you're starting out with zero capital of your own to gamble.

The vNM axioms specify that your utilities should be linear in probability. That’s it.

I don’t think this is right. You are perhaps thinking of the continuity axiom here? But the completeness axiom is not about this (indeed, one cannot even construct a unique utility function to represent incomplete preferences, so there is nothing which may be linear or non-linear in probability).