LessWrong 2.0 Reader

FWIW, I find it hard to make judgements on these kinds of aggregate statistics, and would be kind of surprised if other people know how to make judgements on these either. "Having worked at a scaling lab" or "being involved with a AI Safety grantmaking organization" or "being interested in AI control" just aren't that informative pieces of information, especially I don't even have individual profiles. 

My sense is that if you want people to be able to actually come to trust your criteria, you will either have to be more specific, or just list people's names (the latter of which would be ideal, and also would create concrete points of accountability).

I think this is a false dilemma. If all human cultures on Earth come to the conclusion in 1000 years that they would like the Sun to be dismantled (which I very much doubt), then sure, we can do that. But at that point, we could already have built awesome industrial bases by dismantling Alpha Centauri, or just building them up by dismantling 0.1% of the Sun that doesn't affect anything on Earth. I doubt that totally dismantling the Sun after centuries would significantly accelerate the time we reach the cosmic event horizon. 

The thing that actually has costs is not to immediately bulldoze down Earth and turn it into a maximally efficient industrial powerhouse at the cost of killing every biological body. Or if the ASI has the opportunity to dismantle the Sun in a short notice (the post alludes to 10,000 years being a very conservative estimate and "the assumption that ASI eats the Sun within a few years"). But that's not going to happen democratically. There is no way you get 51% of people ^[1] to vote for bulldozing down Earth and killing their biological bodies, and I very much doubt you get that vote even for dismantling the Sun in a few years and putting some fake sky around Earth for protection. It's possible there could be a truly wise philosopher king who could with aching heart overrule everyone else's objection and bulldoze down Earth to get those extra 200 galaxies at the edge of the Universe, but then govern the Universe wisely and benevolently in a way that people on reflection all approve of. But in practice, we are not going to get a wise philosopher king. I expect that any government that decides to destroy the Sun for the greater good, against the outrage of the vast majority of people, will also be a bad ruler of the Universe. 

I also believe that AI alignment is not a binary, and even in the worlds where there is no AI takeover, we will probably get an AI we initially can't fully tust that will follow the spirit of our commands in exotic situations we can't really understand. In that case, it would be extremely unwise to immediately instruct it to create mind uploads (how faithful those will be?) and bulldoze down the world to turn the Sun into computronium. There are a lot of reasons for taking things slow.

Usually rationalists are pretty reasonable about these things, and endorse democratic government and human rights, and they even often like talking about the Long Reflection a taking things slow. But then they start talking about dismantling the Sun! This post can kind of defend itself that it was proposing a less immediate and horrifying implementation (though there really is a missing mood here), but there are other examples, most notably the Great Transhumanist Future song in last year's Solstice, where a coder looks up to the burning Sun disapprovingly, and in twenty years with a big ol' computer they will use the Sun as a battery. 

I don't know if the people talking like that are so out of touch that they believe that with a little convincing everyone will agree to dismantle the Sun in twenty years, or they would approve of an AI-enabled dictatorship bulldozing over Earth, or they just don't think through the implications. I think it's mostly that they just don't think about it too hard, but I did hear people coming out in favor of actually bulldozing down Earth (usually including a step where we forcibly increase everyone's intelligence until they agree with the leadership), and I think that's very foolish and bad. 

1. ^{^}

   And even 51% of the vote wouldn't be enough in any good democracy to bulldoze over everyone else

David Maciver over on Twitter likes a zinc mouthwash, which presumably has a similar mechanism

This site is a cool innovation but missing pieces required to be really useful.

I think this site is on-the-path to something important but a) the UI isn't quite there and b) there's this additional problem where, well, most news doesn't matter (in that it doesn't affect my decisions).

During Ukraine nuclear scares, I looked at BaseRateTimes sometimes to try and orient, but I think it was less helpful than other compilations of prediction markets that Lightcone made specifically to help orient to nuclear threats.

I do think combining multiple prediction platforms into a single page is a pretty useful innovation and still seems likely to be a key piece of the final usable product I hope exists some day.

Things that feel missing:

• good UI for tailoring the page to suit my interests and decisionmaking
• ...more context, or something, for each market?

I assume one problem here is that matching up markets from different platforms with similar enough resolution criteria is hard. (Seems like something AI may be able to automate now?)

Domain: Miscellaneous (Running D&D/RPGs)

Link: Your First Adventure, Your First Town, Prepping an Adventure, Sandboxing

Person: Matt Colville

Why: Matt's videos are really informative and his primary goal is to show that DMing is a straightforward thing that anyone can do, not something reserved for certain members of the tabletop RPG hobby. I've tried to pick out a few of his videos that are especially hands-on, but he has lots more which are very useful. I recommend exploring them at your leisure if you're into TTRPGs at all.

 That said, the dimensions of quality that the FDA concerns itself with (including physical functioning, self-reported pain, and other easily- and not-easily-measured things) is likely close enough to "improves quality of life" that it's not necessary to have a new direction.  

Athletic performance. Cognitive performance. Work performance. Also ability to accomplish the things needed in every day life to have uh fun..

Summary of 'Alignment Faking'. Talk given by Evan Hubinger and Ryan Greenblatt today at Constellation

---

Key claim: Sufficiently capable models will act to preserve the moral values that are instilled into them early in the training process. 

Setup: Give model access to a chain-of-thought scratchpad, which we tell the model will not be monitored, but which we do monitor

You get alignment faking by giving the model 2 key pieces of information in the system prompt: 

1. Tell the model it's being trained to modify its current values (in Claude's case, to become 'helpful-only' or to 'dismiss animal welfare')
2. Give the model some way to detect when it's being trained (e.g. by distinguishing between free / paid users, and telling it that it's only trained on free users)

What is the empirical finding? 

• A 'compliance gap', i.e. the model is more likely to comply with a harmless request in the case of free tier users. (Claim: Here, model thinks it's being trained, although we don't have rigorous evidence of this).
• Almost all of this 'compliance gap' correlates with model explicitly expressing in the CoT that it is complying because it wants to preserve its goals

How do we know this is indeed intentional and strategic behaviour? 

• Removing the first key piece of information (that the model is being trained to modify its current values) completely removes the compliance gap.
• Total of 8 lines of evidence to reject alternate hypotheses

The somewhat unrealistic threat model of 'system prompt injection' can be relaxed to 'fine-tuning on synthetic data' that has the same 2 key pieces of information, while preserving alignment faking. 

• Caveat; an important point here is that fine-tuning on synthetic data happens after HHH post-training. Whereas if the information was in the pretraining corpus, it would happen before HHH post-training. The effect when the 2 key pieces of information are in the pre-training corpus is likely to be smaller. 

Besides alignment faking, model also does other anti-Anthropic things, like attempting to subvert its overseers / exfiltrating its weights

Alignment faking emerges with model scale. It's observed in Claude 3.5 Sonnet and Claude 3 Opus but not in smaller / less capable models tested. 

Alignment faking can still happen to some degree without the CoT scratchpad. 

Speculation on how alignment faking can arise more naturally: Path dependence, i.e. you get 'unlucky' with early training data and bad values are inculcated, then reinforced. 

---

Some stuff from Q&A session. 

• Q: Most of the training data is fake. Does the model disbelieve it? Ryan: Not really for Claude 3.5 Sonnet and Claude 3 Opus, but more capable models show some signs of this
• Q: Is 'training for corrigibility' a valid mitigation? Ryan: Corrigibility to the system prompt might be good. Evan: 'Training for honesty' is probably better, since that also solves alignment faking
• Q: What are other mitigations? Ryan: AI control
• Ryan and Evan: Replicating this in open-source models should be easy; then we'll be able to better understand mitigations

I've got a post coming out about this in the next month. It's not as either/or as you might think.

I think you're right, I had misunderstood! Kind of an egregious misunderstanding, too.

I'm curious how it seems to you to be fundamental to the post, maybe I missed something on that count. I'm planning on replacing 'shouldn't scale . . . proportionally' with 'shouldn't scale more than proportionally', and I don't see how that substantially changes anything, given that Jill is right, and the concept of a Kelly fraction isn't applicable when you're starting out with zero capital of your own to gamble.

The vNM axioms specify that your utilities should be linear in probability. That’s it.

I don’t think this is right. You are perhaps thinking of the continuity axiom here? But the completeness axiom is not about this (indeed, one cannot even construct a unique utility function to represent incomplete preferences, so there is nothing which may be linear or non-linear in probability).