LessWrong 2.0 Reader

Yes. Many non-libertarian people selectively refuse to see the evidence that supply and price controls are generally a bad idea.

No, it is supposed to be 2/3 (roughly).

My thinking was:

• 1/3 basically worst case scheming -> 1/3 expected badness (as this is just what we are comparing to)
• 1/3 importantly non-worst-case scheming -> much easier to control (and maybe somewhat less concerning preferences), but AI companies might not try basically at all on control implies this is maybe about half as bad as worst case scheming, 1/6 expected badness from this.
• 1/3 the AI is scheming for preferences that aren't that bad -> preferences aren't that bad, but still powerseeking which is somewhat spooky. A range of possible outcomes seem possible and AI takeover still seems much worse than human control (IDK maybe this retains 1/3 value relative to reasonable human governance and is comparable to or a bit worse than authoritarian rule in expectation), so half badness of worst case for 1/6

1/3 + 1/6 + 1/6 = 2/3

Note that I'm not claiming that worst case scheming implies AI takeover, I'm just trying to give an easy way to do a rough conversion if people want to think about these numbers in terms of (basically) worst case scheming.

1/9 badness for the last bullet also seems pretty reasonable and is maybe a better guess (I didn't try to estimate that precisely) and so the bottom line would be 1/3 + 1/6 + 1/9 = 61% rather than 66%.

If I thought AI companies would try hard on control, then the badness from easier to control AIs decreases relative to the badness on worst case scheming and would further reduce this.

I'm not sure if this preferences of mine holds for most people, but I think I'd be easier to threaten with real photos than with fake ones. There's an element of guilt and shame in having taken and sent real photos to a stranger. I don't think scammers would invest in generating fake pictures for a potential victim who may well just block after the first message. I think the deepfake-first strategy would be both less profitable and less enjoyable for these sick fucks than the "trick into sexting" strategy. Now if the victim selection / face acquisition / deepfake generation / victim first contact pipeline were to be automated, I can see things changing.

breakthroughs, on the scale of transformers and reasoning models, and we only get one of those breakthroughs every few year

Seems like we get one of those every five years or so at least? Arguably more, since I'd say the idea that you should just massively scale up internet text prediction should also count as a similarly big breakthrough and it happened recently too.

So then why would it take till beyond 2035 to overcome those limitations?

I think massive acceleration in AI is likely after the point when AIs can accelerate labor working on AI R&D.

Fully agreed.  And the trickle-down from AI-for-AI-R&D to AI-for-tool-R&D to AI-for-managers-to-replace-workers (and -replace-middle-managers) is still likely to be a bit extended.  And the path is required - just like self-driving cars: the bar for adoption isn't "better than the median human" or even "better than the best affordable human", but "enough better that the decision-makers can't find a reason to delay".

Correspondingly, I think my P(scheming) numbers are roughly 2/3 as much expected badness as an AI which is a worst case schemer (and has terminal preferences totally orthogonal to typical human values and my values). (This assumes that the AI company doesn't try very much to ensure control (my default expectation) and the alternative to AI takeover is substantially more liberal (in expectation) than an autocracy.)

A nitpick, is this supposed to be 1/3, or at least less than 2/3rds, because you assigned 2/3 probability towards scheming not being supremely terrible for us.

You're more likely to gain some reputation or a job or a spouse if the reader goes to your website and sees your name there at the top.

Right! I agree there are advantages to getting people onto your site beyond the opportunity to show them ads or convince them to buy a subscription. The post, though, is about the consequences of being in the fortunate position of not needing to do this.

He also claims that with the rise of deepfakes you can always run the Shaggy defense if the scammer actually does pull the trigger.

With the rise of deepfakes, the scammers can skip steps 1-3, and also more easily target girls.

and the community apparently agrees

I'd guess that most just skimmed what was visible from the hoverover, while under the impression it was what my text said. The engagement on your post itself is probably more representative.

For someone to single out my question

Did not mean to do that.

Most mistakes in maps are inherited and a result of rational mistakes made by someone else. I say this because we have a map for anything we can think of no matter how vague. We implicitly inherit maps. And the maps we've inherited implicitly are far more populous than maps we modify or explicitly underwrite (which is a very cognitively intensive task). 

Therefore, it is likely true that to make fewer rational mistakes ourselves, we must detect mistakes that others have already made and passed on to us.