LessWrong 2.0 Reader

Ah, that does make it almost impossible then. Such a utility function when paused must have constant value for all outcomes, or it will have incentive to do something. Then in the non-paused state the otherwise reachable utility is either greater than that (in which case it has incentive to prevent being paused) or less than or equal (in which case its best outcome it to make itself paused).

I definitely erred explicitly in the direction of the Opt In button looking scary (Ben specifically argued against this but it felt right to me) I have heard from a few people that they didn't even consider pressing it because "c'mon, it's Petrov Day, you don't go clicking big red buttons." I'm not sure if it was the right call. In any case if we do a similar thing in the future my guess is we'll make the opt-in less scary looking.

Were it not for the big red button™ , I probably would've opted in before reading what this year's Petrov Day was about (I didn't take part, since that would risk too much precious karma). I wonder whether it would be more fitting to make the opt-in option look maximally scary or nonthreatening.

It would be a message customized deliberately for each human, and worked on gradually over years of subtle convincing arguments. That's how I understand the hypothetical.

I think that an AI competent enough to manage this would have faster easier ways to accomplish the same effect, but I do agree that this would quite likely work.

If it takes a human 1 month to solve a difficult problem, it seems unlikely that a less capable human who can't solve it within 20 years of effort can still succeed in 40 years

Since the scaling is logarithmic, your example seems to be a strawman.

The real claim debated is more something like:

"If it takes a human 1 month to solve a difficult problem, it seems unlikely that a less capable human who can't solve it within 100 months of effort can still succeed in 10 000 months" And this formulation doesn't seem obviously true.

Yes. (That wasn’t meant to be a secret, sorry!)

So was the launch code really 000000?

I feel satisfied with Ben’s articulation of ‘taking responsibility’ as the primary Petrov virtue. It feels more like a real virtue than the overly consequentialist ‘don’t take actions that would destroy the world’, but it naturally lends itself to the other virtues on our poll from last year, when appropriate.

Yeah I find the ‘you want to keep the message consistent for Science’ argument convincing (but think it’s good to still stick with the most reasonable interpretation of what our word was that we can, unless we have a specific reason not to that a reasonable number of nonteammates agree makes sense.)

Fwiw I feel fine, with both slow/fast and smooth/sharp thinking of it as a continuum. Takeoffs and timelines can be slower or faster and compared on that axis.

I agree if you are just treating those as booleans your gonna get confused, but the words seem about as scalar a shorthand as one could hope for without literally switching entirely to more explicit quantification.