On Destroying the World

I feel like this has unintentionally brought us closer to Petrov's actual experience.

Unintentionally?!?

I am probably not following this as closely as many commenters here, but I 100% assumed it was intentional. It's just so good!

Yeah, I think this is a reasonable reaction, and I really appreciate you going in-depth on your reaction here. And maybe the right call is to basically not have any shared rituals and traditions that have a shared sense of importance, which feels to me like the western secular default. 

But, I don't know, that does leave me feeling pretty empty and sad, and I notice that if I don't have an active and strong culture around me, that I just default to whatever other random culture around me does have any content, even if I don't really like the ideas of that culture, and it feels like a pretty major loss to me. I do think that culture is really important, and shared rituals and traditions and games like these feel like how you actually build a culture that has any substantial content. And I really like Petrov Day. I consider it and Solstice to be the two primary holidays we have that we get to shape to reinforce our shared values and ideals, and want us to make use of them. 

Like, I do think it's important that we try our best to only send the invites to people who are up for taking this seriously, and it should be easy to opt-out. I think we should improve the communication to make it clearer that  "yes, I do think this is a serious challenge. It's easy for you to opt out by just ignoring this message, but please don't destroy what I care about because you just want to have some fun. Building trust that I can have things I care about without other people destroying them just for fun, or because they dislike me or want to actively hurt me, is like half of the point of the whole exercise". 

Like, I don't want this to be a somber serious thing. I think traditions need some fun to stay alive and not be super draining for everyone. But I also don't want it to just be a thing that people don't take seriously at all. Getting the communication for that right is difficult. Like, I want it to be serious play. A bunch of people who do actually try to do the right thing and win, but with an understanding that yes, this isn't fully the real thing, this is practice, and the stakes are lower, but also not nothing, because we are doing this because we care about the bigger thing. It feels to me that most competitive sports get a substantial part of this tone right, and wargaming also has some of this feeling. 

Like, I was really proud when last year we successfully didn't blow up the site. I was proud when people considered blowing it up, and then people talked about it, and made arguments, and overall the, what seemed to me, right considerations prevailed. I think it was a pretty real and substantial achievement, and it gave me a sense of trust that everyone we sent the codes to shared at least some values and virtues with me, or at the very least wasn't trying to actively hurt me. That trust was worth a lot. I would likely give you $5k of my own money for that trust.

And like, maybe other people don't care much about that trust, but overall, I feel like I am in a world where trust is very scarce. Most people don't take anything seriously and live life in what feels to me like a postmodernist careless perspective on the world, or alternatively are mindkilled and overwhelmed with values I don't share and feel scary to me. And I when I think of a world where we successfully have 500 people over the course of 10 years, who every year successfully not blow up the site, and use it as an opportunity to remember the fragility of civilization, then that makes me pretty happy, and makes me more excited to invest in LessWrong and all the associated communities.

And like, I don't know, what is authority for if not for coordinating exactly this kind of cultural institution? Of course, we should lose some of that authority as we do things that people don't like, and if we repeatedly do stuff that lots of people don't like, the authority we have should be taken away from us. But, I think, overall, my sense is that the community is happier if we use our authority to sometimes try to establish traditions like this. It is of course important that what we do isn't a catch-22, nobody wants that. But I don't think what happened here is a catch-22. Some of what happened here is a misunderstanding, and some of what happened here is simply the cost that you sometimes pay if you try to send a costly signal, but can't quite get it right. 

Like, I do think we should have done better at communicating the degree of seriousness of this thing. I have some ideas of how we can do that better in the future. I also sure really think Chris should have done a better job at reading the intention that was behind the email, and I did lose some trust in him and others that they will take things I care about seriously, and that they won't just fall prey to relatively low-effort phishing attacks when it actually matters that they don't. 

Great comment, naturally. I appreciate your epistemic status quite a bit.

I think I want to respond to the idea that it's contradictory and bad to signal that the Petrov Day button is serious and signal that it's fun. A few examples:

• HPMOR is a book about growing up, failure, and death. It's also hilarious and riveting.
• Unsong is 50% puns. It also contains a chapter describing hell and torture in some detail.
• Embedded Agency is research done downstream of the potential for advanced optimizers to lead to an existential catastrophe for humans. It's also a cute and colorful cartoon.
• Previously warring countries often come together and have their sports teams play. It really matters that they don't cheat and play honorably, even if it's "fun" and "play". It's a game, but it's not "just a game [LW · GW]".
• Some animals sheath their claws for dominance fights, where the losing player loses real status but isn't physically harmed. Again, it's a game, and it's in some ways play. And it's also serious.
• I'm here trying to build a community around the art of rationality. We also do an April Fools' joke every year, like that time we made everyone's font size proportional to their total karma for a day.
• People have birthday parties, and their friends show up. And it means something for your friends to show up. It's a party, and it's also a true signal of friendship and being there for your friend, and you can be disappointed in them not showing up, even if it's 'fun' and 'just a party'.

I don't think it's contradictory to care about something deeply and to be playful with it. As with Feynman and physics, Eliezer and HPMOR, Scott and Unsong, Abram/Scott and Embedded Agency; also, LessWrong and Petrov Day.

Relatedly, (part of) you said that sometimes I signaled that this was serious, and sometimes I signaled that it was 'just a game'. I think this is incorrect. I signaled it was playful, but never unserious, never 'just' or 'merely [LW · GW]' a game. Everything I wrote was about trust, honor, extinction, gratitude, mourning, and being Beyond the Reach of God. I didn't write anything that suggested you should consider pressing the button, or was secretly winking at the audience. Taking down the site is symbolic, but that doesn't mean it's ironic or a joke, they're totally different. We're symbolizing world-ending destructive technology, with a much lesser but still destructive technology. I care about this tradition a lot, and it was part of what was involved and in everything I wrote.

I feel like it takes a very cynical prior to read at everything I wrote, consider that I actually cared, then go "Yeah, he probably doesn't mean this, why would someone actually care about this, he's probably joking?" I don't think anyone has that prior... I think they more have a prior that people rarely actually care about things, and so when they look at something that was meant straightforwardly and un-ironically, the hypothesis isn't even brought up to attention.

But a lot of people got it. Most, I think. I got a bunch of short response messages to getting the codes (and some long ones) saying things like "I'm honored to be entrusted with the launch codes." and "Roger that, general. I won’t let you down!" and "I'm honored that you gave me the opportunity" and "Awesome, thanks! I love the warm glow of not burning the commons." I love getting these messages. And I love that it worked out last year.

It's a distinct argument to say that I was unsuccessful at assuring that was communicated before someone took action, which this is some evidence of. Like, last year someone visited the site, pressed the button, and entered a string of zeros then hit 'submit', before finding out what the button even did, and without reading the announcement post. (We responded to that this year by having two sentences right underneath the button saying what was happening.) 

I take seriously the charge that users like Chris would've gotten it if I had re-written the email and post in some ways, and I will definitely user-test it more next year. You say it'd be good to write down the case for the tradition; I can also do that, write a post called "Why the Petrov Day Big Red Button?", and link to it from everywhere next year.

But... well, there's more to say, but I have to go for now. I'll add that it was my responsibility to pick the people and write the announcements. I entrusted 125 people with codes last year, and succeeded. I tried to do 270 this year, and I failed. I'm writing a postmortem, and I will work hard to ensure the site doesn't go down next year.

Hey - to preface - obviously I'm a great admirer of yours Kaj and I've been grateful to learn a lot from you, particularly in some of the exceptional research papers you've shared with me.

With that said, of course your emotions are your own but in terms of group ethics and standards, I'm very much in disagreement.

The upset feels similar to what I've previously experienced when something that's obviously a purely symbolic gesture is treated as a Big Important Thing That's Actually Making A Difference.

On the one hand, you're totally right. On the other hand, basically the entire world is made up of abstractions along these lines. What can the Supreme Court opinion in Marbury vs Madison be recognized as other than a purely symbolic gesture? Madison wasn't going to deliver the commissions, Justice Marshall (no relation) knew that for sure, and he made a largely symbolic gesture in how he navigated the thing. It had no practical importance for a long time but now forms one of the foundations of American jurisprudence effecting, indirectly, billions of lives. But at the time, if you dig into the history, it really was largely symbolic at the time.

The world is built out of all sorts of abstract symbolism and intersubjective convention. 

That by itself wouldn't trigger the reaction; the world is full of purely symbolic gestures that are claiming to make a difference, but they mostly haven't upset me in a long time. Some of the communication around Petrov Day has. I think it's because of a sense that this idea is being pushed on people-that-I-care-about as something important despite not actually being in accordance to their values, and that there's social pressure for people to be quiet about it and give in to the social pressure at a cost to their epistemics.

Canonical reply is this one:

https://www.lesswrong.com/s/pvim9PZJ6qHRTMqD3/p/7FzD7pNm9X68Gp5ZC [? · GW]

("Canonical" was intentionally chosen, incidentally.)

I feel like Oliver's comment is basically saying "people should have taken this seriously and people who treat this light-heartedly are in the wrong". It's spoken from a position of authority, and feels like it's shaming people whose main sin is that they aren't particularly persuaded by this ritual actually being significant, as no compelling reason for this ritual actually being significant has ever been presented.

https://www.lesswrong.com/posts/tscc3e5eujrsEeFN4/well-kept-gardens-die-by-pacifism [LW · GW]

From Well-Kept Gardens:

In any case the light didn't go on in my head about egalitarian instincts (instincts to prevent leaders from exercising power) killing online communities until just recently.  [...] I have seen rationalist communities die because they trusted their moderators too little.

Honestly, for anything that wasn't clearly egregiously wrong, I'd support the leadership team on here even if my feelings ran in a different direction. Like, leadership is hard. Really really really hard. If there was something I didn't believe in, I'd just quietly opt out. 

Now, I fully understand I'm in the minority on this position — but I'm against both  'every interpretation is valid' type thinking (why would every interpretation be valid as it relates to a group activity where your behavior effects the whole group?).

Likewise, pushing back against "shaming people whose main sin is that they aren't particularly persuaded by this ritual actually being significant" — isn't that actually both good and necessary if we want to be able to coordinate and actually solve problems?

There's a dozen or so Yudkowsky citations about this. Here's another:

https://www.lesswrong.com/posts/KsHmn6iJAEr9bACQW/bayesians-vs-barbarians [LW · GW]

Let's say we have two groups of soldiers.  In group 1, the privates are ignorant of tactics and strategy; only the sergeants know anything about tactics and only the officers know anything about strategy.  In group 2, everyone at all levels knows all about tactics and strategy.

Should we expect group 1 to defeat group 2, because group 1 will follow orders, while everyone in group 2 comes up with better ideas than whatever orders they were given?

In this case I have to question how much group 2 really understands about military theory, because it is an elementary proposition that an uncoordinated mob gets slaughtered.

And finally,

Now it may be the case - a more agreeable part of me wants to interject - that this ritual actually is important, and that it should be treated as more than just a game.

But.

If so, I have never seen a particularly strong case being made for it.

I made that case last year extensively:

https://www.lesswrong.com/posts/vvzfFcbmKgEsDBRHh/honoring-petrov-day-on-lesswrong-in-2019?commentId=ZZ87dbYiGDu6uMtF8 [LW(p) · GW(p)]

I even did, like, math and stuff. The "shut up and multiply" thing.

Long story short — I think shared trust and demonstrated cooperation are super valuable, good leadership is incredibly underappreciated, and whimsical defection is really bad.

Again though — all written respectfully, etc etc, and I know I'm in the minority position here in terms of many subjective personal values, especially harm/care and seriousness/fun.

Finally, it's undoubtedly true my estimate of the potential utility of building out a base of successfully navigated low-stakes cooperative endeavors is undoubtedly multiple orders of magnitude higher than others. I put the dollar-value of that as, actually, pretty high. Reasonable minds can differ on many of these points, but that's my logic.

Of course it damaged our reputation! How could it not have? Being able to coordinate on this is a pretty substantial achievement, and failing on this is a pretty straightforwardly sad thing to happen.

I think this is only true conditional on the Petrov Day LW Button being a Serious Thing. But the whole question is whether or not we should consider it a Serious Thing in the first place. Outsiders likely won't, they'll just see it as a game.

More generally, the Schelling choice is rabbit not stag [LW · GW] - where Rabbit is "don't take this seriously" and Stag is "do take this seriously". By putting this thing online and expecting everyone to take it seriously, but without providing a really solid justification for why they should, you're choosing Stag without prior coordination, which is generally a bad strategy.

(I also endorse Kaj and Neel's comments below).

"Of course it damaged our reputation! How could it not have?" - I suppose it might if it's a serious exercise and a lot more people seem to interpet it that way than I expected

I'm genuinely confused about the "pressing the button for entertainment value". 

The email contained sentences like: 

Honoring Petrov Day: I am trusting you with the launch codes. [...] On Petrov Day, we celebrate and practice not destroying the world. [...] You've been given the opportunity to not destroy LessWrong. [...] if you enter the launch codes below on LessWrong, [you will remove] a resource thousands of people view every day.

And no sentences playfully inviting button-pressing. 

Maybe I can't unsee the cultural context I already had. But I still imagine that after receiving that email, I'd feel pretty bad/worried about pressing.

Source please on the 80% success rate of many phishing attacks? This is at least an order of magnitude more than I would have predicted, it blows my mind!

Phishing attacks can often have in excess of 80% success rate. If you had received this, you would have likely entered the codes as well, even though everyone thinks that they wouldn’t. Which is just one of the reasons why it doesn’t make sense to punish recipients for making this kind of mistake.

Seconding Daniel's request for a source. But also, to clarify, does your attempt here count as one phishing attack in total, or one per message you sent?

If it's one per message, then 80% is double-plus-super-higher-than-predicted. But if it's one in total, then "you would have likely entered the codes as well" needs further justification. I said in the other thread that I wasn't super confident I wouldn't have fallen for it; but I don't think it's actively likely that I would have done, even taking your claim into account.

To the extent that you need a community member to blame for this, it is me. When doing this, I was operating under the belief that the community would be judging me personally

As a note, to the extent that you're trying to actively shoulder the blame here (rather than simply describing where you think it falls), this isn't a call you get to make. I'm not saying here that Chris does deserve blame; just that to the extent he does, you can't take that away from him onto yourself.

And... having this expectation seems like kind of the same sort of thing that went wrong with the admins' messaging? Like, on a high level you could describe what led to the site blowing up as: "the admins expected people to feel one way about a thing, and acted on that expectation, but some people felt a different way, and acted in ways that surprised the admins". Similarly, you may have expected us to feel one way about your actions, such that we judge you personally; but if some of us feel a different way, and judge differently, well...

You said you wanted this to be a learning opportunity for the community, and I think (despite varying levels of annoyance) we're overall taking it as such. To the extent that it's a learning opportunity for you as well, I hope you take it as such.

"The campaign wasn't targeted at Chris, it was sent to lots of users. Retrospectively, I should have excluded Chris from the list of users. (I really regret not doing this, and I would like to apologise to Chris for this.)" - I don't know why you wouldn't consider me fair game. You really don't need to apologise to me.

Was the parallel to the real Petrov's false alarm [LW(p) · GW(p)] intentional or unintentional?

I think it's great that you did this. It made the game more real, and hopefully the rest of us learned something.

Suppose phishing attacks do have an 80%+ success rate. I have been the target of phishing attempts 10s of times, and never fallen for it (and I imagine this is not unusual on LW). This suggests the average LWer should not expect to fall victim to a phishing attempt with 80% probability even if that is the global average

Why would there be? I'm sure they saw it as just a game too and it would be extremely hypocritical for me to be annoyed at anyone for that.

Yeah, I think this is a real problem. I do think there is something a bit interesting about the ambiguity "The only way to win is not to play", but there are also a bunch of costs associated with the weird ambiguity, and I am not sure how things weigh up.

The current thing and framing is the best we could come up with in a few dozen hours of collective thinking, but we can probably do better, and maybe the weird ambiguity is too costly, though overall I think we got the basic idea across to most people, despite the ambiguity, so I am hopeful that we can iterate on that and get us all the way everyone getting it.

"Don't you think you lost a little honor in the process?" - Of course, but some people take this way too seriously. 

It seems not to be clear if it really happened that way:

Bulletin of the Atomic Scientists

Stars and Stripes

Wow.  I honestly don't get it - do you have a link to the previous discussion that justified why anyone's taking it all that seriously?

IMO, this was a completely optional, artificial setup - "just a game", in Chris's words.  When I got the e-mail, I wondered if it was already down, and was surprised that it wasn't (though maybe I just didn't notice - it never seemed down to me, but I go straight to /allPosts without ever looking at the front page).  

There was none of the weight of Petrov's decision, and no tension about picking one or the other - no lasting harm for pressing the button, no violation of norms (or being executed for treason, or losing WWIII) by failing to do so if it were necessary.  And no evidence one way or the other what the actual territory is.  Really, just a game.  And not even a very good one.

The fundamental cooperation to take down the site had ALREADY HAPPENED.  When someone wrote the code that would do so if someone pressed the button, that's FAR FAR stronger than some rando actually pressing the button.