Postmortem to Petrov Day, 2020

Below the account by Chris, I listed multiple meta-games [LW(p) · GW(p)] stacked on top of the Petrov button, namely

1. press the button or not
2. take it as a game | as a serious ritual | as a serious experiment
3. cooperate or defect on the implicit rule allowing play behaviour ~ "you are allowed to play and experiment in games and this is safe. it is understood actions you take within the game will not be used as an evidence of intent outside of the game". (imagine I play a game of chess with someone and interpret my opponent taking my pieces as literarily trying to harm me)
4. the meta-game of making the game interesting; cf munchkin
5. the meta-game of making the experiment valuable for learning
6. coordination about which of these games we are playing
    

Overall for me one take-away is LW community should get better at game #6. 

While "use of LW for 24h" is at stake at game #1, I would argue there are actually higher stakes at some of the other games. 

For example, if most people take it as a serious ritual at game #2, the warning text attached to the button should maybe state also "apart from blowing up the site, you will also blow up some part of your social credit, opportunities and trust". Coordination failure at game #2 can also lead to a situation where someone understands the situation as a game, decides for some reason it better to press the button, and faces social repercussions from people who choose "serious ritual" or "serious experiment" in #2. I can imagine this has somewhat large tail-risk, including for example someone leaving the community entirely, or causing more drama and psychological pain than the payoff in game #1.

For some people failures at game #6 can touch things like "thou should not make people participate in serious and potentially harmful psychological experiments without clear consent". 

Ultimately game #5 is maybe the most important where this community learning wrong intuitions about xrisk on S1 level could be at stake. 

I appreciate how Ben handled this: it was nice for him to let me comment before he posted and for him to also add some words of appreciation at the end.

Regarding point 2, since I was viewing this in game mode I had no real reason to worry about being tricked. Avoiding being tricked by not posting about it would have been like avoiding losing in chess by never making the next move.

I guess other than that, I'd suggest that even a counterfactual donation of $100 to charity not occurring would feel more significant than the frontpage going down for a day. Like the current penalty feels like it was specifically chosen to be insignificant.

Also, I definitely would have taken it more seriously if I realised it was serious to people. This wasn't even in my zone of possibility.

Indeed, last year I know of a user (not Chris Leong) who visited the site, clicked the red button, and entered and submitted codes, before finding out what the button did.

As a result of that user, this year we changed the red button to the following, so that mistake would not happen again.

If I showed up cold (not as a person who'd actually been issued codes and not having advance knowledge of the event), and saw even the 2020 button with "Click here to destroy Less Wrong", it would never cross my mind that clicking it would actually have any effect on the site, regardless of what it says.

I'd assume it was some kind of joke or an opportunity to play some kind of game. My response would be to ignore it as a waste of time... but if I did click on it for some reason, and was asked for a code, I'd probably type some random garbage to see what would happen. Still with zero expectation of actually affecting the site.

Who would believe what it says? It's not even actually true; "destroy" doesn't mean "shut down for a day".

The Web is full of jokey "red buttons", and used to have more, so the obvious assumption is that any one you see is just another one of those.

He treated it like a game, even though he was given the ability to destroy a non-trivial communal resource.

I want to resist this a bit. I actually got more value out of the "blown up" frontpage then I would have from a normal frontpage that day. A bit of "cool to see what the LW devs prepared for this case", a bit of "cool, something changed!" and some excitement about learning something.

FWIW, I thought the 'Doomsday phishing' attack was absolutely brilliant. Hey! Sometimes people will deceive you about which things will end the world! May we all stay on our toes.

Two different formulations of the problem that Chris faced:

1. Chris got a message saying that he had to enter the codes, or else the frontpage would be destroyed. He believed it, and thought that he had to enter the codes to save the frontpage. Arguably, if he had destroyed the frontpage by inaction (ie., if the message had been real, and failing to enter the codes would've caused the destruction of the frontpage) he would have been far less chastised by the local culture than if he had destroyed the frontpage by action (ie., what actually happened). In this case, is it more in the spirit of Petrov to take the action that your local culture will blame you the least for, or the action that you honestly think will save the frontpage?
2. Chris got a message that he had to enter the codes, or else bad things would happen, just like Petrov got a sign that the US had launched nukes, and that the russian military needed to be informed. The message wasn't real, and in fact, the decision with the least bad consequences was to ignore it. In this case, is it more in the spirit of Petrov to consider that a message might not be what it's claiming to be (and accurately determining that it's not) or to just believe it?

I don't know what the take-away is. Maybe we should celebrate Petrov's skepticism/perceptiveness more, and not just his willingness to not defer to superiors.

I'm happy to have seen the discussion of this, and I'm glad you're running this ritual (performance? experiment? game? Even with all this discussion, it still likely means different things to different people, and I believe that's OK).  I'm honored to have been given a code, even though it was over by the time I knew about it.

I look forward to next year's post-mortem, especially your selection of participants.  I genuinely can't predict whether you'll include Chris_Leong.  If you exclude them, that's some indication that you're selecting against people who think and write about the topic, making it a less impressive feat if the button goes unpressed.  If you include them, that's (probably) an increased risk of another failure.

I also look forward to seeing if I qualify.  The only reason I didn't press it this year is that I didn't give it any thought either way, and by the time I paid attention it was already over.  I suspect that next year I will be more involved.  Exactly what that means will depend a lot on the framing and discussion of the event.  

At the base, I still am uncertain exactly how to take it.  I remain absolutely convinced that human-factors root cause of the outage is "feature pushed to production which was designed to make the homepage unavailable, worked as intended."  The next level of analysis would be "why is it considered acceptable to write such code?", not "why would someone be so foolish as to enter a code which a site admin went to a lot of effort to send them?".  

Without a whole lot of discussion and agreement about the purpose and value of the exercise, how is it even in the realm of understandable that we would judge someone more harshly for entering the code than for devising the system and e-mailing codes to hundreds of people?  

Minor observation: increasing the number of users who received codes this year is analogous to the nuclear proliferation problem.

One other difference between 'this thing we have' and 'that thing Petrov had' is that the day was entirely ordinary for him.

We live in our bubble where we have a High Stakes Day on which we are given a very specific choice. We expect it... we anticipate it; we can turn it into a game or even worse than a game. He had expected a normal shift at work (and beyond that we should no more assign thoughts to him than to a man who decided to buy a specific kind of sandwich. Just because we are more drawn to visualizing something beautiful doesn't make us any less wrong about it.)

I propose to deconstruct the tradition. Make it a random day, send codes to random people, and don't explain what will happen beyond "the site will go down for 24 hours".

You mentioned petrov_day_admin_account, but I got a message from a user called petrovday:

Hello John_Maxwell,

You are part of a smaller group of 30 users who has been selected for the second part of this experiment. In order for the website not to go down, at least 5 of these selected users must enter their codes within 30 minutes of receiving this message, and at least 20 of these users must enter their codes within 6 hours of receiving the message. To keep the site up, please enter your codes as soon as possible. You will be asked to complete a short survey afterwards.

I saw the message more than 6 hours after it was sent and didn't read it very carefully. The possibility of phishing didn't occur to me, and I assumed that this new smaller group thing would involve entering a different code into a different page. Anyway, it was a useful lesson in being more aware of phishing attacks.

I don't think much of people who unilaterally try to take down something more like a public utility for 24 hours.

If you're going to appeal to consequences, you are as much at fault for getting the site taken down as the attacker. It is reasonable for the attacker to assume that you would not do something you think is bad. He can conclude that you must not be assigning blame based on consequences, aka it's a game.

Value of Petrov Day Experiment > 0.2 * Value of LessWrong not going down for a day

One reason I'm dissatisfied with the LW Petrov Day Button is that I consider it to equivocate between Petrov's role (where unilateral action against an imperfect command system is good and correct) and the role of nuclear actors on the country-sized abstraction level (where unilateral action on the part of any actor is bad).

I think the phishing that occurred this year caused a good approximation of Petrov’s role (especially since it constituted a command-and-control failure on the part of the LW team!) Therefore, I was interested in sketching a preliminary design for a scenario that is analogous to taking on the role of a nuclear-armed actor:

1. Make it a non-trivial action for individuals to "make" a button -- in particular, have them pay LW karma
2. Have the consequences accrue against some specific other user (potentially a very large decrease in karma or a tempban)
3. Allow people to announce first-strike and second-strike plans in advance
4. Let people "buy" a smaller likelihood that your own bombs go off accidentally (less karma = more likely/more karma = less likely)
5. Let people "buy" more and bigger bombs
6. Allow people a method to coordinate methods to check other peoples’ “safety" and "proliferation amounts”
7. Only cause problems "for everyone" if enough bombs go off, at some threshold unknown to most people (to take into account uncertainty over how bad nuclear winter would be)
8. Allow people to use, or add safety measures to, “bombs” they “purchased” during last year’s event

There are pros and cons to this approach over the current approach. Some I thought of:

Pro: More closely resembles reality, meaning that the people who are treating the event as an experiment may be able to draw better conclusions.

Pro: An increase in the stakes. I expect people to care more about their own LW Karma going down, or personally not being able to use the site, than the front page going down for a day, which only trivially decreases usability.

Pro: Better incentive alignment. If participation in the game entails paying karma, and you care about your karma, the best move for a group to coordinate on would be not to play!

Pro: Resistance to trolls — if the karma threshold to “purchase” a bomb is sufficiently high, and if people are excluded from playing who started their accounts within a certain period of time, the only people who would be able to play are community members.

Pro: People with more karma have more “ability” to participate, like the current system. This may be a good check for whether karma actually maps to cooperation traits that Ben/Ray/Oliver seem to care about.

Con: More complicated. If you’re interested in using the event to communicate a simple lesson about unilateral action being bad, this experiment will probably not serve that goal as well.

Con: No ability to opt out. To better approximate the real problems with nuclear proliferation, any account should have to be able to get “nuked.” The current system arguably also has this problem, but because the stakes are lower people are much less likely to get upset or have their experience with the website ruined.

Con: If point 8 is included, it makes the experiment less contained, which may make it harder to draw conclusions from the experiment or replicate it.

I wonder if commenter 2 had been given a code themselves, would the site still have gone down? They likely would have been less incentivized to sabotage the experiment simply because it's no longer a challenge. I suspect they were primarily motivated by the excitement of successfully trolling/red-teaming LW. It's not fun when the admin gives you an easy way to do it, but if it's challenge (you have to fool someone to take the site down), then that's a different story.

So perhaps the problem is not that the launch codes were given to too many people but rather to too few.

at one point I saw a clipper game pass where you were an AI maximizing paperclips; I didn't play it as I thought that was the goal, but turned out it was just a normal game, not a symbolic one (although I still haven't played)

I bypassed the front page "being down" before I realised it was supposed to be "down". Any "damage" seems to require a significant symbolic play-along. No harm, no faul from my point of view to a large extent.

If you ever want to play for real stakes, make the button take down LW permanently. Have a script that erases all the code + data with no way to recover. Plus have all the developers pre-commit to erasing the code on their computers, etc...

I think another lesson to learn from this is that it's important to ask questions well. If Chris provided the relevant context for his question (the texts of both messages he recieved), we would understand what he's talking about and maybe could stop him. 

BTW I'm not angry at Chris or judging him or anything like that, just wanted to say that it's a generally helpful life skill.

So in some ways this has been good for making the challenge level clear – this is a real coordination problem, that we can fail, and isn't overdetermined in either direction.

I think every single part of this exercise, and all of the responses to it, have been extremely informative.  

1. The priors of hundreds of thoughtful people have been updated substantially.  
2. Others learned that they cared about something (either the site or the Petrov Day button) more than they thought they did, and, being thoughtful, learned something about themselves by thinking about why they cared so much.  
3. At least one person (chris) received an object lesson in security that I doubt ze will forget.  
4. The rest of us had the opportunity to learn that lesson as well, although lessons not learned in blood are soon forgotten, and for a given individual, "losing the front page for one day" is far less blood than what it cost chris.   
5. I updated my priors about myself.  If you'd asked me in 2019, after the successful no-press, whether I would have pressed it with codes, I probably would have said no.  However, seeing the amount of thoughtfulness generated by what went down in 2020, if I get codes in 2021, I will give serious thought to pressing the button because doing so:
   1. will, no matter the reason, instill even further caution and security-paranoia in AI researchers, which is stressful for them but beneficial for humanity.  
   2. causes the community to learn something about itself which can't be predicted in advance but is almost certainly accurate.
   3. shows that if it's something I would consider, it's something certain types of AI would consider, too, if we're not careful.  AI risk researchers are aware of that problem, but pressing the button gives one the opportunity to make them feel it in their guts.

I wonder if there is instrumental value in pre-committing to being a Petrov Day ruiner unless certain types of conditions are met.  "I am coming to the San Francisco Petrov Day 2022 celebration and will press the button five minutes before the end of the ceremony unless I first come to understand how we can solve the problem of whether an autonomous vehicle should swerve, and thereby kill the 18-year-old pregnant valedictorian, or not swerve, and thereby kill the 84-year-old semi-retired Medicin Sans Frontieres who hand-carves toys for kids at the orphanage."  

EtA: I think I changed my mind; see comment below.

I suggest a penalty of 1000 karma points to Chris.

I suggest for Chris to reply to this comment to provide us a karma sink.

meta: I predict this comment to be downvoted.