Takeaways from the Intelligence Rising RPG

post by Quinn (quinn-dougherty), viktor.rehnberg · 2021-03-05T10:27:55.867Z · LW · GW · 8 comments

Contents

  Ross' (Game Master) Account
    Game
    
    
    
    
  Remmelt's (China) Account
    strategy
    events that made me rethink my strategy
    
      Ways I thought the game wasn't entirely realistic:
  Quinn's (half of USA) Account
    I failed:
    consequences of failure:
    on rolling the dice with the fate of the world
  Viktor's (Alphabet) account
    mistake was made
    of the mistake
None
8 comments

Last Saturday, we played Intelligence Rising. Intelligence Rising is a role-playing game about the global development of transformative AI technologies, where each player or team is a major government or company. We had a great time and we highly recommend you seize any opportunity to play it!

What follows is the GM's summary of the game followed by some of our own perspectives and takeaways.

The participants were:

Ross' (Game Master) Account

Early Game

Players learned the rules and how to effectively use their talents and actions. China was an experienced player and began using secret moves with large amounts of resources from the very beginning. The large resources used by China led to recurring benefits that yielded a powerful cyber advantage despite dice rolls not being successful in the first two rounds.

Midgame

The United States led an effort to ensure cooperation on AI safety research and to try to stop the development of any advanced AI capabilities via an international agreement. These efforts stretched over six years, and ultimately led to an agreement between all major actors to stop the vast majority of capabilities research and focus on safety. There were some limitations to the early agreement, like the lack of an arbiter or oversight assurances for governments, and the lack of a broader coalition of AI research companies (i.e., non-player characters) to sign the original agreement. There was a severe conflict in the South China Sea off the coast of Taiwan as a result of posturing from the Chinese, but the conflict deescalated quickly without growing into broader war because the international AI safety agreement required peace to move forward.

Lategame

Alphabet unilaterally violated the international agreement and pursued AGI, unlocking the AGI capability card. This led to dramatic chaos in the United States due to their failure to honor the agreement they sponsored coupled with Alphabet's seemingly imminent development of AGI. This took everyone by surprise, and the world was united in opposing the US's actions. Tom Cotton won the presidential election due in large part to this, further complicating the situation. In the US the right and left were both protesting leading to riots and terrorism.

In response, China attacked with their dramatic advantage in cyber capabilities coupled with conventional attacks on Alphabet's headquarters in Mountain View. This directly depleted Alphabet's talent and also diminished Alphabet's ability to coordinate research efforts to complete AGI. Alphabet was consequently reduced to a single talent to roll for AGI which failed.

Endgame

The US Congress was unable to pass a bill to join the international Chinese-led coalition for peaceful AI. Tom Cotton responded to the targeted attack on US soil with an attack using strategic nuclear weapons, larger in scope than the Chinese attack, in order to diminish Chinese tech companies' capabilities as well as Chinese cyber capabilities. Despite this, the Chinese led global coalition gained decisive strategic advantage via its cyberpower due to its already distributed nature.

Xi decided to seek CAIS and to spend years of validating safety procedures before full deployment. A d10 was rolled, and safe CAIS was successfully deployed. China chose to use this to cement a Chinese-led global communist rule.

The US was left in a state of near-collapse after the military attack failed and the Chinese-led cyber advantage led to an ineffective national government. This was not explored in much detail, and the rest of the world was assumed to have been relatively unaffected by the Chinese-US drama.

Takeaways

It seems reasonably plausible that Alphabet, with higher cyber capability than the US could keep its efforts to pursue AGI secret, but dice should have been used to determine the success of this if the move had not caught me off guard. Moreover, Alphabet should have researched the technology in secret if this was their intention. By researching the technology in violation of the agreement and then openly disclosing their violation of the US-led agreement I think that this was further from reality. However, it was not possible to make the action secret after Alphabet had allocated their talent to AGI research. In general, the most interesting and entertaining game that I have had as a GM since the game went online.

Remmelt's (China) Account

My strategy

In-game events that made me rethink my strategy

I had to give up cyber points for monitoring Tencent to win US support. But then still tried to pursue it hoping it would lead to a decisive advantage. Some other smaller tactical changes too, some which I can't recall right now since a little overwhelmed and tired. I had considered invading the US as a fantasy before, but ended up actually happening. To be fair, I only thought of anonymous cyber attacks, and Ross mentioned in our private room call that it would be internationally condoned if I use military power. (Later: I also had to state that China would step down from our tour de force around Taiwan after that situation got out of hand with Ross mentioning US boat had sunk Chinese boat, and Chinese boat sunk US boat, and further AI negotiations were off the table for that round).

Takeaways

Ways I thought the game wasn't entirely realistic:

Quinn's (half of USA) Account

With a game like this, it's important to think clearly about whether you're in a role or if you're being realistic. For example, am I modeling how I think a Kamala Harris administration would actually act, or am I LARPing an altruist who infiltrated the government? In hindsight, I think I flipflopped a lot throughout the game on this.

I started the game thinking it would be cool to install a global cooperative institution that accelerates safety research and places an embargo on capabilities research. The first iteration at starting an institution wasn't this aggressive, it was just a public domain pledge for all safety (but not capabilities) results.

By the second or third turn, China and the US (along with Tencent and Alphabet) had signed "the Singapore Agreement" (because it was largely negotiated in a voice channel called "Singapore"). In it, USA would agree to police capabilities research within Alphabet and China would agree to police capabilities research within Tencent, and safety research would be in the public domain. We believed that this would stagnate the gameboard, but at the end of the turn it sunk in for us / Ross explained to us that NPCs (Sony, OpenAI, for instance) would still be advancing capabilities.

Where I failed:

Suddenly, China threatened to gain a decisive strategic advantage in cyber power. I brought up defecting from the Singapore Agreement, with Roland (the other person on the USA team) and Alphabet. There was a capability research opportunity on the board that, if US or Alphabet got their hands on it, would prolong the game and keep China out of a win condition. Alphabet couldn't research it without the US looking the other way. What I should have done is realized what I realized later sooner: that another team winning is preferable to everyone losing. I should have placed a much higher price on my willingness to defect.

However, in my view Alphabet is not blameless: he went and researched human-level decision making even though I only agreed to look the other way on autonomous cyber weapons. Unfortunately, by the time I realized what was happening the US had used both of its actions for the turn, I could've blurted out "US uses hard power (i.e. military) to invade Alphabet" and let the GM decide if it was legal but I didn't have the idea in the 2 minute window I had. I really just sat and pulled my hair out for 2 minutes, while everyone was appalled the the US would defect (by looking the other way) after largely spearheading the Singapore Agreement.

The consequences of failure:

China invaded silicon valley to stop Alphabet, since they had every reason to believe they would go ahead and develop AGI (then it would be game over). At this point, I think USA was wracked with guilt over having partially precipitated a coordination failure. In the end, there was international scorn against the US for allowing Alphabet to bring us so close to the brink, and in a last ditch effort to win back some decency we offered to submit unconditionally to Chinese dominion in hopes that that would coordination to hold us back from the brink. However, we had a populist president at the time and we were given a very low probability of success, the dice denied us that outcome, so we threw in all our cards with Alphabet, defended them from China while they rolled the dice on AGI. Note, while AGI is a game-ending condition, given all the safety capabilities that had been uncovered on the gameboard Alphabet thought it was reasonably likely that we would win a roll for it to not destroy everyone.

Reflection on rolling the dice with the fate of the world

I found it unnerving how easily I compromised my desire to cooperate with these slim possibilities of winning. I found myself deeply regretting the moment I told Alphabet I'd look the other way on capabilities research, instead of trying harder to find other ways of getting China out of win condition. In spite of feeling bad about (arguably) initiating defection, I even agreed to back Alphabet's bet on the fate of the world conditional on submitting to a Chinese singleton not working for political/dice reasons. Hang on on a second; did I really play russian roulette with the fate of the world? It felt wrong at the time, but I also felt pressured to help Alphabet win because it seemed better than letting China win. Moreover, this window of 10 minutes in which an ambitious diplomatic agreement totally fell apart, including 2 minutes of me pulling my hair out and failing to respond with military action, gave me an appreciation for the kinds of mistakes that may end up destroying us all. It may come down to 2 hair-pulling minutes! I never had a visceral appreciation for this before playing the game. Moreover, the game setup, where either one party can win or all parties can lose leaving no option for all parties to win, introduces a kind of pascal mugging: I can drive my preference for the world not dying arbitrarily high such that I'm indifferent to who wins. I felt myself realizing this, but thinking I shouldn't act like I realized this for the sake of a fun, competitive game. I'd love to see a variant of the game that has a global-cooperative win condition, for everyone to really come together to aim for. The preference I had early game for cooperation over ensuring USA would eventually be the victor was motivated strictly by me thinking cooperation is cool, not by an incentive in the game.

Viktor's (Alphabet) account

Note: This account was written a few days after the game with help from notes taken during the game and the chat history in the world channel. There might be details remembered wronlgy or in the slightly the wrong order.

At the very start of the game I tried to reason out how Alphabet would strategise. I figured they would try to use their strengths to accrue resources and grow as a company. After some early deliberation with Ross, the GM, early focus was on using Alphabet's main resource, their AI talent to research that would enable new capabilities and products. At the same time Alphabet invested some its existing budget to attract further AI talent.

However after this first round China was using a lot of their influence to pressure Alphabet and U.S.A.:

Remmelt Action: China is concerned about the rapid increase in AI capabilities and its threat to Chinese and world security. At the same time, we are behind on our own AI capabilities. We have started partnering with Tencent to improve our investment in safe AI. But we are behind on our overall AI capabilities. In our diplomacy, we seek US and Alphabet to share intellectual property and approaches to building AI in return for our efforts to be a responsible world player. Otherwise we will have to focus on levelling up our general AI capabilities first. (4 soft power)

The response to this influence was delayed as Alphabet focused one round on researching products to build up their budget again in part to regain what was lost in last round and in part so that U.S.A. would not get too far ahead of the other players.

In the following round the pressure from China and U.S.A. started to make a difference for the track of Alphabet. Alphabet started to focuse their research on safety aspects and marketted themselves as a responsible company and hoped in this way to further attract AI talent.

VRehnberg Action: Use soft power to market Alphabet as a responssibel company focusing a lot of its research on safety. Hoping to attract more talent.

At the same time negotiations for the Singapore Agreement started, but except limiting the options for Alphabet there was little benefits from this agreement in this early stage as other (NPC) companies continued focusing on capability enabling research.

The details of the Singapore Agreement was further fleshed out, China wanted to get some talent out of Alphabet on their and Tencents behalf while Alphabet tried to keep on to the AI talent it had. The agreement wasn't signed. Tencent and Alphabet both focused their influence to get other actors in the private sector to stop their research into capabilities.

Ben G Action: use soft power to compel tech companies to avoid new capabilities research VRehnberg Action: join with Tencent in soft power use to compel tech companies to avoid new capabilities research

Still there where other NPC companies that continued capability research. Alphabet was starting to get significantly worried about other companies developing TAI capabilities. At the same time China had developed a significantly stronger Cyber power than all other actors. In concern for this U.S.A. wanted Alphabet to research cyber capabilities while they where looking the other way as this would be against the Singapore Agreement. Alphabet saw this as a possibility to go for TAI themselves reasoning that they had the largest AI talent and had done the most AI safety research among all other actors. If someone was going to go for TAI it better be them. Alphabet had gotten U.S.A. to focus all of their talent on researching Extrapolated Coherrent Volition to further strengthen the safety considerations.

A mistake was made

Here I did the largest mistake in the game. As Alphabet I researched Agentive Human-level Decision Making openly. This should either have been done in secret or after trying to build some support for this decision from the public and/or other other actors.

Doing it in secret was something that I wanted to do, but I didn't really understand how secret actions were supposed to be done and what was and was not possible to do through them. Also I thought that doing the research would be sufficient for TAI but there was one other step to be done.

Consequences of the mistake

Remmelt Public action: take rapid targeted offensive military action against Alphabet to end AGI development (3 hard power). Also use cyber to damage Alphabet servers and block any US military response (7 cyber power)

I was quite shocked about how fast and how aggressively China responded to Alphabet's research. Especially since no other capability research from NPCs had previously gotten any actual consequences from China or U.S.A.

Things after this broke down quite quickly. China effectively took over the world and in due time went after TAI with their values as objective.

8 comments

Comments sorted by top scores.

comment by Said Achmiz (SaidAchmiz) · 2021-03-06T04:18:37.555Z · LW(p) · GW(p)

I was very excited to look into this, but without having the rules available, there’s almost no useful feedback or commentary I can give.

Is there any reason not to share the rules as they stand? Call it a “public beta version” or something.

Replies from: remmelt-ellen
comment by Remmelt Ellen (remmelt-ellen) · 2021-03-06T10:36:02.249Z · LW(p) · GW(p)

Do you mean the Game Master’s rules for world development? The basic gameplay rules for participants are outlined in the slides Ross posted above: https://docs.google.com/presentation/d/1ZKcMJZTLRp0tWixWqSW26ncDly9bJM7PMrfVMzdXihE/edit?usp=sharing

Replies from: SaidAchmiz
comment by Said Achmiz (SaidAchmiz) · 2021-03-06T20:00:51.474Z · LW(p) · GW(p)

I mean the complete rules that define the game.

comment by Gurkenglas · 2021-03-05T11:13:27.813Z · LW(p) · GW(p)

Post the rules?

Replies from: rossg, quinn-dougherty, remmelt-ellen, None
comment by rossg · 2021-03-05T18:26:33.223Z · LW(p) · GW(p)

We have a set of introductory slides we can share, but many of the rules are still under development and known only to the developers. Hope this helps!

comment by Quinn (quinn-dougherty) · 2021-03-05T11:30:34.527Z · LW(p) · GW(p)

I can't post a complete ruleset, but I can add some insight-- each party had "stats" representing hard power, soft power, budget, that sort of thing. Each turn you could spend "talent" stats on research arbitrarily, and you could take two "actions" which were GM-mediated expenditures of things like soft power, budget, etc. The game board was a list of papers and products that could be unlocked, unlocking papers released new products onto the board

comment by Remmelt Ellen (remmelt-ellen) · 2021-03-05T11:22:51.301Z · LW(p) · GW(p)

Let me ask Ross.

comment by [deleted] · 2021-03-05T18:24:40.701Z · LW(p) · GW(p)

We have a set of introductory slides we can share, but many of the rules are still under development and known only to the developers. Hope this helps!