Oh, and it's also a slight downwards update on the difficulty of press diplomacy. For example, it might just be possible that you don't need much backstabbing for expert human-level diplomacy?

In a game in which dishonesty is commonplace, it is notable that we were able to achieve human-level performance by controlling the agent's dialogue through the strategic reasoning module to be largely honest and helpful to its speaking partners.

This doesn't mean that Cicero is honest -- notably, it can "change its mind" about what action it should take, given the dialogue of other players. For example, in the video, at 26:12 we see Austria saying to Russia that they will keep Gal a DMZ, but at 27:03 Austria moves an army into Gal.

I think people who are freaking out about Cicero moreso than foundational model scaling/prompting progress are wrong; this is not much of an update on AI capabilities

I think there's a standard argument that goes "You can't just copy paste a bunch of systems that are superhuman in their respective domains and get a more general agent out."  (e.g. here's David Chapman saying something like this: https://mobile.twitter.com/Meaningness/status/1563913716969508864)

If you have that belief, I imagine this paper should update you more towards AI capabilities.  It is indeed possible to duct tape a bunch of different machine learning models together and get out something impressive.  If you didn't believe this, it should update you on the idea that AGI could come from several small new techniques duct taped together to handle each other's weakness.

I don't know anything about Diplomacy and I just watched this video, could someone expand a bit on why this game is a particularly alarming capability gain? The chat logs seemed pretty tame, the bot didn't even seem to attempt psychological manipulation or gaslighting or anything similar. What important real world capability does Diplomacy translate into that other games don't? (People for instance don't seem very alarmed nowadays about AI being vastly superhuman at chess or Go.)

Timeline considerations:

• This is not particularly unexpected if you believed in the scaling hypothesis. (It should be surprising if you continue to take seriously alternative & still-prestigious centrist paradigms like "we need a dozen paradigm shifts and it'll take until 2050".)

  The human range is narrow, so once you reach SBER and GPT-3, you already are most of the way to full-press Diplomacy. The fact that the authors & forecasters thought it would take until 2029 in 2019 (ie 10 years instead of 3 years) is part and parcel of the systematic underestimation of DL, which we have seen elsewhere in eg all the forecasts shattered by inner-monologue techniques - as Eliezer put it, experts can often be the blindest because they miss the Outside View forest for the Inside View trees, having 'slaved over a hot GPU' for so long.

  • From a scaling perspective, the main surprise of the timing is that Facebook chose to fund this research for this long.

    As Diplomacy is unimportant and a pretty niche game even among board games, there's no reason it 'had' to be solved this side of 2030 (when it presumably would become so easy that even a half-hearted effort by a grad student or hobbyist would crack it). Similarly, the main surprise in DeepMind's recent Stratego human-level AI is mostly that 'they bothered'. Keep this in mind if you want to forecast Settlers of Catan progress: it's not hard, and the real question you are forecasting is, 'will anyone bother?' (And, since the tendency of research groups is to bury their failures out back, you also can't forecast using wording like 'conditional on a major effort by a major DL group like FAIR, DM, OA etc' - you won't know if anyone does bother & fails because it's genuinely hard.)

• Deception is deceptive: one interesting aspect of the 'honesty' of the bot is that it might show how deception is an emergent property of a whole system, not just the one part. (EDIT: some Twitter discussion)

  CICERO may be constrained to be 'honest' in each interaction but it still will betray you if you trust it & move into positions where betraying you is profitable. Is it merely opportunistic, or is it analogous to humans where self-deception makes you more convincing? (You sincerely promise to not steal X, but the temptation of X eventually turns out to be too great...) It is trained end-to-end and is estimating expected value, so even if there is no 'deception module' or 'deception intent' in the planning, the fact that certain statements lead to certain long-term payoffs (via betrayal) may influence its value estimates, smuggling in manipulation & deception. Why did it pick option A instead of equally 'honest' option B? No idea. But option A down the line turns out to 'unexpectedly' yield a betrayal opportunity, which it then takes. The interplay between optimization, model-free, model-based planning, and the underlying models is a subtle one. (Blackbox optimization like evolution could also evolve this sort of de facto deception even when components are constrained to be 'honest' on a turn-by-turn basis.)

  I'm not sure of this, because piKL (and the newer variants introduced & used in CICERO) are complex (maybe some causal influence diagrams would be helpful here), but if so, it'd be interesting, and a cautionary example for interpretability & alignment research. Just like 'security' and 'reliability', honesty is a system-level property, not a part-level property, and the composition of many 'honest' components can yield deceptive actions.

• From a broader perspective, this result seems to continue to reinforce the observation "maybe humans just aren't that smart".

  Here's full-press Diplomacy, a game so hard that they don't even have a meaningful AI baseline to compare to because all prior agents were so bad, which is considered one of the pinnacles of social games, combining both a hard board game with arbitrarily complicated social dynamics mediated through unconstrained natural language; and yet. They use a very small language model, and not even that much compute for the CFR planning, in a ramshackle contraption, and... it works well? Yeah, 5 minute rounds and maybe not the very best players in the world, OK, but come on, we've seen how this story goes, and of course, the first version is always the worst, which means that given more R&D it'll become much more powerful and more efficient in the typical DL experience curve. 'Attacks only get better' / 'sampling can show the presence of knowledge but not the absence' - small LMs are already quite useful.

• Gain-of-lambda-function research: yes, this is among the worser things you could be researching, up there with the Codex code evolution & Adept Transformer agents. There are... uh, not many realistic, beneficial applications for this work. No one really needs a Diplomacy AI, and applications to things like ad auctions are tenuous. (Note the amusing wriggling of FB PR when they talk about "a strategy game which requires building trust, negotiating and cooperating with multiple players" - you left out some relevant verbs there...) And as we've seen with biological research, no matter how many times bugs escape research laboratories and literally kill people, the déformation professionnelle will cover it up and justify it. Researchers who scoff at the idea that a website should be able to set a cookie without a bunch of laws regulating it suddenly turn into don't-tread-on-me anarchocapitalists as soon as it comes to any suggestion that their research maybe shouldn't be done.

  But this is far from the most blatantly harmful research (hey, they haven't killed anyone yet, so the bar is high), so we shouldn't be too hard on them or personalize it. Let's just treat this as a good example for those who think that researchers collectively have any fire alarm and will self-regulate. (No, they won't, and someone out there is already calculating how many megawatts the Torment Nexus will generate and going "Sweet!" and putting in a proposal for a prototype 'Suffering Swirlie' research programme culminating in creating a Torment Nexus by 2029.)

Continuing the quote:

... It turns out that with some tweaks to the architecture, you can take a giant pre-trained multimodal transformer and then use it as a component in a larger system, a bureaucracy but with lots of learned neural net components instead of pure prompt programming, and then fine-tune the whole system via RL to get good at tasks in a sort of agentic way.

Worth noting that Meta did not do this: they took many small models (some with LM pretraining) and composed them in a specialized way. It's definitely faster than what Daniel said in his post, but this is also in part an update downwards on the difficulty of full press diplomacy (relative to what Daniel expected).

If we're using Daniel's post to talk about whether capabilities progress is faster or slower than expected, it's worth noting that parts of the 2022 prediction did not come true:

• GPT-3 is not "finally obselete" -- text-davinci-002, a GPT-3 variant, is still the best API model. (That being said, it is no longer SoTA compared to some private models.)
• We did not get giant multi-modal transformers.

He did get the "bureaucracy" prediction quite right; a lot of recent LM progress has been figuring out how to prompt engineer and compose LMs to elicit more capabilities out of them. 

Biggest thing that stood out to me watching this was that while the AI's tactics seemed quite good, its game theory seemed quite poor—e.g. it wasn't sufficiently vindictive if you betrayed it, which made it vulnerable to exploitation by a human aware of that fact.

Had you seen the researcher explanation for the March 2022 "AI suggested 40,000 new possible chemical weapons in just six hours" paper? I quote (paywall):

Our drug discovery company received an invitation to contribute a presentation on how AI technologies for drug discovery could potentially be misused.

Risk of misuse

The thought had never previously struck us. We were vaguely aware of security concerns around work with pathogens or toxic chemicals, but that did not relate to us; we primarily operate in a virtual setting. Our work is rooted in building machine learning models for therapeutic and toxic targets to better assist in the design of new molecules for drug discovery. We have spent decades using computers and AI to improve human health—not to degrade it. We were naive in thinking about the potential misuse of our trade, as our aim had always been to avoid molecular features that could interfere with the many different classes of proteins essential to human life. Even our projects on Ebola and neurotoxins, which could have sparked thoughts about the potential negative implications of our machine learning models, had not set our alarm bells ringing.

Our company—Collaborations Pharmaceuticals, Inc.—had recently published computational machine learning models for toxicity prediction in different areas, and, in developing our presentation to the Spiez meeting, we opted to explore how AI could be used to design toxic molecules. It was a thought exercise we had not considered before that ultimately evolved into a computational proof of concept for making biochemical weapons.

I wouldn't be surprised if there is military funding involved somewhere.

Re 3: Cicero team concedes they haven't overcome the challenge of maintaining coherency in chatting agents. They think they got away with it because 5 minutes are too short, and consider the game with longer negotiation periods will be more challenging.

My brief opinions:
1. Slightly lower probability of s-risk, approximately the same probability of x-risk.
2. Slightly lower probability of s-risk, approximtely the same probability of x-risk.
3. Prior to Cicero I thought full-press diplomacy was significantly more difficult, due to the politics aspect. Now I guess it wasn't actually significantly more difficult.
4. Not sure.
5. No.
6. No.

Cicero is designed to be honest in the sense that all its messages are generated from its intents, where its intents are what moves Cicero in fact intends to play at the moment Cicero said them (Cicero can change its mind after saying things), and at the end of the turn played moves are equal to its last intents.

Not only Cicero uses its true intents to generate messages, it also tries to generate messages that correspond to intents. That is, its dialogue model is trained to imitate humans in WebDiplomacy, but when humans intend to attack Belgium, they will sometimes say things like "I won't attack Belgium". That is, AI can lie by forming intent to attack Belgium, and devising lying intent "won't attack Belgium", and generating lying messages from lying intent. Cicero doesn't do this, its intent input to dialogue model is always truthful. AI can also lie by forming intent to attack Belgium, and generating lying messages like "I won't attack Belgium" by imitating lying humans from truthful intent. Cicero also doesn't do this! Dialogue model is trained to imitate only truthful humans, training data is filtered by the lie detector and 5% of turns are filtered.

That does not mean Cicero does not dissemble or mislead! There are three aspects to this. First, there is messaging model, entirely separate from dialogue model. Messaging model decides whether to send messages at all, trained to imitate humans. When humans intend to attack Belgium, held by France, they may not message France at all. Cicero copies this behavior.

Second, there is topic model, also entirely separate. Topic model decides which intent to talk about, trained to imitate humans. When humans intend to attack Belgium, held by France, and also Norway, held by Russia, they may talk to France about Norway and Russia about Belgium. Cicero also copies this behavior.

Third, there is filtering model, also entirely separate. When Cicero intends to attack Belgium, held by France, maybe messaging model decides to talk to France and topic model decides to talk to France about Belgium and dialogue model decides to say "I will attack Belgium". That does not mean Cicero says "I will attack Belgium", filtering model can veto it. In particular, value-based filtering model estimates how saying something will impact its own utility. Eight messages are sampled from dialogue model, their value impacts are calculated, importance is calculated from value impacts, and in 15% of most important situations, bottom three messages are dropped, and one message is picked at random.

Should Cicero's relative honesty lead us to update toward ELK being easier, or is it too task-specific to be relevant to ELK overall?

I've fine tuned LLMs on chess and it indeed is quite easy for them to learn.

I couldn't find any actual elo estimates (nor code that lets me estimate the elo of a bot), but GPT-3 (at least, davinci and text-davinci-002) can play chess decently well using PGN notation without any finetuning.

Cicero, as it is redirecting its entire fleet: 'What did you call me?'