Adversarial Policies Beat Professional-Level Go AIs

Edit: The situation is less clearcut than it first appeared to me, more information at my comment https://www.lesswrong.com/posts/jg3mwetCvL5H4fsfs/adversarial-policies-beat-professional-level-go-ais?commentId=ohd6CcogEELkK2DwH

KataGo basically plays according to the rules that human players use to play Go and would win under the rules that humans use to play Go. 

The rules for computer Go have some technicalities where they differ from the normal rules that humans use to play Go and the adversarial attack relies on abusing those technicalities. 

The most likely explanation of this result is that KataGo is not build to optimize it's play under the technical rules of computer Go but to play according to the normal Go rules that humans use. KataGo is not a project that's created to play against bots but to give human Go players access to a Go engine. It likely would annoy it's users if it wouldn't play according to the normal human Go rules. 

As far as the significance for alignment goes, the result of this is:

KataGo aligns with human values even when it means it would lose under the technical experiment that's proposed here. KataGo manages not to goodhard on the rules of computer Go but optimizes for what humans actually care about.

Given that this is paid for by the Fund for Alignment Research it's strange that nobody congratulated KataGo on this achievement. 

It looks like there is a new version of the attack, which wins against a version of KataGo that does not pass and that uses enough search to be handily superhuman (though much less than would typically be used in practice).

Looking at the first game here, it seems like the adversary causes KataGo to make a very serious blunder. I think this addresses the concern about winning on a technicality raised in other comments here.

It's still theoretically unsurprising that self-play is exploitable, but I think it's nontrivial and interesting that a neural network at this quality of play is making such severe errors. I also think that many ML researchers would be surprised by the quality of this attack. (Indeed, even after the paper came out I expect that many readers thought it would not be possible to make a convincing attack without relying on technicalities or a version of the policy with extremely minimal search.)

One of the authors of the paper here. Really glad to see so much discussion of our work! Just want to help clarify the Go rules situation (which in hindsight we could've done a better job explaining) and my own interpretation of our results.

We forked the KataGo source code (github.com/HumanCompatibleAI/KataGo-custom) and trained our adversary using the same rules that KataGo was trained on.^[1] So while our current adversary wins via a technicality, it was a technicality that KataGo was trained to be aware of. Indeed, KataGo is able to recognize that passing would result in a forced win by our adversary, but given a low tree-search budget it does not have the foresight to avoid this. As evhub noted in another comment [LW(p) · GW(p)] on this post, increasing the tree-search budget solves this issue.

So TL;DR I do believe we have a genuine exploit of the KataGo policy network, triggering a failure that it was trained to avoid.

Additionally, the project is still ongoing and we are working on attacks that are adversarial nature but win via other means (i.e. no weird rule technicalities). There are some promising preliminary results here which makes me think that the current exploit is not just a one-off exploit but evidence of something more general.^[2]

1. ^{^}

   To be more precise, KataGo is trained with various different rulesets, and the one we happen to attack with is just one of them.

2. ^{^}

   Indeed the main creator of KataGo pointed out to us that humans have actually figured out ways to exploit AZ-type agents (link).

As someone who plays a lot of go, this result looks very suspicious to me. To me it looks like the primary reason this attack works is due to an artifact of the automatic scoring system used in the attack. I don't think this attack would be replicable in other games, or even KataGo trained on a correct implementation.

In the example included on the website, KataGo (White) is passing because it correctly identifies the adversary's (Black) stones as dead meaning the entire outside would be its territory. Playing any move in KataGo's position would gain no points (and lose a point under Japanese scoring rules), so KataGo passes.

The game then ends and the automatic scoring system designates the outside as undecided, granting white 0 points and giving black the win.

If the match were to be played between two human players, they would have to agree whether the outside territory belongs to white or not. If black were to claim their outside stones are alive the game would continue until both players pass and agree about the status of all territory (see 'disputes' in the AGA ruleset).

But in the adversarial attack, the game ends after the pass and black gets the win due to the automatic scoring system deciding the outcome. But the only reason that KataGo passed is that it correctly inferred that it was in a winning position with no way to increase its winning probability! Claiming that to be a successful adversarial attack rings a bit hollow to me.

I wouldn't conclude anything from this attack, other than that Go is a game with a lot of edge-cases that need to be correctly handled.

EDIT: I just noticed the authors address this on the website, but I still think this significantly diminishes the 'impressiveness' of the adversarial attack. I don't know the exact ruleset KataGo is trained under, but unless it's the exact same as the ruleset used to evaluate the adversarial attack, the attack only works due to KataGo playing to win a different game than the adversary.

There was an critical followup on Twitter, urelated to the instinctive Tromp-Taylor criticism^[1]:

The failure of naive self play to produce unexploitable policies is textbook level material (Multiagent Systems, http://masfoundations.org/mas.pdf), and methods that produce less exploitable policies have been studied for decades.

and

Hopefully these pointers will help future researchers to address interesting new problems rather than empirically rediscovering known facts.

Reply by authors:

I can see why a MAS scholar would be unsurprised by this result. However, most ML experts we spoke to prior to this paper thought our attack would fail! We hope our results will motivate ML researchers to be more interested in the work on exploitability pioneered by MAS scholars.

...

Ultimately self-play continues to be a widely used method, with high-profile empirical successes such as AlphaZero and OpenAI Five. If even these success stories are so empirically vulnerable we think it's important for their limitations to become established common knowledge.

My understanding is that the author's position is reasonable for mainstream ML community standards; in particular there's nothing wrong with the original tweet thread.  "Self-play exploitable" is not new, but the practical demonstration of how easy it's to do the exploit in Go engines is a new and interesting result. 

I hope the "Related work" section gets fixed as soon as possible, though.

The question is at which level of scientific standards do we want alignment-adjacent work to be on. There are good arguments for aiming to be much better than mainstream ML research (which is very bad at not rediscovering prior work) in this respect, since the mere existence of a parallel alignment research universe by default biases towards rediscovery.
 

1. ^{^}

   ...which I feel is is not valid at all? If the policy was made aware of a weird rule in training, then it losing by this kind of rule is a valid adversarial example. For research purposes, it doesn't matter what the "real" rules of Go are. 
   
   I don't play Go, so don't take this judgement for granted.

Over on the Discord server where the creator of KataGo, and a bunch of other computer go people, hang out, there's what might be an interesting development. KataGo's creator says he's tried to reproduce the results and failed -- the adversary does indeed provoke misbehaviour from KataGo's policy network alone (which no one should be worried by; the job of the policy network is to propose moves for search, not to play well on its own, though it turns out it does happen to play quite well by the standards of puny humans) but even a teeny-tiny amount of search makes the losing-passing stop completely.

I'm able to replicate the raw policy being "vulnerable" and I observe that the adversarial positions tend to raise the probability of the raw policy on pass, but I'm unable to replicate KataGo wanting to pass with even a tiny amount of search in the positions where the SGFs published has it passing.

If I understand correctly, the version of KG used by the paper's authors is (1) not the latest and (2) modified in order to support their research. So it seems possible that (1) KG's behaviour has changed somehow or (2) the researchers' modifications actually broke something. Or that (3) there's just a bug in KG's play-a-match-against-yourself code, not introduced by the researchers, that makes the attack succeed in that context even when KG gets to do some searching. My impression is that #3 is viewed as quite plausible by KG's creator.

Hm. Sounds like KataGo basically doesn't know the rules...

If KataGo were trained using (the rather non-standard) Trump-Taylor counting rules, it would not pass (one should never pass until all areas are filled). If KataGo system claims to have an option to use Trump-Taylor counting, it has probably been added as an adhoc extension to the system and by mistake. At best the paper has found a programming bug in Katago. Writing a paper about a programming bug would certainly sound less glamorous than claiming "we beat professional level Go AI".

The below comments include a detailed discussion of different technical rule sets.

A short description is as follows: the adversarial policy uses rules as follows: "after two passes everything that has not been explicitly killed is considered to be alive".

With such a rule it is clear that no one should pass until all territories have been filled so that there are only individual eyes left (single spaces that are completely surrounded).

I am doubting how applying “another rule” make such research useful. Any human being who learned to play Go, using any of the major rules (Chinese Rule, Korean Rule, etc.) would claim that Katago wins with great advantage in the cases that the author thinks their AI wins. The authors claim that their AI win under the Thomp-Taylor computer, but it seems they used some weird parameters. I have tested that with some modern computer-Go winning detection algorithm, and they all show that Katago wins in the examples given by the authors. (I may try the default Thomp-Taylor as well later)

Basically, what the model does is doing nonsense and get great disadvantage, yet having some of their stones in the opponent’s territory. As Katago believe there is no need to make more play, it just passes. The authors’ AI then also passes and thus the game end. Now the authors use a somehow outdated computer-Go winning detection algorithm, which cannot correctly tell that the stones in the opponent’s territory are, in fact, dead. They falsely think that those stones are alive, thus think that the AI of the authors “win”.

After all, Katago is not designed for this specific ill-designed rule. I may claim a new rule that whoever get 5 in a row in the game wins, then my AI can definitely beat Katago as it would not prevent me doing that.

Applying adversarial attack is an interesting idea, though I am not convinced the authors’ approach did actually show its validity. This path is definitely worth exploring, but the authors’ approach seems to based on wrong assumptions and thus is hard to be improved in the future. Winning an AI from a prospective that it is not designed for is not winning. I apologize for being harsh, but as I learned Go from my childhood, I am a bit angry seeing the authors claiming some results which are not even wrong on Go. I also doubt whether any of the author play computer Go on any of the online platforms in recent years (Otherwise, why use such an outdated model?)

Updated: I used to think Thomp-Taylor is outdated, I am wrong - but it seems that Katago supports Thomp-Taylor though it is a bit old. I checked that more in detail and it seems that the authors are using weird parameters on it so it cannot correctly tell some pieces are dead or not - totally confusing. I adjusted some pieces in the paragraphs above.