Comment by Zaxiquej on Adversarial Policies Beat Professional-Level Go AIs · 2022-11-10T13:56:08.593Z · LW · GW

I am doubting how applying “another rule” make such research useful. Any human being who learned to play Go, using any of the major rules (Chinese Rule, Korean Rule, etc.) would claim that Katago wins with great advantage in the cases that the author thinks their AI wins. The authors claim that their AI win under the Thomp-Taylor computer, but it seems they used some weird parameters. I have tested that with some modern computer-Go winning detection algorithm, and they all show that Katago wins in the examples given by the authors. (I may try the default Thomp-Taylor as well later)

Basically, what the model does is doing nonsense and get great disadvantage, yet having some of their stones in the opponent’s territory. As Katago believe there is no need to make more play, it just passes. The authors’ AI then also passes and thus the game end. Now the authors use a somehow outdated computer-Go winning detection algorithm, which cannot correctly tell that the stones in the opponent’s territory are, in fact, dead. They falsely think that those stones are alive, thus think that the AI of the authors “win”.

After all, Katago is not designed for this specific ill-designed rule. I may claim a new rule that whoever get 5 in a row in the game wins, then my AI can definitely beat Katago as it would not prevent me doing that.

Applying adversarial attack is an interesting idea, though I am not convinced the authors’ approach did actually show its validity. This path is definitely worth exploring, but the authors’ approach seems to based on wrong assumptions and thus is hard to be improved in the future. Winning an AI from a prospective that it is not designed for is not winning. I apologize for being harsh, but as I learned Go from my childhood, I am a bit angry seeing the authors claiming some results which are not even wrong on Go. I also doubt whether any of the author play computer Go on any of the online platforms in recent years (Otherwise, why use such an outdated model?)

Updated: I used to think Thomp-Taylor is outdated, I am wrong - but it seems that Katago supports Thomp-Taylor though it is a bit old. I checked that more in detail and it seems that the authors are using weird parameters on it so it cannot correctly tell some pieces are dead or not - totally confusing. I adjusted some pieces in the paragraphs above.