When do "brains beat brawn" in Chess? An experiment

You can easily get a draw against any AI in the world at Tic-Tac-Toe. In fact, provided the game actually stays confined to the actions on the board, you can draw AIXI at Tic-Tac-Toe. That's because Tic-Tac-Toe is a very small game with very few states and very few possible actions, and so intelligence, the ability to pick good actions, doesn't grant any further advantage in it past a certain pretty low threshold. 

Chess has more actions and more states, so intelligence matters more. But probably still not all that much compared to the vastness of the state and action space the physical universe has. If there's some intelligence threshold past which minds pretty much always draw against each other in chess even if there is a giant intelligence gap between them, I wouldn't be that surprised. Though I don't have much knowledge of the game.

In the game of Real Life, I very much expect that "human level" is more the equivalent of a four year old kid who is currently playing their third ever game of chess, and still keeps forgetting half the rules every minute. The state and action space is vast, and we get to observe humans navigating it poorly on a daily basis. Though usually only with the benefit of hindsight. In many domains, vast resource mismatches between humans do not outweigh skill gaps between humans. The Chinese government has far more money than OpenAI, but cannot currently beat OpenAI at making powerful language models. All the usual comparisons between humans and other animals also apply. This vast difference in achieved outcomes from small intelligence gaps even in the face of large resource gaps does not seem to me to be indicative of us being anywhere close to the intelligence saturation threshold of the Real Life game.

This is something lc and gwern discussed in the comments here, but now we have clear evidence this is only true for Nash solvers (all typical engines like SF, Lc0, etc.). LeelaQueenOdds, which trained exploitatively against a model of top human players (FM+), is around 2k to 2.9k lichess elo depending on the time controls, so it completely trounces 1.6k elo players (especially 1.2k elo players as another commenter has suggested the author actually is). See: https://marcogio9.github.io/LeelaQueenOdds-Leaderboard/

Nash solvers are far too conservative and expect perfect play out of their opponents, hence give up most meaningful attacking chances in odds games. Exploitative models like LQO instead assume their opponents play like strong humans (good but imperfect) and do extremely well, despite a completely crushing material disadvantage. As some have noted, this is possible even with chess being a super sterile/simple environment relative to real life.

I speculate that the experiment from this post only yielded the results it did because Nash is a poor solution concept when one side is hopelessly disadvantaged under optimal play from both sides, and queen odds fall deep into that category.

See video from 7 minutes. Try it yourself https://lichess.org/@/LeelaQueenOdds :)

I've been having various conversations in private, where I'm quite doomist and my interlocutor is less doomist, and I think one of the key cruxes that has come up several times is that I've applied security mindset to the operation of human governance, and I am not impressed.

I looked at things like the federal reserve (and how you'd implement that in a smart contract) and the congress/president/court deal (and how you'd implement that in a smart contract) and various other systems, and the thing I found was that existing governance systems are very poorly designed and probably relatively easy to knock over.

As near as I can tell, the reason human civilization still exists is that no inhuman opponent has ever existed that might really just want to push human civilization over and then curb stomp us while we thrash around in surprised pain.

For example, in WW2 Operation Bernhard got close to just "ending the money game" explicitly, but the bad guys couldn't bring themselves to make the stupidest and most evil British people rich via relatively secret injections, and then ramp it up more and more, and then as the whole web of market relationships became less and less plausible they could have eventually pumped more and more fake money into the British economy until they were just raining cash on random cities from the air. Part of why they refrained is that they imagined what a retaliatory strike would do to Nazi-controlled Germany... and flinched. The logic of "MAD" (transposed into economics) held them back.

In my view, human civilization is rife with such weaknesses.

I tend to not talk about them much, but I feel like maybe I should talk about them more, because my silence is coming to feel more and more stupid, given that I expect an AI smarter than me to see all the options that I can see, and then more!

So a big part of my model is not that a very very fast nor a very very large "foom" is especially likely... it is just that we are so collectively stupid (as a collective herd of moderately smart monkeys that is not actually very organized, and is mostly held together with prison bars and nuclear blackmail and duct tape) that I expect AIs that are merely slightly smarter than us (and have wildly different weaknesses and a coherent plan to get rid of us), to have a very good chance of succeeding.

Bringing it back to the Chess metaphor...

One can easily imagine starting with Stockfish, and then writing an extra little algorithmic loop that:
1) looks at the board, and 
2) throttles Stockfish's CPU so that it can only "run at full speed" when
3) it has the knights, bishops, queen, and king inside a 4x4 subarea.

The farther from that ideal the board position gets, the more the CPU would be throttled.

A stockfish with this hack to its CPU throttle would start the game already at a "governance disadvantage" because the knights start outside "the zone of effective governance".

Correct early moves for white (taking into account the boost of extra CPU) might very well involve pushing the queen's knight to c3 or d2, and would similarly involve pushing the king's knight to e2 or f3. Only then could Stockfish even operate at "mental full power"!

If you ran AlphaZero from scratch, to expert, using any similar sort of "cpu sabotage based on the board state" I bet it would invent some CRAZY new opening games!

And this "weird board/cpu entangled chess AI constraint"... is actually a pretty good metaphor for how humans make decisions, as a collective, in real life.

Every private CEO is filtered not just by how good of a manager and economic planner they are, but how willing they are to be loyal to the true owners of the company, even at the expense of customers, or country, or moral ideals. Every President is beholden to half-crazy partisan factions, and can't possibly get into power without making numerous entangling deals. Xi and Putin can make moves, but they are each just one guy, and they have to spend a lot of their brain power just preventing coups and assassinations because their advisors aren't that trustworthy. Most governments are simply not arranged to maximize the government's ability to fight something very very very smart from a standing start!

If the human "material in the world" is re-arranged, then the human "ability as a species to coordinate against something that coherently would prefer our extinction" is also going to be affected. And the AI will know this. And the AI will be able to attack it.

If you knew you were playing against a version of Stockfish that got more CPU based on the proximity of various pieces (and less CPU with less material that was more spread out) then I bet that would ALSO be worth a huge amount of material.

That advantage, that you would have against a Stockfish with that additional imaginary weakness, is very very similar to the advantage that AI has over human governments: the "brains" are right there on the table, as part of the stakes, and subject to attack!

If you're smarter than your opponent but have less starting resources, the optimal strategy probably involves some combination of cooperation, making alliances, deception, escaping / running / hiding, gathering resources in secret, and whatever other prerequisites are needed to neutralize such a resource imbalance. Many scenarios in which a smarter-than-human AGI with less resources goes to war with or is attacked by humanity are thus somewhat contradictory or at least implausible: they postulate the AGI taking a less good strategy than what a literal human in its place could come up with.

There's not really an analogue for this to Chess - if I am forced to play a chess game with a grandmaster with whatever handicap, I could maybe flip over the board if I started to lose. But that probably just counts as a forfeit, unless I can also overpower or coerce my opponent and / or the judges.

if a rogue AI is caught early on it’s plot, with all the worlds militaries combined against them while they still have to rely on humans for electricity and physical computing servers. It’s somewhat hard to outthink a missile headed for your server farm at 800 km/h.

Breaking it down by cases:

• If deception / misalignment is caught early enough (i.e. in the lab, by inspecting the system statically before it is given a chance to execute on its own), then you don't need any military, you just turn the system off or don't run it in the first place.
• If the deception / misalignment is not detected until the AI is already "loose in the world", such that it would take all the world's militaries (or any military, really) to stop, it's already too late. The AI has already had an opportunity to hide / copy itself, hide its true intentions, make alliances with powerful and sympathetic (or naive) humans, etc. the way any smart human actor would do.
• If the detection happens somewhere in between these points, there might be a situation in which "brawn" is relevant. This seems like a really narrow time frame / band of possibilities to me though, and when people try to postulate what scenarios in this band might look like, they often rely on a smarter-than-human system making mistakes a literal human would know to avoid. Not saying it's not possible to come up with more realistic scenarios, but "is a supposedly smarter-than-human AI making mistakes or missing strategies that I myself would not miss" is a good basic sanity check on whether your scenario is plausible or not.

This kind of experiment has been at the top of my list of "alignment research experiments I wish someone would run". I think the chess environment is one of the least interesting environments (compared to e.g. Go or Starcraft), but it does seem like a good place to start. Thank you so much for doing these experiments!

I do also think Gwern's concern about chess engines not really being trained on games with material advantage is an issue here. I expect a proper study of this kind of problem to involve at least finetuning engines.

↑ comment by Archimedes · 2024-11-22T04:43:13.525Z · LW(p) · GW(p)

@gwern [LW · GW]  and @lc [LW · GW]  are right. Stockfish is terrible at odds and this post could really use some follow-up.

As @simplegeometry [LW · GW]  points out in the comments, we now have much stronger odds-playing engines that regularly win against much stronger players than OP.

https://lichess.org/@/LeelaQueenOdds

https://marcogio9.github.io/LeelaQueenOdds-Leaderboard/

Replies from: habryka4

↑ comment by habryka (habryka4) · 2024-11-22T04:47:57.888Z · LW(p) · GW(p)

That's really cool! Do you have any sense of what kind of material advantage these odd-playing engines could use against the best humans?

Replies from: lc, simplegeometry, Archimedes

↑ comment by lc · 2024-11-22T05:06:56.591Z · LW(p) · GW(p)

As the name suggests, Leela Queen Odds is trained specifically to play without a queen, which is of course an absolutely bonkers disadvantage against 2k+ elo players. One interesting wrinkle is the time constraint. AIs are better at fast chess (obviously), and apparently no one who's tried is yet able to beat it consistently at 3+0 (3 minutes with no timing increment)

↑ comment by simplegeometry · 2024-11-22T05:47:32.429Z · LW(p) · GW(p)

At rapid time controls, it seems like we could maybe go even against Magnus with knight odds? If not Magnus, perhaps other high-rated GMs.

There was a match with the most recently updated LeelaKnightOdds and GM Alex Lenderman but I don't recall the score exactly. EDIT: which was 19-3-2 win draw loss.

Replies from: andrei-alexandru-parfeni

↑ comment by sunwillrise (andrei-alexandru-parfeni) · 2024-11-22T05:51:18.350Z · LW(p) · GW(p)

rapid time controls

I am very skeptical of this on priors, for the record. I think this statement could be true for superblitz time controls and whatnot, but I would be shocked if knight odds would be enough to beat Magnus in a 10+0 or 15+0 game. That being said, I have no inside knowledge, and I would update a lot of my beliefs significantly if your statement as currently written actually ends up being true.

Replies from: Veedrac, simplegeometry

↑ comment by Veedrac · 2024-12-17T11:45:32.396Z · LW(p) · GW(p)

LeelaKnightOdds has convincing beaten both Awonder Liang and Anish Giri at 3+2 by large margins, and has an extremely strong record at 5+3 against people who have challenged it.

I think 15+0 and probably also 10+0 would be a relatively easy win for Magnus based on Awonder, a ~150 elo weaker player, taking two draws at 8+3 and a win and a draw at 10+5. At 5+3 I'm not sure because we have so little data at winnable time controls, but wouldn't expect an easy win for either player.

It's also certainly not the case that these few-months-old networks running a somewhat improper algorithm are the best we could build—it's known at minimum that this Leela is tactically weaker than normal and can drop endgame wins, even if humans rarely capitalize on that.

↑ comment by simplegeometry · 2024-11-22T05:58:10.676Z · LW(p) · GW(p)

Hissha from the Lc0 server reports 19 wins, 3 draws, and 2 losses against Lenderman (currently ~2500 FIDE) at 15+10 from a knight odds match 2 months ago -- with the caveat that Lenderman started playing too fast after 10 games. I haven't run the numbers but suspect this would be enough to go even against a 2750, if not Magnus?

I was surprised too. I think it's an exciting development :)

Replies from: andrei-alexandru-parfeni

↑ comment by sunwillrise (andrei-alexandru-parfeni) · 2024-11-22T06:02:33.060Z · LW(p) · GW(p)

Hmm, that sounds about right based on the usual human-vs-human transfer from Elo difference to performance... but I am still not sure if that holds up when you have odds games, which feel qualitatively different to me than regular games. Based on my current chess intuition, I would expect the ability to win odds games to scale better than ELO near the top level, but I could be wrong about this.

↑ comment by Archimedes · 2024-11-22T23:29:13.945Z · LW(p) · GW(p)

Knight odds is pretty challenging even for grandmasters.

Some nitpicks:

• You write like Stockfish 14 is a probabilistic function from game-state to next-move, the thing-which-has-an-ELO. But I think Stockfish 14 running on X hardware for Y time is the real probabilistic function from game-state to next-move (see e.g. the inclusion of hardware in ELO ranking here). And you probably played with hardware and time such that its ELO is substantially below 3549.
• I think a human with Stockfish's ELO would be much better at beating you down odds of a queen, since (not certain about these):
  • Stockfish is optimized for standard chess and human grandmasters are probably better at transferring to odds-chess.
  • Stockfish roughly tries to maximize P(win) against optimal play or Stockfish-level play, or maximize number of moves before losing once it knows you have a winning strategy. Human grandmasters would adapt to be better against your skill level (e.g. by trying to make positions more complex), and would sometimes correctly make choices that would be bad against Stockfish or optimal play but good against weaker players.

While I think your overall point is very reasonable, I don't think your experiments provide much evidence for it. Stockfish generally is trained to play the best move assuming its opponent is playing best moves itself. This is a good strategy when both sides start with the same amount of pieces, but falls apart when you do odds games. 

Generally the strategy to win against a weaker opponent in odds games is to conserve material, complicate the position, and play for tricks - go for moves which may not be amazing objectively but end up winning material against a less perceptive opponent. While Stockfish is not great at this, top human chess players can be very good at it. For example, a top grandmaster Hikaru Nakamura had a "Botez Gambit Speedrun" (https://www.youtube.com/playlist?list=PL4KCWZ5Ti2H7HT0p1hXlnr9OPxi1FjyC0), where he sacrificed his queen every game and was able to get to 2500 on chess.com, the level of many chess masters. 

This isn't quite the same as your queen odds setup (it is easier), and the short time format he is on is a factor, but I assume he would be able to beat most sub-1500 FIDE players with queen odds. A version of Stockfish trained to exploit a human's subpar ability would presumably do even better.

The post studies handicapped chess as a domain to study how player capability and starting position affect win probabilities. From the conclusion:

In the view of Miles and others, the initially gargantuan resource imbalance between the AI and humanity doesn’t matter, because the AGI is so super-duper smart, it will be able to come up with the “perfect” plan to overcome any resource imbalance, like a GM playing against a little kid that doesn't understand the rules very well. 

The problem with this argument is that you can use the exact same reasoning to imply that’s it’s “obvious” that Stockfish could reliably beat me with queen odds. But we know now that that’s not true.

Since this post came out, a chess bot (LeelaQueenOdds) that has been designed to play with fewer pieces has come out. simplegeometry's comment [LW(p) · GW(p)] introduces it well. With queen odds, LQO is way better than Stockfish, which has not been designed for it. Consequentially, the main empirical result of the post is severely undermined. (I wonder how far even LQO is from truly optimal play against humans.)

(This is in addition to - as is pointed out by many commenters - how the whole analogue is stretched at best, given the many critical ways in which chess is different from reality. The post has little argument in favor of the validity of the analogue.)

I don't think the post has stood the test of time, and vote against including it in the 2023 Review.

I'm surprised by how much this post is getting upvoted. It gives us essentially zero information about any question of importance, for reasons that have already been properly explained by other commenters:

• Chess is not like the real world in important respects. What the threshold is for material advantage such that a 1200 elo player could beat Stockfish at chess tells us basically nothing about what the threshold is for humans, either individually or collectively, to beat an AGI in some real-world confrontation. This point is so trivial that I feel somewhat embarrassed to be making it, but I have to think that people are just not getting the message here.

• Even focusing only on chess, the argument here is remarkably weak because Stockfish is not a system trained to beat weaker opponents with piece odds. There are Go AIs that have been trained for this kind of thing, e.g. KataGo can play reasonably well in positions with a handicap if you tell it that its opponent is much weaker than itself. In my experience, KataGo running on consumer hardware can give the best players in the world 3-4 stones and have an even game.

If someone could try to convince me that this experiment was not pointless and actually worth running for some reason, I would be interested to hear their arguments. Note that I'm more sympathetic to "this kind of experiment could be valuable if ran in the right environment", and my skepticism is specifically about running it for chess.

Thanks for the post! It was a good read. One point I don't think was brought up is the fact that chess is turn-based whereas real life is continuous.

Consequently, the huge speed advantage that AIs have is not that useful in chess because the AI still has to wait for you to make a move before it can move.

But since real life is continuous, if the AI is much faster than you, it could make 1000 'moves' for every move you make and therefore speed is a much bigger advantage in real life.

I'm not familiar with how Stockfish is trained, but does it have intentional training for how to play with queen odds? If not, then it might be able to start trouncing you if it were trained to play with it, instead of having to "figure out" new strategies uniquely. 

(My current fide rating is ~1500 elo (~37 percentile) and my peak rating was ~1700 elo (~56 percentile)).

While I'm not that good at chess myself, I think you got some things wrong, and on some I'm just being nitpicky.

My rating on lichess blitz is 1200, on rapid is 1600, which some calculator online said would place me at ~1100 ELO on the FIDE scale.

I’m quite skeptical of such conversions, but I understand you had nothing better to go on. This website (made from surveying a bunch of redditors ^[1]) converts your lichess blitz rating into 1005, 869 ^[2], 828 elo for fide standard, rapid and blitz respectively. Your rapid lichess rating would indicate 1210, 1125, 1194. Make of that, what you will.

In comparison, a chess master is ~2200, a grandmaster is ~2700.

People above 2700 are customarily considered super GM’s. I wasn’t able to download the latest rating list of the Fide website, but according to the standard rating list from september 2022 the average GM fide rating ^[3] is 2498.

I also have some squabbles with the way you wrote about piece relative value, but I understand you’re just oversimplifying for a layman audience. 

1. ^{^}

   I wasn’t able to find the original post.

2. ^{^}

   The minimum fide rating is a 1000, so anything below that is Fide unrated.

3. ^{^}

   Only considering players that played at least a single rated game in the last 12 months, otherwise their fide rating becomes inactive.

Although realistically, the real odds would be less about the ELO and more on whether he was drunk while playing me.

Related

Curated. The question beneath feels really quite interesting. As the OP have said, even if it's the case that a vastly superhuman intelligent AI could defeat even at extreme disadvantage, this doesn't mean there isn't some advantage that would let humans defeat a more nascently powerful AGI, and it's pretty interesting to understand the how that works out. I'm excited to see more work on this, especially in domains resembling more and more real life* (e.g. Habryka suggests Starcraft).

*Something about chess is it feels quite "tight" in terms of not admitting exploits or hacks the way I could imagine other games have hidden exploitable bugs that can be mined – like reality.

I intend to write a lot more on the potential “brains vs brawns” matchup of humans vs AGI. It’s a topic that has received surprisingly little depth from AI theorists.

I recommend checking out part 2 of Carl Shulman's Lunar Society podcast [? · GW] for content on how AGI could gather power and take over in practice.

Leela now has a contempt implementation that makes odds games much more interesting. See this Lc0 blog post (and the prior two) for more details on how it works and how to easily play odds games against Leela on Lichess using this feature.

GM Matthew Sadler also has some recent videos about using WDL contempt to find new opening ideas to maximize chances of winning versus a much weaker opponent.

I'd bet money you can't beat LeelaQueenOdds at anything close to a 90% win rate.

On the other hand, the potential resource imbalance could be ridiculously high, particularly if a rogue AI is caught early on it’s plot, with all the worlds militaries combined against them while they still have to rely on humans for electricity and physical computing servers. It’s somewhat hard to outthink a missile headed for your server farm at 800 km/h. ... I hope this little experiment at least explains why I don’t think the victory of brain over brawn is “obvious”. Intelligence counts for a lot, but it ain’t everything.

While this is a true and important thing to realise, I don't think of it as the kind of information that does much to comfort me with regards to AI risk. Yes, if we catch a misaligned AI sufficicently early enough, such that it is below whatever threshold of combined intelligence and resources that is needed to kill us, then there is a good chance we will choose to prevent it from doing so. But this is something that could happen thousands of times and it would still feel rather besides the point, because it only takes one situation where one isn't below that threshold and therefore does still kill us all. 

If we can identify even roughly where various thresholds are, and find some equivalent of leaving the AI with a king and three pawns where we have a ~100% chance of stopping it, then sure, that information could be useful and perhaps we could coordinate around ensuring that no AI that would kill us all should it get more material from indeed ever getting more than that. But even after clearing the technical challenge of finding such thresholds with much certainty in such a complex world, the coordination challenge of actually getting everyone to stick to them despite incentives to make more useful AI by giving it more capability and resources, would still remain.

Still worthwhile research to do of course, even if it ends up being the kind of thing that only buys some time.

I think this is a great article, and the thesis is true.

The question is, how much intelligence is worth how much material?

Humans are so very slow and stupid compared to what is possible, and the world so complex and capable of surprising behaviour, that my intuition is that even a very modest intelligence advantage would be enough to win from almost any starting position. 

You can bet your arse that any AI worthy of the name will act nice until it's already in a winning position.

I would.

If you're open to more experimentation, I'd recommend trying playing against Leela Chess Zero using some of the newer contempt parameters introduced in this PR and available in the latest pre-release version. I'm really curious if you'd notice significant style differences with different contempt settings.

Update: The official v0.30.0 release is out now and there is a blog post detailing the contempt settings. Additionally, there is a Lichess bot set up specifically for knight odds games.

Further update: There are now three Lichess bots set up to play odds games on Lichess: LeelaKightOdds, LeelaQueenForKnight, and LeelaQueenOdds. They are currently featured bots on https://lichess.org/player/bots

Probably not relevant to any arguments about AI doom, but some notes about chess material values:

You said a rook is "ostensibly only 1 point of material less than two bishops".  This is true in the simplified system usually taught to new players (where pawn = 1, knight = bishop = 3, rook = 5, queen = 9).  But in models that allow themselves a higher complexity budget, 2 bishops can be closer to a queen than a rook (at the start of the game):

• Bishops are usually considered slightly better than knights; a value of 3 + 1/3 is typical
• There is a "pair bonus" of ~1/2 point for having 2 bishops on opposite colors.  (Bishops are a "color-bound" piece: a bishop that starts on a dark square can only reach other dark squares, and vice-versa.  Having 2 on opposite colors mitigates this disadvantage because an opportunity that is on the wrong color for one bishop will be exploitable by the other; the "Jack Sprat" effect.)
• Rooks are weaker in crowded boards (early game) where their movement options are often blocked, and stronger in open boards (endgames).  5 is an average across the whole game.  I've seen estimates <4.5 for early-game and >6 for endgame.
• (Queen is also often a bit higher than 9, especially for AI players; e.g. 9.25 or 9.5)

If you're interested in a deeper analysis of material values, I recommend these articles by Ralph Betza.  Betza is both an international master chess player and a prolific designer of chess variants, so he's interested in models that work outside the distribution of standard chess.

A related thought: an intelligence can only work on the information that it has, regardless of its veracity, and it can only work on information that actually exists.

My hunch is that the plan of "AI boostraps itself to superintelligence, then superpower, then wipes out humanity" relies on it having access to information that is too well hidden to divine through sheer calculation and infogathering, regardless of its intelligence (ex: the location of all the military bunkers, and nuclear submarines humanity has), or simply does not exist (ex: future Human strategic choices based on coin-flips).

Most AI Apocalypse scenarios depend not only on the AI being superhumanly smart, but being inexplicably Omniscient about things that nobody could be plausibly Omniscient about. 

This might actually be a case where a chess GM would outperform an AI: they can think psychologically, so they can deliberately pick traps and positions that they know I would have difficulty with.

Emphasis needed. I expect a GM to beat you down a rook every time, and down a queen most times.

Stockfish assumes you will make optimal moves in planning and so plays defensive when down pieces, but an AI optimized to trick humans (i.e. allowing suboptimal play when humans are likely to make a mistake) would do far better. You could probably build this with maiachess, I recall seeing someone build something like this though I can't find the link right now.

Put another way, all the experiments you do are making a significant type error, Stockfish down a rook against a worse opponent does not play remotely like a GM down a rook. I would lose to a GM every time, I would beat Stockfish most times.

I think the assumptions that.

1. Humans realize the AI exists early on.
2. Humans are reasonably coordinated and working against the AI.

Are both dubious. 

What is stopping someone sending a missile at GPT-4's servers right now. 

1. OpenAI hasn't anounced a list of coordinated for where those servers are (as far as I know) This is because
2. OpenAI doesn't want you to missile strike their servers because
3. OpenAI thinks their AI is safe and useful not dangerous.

I think seeing large numbers of humans working in a coordinated fashion against an AI is unlikely. 

If a rogue AI is discovered early, we could end up in a war where the AGI has a huge intelligence advantage, but humans have a huge resource advantage.

In that scenario, it seems to me that enough abstractions break down that the analogy to the Stockfish experiment no longer works. Like talking about a conflict of AGI vs. "humans" as two agents in a 2-player game, rather than AGI vs. a collection of exploitable agents.

But I want to focus on the "resource" abstraction here. First of all, "ownership" of resources seems irrelevant; that's mostly a legal concept, and seems like an irrelevant fiction in this scenario. What matters more in a conflict is possession of resources, i.e. who can actually control and command resources.

And here things get tricky. Maybe humans ostensibly possess nukes, but it's not clear whether they would in practice be able to employ them against the AI (what would you even target?), or rather see them employed against them via hacks or social engineering. Humans certainly possess a lot of computers, but can quickly lose possession over a significant chunk of them when they're turned into botnets. And so on.

Overall, in terms of security mindset, it does not make too much sense to me to think in terms of a conflict of Team AGI vs Team Humanity, where one side has an enormous resource advantage.

Enjoyed this post, thanks. Not sure how well chess handicapping translates to handicapping future AGI, but it is an interesting perspective to at least consider.

Thank you for doing the experiment. Someone could run a similar set of tests for Go.

Just to prime your thinking: what's war winning for most wars on earth?

Probably whoever can use the majority of physical resources and turn them into weapons. We had several rounds of wars and the winner had a vast material advantage.

It occurred to me that the level of AI capabilities needed to reach exponential growing levels of resources is essentially a general robot system, trained on all videos in existence of humans taking actions in the real world and a lot of reinforcement learning tasks.

The general robot AGI isn't designed to be particularly smart, only thinks of the short term (myopia in cognitive architecture), can't learn as an individual, and can do nearly all mining, logistics, and manufacturing tasks a human can do if given a detailed prompt on the goal of a task. This is done through separate instances given separate prompts and they can only communicate with peers in the network they have a physical connection to.

Basically the idea is that humans could gain an insurmountable advantage built on (exponential) numbers of AGI limited to the point humans can remain in full control.

Anecdotally, I remember seeing analyses of Stockfish v. Alpha Zero (I think) where AlphaZero would fairly consistently trade absurd amounts of materiel for position. While there is obviously still a tipping point at which a materiel advantage will massively swing the odds I feel that the thrust of this essay kind-of understates the value of a lot-a lot of intelligence in light of those matches.

With that said, I haven't seen any odds-games with AlphaZero, so perhaps my point is entirely moot and it does need that initial materiel as badly as Stockfish.

The LessWrong Review [? · GW] runs every year to select the posts that have most stood the test of time. This post is not yet eligible for review, but will be at the end of 2024. The top fifty or so posts are featured prominently on the site throughout the year.

Hopefully, the review is better than karma at judging enduring value. If we have accurate prediction markets on the review results, maybe we can have better incentives on LessWrong today. Will this post make the top fifty?

I suspect this is a lack of flexibility in Stockfish.  It was designed (trained?) for normal equal-forces chess and can't step back to think "How do I best work around this disadvantage I've been given?"  I suspect something like AlphaZero, given time to play itself at a disadvantage, would do better.  As would a true AGI.

I have a habit of reading footnotes as soon as they are linked, and your footnote says that you won with queen odds before the call to guess what odds you'd win at, creating a minor spoiler.

I think this is a really useful and thought provoking experiment. One thing that worries me, is that large corporations may find it easier and faster to give the AI brawn than brains. Why play fair when in competition when you have a money and machine advantages? I think this will be especially so with not so good AIs, and the advantages will remain after the brains part improves. So in your analogy, what about giving stockfish 3 extra queens? A second question is how does it do against stockfish with just 2 extra queens?

It's maybe worth noting that Stockfish 14 NNUE still has some failure modes. Take this position for example: positionOnLichess. The position is a complete draw, as Black can't make any progress, and White cannot lose as long as he only moves his king. Despite this, Stockfish 14 NNUE evaluates it as a -15 advantage for Black, which should typically indicate a decisive advantage. Even a human player with relatively low Elo should be able to quickly assess this position as a draw.

Thanks for the insights. Actually, board game models don't play very well when they are so heavily loosing, or so heavily winning that it doesn't seem to matter. A human player would try to trick you and hope for a mistake. This is not necessarily the case with these models that play as if you were as good as them, which makes their situation look unwinnable. 

It's quite the same with AlphaGo. AlphaGo plays incredibly well until there is a large imbalance. Surprisingly, AlphaGo also doesn't care about winning by 10 points or by half a point, and sometimes plays moves that look bad to humans just because it's winning anyway. And when it's loosing, since it assumes that its opponent is as strong, it can't find a leaf in the tree search that end up winning. Moreover, I suspect that removing a piece is prone to distribution shift.

A somewhat related point: it's only very recently (2023) that chess engines have begun competently mimicking the error patterns of human play. The nerfings of previous decades were all artificial.

I'm an FM and play casual games vs. the various nerfed engines at chess.com. The games are very fast (they move instantly) but there's no possibility of time loss. Not the best way to practice openings but good enough.

The implication for AI / AGI is that humans will never create human-similar AI. Everything we make will be way ahead in many areas and way behind in others, and figuring out how to balance everything to construct human-similar is far in the future. Unless we get AIs to help...

I predicted your odds of winning to be 50% with queen+rook odds, 1% with queen odds, 0.2% with 2 bishops odds, and 0.1% with rook odds. When you started describing strategies tailored to odds games that you were going to use, I felt cheated! I thought you were just going to play your normal 1100-rated game, but I made a big mistake. I forgot that you're a general intelligence, not a narrow, 1100-rated chess AI. Stockfish's NNUE was never trained on positions like the ones at the start of your odds games since they can't be reached from a normal 32-piece start, so its ability to generalize to these new board states is anybody's guess. What you intended as a conflict between a less intelligent^[1] agent with more resources and a more intelligent agent with less resources turned out to be (more intelligent + less resources + less general) vs. (less intelligent + more resources + more general). If the generalization gap helped you at all, then the amount of resources required to overcome a given intelligence gap is higher than your experiment suggests. Bad news for our species.

I expect the chess analogy would break down when considering agents that can obfuscate their intentions and actions better than a human can. If it could not win in a direct conflict, why wouldn't it just wait until it could?

I do love the "brains vs. brawns" framing, though, and I'm looking forward to what you write about asymmetrical conflicts. If you or anyone else is interested in repeating this experiment without the confounding variable of generality, I suggest using a handicapped version of SF, set to some arbitrary rating. A truly enterprising person could probably automate many matches of of SF vs. SF at different ratings and make a series of 2D plots, each plot being of a different material imbalance, with their ratings plotted against each other, showing the odds of the higher rated one winning for a given rating pair. Unfortunately, the result of that might be largely determined by how it's handicapped, since doing it by reducing search depth would mean the deeper seeing one would see strictly more than the other. I suspect this handicapping would much more strongly favor the more intelligent SF than handicapping by occasionally playing random legal moves.

1. ^{^}

   measuring intelligence as Elo rating at standard chess, without odds

The problem is that true AGI is self-improving and that a strong enough intelligence will always either accrue the resource advantage or simply do much more with less. Chess engines like Stockfish do not serve as good analogies for AGI since they don't have those self-referential self-improvement capabilities that we would expect true AGI to have.

Odds games against engine are played with contempt equal to matherial difference.

Sorry you didn't know that beforehand.