Kidnapping and the game of Chicken

You might want to analyze this game according to standard game theory before trying to reinvent the wheel (and risking to make it square).
In particular, when your proposed strategy involves guessing what the other players do, and this doesn't converge when the other players do the same to you, it is usually a bad sign (sometimes it's not possible to do anything better, but sometime it is).

Let's start with the standard version:

There are two pure strategy Nash equilibria: (A, B) and (B, A). Both are Pareto-optimal, but they benefit the player who played A more than the other, thus they are "unfair", in the sense that they yield asymmetric payoff in a symmetric game.
There is also a mixed strategy Nash equilibrium which is symmetric but sub-optimal. The game is a bargaining problem and a coordination game as well. In particular, it is a variation of the battle of the sexes game.
Unfortunately, there doesn't seem to be any truly satisfactory solution to this game. There is, however, a correlated equilibrium solution. If both players have access to a public random bit generator, they can obtain a Pareto-optimal symmetric payoff equilibrium, assuming that they can communicate to coordinate a strategy (they don't have to make a contract).

Now, let's consider the program-swap variant of the game.

First of all, a remark on the wording:

what if we allow players access to each others' source code?

Players don't have access to each others' source code. Players submit 'bots': programs which have access to each others' source code. The submitted programs are (pure) strategies, not players. I think it is important to always make the distinction clear, otherwise there is a risk of confusion between the decision makers and the decisions themselves, which may lead to flawed proposed solutions.

In the program-swap variant of this game we (players) can achieve something eqiivalent to the correlated equilibrium by using clique-like mixed strategies:

Each player flips a coin and submits either CliqueBot_0 or CliqueBot_1, which are defined as:
def CliqueBot_X(otherBot):
if otherBot is not a CliqueBot_X then output 'A'
Y = extract 'X' from otherBot
ID = player id (either 0 or 1)
if X xor Y xor ID then output 'A'
otherwise output 'B'

This achieves a Pareto-optimal symmetric equilibrium against itself and, being a clique, it also achieves stability: defectors from the clique are immediately punished. It also achieves maximum payoff against B-bots and PredictorBots, though it performs poorly against A-bots.

The main issue with this strategy is, of course, that there are infinitely many cliques, each of them doesn't cooperate with the others. This becomes to a "choosing sides" coordination game, which is easier than the original game: you just need a single way communication channel between the players to achieve coordination. Even if there is no communicaton channel, it might possible to achieve coordination using a prior, without lack of convergence.

The best "A-players" and "B-players" now choose moves against their copies by flipping coins, so that half the time they get at least one cookie.

Waidaminute - those aren't "A-players" or "B-players" any more, but variants. And if two "A-player variants" vary in a slightly different way, will they recognize each other as such? Doing so is not a trivial problem!

Anyway, my algorithm would be something like:

"Calculate a hash of my code, hashMe, and a hash of my opponent, hashYou. If (hashMe + hashYou) is odd, the player with the highest hash shall be the Chosen One, otherwise it's the player with the lowest hash (if the hashes are identical, find some other unambiguous higher/lower criteria both players share, player IDs or something or lexicographic ordering of source code, if none exists, flip a coin). If I'm the Chosen One, play A, otherwise play B. Also, random string of comments (to ensure different instances of this algorithm have different hash functions)."

This should have an expected value 1.5 against predictors (it can be simulated safely without recursion problems), 0.5 against A-players, 1 against B-players and 0.75 against coin-flippers. And more importantly, it's resistant to exploitation/blackmail, and gets the max utility of 1.5 against itself.

The only problem with this algorithm is that it's not good at coordinating with similar algorithms that have a different "Chosen One" criteria (e.g. "look at the parity of the (hashMe+HashYou)th digit of pi"). That could be fixed, but is considerably more complicated.

It's not directly relevant, but here's my favourite fact about Chicken, which I think I found in a book by Steven Brams: If you play a game of Chicken against God (or Omega, or any other entity able to read your mind or otherwise predict your behaviour), God loses. (Because all you have to do is decide not to flinch, and your omniscient opponent knows you will not flinch, and It then has no better option than to flinch and let you win.)

Of course the correct next inference is that Omega either doesn't play Chicken, or cheats (at which point It is in fact no longer playing Chicken but some other game). Seems reasonable enough.

Note: If you understand up to Stage 1 of this post, you now understand how allele frequencies are determined in a population that has random mating and a few other biology related assumptions. The invisible "Players" choose which allele it is better to be, and they will settle onto an equilibrium as Manfred describes.

The above post contains everything you need to do some intro-level population genetics problems. If you learned something from the above most and would now like to test your shiny new genetics skills, here's a problem set:

Suppose A and B are the only two alleles in a diploid genome, and AA has fitness 2, BB has fitness 1, and AB has fitness 3

Problem: What proportion of alleles are A? What proportion of alleles are B?

Solution

Bonus question: What will the final distribution of AA, AB, and BB prototypes look like?

Solution

In practice for biologists, you'd be doing this problem backwards - looking at the percent of the population that expresses each phenotype, and using that to infer allele frequencies and relative fitness. If I tell you that 25% of a population has sickle cell, if you know that sickle cell disease comes from a homozygous recessive allele, and if you are willing to assume various things, that's all you need to work backwards all the way to the payoff matrix.

In practice, of course, mating isn't random. If AB is inferior to both AA and BB, then there is an incentive for A carriers to avoid B carriers and it is a trend towards speciation. If AB is superior to both AA and BB, there is an incentive for heterozygosity.) Also in practice, the allele frequencies are constantly shifting so you are following a moving target, and sometimes sampling is biased (for example, carriers of low fitness alleles die before you can measure them) etc.

This strikes me as yet another example of the value of being able to precommit in Newcomblike problems. The dumb predictor is playing the role of the box-chooser; the smarter predictor is Omega.

At the meta-level, it seems to me like I aught to use a fair coin flip to decide who gets A and who gets B, and that I further aught to punish any defectors who don't go along with this and try to just pick A and kidnap the cookies.

But this only "wins" to the extent that it reduces the ratio of non-negotiating-A-players, so it only works in the iterated case...