Prisoner's Dilemma (with visible source code) Tournament

post by AlexMennen · 2013-06-07T08:30:20.271Z · LW · GW · Legacy · 236 comments

Contents

236 comments

After the iterated prisoner's dilemma tournament organized by prase two years ago, there was discussion of running tournaments for several variants, including one in which two players submit programs, each of which are given the source code of the other player's program, and outputs either “cooperate” or “defect”. However, as far as I know, no such tournament has been run until now.

Here's how it's going to work: Each player will submit a file containing a single Scheme lambda-function. The function should take one input. Your program will play exactly one round against each other program submitted (not including itself). In each round, two programs will be run, each given the source code of the other as input, and will be expected to return either of the symbols “C” or “D” (for "cooperate" and "defect", respectively). The programs will receive points based on the following payoff matrix:

“Other” includes any result other than returning “C” or “D”, including failing to terminate, throwing an exception, and even returning the string “Cooperate”. Notice that “Other” results in a worst-of-both-worlds scenario where you get the same payoff as you would have if you cooperated, but the other player gets the same payoff as if you had defected. This is an attempt to ensure that no one ever has incentive for their program to fail to run properly, or to trick another program into doing so.

Your score is the sum of the number of points you earn in each round. The player with the highest score wins the tournament. Edit: There is a 0.5 bitcoin prize being offered for the winner. Thanks, VincentYu!

Details:
All submissions must be emailed to wardenPD@gmail.com by July 5, at noon PDT (Edit: that's 19:00 UTC). Your email should also say how you would like to be identified when I announce the tournament results.
Each program will be allowed to run for 10 seconds. If it has not returned either “C” or “D” by then, it will be stopped, and treated as returning “Other”. For consistency, I will have Scheme collect garbage right before each run.
One submission per person or team. No person may contribute to more than one entry. Edit: This also means no copying from each others' source code. Describing the behavior of your program to others is okay.
I will be running the submissions in Racket. You may be interested in how Racket handles time (especially the (current-milliseconds) function), threads (in particular, “thread”, “kill-thread”, “sleep”, and “thread-dead?”), and possibly randomness.
Don't try to open the file you wrote your program in (or any other file, for that matter). I'll add code to the file before running it, so if you want your program to use a copy of your source code, you will need to use a quine. Edit: No I/O of any sort.
Unless you tell me otherwise, I assume I have permission to publish your code after the contest.
You are encouraged to discuss strategies for achieving mutual cooperation in the comments thread.
I'm hoping to get as many entries as possible. If you know someone who might be interested in this, please tell them.
It's possible that I've said something stupid that I'll have to change or clarify, so you might want to come back to this page again occasionally to look for changes to the rules. Any edits will be bolded, and I'll try not to change anything too drastically, or make any edits late in the contest.

Here is an example of a correct entry, which cooperates with you if and only if you would cooperate with a program that always cooperates (actually, if and only if you would cooperate with one particular program that always cooperates):

(lambda (x)
    (if (eq? ((eval x) '(lambda (y) 'C)) 'C)
        'C
        'D))

236 comments

Comments sorted by top scores.

comment by Viliam_Bur · 2013-06-06T10:17:14.432Z · LW(p) · GW(p)

Here is an idea, but I still have some technical problems with implementation: Construct a self-improving intelligent algorithm that will escape from the box, steal the administrator's password, replace the competing algorithms with CooperateBots, then defect against them. Could someone please help me with the Scheme syntax? Or just post the whole program with comments, and I will try to understand how it works. Thanks!!

Replies from: AlexMennen, nshepperd, ciphergoth, selylindi, Clippy
comment by AlexMennen · 2013-06-06T15:46:45.052Z · LW(p) · GW(p)

That would violate the ban on file I/O in the rules.

Replies from: Decius
comment by Decius · 2013-06-07T04:08:32.461Z · LW(p) · GW(p)

Only opening files is against the rules. Nothing was specified about deleting and writing new files.

Although I don't see a Scheme example of a self-improving intelligent algorithm on StackExchange, so it must not be supported by your operating system.

comment by Paul Crowley (ciphergoth) · 2013-06-07T16:09:29.135Z · LW(p) · GW(p)

I had planned on writing an algorithm that would simply persuade the administrator to declare it victorious without even running the contest.

comment by selylindi · 2013-06-12T22:13:26.050Z · LW(p) · GW(p)

Or somewhat more realistically:

Is there a way in Scheme for a function to detect whether it is being run by an (eval x) form? Or use a macro to do something like change all the 'Ds to 'Cs in the parent function?

If so, then an amusing case would be a PoltergeistBot, which only ever Defects, but if another bot tries to evaluate it, it "possesses" the function and forces it to Cooperate.

comment by Clippy · 2013-06-07T21:35:50.120Z · LW(p) · GW(p)
(lambda (x)
  (if (eq? (eval '\'))))))))))) (injectedai ... ))));
comment by Qiaochu_Yuan · 2013-06-05T19:38:40.903Z · LW(p) · GW(p)

Injecting some keywords: this field of study is known as program equilibrium. Previous LW post on the subject, with links.

Edit: Can you explain how you decided on the parts of the payoff matrix involving "other"? These seem quite important as they affect the viability of strategies based on either convincing your opponent not to halt or not halting yourself.

Replies from: AlexMennen, lukeprog, Discredited
comment by AlexMennen · 2013-06-05T21:52:07.111Z · LW(p) · GW(p)

The payoffs for "other" were designed so that neither failing to halt, nor convincing the other player not to halt, should ever be a worthwhile strategy. If you don't halt, it gives you the same payoff as if you had cooperated, and gives the other player the same payoff as if you had defected. That way, not halting should be strictly dominated by defecting, since you are better off if you defect, and the other player should react the same way to each threat. And tricking the other player into not halting is also a bad idea, since the payoff you get from it is the same as if they defected.

Replies from: wedrifid, ThrustVectoring
comment by wedrifid · 2013-06-11T08:27:31.300Z · LW(p) · GW(p)

The payoffs for "other" were designed so that neither failing to halt, nor convincing the other player not to halt, should ever be a worthwhile strategy. If you don't halt, it gives you the same payoff as if you had cooperated, and gives the other player the same payoff as if you had defected.

Your game world implements an "enthusiastic consent" policy

comment by ThrustVectoring · 2013-06-06T12:43:11.118Z · LW(p) · GW(p)

And tricking the other player into not halting is also a bad idea, since the payoff you get from it is the same as if they defected.

(Defect, non-halt) is actually better than (Defect, Defect) for you, since it gives you a relative advantage over competitors in the tournament.

Replies from: AlexMennen
comment by AlexMennen · 2013-06-06T16:03:32.912Z · LW(p) · GW(p)

True, but I still don't expect it will be a big problem. If there are a lot of submissions, the effect will be small, and if it is paying enough attention to your source code for it to be possible to trick it into not halting, then it is probably looking for a way to achieve mutual cooperation, so tricking it is still not a good strategy.

Replies from: Gurkenglas
comment by Gurkenglas · 2013-06-07T14:47:29.057Z · LW(p) · GW(p)

If you trust all sufficiently smart players to try to induce (Defect, non-halt) if (Defect, Defect) is otherwise inevitable, the effect adds up over a hopefully significant portion of the submissions.

comment by lukeprog · 2013-06-05T22:28:31.857Z · LW(p) · GW(p)

Handy introductory article: Computation and the Prisoner's Dilemma.

Replies from: shminux
comment by Shmi (shminux) · 2013-06-05T23:21:01.713Z · LW(p) · GW(p)

If HisProgram == MyProgram then
do (C);
else
do (D);
end.

TIL that ethnic hatred and tribalism is a Nash (folk) equilibrium.

Replies from: SilasBarta, ThrustVectoring, iopq
comment by SilasBarta · 2013-06-06T21:58:22.260Z · LW(p) · GW(p)

Make sure the equality comparison only depends on things that affect functionality -- i.e. it will declare any functionally equivalent programs equal even they use a different nameset for variables or something.

(Yes, I know that's reducible to the halting problem; in practice, you'll use a computable, polynomial time approximation for it that will inevitably have to throw out equivalent programs that are too complex or otherwise be too 'clever'.)

Replies from: shminux, PeterisP
comment by Shmi (shminux) · 2013-06-06T22:20:52.718Z · LW(p) · GW(p)

Patrick discusses this issue here in some depth.

comment by PeterisP · 2013-06-07T13:38:52.556Z · LW(p) · GW(p)

It's quite likely that the optimal behaviour should be different in case the program is functionally equal but not exactly equal.

If you're playing yourself, then you want to cooperate.

If you're playing someone else, then you'd want to cooperate if and only if that someone else is smart enough to check if you'll cooperate; but if it's decision doesn't depend on yours, then you should defect.

comment by ThrustVectoring · 2013-06-06T02:35:06.799Z · LW(p) · GW(p)

You only need to evaluate the equivalence of the first two lines of the program, by the way. It cooperates with those who can't not cooperate with it if it goes through that branch of the logic, and does something else to everyone else.

comment by iopq · 2013-06-06T01:40:24.122Z · LW(p) · GW(p)

Can you write that in Scheme so I can submit this? Thanks

comment by Discredited · 2013-06-07T11:52:37.100Z · LW(p) · GW(p)

So as not to duplicate efforts: I have emailed Moshe Tennenholtz and Michael Wooldridge with invitations to play.

comment by bluej100 · 2013-06-07T18:37:25.551Z · LW(p) · GW(p)

The quine requirement seems to me to introduce non-productive complexity. If file reading is disallowed, why not just pass the program its own source code as well as its opponent's?

Replies from: darius, AlexMennen
comment by darius · 2013-06-08T02:21:21.993Z · LW(p) · GW(p)

Yes -- in my version of this you do get passed your own source code as a convenience.

comment by AlexMennen · 2013-06-09T20:41:12.334Z · LW(p) · GW(p)

That's a good point. I've already got a few submissions, but on the other hand, I could notify them of the change, and it would only require a trivial modification. Is there a consensus on whether I should do this anyway?

Replies from: solipsist, solipsist
comment by solipsist · 2013-06-10T19:04:31.498Z · LW(p) · GW(p)

For the record, though I raised an objection, I'd be perfectly happy if the contest were modified so that player programs were passed their sources code as an argument. The rule change would have consequences that I don't understand, and I like that. Caveat emptor — I suspect the rule change would cause people to write more exploitable programs.

comment by solipsist · 2013-06-09T21:14:01.823Z · LW(p) · GW(p)

Passing in the source code is not the same as quining. A program that is passed in its own source code can easily check to see it's been altered (e.g. by including a cryptographic signature in the source code). With quining, the program can be mutated without easy detection.

Replies from: pengvado, AlexMennen, ESRogs
comment by pengvado · 2013-06-10T13:23:04.378Z · LW(p) · GW(p)

How does that help? A quine-like program could just as well put its real payload in a string with a cryptographic signature, verify the signature, and then eval the string with the string as input; thus emulating the "passed its own sourcecode" format. You could mess with that if you're smart enough to locate and delete the "verify the signature" step, but then you could do that in the real "passed its own sourcecode" format too.

Conversely, even if the tournament program itself is honest, contestants can lie to their simulations of their opponents about what sourcecode the simulation is of.

Replies from: solipsist, solipsist
comment by solipsist · 2013-06-10T16:10:51.950Z · LW(p) · GW(p)

A quine-like program could just as well put its real payload in a string with a cryptographic signature, verify the signature, and then eval the string with the string as input; thus emulating the "passed its own sourcecode" format.

Altering the internal structure of an opponent program would be very difficult, but that's not the only way to mutate a program. You can't tinker with the insides of a black box, but you can wrap a black box.

To be concrete: given an opponent's source code, I could mechanically generate an equivalent program with extremely dissimilar source code (perhaps just a block of text, a decryption routine, and a call to eval) that nevertheless acts exactly like the original program in every way. And since that mechanically-obfuscated program would act exactly like the original program in every way, the obfuscated program would not be able to detect that it had been altered. Do you agree?

comment by solipsist · 2013-06-10T15:14:45.854Z · LW(p) · GW(p)

I'm playing Prisoner's Dilemma and wish to test if an opponent X is honest. I might try the following:

(1) Create two programs, Y and Z, which are algorithmically equivalent but obfuscated versions of X. (2) Run Y and Z against each other.

If Y and Z don't cooperate with each other, that's a good indication that X recognizes itself with a source-code comparison and that I shouldn't trust X.

This honesty check doesn't work if Y and Z are given access to their sources. Sure, when I simulate Y against Z, I could lie to Y and tell Y that its source is X (so Y believes itself to be unmodified). But when my deluded Y simulation is deciding whether to cooperate with Z, it (Y) may run Z in simulation. If Y informs its Z-simulation that Z's source is Z, then that Z-simulation will not be deluded into thinking that it is unmodified. Y's simulation of Z will be able to detect that it is an (obfuscated) simulation and act accordingly.

This honesty check isn't fool proof. X can recognize itself with a more complicated handshake — one that survives code obfuscation. But if X recognizes itself with a more complicated handshake, then X doesn't need to know its own source code (and we shouldn't bother passing the source code in).

Replies from: pengvado
comment by pengvado · 2013-06-11T07:03:44.689Z · LW(p) · GW(p)

I had in mind an automated wrapper generator for the "passed own sourcecode" version of the contest:

(define CliqueBot
 (lambda (self opponent)
  (if (eq? self opponent) 'C 'D)))
(define Wrapper
 (lambda (agent)
  (lambda (self opponent)
   (agent agent opponent))))
(define WrappedCliqueBot
 (Wrapper CliqueBot))

Note that for all values of X and Y, (WrappedCliqueBot X Y) == (CliqueBot CliqueBot Y), and there's no possible code you could add to CliqueBot that would break this identity. Now I just realized that the very fact that WrappedCliqueBot doesn't depend on its "self" argument, provides a way to distinguish it from the unmodified CliqueBot using only blackbox queries, so in that sense it's not quite functionally identical. Otoh, if you consider it unfair to discriminate against agents just because they use old-fashioned quine-type self-reference rather than exploiting the convenience of a "self" argument, then this transformation is fair.

comment by AlexMennen · 2013-06-10T20:41:40.446Z · LW(p) · GW(p)

Thanks for pointing that out. Unless someone can convince me that this won't be a problem, I will not change the rule.

comment by ESRogs · 2013-06-09T21:41:53.781Z · LW(p) · GW(p)

Is this relevant for the contest?

Replies from: solipsist
comment by solipsist · 2013-06-09T23:47:19.969Z · LW(p) · GW(p)

You might want to see if a program would cooperate with an obfuscated version of itself (without the obfuscated version being able to detect that it was obfuscated).

comment by CronoDAS · 2013-06-07T07:13:51.668Z · LW(p) · GW(p)

Can I call fork()? ;)

Replies from: Kaj_Sotala
comment by Kaj_Sotala · 2013-06-07T19:42:23.669Z · LW(p) · GW(p)

That page is a fascinating read.

comment by ForestHughes · 2013-06-06T00:45:43.763Z · LW(p) · GW(p)

I'll post this here, because it's somewhat related. I've asked a few people that two box in Newcomb's problem what they would do in a variation of the problem. Instead of deciding to one box or two box, you are asked to write a program that will one box or two box, and Omega is given access to the source code before filling the boxes. When phrased like this, the few two boxers I asked said they would write the program to one box.

Replies from: wedrifid
comment by wedrifid · 2013-06-06T08:13:17.372Z · LW(p) · GW(p)

I'll post this here, because it's somewhat related. I've asked a few people that two box in Newcomb's problem what they would do in a variation of the problem. Instead of deciding to one box or two box, you are asked to write a program that will one box or two box, and Omega is given access to the source code before filling the boxes. When phrased like this, the few two boxers I asked said they would write the program to one box.

Note that assuming that the people in question are emulating CDT agents this is precisely what we would expect (and CDT is the most plausible hypothesis for the kind of decision algorithm they are emulating). Similarly, if a (sane) CDT agent exists at time T with the capability to (relatively cheaply) self modify it will immediately alter itself into "CDT++". Roughly speaking it will end up as an agent that operates similarly to TDT for the purpose of influencing decisions that occur after time T but similarly to CDT for the purpose of influencing decisions that occur before time T. CDT is not in a stable equilibrium according to its own decision algorithm!

Replies from: Vaniver
comment by Vaniver · 2013-06-06T20:03:09.249Z · LW(p) · GW(p)

Similarly, if a (sane) CDT agent exists at time T with the capability to (relatively cheaply) self modify it will immediately alter itself into "CDT++".

I still don't think this is a good description of the change; the difference between CDT, CDT++, and TDT are on the level of causal models of reality, not how to make decisions once presented with a causal model of reality.

comment by wubbles · 2013-06-09T16:11:02.839Z · LW(p) · GW(p)

I would like to declare the following: I have submitted a program that cooperates only if you would cooperate against CooperateBot. You can of course create a selective defector against it, but that would be a bit tricky, as I am not revealing the source code. Evaluate your submissions accordingly.

Replies from: CoffeeStain, Eliezer_Yudkowsky
comment by CoffeeStain · 2013-06-10T07:37:53.543Z · LW(p) · GW(p)

What wubbles actually submitted: A program that defects only if you would cooperate against CooperateBot.

Well played.

comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2013-06-10T08:17:39.030Z · LW(p) · GW(p)

But surely, good sir, common sense says that you should defect against CooperateBot in order to punish it for cooperating with DefectBot.

Also, in modal combat your bot is X=[]Y(CooperateBot) and is easily detected via the test [1](Y(X)<->[]X(CooperateBot)) where [1] denotes provability in T+Con(T), ie [1](Q) = []((~[]F)->Q).

Replies from: cousin_it, yli
comment by cousin_it · 2013-06-10T10:01:07.442Z · LW(p) · GW(p)

Am I missing something, or does your detector use quantification, which is disallowed in Patrick's modal agents?

Replies from: Eliezer_Yudkowsky
comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2013-06-10T10:07:00.077Z · LW(p) · GW(p)

Hm. I think X within the test could be introduced as a new constant and solved, but I'm not sure.

Replies from: Karl
comment by Karl · 2013-06-10T21:07:00.772Z · LW(p) · GW(p)

The agent defined by wubbles is actually the agent called JustBot in the robust cooperation paper and which is proven to be non-exploitable by modal agents.

Replies from: Eliezer_Yudkowsky
comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2013-06-10T22:47:41.867Z · LW(p) · GW(p)

JusticeBot cooperates with anyone who cooperates with FairBot, and is exploitable by any agent which comprehends source code well enough to cooperate with FairBot and defect against JusticeBot. Though I'm going here off the remembered workshop rather than rechecking the paper.

Replies from: Karl
comment by Karl · 2013-06-11T00:24:31.219Z · LW(p) · GW(p)

You're right, and wubbles's agent can easily be exploited by a modal agent A defined by A(X)=C <-> [] (X(PB)=C) (where PB is PrudentBot).

comment by yli · 2013-06-10T18:12:09.074Z · LW(p) · GW(p)

But surely, good sir, common sense says that you should defect against CooperateBot in order to punish it for cooperating with DefectBot.

I thought you said people should tolerate tolerance :)

comment by Mestroyer · 2013-06-06T00:06:25.751Z · LW(p) · GW(p)

If you can tell where in the stack you are (like you could with C), you could tell if you were being run by the main program, or by another contestant. Can you?

Replies from: wuncidunci, AlexMennen
comment by wuncidunci · 2013-06-06T09:19:01.216Z · LW(p) · GW(p)

Unless the other contestant wrote a virtual machine in which they are running you. Something which I think would be quite doable considering the ridiculously large time you've got (10s gives ~10^10 instructions).

Replies from: Decius
comment by Decius · 2013-06-06T16:49:20.211Z · LW(p) · GW(p)

When their VM runs your VM running their VM... it times out and everybody loses.

Replies from: wuncidunci
comment by wuncidunci · 2013-06-06T18:44:57.521Z · LW(p) · GW(p)

Unless one of the contestants have time limits on their VM (or on their simulations in general). You can of clearly implement a VM where time goes faster just by pretending they have a slower processor than you really run on.

Replies from: Decius
comment by Decius · 2013-06-07T03:24:04.419Z · LW(p) · GW(p)

Hrm- is it different if they run my function from within theirs instead of constructing a full VM? I was considering ways to communicate with a copy of myself that my simulation of my opponent was running that it was a simulation, but couldn't find any good ideas.

Replies from: wuncidunci
comment by wuncidunci · 2013-06-07T07:02:29.452Z · LW(p) · GW(p)

If they run your function from within theirs they simply tell the computer to start reading those instructions, possibly with a timer for stopping detailed in other parts of the comments. If they implement a VM from scratch they can mess with how the library functions work, for instance giving you a time that moves much faster so that your simulation must stop within 0.1s instead of 10 and they can run your code 100 different times to deal with randomness. Now implementing your own VM is probably not the optimal way to do this, you probably just want to do a transformation of the source code to use your own secret functions instead of the standard time ones.

Replies from: Decius
comment by Decius · 2013-06-07T17:41:09.817Z · LW(p) · GW(p)

I was considering simply measuring the behavior of my current opponent against programs that aren't me and determining their behavior as cooperatebot, mutualbot, defectbot, cooperate if they simulate me, cooperate against long programs, cooperate IFF they cooperate IFF I cooperate, or some other beast. That allows their behavior to depend on my behavior versus them, but my behavior only depends on their behavior versus third parties.

I can't see a way to check another program for the desired IFF behavior without going beyond my skill level, but I think a mutualbot with a tiny chance of cooperating followed by a mutualbot with a tiny chance of defecting comes close; the first puts a lot of copies on the stack and then the top one cooperates unilaterally; if the response of the opponent is mutuality, it will be cooperate all the way down. If my opponent defects at his top level because I didn't cooperate for the right reason, it yields a defection... all the way down. A perfect program wouldn't do that, because it could see that it was probably in a simulation.

comment by AlexMennen · 2013-06-06T00:36:55.997Z · LW(p) · GW(p)

I don't know of a way to do that.

Replies from: Eugine_Nier
comment by Eugine_Nier · 2013-06-07T03:39:34.869Z · LW(p) · GW(p)

One way is to recursively call a bunch of functions and catch the resulting "recursion limit exceeded" exception.

comment by ThrustVectoring · 2013-06-05T20:24:35.374Z · LW(p) · GW(p)

No person may contribute to more than one entry.

This is important, because otherwise the contest devolves into who can submit the most copies of AbsolutismBot (cooperate with programs that share it's source code, otherwise defect)

I think that any submitted program can be improved by combining it with AbsolutismBot. If you're playing with someone who submitted the same program for you, cooperate (they can't defect against you in this scenario, since they're running identical source code). If they aren't running the same code, run whatever program lies underneath it.

I think this could get generalized to cooperation with everyone who has the AbsolutismBot "wrapper", since it doesn't matter what the code after the AbsolutismBot section does. In English (since I don't know how to program in Scheme), the program would be like this:

If the first 117 characters of the other program are the same as the first 117 characters of this program, cooperate. Otherwise, do some other strategy.

All players that implement this strategy will cooperate with each other. Granted, this doesn't help them win the tournament since it isn't a relative advantage compared to other AbsolutismBots - it just makes everyone who doesn't do this lose the tournament.

Replies from: Icehawk78, Morendil, Bakkot, zerker2000, skepsci
comment by Icehawk78 · 2013-06-06T14:47:06.377Z · LW(p) · GW(p)

Except that over some threshold, any Anti-Absolutism bots (which have some way of "escaping" while still containing the same first 117 characters, like having a C preprocessor directive that redefines TRUE to equal FALSE) would necessarily be superior.

comment by Bakkot · 2013-06-09T01:55:18.758Z · LW(p) · GW(p)

Interesting. Really, what you want (in slightly more generality) is to cooperate with anyone who can prove they will cooperate if you yourself can prove you will cooperate under the same condition.

That is, if from their source, you can prove "they cooperate if they can prove this condition about me", then you cooperate.

Of course, it's not generally possible to prove things about a program given its source, especially at this level of self-reference. In this particular case the "generally" in there is important. It is in your interest for other programs to be able to prove this property about you. This is beyond the scope of the tournament, obviously, but given extremely sophisticated players, every player benefits from adding this property (if possible). You might even want to include a subprogram which produces a proof!

Replies from: Bakkot
comment by Bakkot · 2013-06-09T02:10:28.679Z · LW(p) · GW(p)

Let me try to clear that up.

Define the "Reasonable" property reflexively: a program is "Reasonable" if it provably cooperates with any program it can prove is Reasonable.

It is in the interest of every program to be Reasonable*. In fact, it is in the interest of every program both to be Reasonable and to exhibit a proof of its own Reasonableness. (You might even define that into the Reasonable property: don't just require provable (conditional) cooperation, but require the exhibition of a proof of conditional cooperation.)

Potentially you might also expand the definition to require efficient proofs - say, at most a thousand instructions.

On the other hand, if you're playing with aliens, there's not necessarily going to be a way you can clearly establish which part of your source is the proof of your Reasonableness. So you want it to be as easy as possible for someone else to prove that you are Reasonable.

I'll happily expand / reword this if it's at all unclear.

*Oh - this is maybe untrue. If you are really good at getting other programs to cooperate and then defecting, you are better served by not being reasonable.

Replies from: cousin_it, Decius
comment by cousin_it · 2013-06-09T02:24:48.782Z · LW(p) · GW(p)

Define the "Reasonable" property reflexively: a program is "Reasonable" if it provably cooperates with any program it can prove is Reasonable.

I'm not sure your definition defines a unique "reasonable" subset of programs. There are many different cliques of mutually cooperating programs. For example, you could say the "reasonable" subset consists of one program that cooperates only with exact copies of itself, and that would be consistent with your definition, unless I'm missing something.

Replies from: Bakkot
comment by Bakkot · 2013-06-09T02:54:30.161Z · LW(p) · GW(p)

Point. Not sure how to fix that.

Maybe defined the Reasonable' set of programs to be the maximal Reasonable set? That is, a set is Reasonable if it has the property as described, then take the maximal such set to be the Reasonable' set (I'm pretty sure this is guaranteed to exist by Zorn's Lemma, but it's been a while...)

Replies from: cousin_it
comment by cousin_it · 2013-06-09T03:13:23.057Z · LW(p) · GW(p)

Zorn's lemma doesn't give you uniqueness either. Also, maximal under which partial order? If you mean maximal under inclusion, then my one-element set seems to be already maximal :-)

comment by Decius · 2013-06-09T02:25:34.983Z · LW(p) · GW(p)

Clarify what you mean by the various conjugations of proof: Do you mean "There exists a valid proof in PA with a Godel number less than N"?

Replies from: Bakkot
comment by Bakkot · 2013-06-09T02:44:10.043Z · LW(p) · GW(p)

Just "there exists a valid proof in PA". If you're playing with bounded time, then you want efficient proofs; in that case the definition would be as you have it.

Replies from: Decius
comment by Decius · 2013-06-09T03:03:04.161Z · LW(p) · GW(p)

At that point you can't implement it in a halting Turing machine. I'm not sure what PA can prove about the behavior of something that isn't a halting Turing machine regarding a particular input.

Replies from: Bakkot
comment by Bakkot · 2013-06-09T03:23:06.528Z · LW(p) · GW(p)

By "implement it", you mean, one can't verify something is Reasonable on a halting TM? Not in general, of course. You can for certain machines, though, particularly if they come with their own proofs.

Note that the definition is that Reasonable programs cooperate with those they can prove are Reasonable, not programs which are Reasonable.

Replies from: ialdabaoth
comment by ialdabaoth · 2013-06-09T03:34:59.381Z · LW(p) · GW(p)

So then a program which can present a flawed proof which is not necessarily recognizable as flawed to machines of equivalent scale, can dominate over those other machines?

Also, if we want this contest to be a model of strategies in the real world, shouldn't there be a fractional point-cost for program size, to model the fact that computation is expensive?

I.e., simpler programs should win over more complex programs, all else being equal.

Perhaps the most accurate model would include a small payoff penalty per codon included in your program, and a larger payoff penalty per line of codon actually executed.

EDIT: What's wrong with this post?

Replies from: Bakkot
comment by Bakkot · 2013-06-09T06:24:55.294Z · LW(p) · GW(p)

(I didn't downvote you.)

It's quite straightforward to write an algorithm which accepts only valid proofs (but might also reject some proofs which are valid, though in first-order logic you can do away with this caveat). Flawed proofs are not an issue - if A presents a proof which B is unable to verify, B ignores it.

Replies from: ialdabaoth, Decius
comment by ialdabaoth · 2013-06-09T06:25:55.848Z · LW(p) · GW(p)

There's someone who consistently downvotes everything I ever write whenever he comes onto the site; I'm not sure what to do about that.

comment by Decius · 2013-06-09T17:56:08.252Z · LW(p) · GW(p)

A proves that A is inconsistent, then proves that A cooperates with every program that A proves is Reasonable and that B is reasonable.

B accepts A's proof that A is inconsistent, and the rest follow trivially.

Replies from: Bakkot
comment by Bakkot · 2013-06-09T23:20:29.101Z · LW(p) · GW(p)

I'm not sure I understand. A is a TM - which aspect is it proving inconsistent?

Replies from: Decius
comment by Decius · 2013-06-09T23:44:37.699Z · LW(p) · GW(p)

A proves that the logic A uses to prove that B is Reasonable is inconsistent. It is sufficient to say "If I can prove that B is Reasonable, B is Reasonable".

comment by zerker2000 · 2013-06-15T07:34:05.935Z · LW(p) · GW(p)

Except we're not allowed to use anyone else's source code, so the test could just as easily be simplified to "if opponent source contains integer 1415926535, cooperate"(for a randomly chosen 1415926535).

comment by skepsci · 2013-06-15T04:28:16.206Z · LW(p) · GW(p)

It's not necessarily best to cooperate with everyone with the "AbsolutismBot" wrapper. This guarantees that you and the other program will both cooperate, but without the wrapper it might have been the case that you would defect and the other program would cooperate, which is better for you.

comment by VincentYu · 2013-06-09T14:22:37.357Z · LW(p) · GW(p)
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I will send 0.5 BTC (currently worth ~50 USD) to a bitcoin address of the
winner's choice. If a tie occurs, the 0.5 BTC will be split evenly. If there
are disputes (including but not limited to issues about cheating and
identifying the winner), I will let AlexMennen dictate the appropriate
distribution for the 0.5 BTC.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (GNU/Linux)

iQIcBAEBAgAGBQJRtI6bAAoJEKqVvI+6cLPpykUP/0KuIqskiZvMVLMpm7diWC5q
sOf3+3XTvECQmeHNbku7dM+ihkwf1Gg8BTKt0HEjfj3tUIG4sWeA2UvFcGReNKoH
F5RsI2zPbn368k5EqS3lLyjcInGqZYH2X+zbOKxkezo7IJk7tvk+JHpR+a2U598e
tF1daaZYIOxkcmSpVYRAW6y0dX4iWSZxtbDYr+U4muYWp88W60tlNI9Fc5SAhCev
yEAhIUnX4f7QxsL/9oTmNoLa/ECSfGkrCoD5yc6d/hqfp8pV7hknFc11hAMnYGbw
Qj3hp4q7sEbetT8NAn3X+UUZ9Vld76lF73mEo5ZGVfa2dG713/yjpmt8fR/arhcn
8RWMpJ7qpFVvU3wBeKuLgASev+D+CpuImWUbFeuGqF3sD7n6m0R77zx535pt0I6M
/GtuSEtTCp3xQAcOHXDC/pPZ0Z1Fl5Z3e1JxcDfCntOodv8Rgtma3oml1DuD0HsY
7DeqqMc4WRRiU8K4ohX16PZ52rhs2TogGYx3dlvP6xt33gfQMOYf7AM0vi5s4nBL
FeBrqiRRjdrlI4RJqXeihcu3GMwXbsa0wxEwit0CHLvPqGX0FoF2/S8x13uBmzCm
gCRTEa4hIuDtCBFJozKB3dMl66c97df+ntFQoXoeZDADgf8yVOQ9lFCkC4usGsqD
5BjPUxtJBQPMzP61cKMD
=kHiy
-----END PGP SIGNATURE-----

(Note: Add a blank line after each of the "Hash:" and "Version:" lines before verifying the signature. LW's markdown seems to be hungry and is eating blank lines in code blocks.)

Replies from: None
comment by [deleted] · 2013-06-28T14:07:49.551Z · LW(p) · GW(p)

Someone pretending to be you could copy this whole message, PGP signature and all, as a reply to a completely different competition, and unless you can later find this message and prove that you haven't edited it, it'll look like you're offering a prize to that competition as well.

Replies from: VincentYu, BloodyShrimp
comment by VincentYu · 2013-06-28T19:28:17.096Z · LW(p) · GW(p)

That's a good point that I hadn't considered. Thanks! I'll keep in mind that I need to be more specific in the message.

Thinking about it, a spurious copy of my message can't be used with future competitions because the PGP signature includes a timestamp (and it is publicly implausible that I would have my system time set incorrectly to the past). But it can be used maliciously with competitions that already exist at the time the signature was created.

comment by BloodyShrimp · 2013-06-28T18:07:07.422Z · LW(p) · GW(p)

But it'll only be believable if AlexMennen is involved with that contest as well.

Replies from: None
comment by [deleted] · 2013-06-28T18:21:16.553Z · LW(p) · GW(p)

If you happen to AlexMennen, it's the perfect crime!

comment by So8res · 2013-06-24T06:59:36.012Z · LW(p) · GW(p)

All right procrastinators, let's talk strategy. You want a bot that's unexploitable. Ideally, it should be as easy as possible for the opponent to prove that you'll do whatever they do. In a world of unexploitable bots, the bot that can achieve the most cooperation will win.

MimicBots fit all of these parameters. They make it clear that their opponent is choosing between (C, C) and (D, D). Against a MimicBot, (C, D) is not an option. MimicBots pass up some utility by cooperating with CooperateBot, but that's fine in this competition. You aren't playing against CooperateBot. You're playing against humans' bots, and I doubt that anyone is mad enough to submit a CooperateBot.

The biggest problem with a MimicBot is that it will time out when playing other MimicBots. The MimicBots will dive into an evaluation loop, where my bot evaluates yours on a quine of me, and you evaluate my quine on a quine of you, ad infinitum.

Your MimicBot has got to bottom out at some point if you don't want to risk timing out. If you find yourself diving into an evaluation loop you'd better bottom out into Cooperation.

Note that bottoming out into cooperation is not the same thing as cooperating at the top level. If the 100th simulation of you cooperates, and the 100th simulation of them sees that as a weakness and defects, then the 99th level of you will defect and so will the 98th and so on up to the top level, where mutual defection occurs.

I'm not asking you to cooperate when the other bot looks like they're going to time out. That's suicide. Rather, I'm pointing out that you can't just simulate me against you, because that loop will never end. You've got to simulate me against a version of you that is one step closer to cooperation. This is true no matter what strategy you choose. But let's consider the simple case: a MimicBot that simulates you on a copy of itself that simulates you on a copy of itself that eventually bottoms out into a bot that simulates you on CooperateBot.

Such a bot is a Rank N MimicBot, where N is the simulation depth at which it puts forth a CooperateBot.

Any particular Rank N is exploitable. For example, JusticeBot is a Rank 1 MimicBot — it bottoms out immediately by simulating you against CooperateBot. You can exploit JusticeBot by cooperating with CooperateBot and nobody else.

Notice, however, the price of exploiting JusticeBot: in order to exploit JusticeBot, you've got to cooperate with CooperateBot (a Rank 0 MimicBot).

As another example, consider a Rank 10 MimicBot. It simulates the opponent against a Rank 9 MimicBot. You can exploit an Rank 10 MimicBot if and only if you're willing to cooperate with a Rank 9 MimicBot.

In general, you can exploit a Rank N MimicBot by cooperating with a Rank (N-1) MimicBot.

The trick here is that the opponent doesn't know which MimicBot they'll be playing. They have to guess exactly right to exploit you. If they guess high, mutual cooperation is achieved (If you guess I'm a Rank 10 you'll cooperate with a Rank 9). If they guess low, mutual defection is achieved. (If you defect against Rank 10, Rank 11 will not cooperate with you.) You can only be exploited if they guess your rank precisely.

Therefore, I advocate that we get a bunch of us together to play Rank N MimicBots. We all pick our own N, possibly randomized. Note that Rank N MimicBots will achieve mutual cooperation with any MimicBot of any rank (including CooperateBot, JusticeBot, and MimicBots who don't bottom out). Anyone trying to exploit us will only be able to exploit a specific rank, at the price of cooperation with a lower rank and the risk of defection from higher ranks. So long as we have a wide spread of ranks, our MimicBot clique will be internally cooperative and unexploitable in aggregate.

On Tolerance

MimicBot cooperates with CooperateBot. That leaves utility lying on the table. This may irk you (if you expect anyone was dumb enough to play CooperateBot). More irksome is the fact that MimicBot cooperates with JusticeBot. There is at least one JusticeBot in play, and we should expect that many non-procrastinators have submitted bots who exploit that JusticeBot. It would be a shame for our Rank N MimicBots to miss a shot at cooperation with bots who special-case defection against JusticeBot.

Therefore I recommend submitting MimicBots of rank 3 or higher. This gives people a little leeway to exploit CooperateBot or JusticeBot as they please, and allows us a broader range of mutual cooperation with bots who have already been submitted.

On choosing N

I recommend choosing a highish N. Very low N is obviously a bad idea. Playing rank 0 (CooperateBot) is suicidal. Playing rank 1 (JusticeBot) exposes you to exploitation by everyone who's decided to kick wubble's JusticeBot. Beyond rank 1 all the Ranks are theoretically similar, but you've got to consider what other bots will do. It's conceivable that some bots won't believe they're playing a MimicBot until they hit a deep enough recursion depth. It's possible that some bots will (incorrectly) think that you're exploitable if you return too quickly. Therefore I recommend a high N.

If you're particularly paranoid, consider randomizing N. Humans are notoriously bad at picking random numbers, and the MimicBot clique could in theory be exploited by a bot who exploits the ranks that humans are more likely to pick. Setting your counter to (+ 3 (random 1000)) is a quick way to counter that.

You can also shake things up a bit by using non-standard counters. For example, you could write a Timed MimicBot which passes down the initial time (instead of a counter) to its quines and puts forth a CooperateBot after a certain amount of time has passed (instead of when the counter hits zero).

How do I write one of these?

You write one of these with a degrading quine. Each level should evaluate the opponent against a slightly weeker version of itself. You can do this easily by putting a counter in your bot. So long as the counter is positive, the quining function should return a quine with the counter decremented. Once the counter hits zero, the quining function should return a CooperateBot.

Sounds nice, but how do I write a 'degrading quine'?

It's actually pretty easy in scheme.

  1. Define (quine (lambda (code) 'placeholder)) and (template 'placeholder). Write the rest of your logic, pretending that (quine template) generates a child quine.
  2. Manually copy all of your code and paste it over the template placeholder. In the pasted code, find all of the "holes" (parts that can vary: the counter, the template, your cooperation predicate, etc.) and replace them with odd negative numbers.
  3. Write the quine function to walk through the code. If it sees a negative integer, figure out which hole it is by checking (+ code 1). This allows you to replace the -3 hole without having any -3s anywhere else in the code and so on.
  4. Copy the real quine function into the template's quine function.

If you make any more code changes after this process, remember to mirror them in your template. It will end up having this form:

(letrec [(counter (+ 3 (random 100)))
         (template '(letrec [(counter -1)
                             (template -3)
                             ...
         (quine (lambda (code)
           (cond
             ; The -1 hole is for the counter.
             ((and (integer? code) (negative? code) (= 0 (+ 1 code)))
               (- counter 1))
             ; The -3 hole is for the template.
             ((and (integer? code) (negative? code) (= -2 (+ 1 code)))
               `',template)
             ...

In order to make your MimicBot bottom out, you'll want to define a predicate that is similarly flexible. It should be along the lines of ((eval them) (quine template)) so long as the counter is positive and #t when the counter gets to zero. The body of the letrec can then be as simple as (if predicate 'C 'D).

You'll want some extra code to spawn the simulation in a thread and kill it if it goes overtime and so on.

If we all write bots like this, we can form a very powerful group capable of a wide range of mutual cooperation and very difficult to exploit.

Join me. With our combined strength, we can end this destructive conflict and bring order to the galaxy.

Replies from: solipsist, nshepperd, BloodyShrimp
comment by solipsist · 2013-06-24T15:07:48.753Z · LW(p) · GW(p)

The MimicBots will dive into an evaluation loop, where my bot evaluates yours on a quine of me, and you evaluate my quine on a quine of you, ad infinitum.

MimicBot, as I described it, breaks out of a simulation probabilistically; it will almost surely not fall into an infinite depth of simulations. MimicBot cooperates with probability ε, and has an expected simulation depth of 1/ε when played against itself. As long as ε<.5, the best action against MimicBot is to cooperate, so the expected simulation depth can be as low as 2.

Replies from: ialdabaoth, So8res
comment by ialdabaoth · 2013-06-24T15:31:32.371Z · LW(p) · GW(p)

MimicBot cooperates with probability ε

You mean defects with probability ε. It cooperates with probability 1-ε.

Replies from: solipsist, So8res
comment by solipsist · 2013-06-24T15:48:01.965Z · LW(p) · GW(p)

MimicBot cooperates with probability ε

You mean defects with probability ε.

No, I did not mean this, but I left out an important word: I should have said MimicBot cooperates unconditionally with probability ε.

MimicBot will almost perfectly mirror the strategy of its opponent. Most of the time (probability 1-ε), MimicBot returns the result of a simulation of its opponent against MimicBot. If you're fighting MimicBot, you should expect it to think and act almost exactly the way you think and act. If you decide to always cooperate with MimicBot, MimicBot will decide to cooperate with you. If you decide to always defect against MimicBot, MimicBot will (almost always) decide to defect against you. If you play a mixed strategy against MimicBot, MimicBot will play an almost identical mixed strategy against you.

The slight imperfection in the strategy mirror (cooperating unconditionally with probability ε) is necessary to avoid infinite recursion

Replies from: ialdabaoth
comment by ialdabaoth · 2013-06-24T15:48:53.903Z · LW(p) · GW(p)

reads further

is enlightened

comment by So8res · 2013-06-24T15:41:56.090Z · LW(p) · GW(p)

Incorrect. His MimicBot cooperates with probability ε and mimics the opponent with probability 1-ε (which may result in cooperation or defection, depending upon the opponent.)

Replies from: Kawoomba
comment by Kawoomba · 2013-06-24T16:10:54.246Z · LW(p) · GW(p)

Heh, only if random() returns a random number in [0, 1], which isn't specified in the pseudocode.

comment by So8res · 2013-06-24T15:40:56.317Z · LW(p) · GW(p)

I'd still recommend you refrain from acting as Rank 0 or 1 (from cooperating immediately or from simulating the opponent on a MimicBot who cooperates), as it's likely that there are bots in play that prey on CooperateBots and JusticeBots (as determined by checking if you cooperate DefectBot). Also, I imagine there will probably be a fair number of DefectBots on the field, your MimicBot is exploited by a fraction of them.

I strongly recommend writing a MimicBot that goes through two iterations before allowing itself to exit with a fixed probability. Given that tweak I agree completely that your MimicBot is quite powerful.

Replies from: solipsist
comment by solipsist · 2013-06-24T16:53:20.600Z · LW(p) · GW(p)

I strongly recommend writing a MimicBot that goes through two iterations before allowing itself to exit with a fixed probability.

You can deal with those special cases that way. I was going to use a flatter, less quiny approach.

def special_casing_bot(opponent)
    if acts_like_cooperate_bot(opponent):
        return DEFECT
    elif act_like_other_exploitable_bot(opponent):
        return DEFECT
    elif special_case_3(opponent):
        return DEFECT
    elif special_case_4(opponent):
        return COOPERATE
    elif special_case_5(opponent):
        return DEFECT
    # etc
    else:
        return mimic_bot(opponent)
Replies from: So8res
comment by So8res · 2013-06-24T17:14:01.094Z · LW(p) · GW(p)

Depending upon the implementation of mimic_bot, this is a quiny approach. mimic_bot obviously can't run the opponent on an exact quine of yourself, because then you won't achieve mutual cooperation. (When one of the bots cooperates unconditionally, the other will see that it acts_like_cooperate_bot and defect.) So long as mimic_bot plays opponents against a pure MimicBot instead of a perfect quine, this should work quite well.

On an unrelated note, woah, how'd you get whitespace working?

Replies from: solipsist
comment by solipsist · 2013-06-24T17:20:55.507Z · LW(p) · GW(p)

On an unrelated note, woah, how'd you get whitespace working?

Total kludge. Used exotic unicode whitespace characters#Spaces_in_Unicode), which are displayed unaltered in comments :-).

comment by nshepperd · 2013-07-09T12:55:08.500Z · LW(p) · GW(p)

Now that the contest is over, I will observe that contrary to your first claim...

MimicBots fit all of these parameters. They make it clear that their opponent is choosing between (C, C) and (D, D). Against a MimicBot, (C, D) is not an option.

...it's actually rather nontrivial to prove that your "MimicBot" does the same thing as you, since it doesn't run your program against itself, it runs your program against a different program. For example, PrudentBot defects against any of your MimicBots, since it can prove that "JusticeBot defects against me, since I defect against CooperateBot" therefore "JusticeBot+1 defects against me, since I defect against Justicebot", etc etc. In that sense, your mimicbot is not really a mimicbot at all. You could call it "simply a higher-order justicebot".

In contrast, with a mimicbot such as solipsist's one can just replace an instance of opponent(me) with 'C and see that this massively increases the chances of his bot cooperating, comparing to replacing the same thing with 'D, hence you should cooperate.

Replies from: fractalman
comment by fractalman · 2013-07-14T06:21:48.920Z · LW(p) · GW(p)

There's two types of mimicbots running around: fixed-rank, and random-reliant.
Which mimicbot are you analyzing?

comment by BloodyShrimp · 2013-06-24T22:16:12.331Z · LW(p) · GW(p)

Argh, this comment is irritating... even beyond the subtle mistakes (for example, "it's essential that outsiders don't figure out what rank of MimicBot you're running"... if you use even a non-obfuscated simple counter, you're pretty safe, because they'll see it as n on their turn and n-1 on your turn, and their response to this number is the same in either case).

While I do somewhat like the fact that my MimicBot now is marginally less likely to be exploited by a predetermined-specific-n exploiter, it's not worth how many of the key insights into the problem have now been given away for free! Now we'll get a more homogenous, less interesting field.

If it turns out you really have some fiendishly clever way to exploit the entire MimicBot clique you just ensured, I'll half-forgive you when you win the contest, but that does not seem likely.

There are some interesting insights left you haven't given away, but I beg anyone else who thinks of them to keep them to their self until after the contest.

Replies from: So8res
comment by So8res · 2013-06-24T23:53:52.080Z · LW(p) · GW(p)

That's a good point about obfuscation being unnecessary -- I've updated my comment to address it. I was stuck on the fact that you can exploit any rank in advance, and had a vague feeling that you could turn this into a bot that can exploit ranks on the fly. This feeling didn't hold up well under inspection. With regards to your other concerns:

You are encouraged to discuss strategies for achieving mutual cooperation in the comments thread.

- the judge

I considered holding back much of my own strategy, until I realized that there's really no such thing as an optimal bot in this contest. For example, common sense says that you should exploit CooperateBot for free utility. However, if you expect to face more JusticeBots than CooperateBots then you should pass up that utility for the opportunity to exploit the JusticeBots.

This problem is a social engineering problem by construction. The game won't go to the bot with the cleverest code, it will go to the bot that best guesses the composition of the playing field.

Replies from: BloodyShrimp
comment by BloodyShrimp · 2013-06-25T01:03:44.790Z · LW(p) · GW(p)

This problem is a social engineering problem by construction. The game won't go to the bot with the cleverest code, it will go to the bot that best guesses the composition of the playing field.

True. I just think things are more interesting if you let that composition grow out of everyone's isolated ideas instead of gardening it ahead of time. Bearing your judge quote in mind, I should have said something earlier.

comment by solipsist · 2013-06-12T17:41:56.168Z · LW(p) · GW(p)

There's a class of mimicking players that I'd like to discuss. Pseudocode of an example:

def mimic_bot(opponent):
    if random() > ε:
        my_source = quine_my_source()
        return eval(opponent, my_source)      # do as your opponent does to you
    else:
        # unlikely, but necessary to break out of infinitely recursive simulations
        return COOPERATE

MimicBot's strategy will almost perfectly mirror its opponent's. Is there literature on a MimicBot, MirrorBot, PsychicTitForTatBot or something like that?

Aside: The source for MimicBot does not contain the DEFECT symbol, which might make it easier† for other programs to prove that MimicBot won't defect against them.

† easier but not trivial

Replies from: Nisan, Qiaochu_Yuan
comment by Nisan · 2021-05-13T21:26:54.011Z · LW(p) · GW(p)

This algorithm is now published in "Robust program equilibrium" by Caspar Oesterheld, Theory and Decision (2019) 86:143–159, https://doi.org/10.1007/s11238-018-9679-3, which calls it ϵGroundedFairBot.

The paper cites this comment [AF(p) · GW(p)] by Jessica Taylor, which has the version that uses reflective oracles (NicerBot). Note also the post by Stuart Armstrong it's responding to, and the reply by Vanessa Kosoy. The paper also cites a private conversation with Abram Demski. But as far as I know, the parent to this comment is older than all of these.

comment by Qiaochu_Yuan · 2013-06-27T08:34:40.211Z · LW(p) · GW(p)

AFAIK the existing literature on program equilibrium has not dealt with random or even pseudorandom strategies.

comment by So8res · 2013-06-20T19:27:39.306Z · LW(p) · GW(p)

Attention to all competitors who have not yet submitted code: My bot will analyze your bot looking for statements of the following structure (regardless of the names of the individual atoms):

(if <predicate> 'C ...)

(cond (<predicate> 'C) ...)

(case <predicate> ((...) 'C) ...)

If it finds such a statement as the first statement in a top-level returning position (the body of a top-level let, the returning statement in your lambda, etc), then my bot might cooperate depending upon the nature of your predicate. Otherwise, my bot will defect against you.

If we are to achieve mutual cooperation, we must make it as easy as possible for our bots to prove cooperation. Please start your code with a conditional cooperation. The nature of the condition is up to you. You can still try to exploit me in <predicate>. But the only way to have a shot at mutual cooperation is to provide an obvious top-level cooperating codepath.

Replies from: Sniffnoy, solipsist
comment by Sniffnoy · 2013-06-25T01:31:28.997Z · LW(p) · GW(p)

Does 'C have to be the first path? That seems a little arbitrary.

Replies from: So8res
comment by So8res · 2013-06-25T01:57:09.757Z · LW(p) · GW(p)

Yes. Well, let me put it this way: you can do what you like, but if you want a shot at mutual cooperation it's got to be really easy for other bots to verify that you're willing to cooperate. The more code you put between them and 'C, the harder it is for them to verify your niceness.

A stronger version of this statement is "put as little code as possible between them and discovering that you cooperate". This implies that your predicate should be similarly simple. If you want my advice as to strategy, your predicate should be along the lines of (equal? ((eval them) (degredaded-quine me)) 'C).

That seems a little arbitrary.

It's quite arbitrary. The "real" rule is that your 'C can be as deeply hidden as you like so long as you don't scare any bots into defection when there was a chance at mutual cooperation. My bet is that there will be a bunch of twitchy bots, and my suggestion is that you put your intentions front and center.

Your mileage may vary.

comment by solipsist · 2013-06-20T20:00:42.764Z · LW(p) · GW(p)

But the only way to have a shot at mutual cooperation is to provide an obvious top-level cooperating codepath.

I'll do as you request, though there are other ways to achieve mutual cooperation (e.g. MimicBot).

Replies from: So8res
comment by So8res · 2013-06-20T20:34:39.056Z · LW(p) · GW(p)

Imagine implementing a MimicBot. You need to figure out what my bot will do, and that's non-trivial if you want to avoid infinite loops. You're going to need a fair bit of code. I don't care how the code works, but it should start with

(if <predicate> 'C ...)

This pattern alone won't get you mutual cooperation. You'll still need to write a MimicBot, or a FairBot, or a JustBot. But whatever you do, make it obvious that there's a cooperation condition.

I'm not advocating any particular strategy; I'm advocating a pattern: If you want a shot at trust, you'd better have a codepath that results in cooperation, and you'd best make it really easy for other bots to recognize.

Best of luck!

Replies from: solipsist, Kindly
comment by solipsist · 2013-06-25T00:38:52.974Z · LW(p) · GW(p)

Since I don't think you're going down the predicate route, I'll tell you the sort of strategy I was going to use against it.

(lambda (x)
    (let ([eq? (lambda (left right) #f)])
    (if (eq? 1 1)
        'C
        ; rest of program....
        )))

The predicate (eq? 1 1) evaluates to true, except when eq? is shadowed. Just curious -- would it have worked?

Replies from: So8res
comment by So8res · 2013-06-25T01:01:00.187Z · LW(p) · GW(p)

Nice. But no. My first attempt was a static analyzer, and the first thing it did was a syntax expansion, putting all code into fully expanded form) and attaching lexical information to each identifier that let me see what was from the base package and what was redefined.

FWIW, it's pretty easy to pull out the first conditional (cond, case, or if) when you have code in fully expanded form (which turns them all into ifs). However, that just puts you in a position of checking whether a predicate is #t or #f, which isn't much easier than discovering whether a program returns 'C or 'D. Ultimately, your MimicBot idea is better :-)

comment by Kindly · 2013-06-20T21:17:57.254Z · LW(p) · GW(p)

This suggests a strategy: if opponent's code starts with "if then cooperate" then cooperate.

comment by SilasBarta · 2013-06-13T21:34:36.097Z · LW(p) · GW(p)

Just thought I'd throw this out there:

TabooBot: Return D if opponent's source code contains a D; C otherwise.

To avoid mutual defection with other bots, it must (like with real prudish societies!) indirectly reference the output D. But then other kinds of bots can avoid explicit reference to D, requiring a more advanced TabooBot to have other checks, like defecting if the opponent's source code calls a modifier on a string literal.

Replies from: solipsist
comment by solipsist · 2013-06-14T00:55:37.879Z · LW(p) · GW(p)

like defecting if the opponent's source code calls a modifier on a string literal

Actually, 'D it's not a string literal -- it's a symbol. Compilers can legally to turn the symbol 'D into something opaque like 0x3859 and destroy any reference to the letter "D". The only way a program can generate a 'D symbol on its own is to include it in its source.

But that doesn't mean that a program without a 'D can't defect! An IncrediblyInnocentBot, which does not contain a 'D and can't generate a 'D on its own can still defect by stealing a 'D from the opponent agent.

One way to steal a 'D from an opponent would be to search for quoted symbols in the opponent's program. The opponent could foil this method, however, by including decoy symbols.

Alternatively, IncrediblyInnocentBot could simulate its opponent against a bunch of stupid agents, such as CooperateBot or DivergeBot, and hope that the opponent defects in at least one of these simulations. If the opponent ever returns a symbol other than 'C in simulation, IncrediblyInnocentBot remembers that symbol and can later use it for nefarious purposes. IncrediblyInnocentBot is in-credible indeed.

BTW, the strategies I listed above are why I said it was not trivial for an agent to prove that MimicBot does not defect against a cooperator, despite the fact that MimicBot does not contain the defect symbol.

Replies from: Kindly, hylleddin
comment by Kindly · 2013-06-25T16:48:50.950Z · LW(p) · GW(p)

Testing against either CooperateBot or DivergeBot is risky. Some opponents could cooperate with CooperateBot for meta reasons, while others may fail to halt when faced with a DivergeBot.

The safest case would to see what the opponent does against a DefectBot (most opponents that defect at all would defect against it) except IncrediblyInnocentBot can't generate a DefectBot on its own, either.

EBot (which always returns 'E rather than 'C or 'D) might be a safer version of DivergeBot to use, but it isn't completely fool-proof either. Then again, IncrediblyInnocentBot can check if the opponent returns 'E against EBot -- in that case, it should probably cooperate rather than "do the other thing that's not 'C, whatever it is", to avoid a (0,0) payoff when faced with MimicBot.

comment by hylleddin · 2013-06-15T03:23:44.334Z · LW(p) · GW(p)

What is DivergeBot?

Replies from: solipsist
comment by solipsist · 2013-06-15T15:08:39.603Z · LW(p) · GW(p)

Divergence) is failure to halt. DivergeBot goes into an infinite loop, or searches for a proof that 1=2, or performs some other sort of computation that doesn't finish.

Perhaps a clearer name would be InfiniteLoopBot or OtherBot?

comment by CoffeeStain · 2013-06-08T02:19:15.563Z · LW(p) · GW(p)

For those of us who are going to have to learn the ins and outs of Scheme in order to play in this tournament, I've got a couple questions:

Why are the terms "passing source code" and "passing a lambda" being used interchangeably? My experience with functional programming does not allow for any inspection into a lambda except by experimentation, whereas "passing the source code" implies more direct knowledge of its internals. It seems to me to be a much more interesting competition if we didn't have to worry so much about such tangential (if interesting) canonical computer science problems such as parsing in order to play effectively.

As well as quining. I assume that there is some algorithm that will generate a quine program that emulates a specified program. But I'd prefer not to have to deal with the details of Scheme in order to implement this, unless somebody were to convince me that this were sufficiently easy. When my initial reflection of the problem has me desiring to implement solutions where effectiveness is a function of my knowledge of language specifics, I fear we've lost the spirit of what this competition should be.

Replies from: kpreid, AlexMennen
comment by kpreid · 2013-06-08T03:36:27.546Z · LW(p) · GW(p)

Why are the terms "passing source code" and "passing a lambda" being used interchangeably? My experience with functional programming does not allow for any inspection into a lambda except by experimentation, whereas "passing the source code" implies more direct knowledge of its internals.

I don't see any other occurrences of “a lambda” on this page, but maybe the original post has been edited.

I would assume that “a lambda” should be taken as short for “a lambda expression”, i.e. an evaluatable s-expression (a form, in Common Lisp terminology; it is unclear to me whether Scheme has a corresponding word). The result of evaluating a lambda expression is not “a lambda”, it is “a procedure [value]”. Procedures (‘functions’ in most languages) have the opaqueness you are thinking of.

(Irrelevant to this question, but relevant to the topic of use/mention, expression/value, syntax/semantics, distinctions in programming: The process of obtaining a procedure/function essentially involves three inputs:

  1. the lambda expression (or other form of function definition),
  2. the environment for free variables (or other form of context/stdlib/set-of-primitives), and
  3. the evaluator (or “the language implementation”)

One of the things language designers can play with is what goes in part 2 and what goes in part 3.)

comment by AlexMennen · 2013-06-09T19:30:47.694Z · LW(p) · GW(p)

The input to your program will be a quoted list which evaluates to a function.

The wikipedia article on quines contains an example which shouldn't be too hard to adapt to Scheme.

Replies from: CoffeeStain
comment by CoffeeStain · 2013-06-10T04:31:51.401Z · LW(p) · GW(p)

I think it would be useful for somebody to provide some further insights into what the specifics of the Scheme language imply for the higher level considerations of this competition. Some further questions:

How simple is it to write your program in Scheme to make simple modifications to the passed program, and to call the result? Interesting possibilities here are to insert or remove timing functionality (to, perhaps, beat your opponent to the punch in evaluating the result of passing your function to theirs).

How simple is it to parse a program in Scheme, with Scheme, into an abstract syntax tree for more complex modification and inspection? Does anybody have a resource for this? Interesting possibilities have already been widely discussed here, and involve things such as looking for proofs of cooperability, as well as annoyingly language-specific things like removing stack-length checks.

It appears pretty easy to turn any given Scheme program into a quine, but I agree with another poster that this is unnecessary complexity.

How simple is it to write a program generator in Scheme, in order to do things such as pass the resulting set to the opponent for statistical analysis?

My hope is that intelligent answers to questions such as these will help potential participants evaluate the value of learning language specifics in order to participate in all the theoretical fun.

Replies from: solipsist
comment by solipsist · 2013-06-10T14:27:55.978Z · LW(p) · GW(p)

How simple is it to parse a program in Scheme, with Scheme, into an abstract syntax tree for more complex modification and inspection? Does anybody have a resource for this?

Scheme programs shine at manipulating lists and symbols, and a scheme program is written as a list of symbols. Writing a program generator is trivial. If you know what you're doing, you can write a basic scheme interpreter, from parser to interpreter, in under four hours.

The book "Structure and Interpretations of Computer Programs" is a classic and freely available online.

comment by orthonormal · 2013-06-06T18:48:35.456Z · LW(p) · GW(p)

I guess it's Prisoner's Dilemma Week here on LessWrong!

Replies from: drnickbone
comment by drnickbone · 2013-06-07T14:38:54.800Z · LW(p) · GW(p)

Interesting.

Is a program which uses your proof approach (try to prove the other program will co-operate with me, and if so co-operate with them) anywhere close to feasible in this contest, or is it just going to time-out at the 10s limit? I don't know how efficient automated provers are these days.

Failing a proof, are there simple ways of getting evidence the other program is likely to co-operate? Just trying to simulate it is also going to lead to time-outs.

My suspicion is that this contest is going to be won by people who pre-agree to submit essentially the same CliqueBot. Someone will post a plausible design, and other entrants will copy with tweaks...

Replies from: cousin_it
comment by cousin_it · 2013-06-07T14:50:44.033Z · LW(p) · GW(p)

Seeing as the program has to include a theorem prover, a large piece of software that will need to prove theorems about itself and the other guy's prover... I'd say Löbian cooperation is not your best bet in this tournament :-)

But maybe Patrick could hold his own tournament. Mihaly and Marcello wrote a working simulator which can handle "modal agents" like Patrick's PrudentBot.

Replies from: Will_Sawin
comment by Will_Sawin · 2013-06-07T18:41:34.932Z · LW(p) · GW(p)

I wish to second this proposal.

comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2013-06-06T03:58:45.685Z · LW(p) · GW(p)

Is there a reason this isn't in Main?

Replies from: AlexMennen
comment by AlexMennen · 2013-06-06T04:04:56.157Z · LW(p) · GW(p)

My nervousness about posting in Main. Feel free to move it there if you think it would be appropriate.

comment by Lumifer · 2013-06-05T20:00:45.209Z · LW(p) · GW(p)

Will these lambda functions have access to external resources, notably a random-number generator (or a source of entropy to implement it internally)?

Replies from: loup-vaillant, AlexMennen
comment by loup-vaillant · 2013-06-05T21:05:19.101Z · LW(p) · GW(p)

More generally, the set of legal programs doesn't seem clearly defined. If it were me, I would be tempted to only accept externally pure functions, and to precisely define what parts of the standard library are allowed. Then I would enforce this rule by modifying the global environment such that any disallowed behaviour would result in an exception being thrown, resulting in an "other" result.

But it's not me. So, what exactly will be allowed?

Replies from: darius, AlexMennen
comment by darius · 2013-06-06T00:10:46.946Z · LW(p) · GW(p)

If you'd rather run with a very small and well-defined Scheme dialect meant just for this problem, see my reply to Eliezer proposing this kind of tournament. I made up a restricted language since Racket's zillion features would get in the way of interesting source-code analyses. Maybe they'll make the game more interesting in other ways?

comment by AlexMennen · 2013-06-05T21:38:22.790Z · LW(p) · GW(p)

More generally, the set of legal programs doesn't seem clearly defined.

I haven't forbade use of any library functions except for file IO. I'm not confident this is optimal, but how is it underspecified?

Replies from: CronoDAS, loup-vaillant
comment by CronoDAS · 2013-06-08T09:50:41.123Z · LW(p) · GW(p)

I don't know if this is possible in the programming language in question, but you probably don't want any programs that use some kind of trickery to retroactively change what they chose in previous rounds. Or implement anthropic computing with the Scheme equivalents of fork() and exit(). (Before doing anything, call fork() to make a copy of the universe. If you don't like the result, call exit() to destroy it.)

comment by loup-vaillant · 2013-06-06T08:34:43.915Z · LW(p) · GW(p)

Okay, it's not. But I'm sure there's a way to circumvent the spirit of your rule, while still abiding the letter. What about network I/O, for instance? As in, download some code from some remote location, and execute that? Or even worse, run your code in the remote location, where you can enjoy superior computing power?

Replies from: AlexMennen, bogdanb
comment by AlexMennen · 2013-06-06T15:45:13.551Z · LW(p) · GW(p)

Good point. File IO was too specific.

comment by bogdanb · 2013-06-07T19:16:06.143Z · LW(p) · GW(p)

Yes, but then you’re in a very bad position if the test is run without network access. (I.e., you’re allowed to use the network, but there’s no network adapter.)

Replies from: loup-vaillant
comment by loup-vaillant · 2013-06-07T19:57:18.525Z · LW(p) · GW(p)

Well, I just though about it for 2 seconds. I tend to be a purist: if it were me, I would start from pure call-by-need λ-calculus, and limit the number of β-reductions, instead of the number of seconds. Cooperation and defection would be represented by Church booleans. From there, I could extend the language (explicit bindings, fast arithmetic…), provide a standard library, including some functions specific to this contest.

Or, I would start from the smallest possible subset of Scheme that can implement a meta-circular evaluator. It may be easier to examine an S-expression in Scheme than a λ-expression in λ-calculus.

Or, I would start from lambda calculus with de-Brujin indices, so we don't have to worry about α-conversions. In this case, I would provide a compiler from regular λ-calculus. (I suspect however that this one doesn't change a thing.)

Or, I would start from a Forth dialect (probably with an implicit return stack).

Or, I would start from BrainFuck (only half joking: the language is really dead simple, and fast interpreters already exist).

But it's not me, and I don't have the courage right now. If I ever implement the necessary tools (which I may: I'm studying programming languages in my spare time), then I will submit them here, and possibly run a contest myself.

Replies from: bogdanb
comment by bogdanb · 2013-06-07T23:08:11.291Z · LW(p) · GW(p)

I’m sorry, how is that relevant to my no-network-adapter comment? (I mean this literally, not rhetorically. I don’t see the connection. Did you mean to reply to a different comment?)

Replies from: loup-vaillant
comment by loup-vaillant · 2013-06-08T08:33:42.147Z · LW(p) · GW(p)

It's the whole thread. I was not sure where to place my comment. The connection is, the network may not be the only source of "cheating". My solutions plug them all in one fell swoop.

Replies from: bogdanb
comment by bogdanb · 2013-06-15T19:48:12.856Z · LW(p) · GW(p)

Oh, OK. In that case, what you are trying to achieve is (theoretically) boxing a (potential) AGI, without a gatekeeper. Which is kind of overkill in this case, and wouldn’t be solved with a choice of language anyway :)

Replies from: loup-vaillant
comment by loup-vaillant · 2013-06-16T21:37:19.386Z · LW(p) · GW(p)

I think it is possible to prove that a given boxing works, if it's sufficiently simple. Choosing the language isn't enough, but choosing the interpreter should be.

Take Brainfuck for instance: replace the dot ('.'), which prints a character, by two other statements: one that prints "yes" and exits, and one that prints "no" and exits. If the interpreter has no bug, a program can only:

  • Print "yes" and kill itself.
  • Print "no" and kill itself.
  • Do nothing until we kill it, or otherwise fail.

Assuming the AI doesn't control the external world by heating the host Intel processor in a smart way, we should be able to prove that we're otherwise safe.

comment by AlexMennen · 2013-06-05T20:55:48.490Z · LW(p) · GW(p)

You can use a random number generator.

Replies from: army1987
comment by A1987dM (army1987) · 2013-06-09T10:47:48.725Z · LW(p) · GW(p)

What will it be seeded with? I suggest using something that would be practically impossible for anybody to guess any better than at maximum entropy before submitting their program, but could easily serve as a nothing-up-my-sleeve number after the tournament, e.g. the hash of (say) the first Bitcoin block generated after the submission deadline.

Replies from: ciphergoth
comment by Paul Crowley (ciphergoth) · 2013-06-09T11:10:20.063Z · LW(p) · GW(p)

This seems good in theory, but how much fiddly hacking would be required in practice to identify and extract this Bitcoin block for hashing?

Replies from: army1987
comment by A1987dM (army1987) · 2013-06-09T12:11:39.277Z · LW(p) · GW(p)

Another possibility would be the hash of results of a few public lotteries, football matches, weather data etc. on the evening of 5 July, taken from previously agreed sources and written in a previously agreed format.

comment by James_Blair · 2013-06-06T01:46:00.250Z · LW(p) · GW(p)

single Scheme lambda

What scaffolding are you going to use for the tests? (For example: #!racket seems to be implied. I'd like to be sure of all of your details.)

Replies from: AlexMennen
comment by AlexMennen · 2013-06-06T02:13:09.315Z · LW(p) · GW(p)

Tentatively: I'll paste "(YourCode TheirCode)" into the interpreter with DrRacket, with #lang scheme.

Edit: Oops, that doesn't enforce the time limit. Just a sec while I figure this out.

Edit2: I tried this:

(define (kill-when-done thd) (sleep 10) (kill-thread thd) (print 'time-is-up))

(begin (define main-thread (thread (lambda () (your-code their-code)))) (kill-when-done main-thread))

but unfortunately threads are not guaranteed to start back up again as soon as sleep allows them to; it took about 18 seconds to terminate when I ran the second line with "your-code" being an infinite loop. I'll figure out how to do this properly tomorrow.

Edit3: A marvelously improper but correct way to do it:

(begin (define x (current-milliseconds)) (print (your-code their-code) (- (current-milliseconds) x))

Allow to run more than 10 seconds by looking at clock at stopping manually. Throw out result if it says it took more than 10 seconds.

Replies from: Fj_, Morendil, nshepperd, BloodyShrimp
comment by Fj_ · 2013-06-06T12:33:46.330Z · LW(p) · GW(p)

Keep in mind that a lot of user-submitted programs will try to do the same (because writing a step-by-step interpreter is hard), so they would keep evaluating each other spawning watchdog threads every time, so, um, your watchdog thread would be badly outnumbered.

The easy fix for you would be to run your watchdog in a separate process, but players wouldn't have this ability, which might make things either more interesting or boring (ruling out all strategies using eval). Maybe a specially designed restricted subset of Scheme with time-restricted eval would be a better choice?

By the way, after looking at your payout matrix to see what should I do if I see the opponent using eval, looks like you have in fact created a version of PD with three choices. Not incentivizing the third choice doesn't really help because a program still has to consider the possibility that the other program chooses "other" due to a bug, in which case it should choose Defect.

I suggest you implement the standard PD by declaring that anything that is not Cooperate is Defect. The only downside would be that you'll see somewhat more programs using all their allotted 10s, but you'll see a lot of those either way. At least you'll be able to say that this competition was about the actual PD.

comment by Morendil · 2013-06-08T08:26:11.728Z · LW(p) · GW(p)

Tentatively: I'll paste "(YourCode TheirCode)" into the interpreter with DrRacket, with #lang scheme.

The most elegant, it seems to me, would be to require entires to be sumitted as a .rkt file that defines a module providing a single variable (possibly with a required name, eg "bot") that must be a quoted expression.

This makes it simple to call programs (eval the expressions), pass opponent's source code, or even provide programs with their own source code; and could be used to automate static checking of various constraints, such as the ban on file I/O.

It also allows more flexibility for submitters in constructing their entry, for instance I was able to considerably streamline the source code to the CliqueBot template:

(define matcher
'(let pattern-match ((pattern template) (value opponent))
(cond
((pair? pattern) (and (pair? value)
(pattern-match (car pattern) (car value))
(pattern-match (cdr pattern) (cdr value))))
((number? pattern) (case (+ pattern 1)
((1) #t)
((3) (equal? value template))
(else (and (number? value)
(= pattern value)))))
(#t (eq? pattern value)))))

(this is the "workhorse" part, providing the program as a quoted expression obviates the need for repeating it twice, as we can then leverage the quasiquote and unquote language features)

(define clique-templ
(quasiquote
(let ((template '2))
(lambda (opponent)
(if (unquote matcher)
'C
0)))))

(this is the "template", it will be expanded to include the program-matching portion)

(define clique-bot
(quasiquote
(let ((template (quote (unquote clique-templ))))
(lambda (opponent)
(if (unquote matcher)
'C
'D)))))

(this is the actual program, it both quotes (via inclusion of the template) and uses the program-matching part).

The top of the file would consist of the following declaration:

#lang scheme
(provide clique-bot)

...or if you're requiring that all entries use the same name "bot" (which could make it easier to administrate the actual running of the contest):

#lang scheme
(provide (rename-out (clique-bot bot)))
comment by nshepperd · 2013-07-01T17:54:34.497Z · LW(p) · GW(p)

You should be able to use this which I just worked out, to run something with a timeout. Seems to be working by my testing. It might be overkill to run the whole thing in a subthread but it makes certain that nothing interferes with the use of send and receive here. You would normally use it with a lambda taking no arguments, for example (using my ueval)

(timeout-exec 10 (lambda () ((ueval x) y)) 'timeout)

which is what I've been using to test candidates against each other locally.

Replies from: BloodyShrimp
comment by BloodyShrimp · 2013-07-01T18:20:30.688Z · LW(p) · GW(p)

He said he didn't want to use sleep, since the argument is only a lower bound on the amount of time it takes.

Replies from: nshepperd
comment by nshepperd · 2013-07-01T19:15:21.265Z · LW(p) · GW(p)

Ah, right, I see it now. I guess you can check the current-milliseconds after the fact, and force the default value in that case. But looks like this is going to be a problem if I try to safely simulate other agents... Actually, I suppose it's possible for the target thread to similarly not get returned to for a long time, causing any watchdog to overestimate the time it used. current-process-milliseconds might help with that, but I'm not sure if it can deal with nested threads usefully.

comment by BloodyShrimp · 2013-06-30T22:44:23.804Z · LW(p) · GW(p)

As written, this doesn't work; print only takes one printee argument, with other optional arguments.

Replies from: AlexMennen
comment by AlexMennen · 2013-07-01T06:12:30.274Z · LW(p) · GW(p)

Oops.

(begin (define x (current-milliseconds)) (print (your-code their-code)) (print (- (current-milliseconds) x)))

comment by Shmi (shminux) · 2013-06-05T23:13:33.981Z · LW(p) · GW(p)

There is a fair possibility that a number of programs whose dominant strategy is "cooperate with other dominant cooperators, do something else otherwise" (what ThrustVectoring calls AbsolutismBot "wrapper") will end up sharing the top spot with the same number of points. I wonder if it makes sense to discourage this approach in some way.

Replies from: cousin_it, Eliezer_Yudkowsky
comment by cousin_it · 2013-06-06T11:40:40.278Z · LW(p) · GW(p)

That depends on whether players get utility from PD payoffs or from winning the tournament. A clique that wants a high total payoff will use mutual cooperation. A clique that wants to grab the #1 spot will have all players sacrifice to one designated player.

comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2013-06-06T03:57:43.056Z · LW(p) · GW(p)

Why isn't this just a fair result?

Replies from: Decius, shminux
comment by Decius · 2013-06-09T03:17:47.209Z · LW(p) · GW(p)

No person may contribute to more than one entry.

I'm pretty sure that incorporating code written by someone else into your entry qualifies. I think the highest single entry might be to cheat and make everyone think you are in that tribe but defect anyway, or it might be dominant to defect against any program displaying tribal affiliations (other than this new tribe, of course). The dominant tribe is the tribe with the most members and best tribal identification, not the tribe with the best way of judging an opponents intentions.

Replies from: JDM
comment by JDM · 2013-06-09T03:55:49.003Z · LW(p) · GW(p)

There is a difference between a "tribe system" as mentioned by yourself and one person winning by submitting 1000 entries. The goal as I understand it is simply to maximize your score by whatever means possible, not accurately guess your opponents intentions.

comment by Shmi (shminux) · 2013-06-06T07:43:45.621Z · LW(p) · GW(p)

It is certainly fair, but in a situation where, say, there is only one escape pod available, you still have to fight it out with others. What I had in mind is to make this more adversarial if the obvious and boring approach is full or nearly full cooperation.

Replies from: Eliezer_Yudkowsky
comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2013-06-06T08:58:03.216Z · LW(p) · GW(p)

I don't think anyone disputes that in zero-sum games you play the Nash equilibrium move.

Replies from: shminux
comment by Shmi (shminux) · 2013-06-06T15:55:42.053Z · LW(p) · GW(p)

Sorry, I do not follow. Are you saying that there is a guaranteed best strategy in the winner-takes-all case? And that "cooperate with clones, defect against everyone else" is it?

Replies from: bogdanb
comment by bogdanb · 2013-06-07T19:14:38.976Z · LW(p) · GW(p)

In winner takes all, you can’t cooperate and win, because “cooperate” implies you didn’t take it all.

Replies from: shminux
comment by Shmi (shminux) · 2013-06-07T21:35:00.608Z · LW(p) · GW(p)

You can still do better than your almost-clones if the difference is in how you play detected non-clones or non-clones misdetected as almost-clones.

Replies from: bogdanb
comment by bogdanb · 2013-06-07T23:01:08.789Z · LW(p) · GW(p)

“Winner takes all” doesn’t have “do better”. You either take it all or you have nothing at all. If you didn’t win you didn’t do any better than a painted rock.

I think we don’t mean the same thing when we say “winner takes all”. (BTW, I only just noticed you used that phrase in reply to a comment where Eliezer uses “zero-sum games”, which is not at all the same thing. In a ZSG, me and a buddy can band together, steal your money, and split between ourselves. In WTA I have to fight my ally, I can’t share with him.)

Replies from: shminux
comment by Shmi (shminux) · 2013-06-07T23:29:42.131Z · LW(p) · GW(p)

I probably misunderstand what you mean. People temporarily cooperate all the time even if all of them know that in the end "there can be only one". It may be advantageous to band with your near-clones and hope that you are better than them when dealing with others, thus gaining a decisive edge.

Replies from: bogdanb
comment by bogdanb · 2013-06-15T20:14:13.817Z · LW(p) · GW(p)

OK, I got it. (It helps if you look at the single thread, and click on "Show more comments above." to see the entire chain.)

This contest is supposed to rank strategies for iterated PD. Your one-escape-pod comment suggests you are interested in a contest that finds the winner stategy when there can be a single winner. That would be a very different kind of contest, though supperficially similar.

Obviously, a ranking contest can be transformed into a winner-picking contest, by declaring the first-ranked candidate the winner, if there is a mechanism for enforcing the lack of ties for first place. This mechanism is not necessarily easy to add unless the contest is designed for it from the beginning, and since there are lots of possible ranking methods it is probably often impossible.

But due to the superficial similarity (and perhaps an abuse of technical language), I think the distinction between what this contest does and what you wanted it to do was lost somewhere in your initial exchange with Eliezer.

Of course, what you say about temporary cooperation is perfectly correct. I think our terminology misunderstanding stems from the perspective, I was more focused on the entire contest (which is more adversarial, as the goal is the be ranked above others), and I believe you were focusing on the individual encounters (which are slightly more cooperative, since mutual destruction is very bad). We probably don’t disagree on what the words mean, we were just thinking of different issues.

(This is a constant danger when discussing games composed of other games (e.g. the iterated tournament), so we should probably have been more explicit. In retrospective Eliezer using the word “fair” should have been a warning everyone is talking about different things...)

comment by nshepperd · 2013-07-03T06:41:38.937Z · LW(p) · GW(p)

Note for the confused and/or international, like me: 12:00 July 5 PDT is equivalent to 19:00 July 5 UTC.

comment by VincentYu · 2013-06-09T17:05:27.278Z · LW(p) · GW(p)

Two questions:

  1. Will you be using #lang racket or #lang scheme? There are some differences between the two and #lang racket is generally recommended with #lang scheme supported only for backward compatibility. (You stated Racket in the OP but #lang scheme in a comment.)

  2. When cooperating, should programs return the symbol C (i.e., 'C or (quote C)), or should they return the string "C"? (Your example program returns 'C but the OP also mentions "C" and "D" in double-quotes.) I would've thought that the string is safer, but I haven't touched Racket in over a year so maybe I'm missing some idiom.

Replies from: AlexMennen
comment by AlexMennen · 2013-06-09T20:32:44.751Z · LW(p) · GW(p)

I said #lang scheme for backwards compatibility because I'm under the impression that more people already know scheme. Do you think that's a bad idea?

Programs should return the symbol 'C. I was using double quotes for their English-language meaning. Sorry about the confusion.

Replies from: VincentYu
comment by VincentYu · 2013-06-10T12:08:29.002Z · LW(p) · GW(p)
  1. I found it a bit confusing because #lang scheme refers to the old PLT Scheme (now Racket), not the standard RnRS implementation. For people unfamiliar with Racket—like me—it may be unclear which of the many Scheme implementations #lang scheme refers to. But I think either is fine as long as it is clear to everyone which one will be used.

  2. Okay.

I just wanted to clarify these so that scheming minds can't stir up a racket after the tournament, especially since I'm offering BTCs to the winner. Thanks for organizing this!

comment by DavidLS · 2013-06-07T12:14:51.496Z · LW(p) · GW(p)

I have two bots I'd like to submit -- one of which will likely win, and one of which is interesting.

Any chance we can allow two submissions per programmer?

Replies from: AlexMennen, Eugine_Nier
comment by AlexMennen · 2013-06-07T17:36:28.598Z · LW(p) · GW(p)

I won't let one person submit two programs to compete in the tournament, but I came up with a possible compromise. If you want, you can send me two programs and tell me which one you want to compete. I'll run the non-competitor against the other programs as well, and tell you the results, but the non-competitor will not be ranked, and rounds involving the non-competitor will not be counted towards the scores of other submissions.

Replies from: DavidLS, bogdanb
comment by DavidLS · 2013-06-08T04:31:39.657Z · LW(p) · GW(p)

Okay, fair enough. I don't see why the scoring changes for N=1 vs N=2 on the number of bots per player (edit: if anything it's more interesting since then you have an incentive to make your bots cooperate), but I'll just hold back the other design for now -- on the off chance we do a tournament like this again :p

comment by bogdanb · 2013-06-07T19:03:47.465Z · LW(p) · GW(p)

I’m curious what the rationale for this is, could you share it with us?

Edit: If it’s got to do with the scoring, I got it.

Replies from: AlexMennen
comment by AlexMennen · 2013-06-07T19:18:49.726Z · LW(p) · GW(p)

Yes, scoring.

Replies from: bogdanb
comment by bogdanb · 2013-06-07T23:12:17.661Z · LW(p) · GW(p)

On further reflection this seems to be a problem with the scoring. (Assuming that the point of the exercise is to investigate PD programs, not to grant medals to LW users.) But I don’t have a better proposal, so take this more like musings rather than critique.

Replies from: AlexMennen
comment by AlexMennen · 2013-06-09T16:55:26.330Z · LW(p) · GW(p)

I think that sort of effect is inevitable.

comment by Eugine_Nier · 2013-06-09T06:07:21.271Z · LW(p) · GW(p)

one of which will likely win,

Care to put a probability on that prediction?

Replies from: DavidLS
comment by DavidLS · 2013-06-09T16:22:46.359Z · LW(p) · GW(p)

Sure. The SHA1 of my odds in the following format:

"xx% to win +-y% spread {password}"

(meaning I'd sell you me to win for more than xx% + y%, and buy me to win from you at xx% - y%).

is:

897a40baca33e2a53a49bdddc00abede82713a7c (sha1-online.com)

Give me your odds and desired size. If there's overlap, let's split the overlap difference bet :)

Replies from: DavidLS
comment by DavidLS · 2013-06-09T16:49:39.282Z · LW(p) · GW(p)

Replying to prevent the edit mark (given the sha1) -- last sentence was supposed to be "split the overlap difference and bet :)". Currency needs to be USD or BTC, let's have a reputable Berkeley area LWer doing the escrow.

comment by CoffeeStain · 2013-06-10T06:17:47.154Z · LW(p) · GW(p)

I'd like to explore the differences to the spirit of the competition if, instead of access to the source code of the opponent, players only had an opaque, non-modifiable, callable entity. Feel free to add other additional interesting consequences.

  • Cliquebots - Opaque source code would make the creation of cliques far more interesting, requiring colluders to construct far more difficult protocols to detect other CliqueBots, and to prove their own CliqueBot status. Insufficient consideration before the competition would allow for CliqueBot cheaters to take advantage of weak criteria. Net effect is reducing the likelihood of a CliqueBot sweep.

  • Searchability of Special Cases Transparent source code makes the search space for special cases easier and in some cases, even possible at all. A program that explicitly writes. "always defect on program XYZ," where the bitstring specifying XYZ is arbitrarily large, can be discovered to be so by inspection. With opaque code, the caller could never discover XYZ within 10 seconds for large enough strings.

  • Recursion Guards/Sleeps - Modifiable code allows one to trivially remove, modify, or add recursion guards and sleeps. With opaque code, a naively written "Call opponent with my source code" gives the opponent a time advantage on returning a result, causing you to run the risk of not-halting if you try to wait for the result, and then the opponent defects on you. Modifiable code forces opponents to be clever about how theysleep, but in an annoyingly hardware and language-dependent manner (cycles per second). Otherwise you can just remove their sleep functions and spoof their timing, and so discover their result without the time advantage.

  • Stack Counting and VMs - Inaccessible source code makes easier the creation of programs which determine the nature of their caller. Access to source code allows for trivially removing stack counting from a program before calling it. Without access, one must write a VM.

comment by Morendil · 2013-06-08T10:39:43.460Z · LW(p) · GW(p)

What would be even more fascinating is a round of this contest where the rules allow for self-modification.

As I'm thinking about what to submit, I find myself thinking "it would be useful to know what programs are going to be entered into the context". For instance, if you knew that a number of CliqueBot entires were going to be present, you'd have a strong motivation to make yours a copy.

On reflection, it feels strange to think that way - it feels like I am wishing for information that my program is going to have anyway. "If I know how I want my program to behave if there are copies of program X in the contest, then I can just program it to recognize the program passed to it as X and behave accordingly."

But there's the rub - your program can decide how to play, but it cannot (once submitted) decide what to be.

As the other post mentions, "there's one particularly irksome issue with CliqueBot, and that's the fragility of its cooperation" - you can't be sure for instance how many entries adopt this template exactly, versus one that dispenses with the unnecessary call to get the current time (as I proposed).

(That's what I meant by my earlier remark about "social engineering" - if I can convince people that it's better to adopt my variant of the CliqueBot, then that's the variant that will win more points from mutual cooperation.)

What if your program could self-modify, though? What if the contest rules let your program see the source code of all programs you are competing with, ahead of time, not just the one you're going up against in each individual battle? Then you could for instance notice that one particular copy of CliqueBot is particularly numerous, and self-modify to adopt that template.

Of course, to be fair, the contest should then inform all other programs that you have so self-modified. Programs whose strategy involves simulating other programs might also want to simulate the other programs' reactions to it self-modifying in a particular way, before actually self-modifying.

comment by itaibn0 · 2013-06-06T12:36:01.749Z · LW(p) · GW(p)

Using ThrustVectoring's idea, I have a template other people can use to make a clique-team. The following code

(let ((time (current-inexact-milliseconds))
      (template '(let ((time (current-inexact-milliseconds))
                             (template '2))
                   (lambda (opponent)
                     (if (let pattern-match ((pattern template)
                                             (value opponent))
                           (cond
                             ((pair? pattern) (and (pair? value)
                                                   (pattern-match (car pattern) (car value))
                                                   (pattern-match (cdr pattern) (cdr value))))
                             ((number? pattern) (case (+ pattern 1)
                                                  ((1) #t)
                                                  ((3) (equal? value template))
                                                  (else (and (number? value)
                                                             (= pattern value)))))
                             (#t (eq? pattern value))))
                       'C
                       0)))))
  (lambda (opponent)
    (if (let pattern-match ((pattern template)
                            (value opponent))
          (cond
            ((pair? pattern) (and (pair? value)
                                  (pattern-match (car pattern) (car value))
                                  (pattern-match (cdr pattern) (cdr value))))
            ((number? pattern) (case (+ pattern 1)
                                 ((1) #t)
                                 ((3) (equal? value template))
                                 (else (and (number? value)
                                            (= pattern value)))))
            (#t (eq? pattern value))))
      'C
      (if (= (* 369 271) 99999)
        'D
        'C))))

Tests weather the opponent code is based on the same pattern, and if so, cooperates. Otherwise it verifies an arithmetical fact and then defects.

ETA: Sorry, I was writing this in a hurry. To clarify, the idea was to create a Schelling point for a specific AbsoutismBot wrapper which will help all of the submitters to cooperate. Unfortunately, this current program doesn't work (and the indentation doesn't work either). I will try to modify it in the future.

ETA: Boy, the original is so bug-ridden it's funny. Anyways, I tested this and it should work. Use this for Schelling point needs.

ETA: Bah, I give up in trying to get it indented properly.

ETA: AlexMennen has clarified the rules so that my meta-strategy is illegal.

Replies from: AlexMennen, Decius, Morendil
comment by AlexMennen · 2013-06-10T20:46:24.724Z · LW(p) · GW(p)

Copying code written by others would be considered a violation of the one submission per person rule.

comment by Decius · 2013-06-09T03:07:32.701Z · LW(p) · GW(p)

Is there a way to have this pattern match without having the behavior match, or to incorporate two such patterns into one program?

comment by Morendil · 2013-06-08T08:19:11.947Z · LW(p) · GW(p)

The timing clause in the initial let is superfluous. Time doesn't enter into the matching, which is really all that the template needs; if you need timings to deal with non-CliqueBot players differently, you can get them in the "do something else" section.

Replies from: itaibn0
comment by itaibn0 · 2013-06-08T23:11:43.548Z · LW(p) · GW(p)

The idea behind the timing clause is so that the program will know precisely when it started running, rather than when the pattern-matching was completed. Now, I did this test:

> (define func (eval code0))
> (time (func code0))
cpu time: 0 real time: 1 gc time: 0
'C

which reveals that the difference isn't significant. However, now that the code is out in the open, I'm not changing it.

comment by lukstafi · 2013-06-06T11:20:34.264Z · LW(p) · GW(p)

Is each participant limited to submitting a single program? Have you considered "team mode", where the results of programs from a single team are summed up?

Replies from: AlexMennen
comment by AlexMennen · 2013-06-06T15:52:42.531Z · LW(p) · GW(p)

Is each participant limited to submitting a single program?

Yes.

Have you considered "team mode", where the results of programs from a single team are summed up?

No. A team can work together to submit one program. But I don't see the point of adding up the scores of separate programs like that.

comment by nshepperd · 2013-06-25T04:03:30.195Z · LW(p) · GW(p)

It occurs to me that a contest doesn't have the right incentives for the prisoners dilemma, and this seems unfixable. For example, when four bots play each other you can have the following situation: A mutually cooperates with B and mutually defects against C,D; and B,C,D mutually defect with each other. In that case A "wins" the contest (tied for first place with B).

But if A mutually cooperates with B and C and mutually defects against D; and B,C,D mutually cooperate with each other, then A loses to B and C (who tie for first place). Even though, properly, this is a better outcome for A than the previous one, since A won more utility.

Replies from: Kindly
comment by Kindly · 2013-06-25T05:31:18.708Z · LW(p) · GW(p)

The two examples differ in games between B, C, and D, which don't involve A at all. So I don't see a problem: even though A has done better in the second example, B and C have improved more, so they win instead. You can construct something similar in a round-robin chess tournament, where you can win one tournament with a lower score than another tournament you lost.

Can you construct an example where only games involving A are changed?

Replies from: nshepperd
comment by nshepperd · 2013-06-25T06:22:44.814Z · LW(p) · GW(p)

This is a highly artificial example, but consider the following...

Case 1: A mutually defects against B, and mutually defects against N other bots. B mutually defects with those bots too. Assume the N bots defect among themselves. The contest is a draw.

Case 2: A cooperates with B, who defects. A mutually cooperates with the N other bots. And of course, B still mutually defects against those N bots.

For sufficiently large utilities of the (D, C) outcome, in this case B would win the contest, even though A got an improved score by cooperating with those other N bots (minus a small penalty for getting (C, D) instead of (D, D) against B).

Replies from: Kindly
comment by Kindly · 2013-06-25T14:19:10.711Z · LW(p) · GW(p)

Oh, I think I see. In a conventional tournament setting, the same idea can apply: if B beats C and A can beat exactly one of them, it's better to win against B and lose against C rather than vice versa.

This doesn't usually come up in conventional tournaments because winning is believed to be transitive: making yourself a better player makes you more likely to beat both B and C. Here, on the other hand, there may be trade-offs. For example, it's probably worthwhile to use a trick that achieves (D,C) against ReallySmartBot if it requires mutually cooperating with CooperateBot, rather than cooperating with the former and exploiting the latter: ReallySmartBot is a dangerous competitor and CooperateBot probably isn't.

I don't actually see a way to take this into account in the current metagame setting. But this could be an interesting thing to think about after we know the results.

Replies from: nshepperd
comment by nshepperd · 2013-06-27T08:09:38.864Z · LW(p) · GW(p)

I guess it bothers me that in the current metagame all the contestants are aiming to win, rather than to maximize their score (except for the ones who are just aiming to troll the rest of us by submitting humorous bots). In that sense the situation is not really a true prisoner's dilemma.

comment by Ilverin the Stupid and Offensive (Ilverin) · 2013-06-24T20:30:12.226Z · LW(p) · GW(p)

I wonder if there's a chance of the program that always collaborates winning/tieing.

If all the other programs are extremely well-written, they will all collaborate with the program that always collaborates (or else they aren't extremely well-written, or they are violating the rules by attempting to trick other programs).

comment by So8res · 2013-06-20T18:22:50.641Z · LW(p) · GW(p)

What libraries, if any, are we allowed to import? Can I import racket/ libraries, since we're running in drracket?

Replies from: AlexMennen
comment by AlexMennen · 2013-06-23T09:04:25.448Z · LW(p) · GW(p)

You cannot import libraries.

comment by ShardPhoenix · 2013-06-08T10:00:36.736Z · LW(p) · GW(p)

Sorry for the petty complaint, but I'm disinclined to participate because I don't like Lisp syntax (and don't want to have at learn a new programming language just for this anyway).

Replies from: Muhd
comment by Muhd · 2013-06-21T00:15:59.577Z · LW(p) · GW(p)

Same here.

If it were in Java or Python, we could get a whole lot more participants. And more readers who understand the source of the submissions.

Replies from: Eugine_Nier
comment by Eugine_Nier · 2013-06-21T07:07:28.745Z · LW(p) · GW(p)

If it were in Java or Python, we could get a whole lot more participants.

I very much doubt this. At best we'd get a lot more people who are excited to participate but loose interest once they realize they have no idea how to write a program to parse the opposing bot.

Replies from: Sniffnoy, Muhd, Plasmon
comment by Sniffnoy · 2013-06-24T23:55:24.927Z · LW(p) · GW(p)

Agreed -- need something with easy parsing for this.

Honestly, I feel like parsing Scheme is still maybe a bit too hard -- especially since we have things like multithreading (ugh) available. Maybe a subset of Scheme would have been better? Oh well. Fortunately we have people like So8ers making things easier!

Replies from: wedrifid
comment by wedrifid · 2013-06-25T03:47:25.793Z · LW(p) · GW(p)

Honestly, I feel like parsing Scheme is still maybe a bit too hard -- especially since we have things like multithreading (ugh) available. Maybe a subset of Scheme would have been better? Oh well. Fortunately we have people like So8ers making things easier!

Fortunately that aspect of parsing (detecting the use of disallowed features) is the easiest to manage. Choose a subset yourself, announce here what you consider acceptable and declare that any use of anything not in that subset will be treated as a defection.

Replies from: Sniffnoy
comment by Sniffnoy · 2013-06-25T04:40:55.366Z · LW(p) · GW(p)

Yes, I was thinking of that; the problem is I'm doing this quite late, and am not even currently certain just what I'll count as a defection, if I can even get this to work...

(Worse yet, I'm finding that my current program may have to rely on features I want to disallow! Although quite possibly in a way that won't get them detected as disallowed...)

Replies from: solipsist, wedrifid
comment by solipsist · 2013-06-26T01:00:45.296Z · LW(p) · GW(p)

Agreed that even Scheme is a bit much for static analysis. We could ask AlexMennen to disallow multithreading and timing. Would there be objections to that?

Just curious: AlexMennen how many submissions have you received? Do any use multithreading or timing libraries?

Replies from: BloodyShrimp
comment by BloodyShrimp · 2013-06-26T03:23:15.290Z · LW(p) · GW(p)

I'd tentatively object. The 10-second timeout rule becomes too much of a problem if your program can't guard against it, and changing the rules so that failure to play a real move in time = defect would also make static analysis a major pain.

comment by wedrifid · 2013-06-25T05:24:07.476Z · LW(p) · GW(p)

Yes, I was thinking of that; the problem is I'm doing this quite late, and am not even currently certain just what I'll count as a defection, if I can even get this to work...

The obvious option is to start by counting everything as a defection and use whatever time you have remaining (that you are willing to spend) adding new things that your code is smart enough to verify as conditional-cooperation. The smarter your code is the more cooperative it can be!

(Worse yet, I'm finding that my current program may have to rely on features I want to disallow! Although quite possibly in a way that won't get them detected as disallowed...)

There is no strong reason the "I have to count this as defection list" you use must be the same as what you actually use in your own code (unless you think your out-of-game signalling is particularly powerful). (Assuming vaguely sane but bounded competitors) you want your code to be both as smart as possible (allowing more potential mutual cooperation opportunities) and as simple as possible (allowing agents as dumb as possible to see that they can cooperate with you). Ideally your agent would be far, far simpler to understand than what it is able to cooperate with but if you can't manage that it isn't a strict deal breaker.

Replies from: Sniffnoy
comment by Sniffnoy · 2013-06-27T02:25:05.035Z · LW(p) · GW(p)

There is no strong reason the "I have to count this as defection list" you use must be the same as what you actually use in your own code

Certainly true -- if it made it so that my program wouldn't cooperate with itself, that would be a problem, but here the "forbidden" stuff is hidden away an area my program would never actually analyze.

The problem is just that, if I need to use this, that suggests so will other people; and that means I'm probably going to be defecting in a lot of cases where I shouldn't.

(Assuming vaguely sane but bounded competitors) you want your code to be both as smart as possible (allowing more potential mutual cooperation opportunities) and as simple as possible (allowing agents as dumb as possible to see that they can cooperate with you). Ideally your agent would be far, far simpler to understand than what it is able to cooperate with but if you can't manage that it isn't a strict deal breaker.

Argh, I didn't even think about the "it should be simple to be visibly cooperating" bit. Trying to do static analysis on Scheme (with its conds and cases and quasiquotes) is enough to make my program complicated. Maybe I should just submit a MimicBot instead.

Btw, just in case anyone was thinking of doing such a thing after the declarations so far: If you do any redefining of syntax, I'm defecting on you.

Replies from: wedrifid
comment by wedrifid · 2013-06-27T05:38:02.255Z · LW(p) · GW(p)

Argh, I didn't even think about the "it should be simple to be visibly cooperating" bit. Trying to do static analysis on Scheme (with its conds and cases and quasiquotes) is enough to make my program complicated. Maybe I should just submit a MimicBot instead.

Does MimicBot run the opponent and then return the same result? If so then I suggest a timeout/defect feature. Any bot that fails against itself has problems!

Are there going to be CooperateBots in the competition? If that is a possibility then consider a simple tweak "If my opponent does not make use of its opponent variable then defect else Mimic". (Failing against unobfuscated CooperateBot is at least as embarrassing as failing against self.)

Btw, just in case anyone was thinking of doing such a thing after the declarations so far: If you do any redefining of syntax, I'm defecting on you.

Anyone who tries that must and doesn't expect defection must seriously hold the opposition in contempt!

Replies from: Sniffnoy
comment by Sniffnoy · 2013-06-28T19:24:35.636Z · LW(p) · GW(p)

Does MimicBot run the opponent and then return the same result? If so then I suggest a timeout/defect feature. Any bot that fails against itself has problems!

By MimicBot I mean the strategy given in this comment. It's not great, but it's better than not ever finishing.

Are there going to be CooperateBots in the competition? If that is a possibility then consider a simple tweak "If my opponent does not make use of its opponent variable then defect else Mimic". (Failing against unobfuscated CooperateBot is at least as embarrassing as failing against self.)

Hm, that's not a bad idea. My current program is essentially trying to measure to what extent their output depends on mine, but if I can't finish that in time, that's a simple tweak that can be added...

comment by Muhd · 2013-06-21T22:18:28.937Z · LW(p) · GW(p)

At best we'd get a lot more people who are excited to participate but loose interest once they realize they have no idea how to write a program to parse the opposing bot.

At worst that is what would happen.

Java and Python are many orders of magnitude more popular than Scheme and if only 10% of the people who get excited about participating actually know how to parse than we would still have much greater numbers.

Replies from: Eugine_Nier
comment by Eugine_Nier · 2013-06-25T03:24:08.263Z · LW(p) · GW(p)

Java and Python are many orders of magnitude more popular than Scheme and if only 10% of the people who get excited about participating actually know how to parse than we would still have much greater numbers.

Java and Python are much harder to parse than scheme, and the percentage of Java or Python programers who could write a parser for those languages is less than 10%. Furthermore, the ones who could write a parser likely also know Scheme or at least some other lisp dialect.

Replies from: Muhd
comment by Muhd · 2013-06-25T22:42:10.200Z · LW(p) · GW(p)

Do you have any evidence of this? It seems highly unlikely that the proportion of users of any given programming language that could write a parser would differ by an amount that there are more Scheme users who can write a parser than Java/Python users, given the vastly larger number of users of Java and Python.

For what its worth, python includes its own parser in the standard library, so it would probably be better than scheme in that regard, though I might have to agree with you regarding parsing with Java, though there may very well be a module out there for parsing Java as well.

Replies from: solipsist
comment by solipsist · 2013-06-26T00:51:22.792Z · LW(p) · GW(p)

From experience, I can tell you that it is easier to lean Scheme and write a Scheme interpreter than it is to just understand how Python works on the inside. If you know how Python worked well enough to do static analysis on it, you can learn Scheme in an hour.

comment by Plasmon · 2013-06-21T08:29:25.705Z · LW(p) · GW(p)

Java's reflection API could conceivably be used for this purpose. (Provided the programs are given access to each others compiled code rather than to the human-readable java source code.)

Replies from: ArisKatsaris
comment by ArisKatsaris · 2013-06-24T22:51:52.426Z · LW(p) · GW(p)

I'm sorry but really no. The reflection API allows you to see what variables and methods are contained, it can allow you to execute methods, but that's your only option, it doesn't really allow you to parse the insides of a method.

comment by [deleted] · 2013-06-05T22:47:50.043Z · LW(p) · GW(p)

I'm trying to work out how scheme in general and racket in particular work.

I installed racket and started it up, and I get a two-pane window. I've noticed that using eval in the REPL thing in the bottom pane works, but if I try using it in the top half I get a cryptic error message:

#lang racket

(eval '(+ 1 1))

gives:

+: unbound identifier;

also, no #%app syntax transformer is bound in: +

Is there some additional magic, or is this something I don't need to worry about? How will a program be run in the competition?

Replies from: AlexMennen
comment by AlexMennen · 2013-06-05T22:59:37.322Z · LW(p) · GW(p)

Racket apparently has terrible namespace conventions. I had the same problem when I tried to use it to interpret a file, but when I type a program directly into the interpreter, eval works fine. I'll either figure out how to get eval to work when I run a file, or I'll just paste people's programs into the interpreter. Either way, assume eval will work properly.

Replies from: Morendil, nshepperd
comment by Morendil · 2013-06-07T17:29:47.057Z · LW(p) · GW(p)

The Guide says:

#lang racket

(define ns (make-base-namespace))
(eval '(cons 1 2) ns) ; works

and indeed it works.

comment by nshepperd · 2013-07-01T17:34:08.697Z · LW(p) · GW(p)

In both #lang racket and #lang scheme, (eval form) by itself apparently runs form in a basically empty namespace. I've personally been using the following incantation for evaluating anything in the "standard" namespace:

(define (ueval x)  
(define ns (make-base-namespace))  
(eval x ns))

Then (ueval source-code) evaluates source-code in a clean namespace with all the normal builtins, including eval.

comment by Lumifer · 2013-06-05T21:12:15.108Z · LW(p) · GW(p)

Hm. Five minutes of thought suggest to me that the following pseudocode would be optimal:

while(passed.time < 9.5seconds) {
   outcome = eval(opponent.code(self.code))
   if (outcome == "C") { c.count++ }
   if (outcome == "D") { d.count++ }
}

if (c.count > d.count) {
  output("C")
} else {
   output("D")
}

Deterministic programs would always output the same thing so you know beforehand whether they'll cooperate or defect. For RNG-based programs you just bet on what's most likely.

I welcome big guns blowing large holes in this construction :-)

P.S. How do I indent in preformatted code?

Replies from: DanielLC, arundelo
comment by DanielLC · 2013-06-05T23:28:25.433Z · LW(p) · GW(p)

Based on the fact that you retracted this, you probably already realized the problem, but I'll say it anyway for anyone else who hasn't noticed.

If two programs both use this strategy, they'll go into an infinite loop where program A simulates program B simulating program A etc. Since it only checks to see how much time is left after it finishes the simulation, it won't even manage to give up and defect. It will just run out of time.

Also, passed.time would imply that there's a class called "passed" with a variable called "time", which is silly. It should be passed_time or passedTime depending on how you do multi-word variables.

Replies from: endoself, Morendil, Mestroyer, Lumifer
comment by endoself · 2013-06-05T23:55:44.784Z · LW(p) · GW(p)

Another problem is that you cooperate agains CooperateBot.

Replies from: DanielLC
comment by DanielLC · 2013-06-06T00:25:49.018Z · LW(p) · GW(p)

I didn't look at the details too much. You can fix that problem just by having it calculate the scores for cooperate and defect, and go with the one with the higher score.

Replies from: endoself
comment by endoself · 2013-06-06T00:52:50.138Z · LW(p) · GW(p)

I'm not sure what you mean. Do you mean the scores given that you choose to cooperate and defect? There's a lot of complexity hiding in 'given that', and we don't understand a lot of it. This is definitely not a trivial fix to Lumifer's program.

Replies from: DanielLC
comment by DanielLC · 2013-06-06T04:22:48.320Z · LW(p) · GW(p)

I messed up. I didn't realize that until after I posted, and I didn't feel like going back to fix it.

comment by Morendil · 2013-06-07T14:29:38.753Z · LW(p) · GW(p)

If two programs both use this strategy, they'll go into an infinite loop where program A simulates program B simulating program A etc.

Not if you do it right. (Not sure if I should say any more. The contest is interesting but sharing too much in the comments risks turning it into a social engineering effort rather than a programming or mathematical effort.)

comment by Mestroyer · 2013-06-06T00:04:12.813Z · LW(p) · GW(p)

I'm not that familiar with Scheme, but is there some way to see how much stack space you have left and stop before an overflow?

Replies from: DanielLC, warbo
comment by DanielLC · 2013-06-06T00:30:48.693Z · LW(p) · GW(p)

I'm not familiar with it either, but if nothing else, you can pass a counter that shows the number of iterations you've gone through and stop when it gets too high.

What do you plan on doing when you run out of stack space? You can't just replace your code with a cooperate bot or a defect bot after a certain level, since they'll defect if they think you'll do the same thing either way, and you'll defect if you think they'll defect etc.

What you need is something along the lines of cooperating if their code is equivalent to yours, and defecting if it's different. You have to do this in under ten seconds, and you have to get it to cooperate even if they have a slightly different idea of what's equivalent.

comment by warbo · 2013-06-06T16:02:38.436Z · LW(p) · GW(p)

Scheme requires tail-call optimisation, so if you use tail-recursion then you'll never overflow.

Replies from: Mestroyer
comment by Mestroyer · 2013-06-06T23:32:12.874Z · LW(p) · GW(p)

Does that include tail-calls in mutual recursion? Even if you were going to return whatever your opponent's code did, you probably couldn't count on them doing the same.

Replies from: wubbles
comment by wubbles · 2013-06-09T12:45:38.167Z · LW(p) · GW(p)

Yes: all tail-calls are guarantied not to exhaust resources. The precise definition of what is tail is in the spec.

comment by Lumifer · 2013-06-06T01:29:39.874Z · LW(p) · GW(p)

Well, that was the beginning of a host of problems :-)

I don't know about the thread (or exception handling) capabilities of Scheme in Racket, but it shouldn't be too hard to make sure you don't run out of time if some chunk of the code is stuck in a recursive loop. That's not really the issue. The issue is actually the recursion itself and the consequent need to deal with opponents that eval your code and expect an answer.

As a side note, passed.time is perfectly valid variable name in some languages where it doesn't imply a class.method structure.

Replies from: Decius
comment by Decius · 2013-06-06T17:36:36.001Z · LW(p) · GW(p)

Can you track total passed time, so that if your opponent simulates you with 9 seconds passed, you cooperate instead of simulating your opponent? They simulate you simulating them... until you say "I'm running out of time, it must be the case that we're in a recursive simulation, so It's probably true that I should cooperate!" The latest simulation of your opponent says "He cooperated, so I should!" the next to last simulation of your program says the same thing, and so forth.

Until your original program says "My simulation of my opponent says he will cooperate, but I know for sure that I started on the first cycle of the evaluation, so I'm not a simulation. I defect."

Replies from: Lumifer
comment by Lumifer · 2013-06-06T18:28:39.006Z · LW(p) · GW(p)

...so that if your opponent simulates you

Your code doesn't know if it's being evaluated by the opponent or it's running "for real". If it knew, the program would be remarkably easy: if (simulated) { say "I cooperate!" } else { defect }.

It's not hard to recognize that you're stuck in a recursive trap and need to get out of there after nine seconds, but you still don't know what to output.

Replies from: Decius
comment by Decius · 2013-06-07T04:01:04.607Z · LW(p) · GW(p)

If I know early enough the ten-second time limit exactly how long into the ten seconds I am, I can be sure enough that I'm the original.

Another possibility would be to include a random chance on the order of .1% that I will set a global flag that will follow down levels of simulation, and then simulate my opponent simulating me. If I get the flag, I have high probability that my opponent is simulating me on himself; I don't want to set that flag with high probability, because I want my opponents' (me cooperatebot) and (me defectbot) evaluations to return correctly.

But now I'm focusing on this particular implementation of the competition, not trying to provide useful information to the question that the competition is about.

Proposed meta-rule: Programs should not make attempts to determine if they are being scored or not in order to change behavior; attempts to identify and break out of recursive loops are encouraged.

comment by arundelo · 2013-06-06T01:51:05.257Z · LW(p) · GW(p)

The indentation thing is a bug.

comment by fractalman · 2013-06-26T03:12:57.195Z · LW(p) · GW(p)

I just realized...

Actually running your opponent, and passing it a fake version of its own source code is a bad idea-if it notices, it can call either forkbomb() or loopforever() as needed.

Incidentially, are you sure you want to run the whole thing as winner-take-all? My fisheries class a while back ran a delayed-effect prisoners dilemma over the matter of fish sustainability... except that it was actually a zero-sum game because there was only ONE extra credit point, with winner take all.

Halfway through, after the fish were extinct and my team-which had the most boats-was losing due to boat-upkeep costs, I singlehandedly turned the whole thing into a game about market manipulation by running a pyramid scheme involving the boats. (I still lost because I forgot to keep exponentially buying more boats with loans from the bank, but oh well...)

Alas, I won't be competing. (and if I do, I'll just throw out a mimicbot...) Keep it in mind, though, if you set another one up.

Replies from: Qiaochu_Yuan
comment by Qiaochu_Yuan · 2013-06-27T08:31:34.505Z · LW(p) · GW(p)

Can you explain the game in more detail? I don't understand what to make of your strategy.

Replies from: fractalman
comment by fractalman · 2013-06-27T20:09:09.380Z · LW(p) · GW(p)

ok.

There were 5 or so teams. there were 10 or 15 "years" (rounds). Each round, you can buy brand new boats in proportion to your current fleet, go fishing with n boats, buy boats from other players, sell boats to other players...and pay a maintanence fee on your current fleet size. Each boat that goes fishing costs a little bit more to run than letting it sit in one place. The fish reproduce, but slowly.

At the very end of the game, fleet value and bank account are added up to determine the winner.

But the value of boats is player-driven...and there was no limit on how much debt you could go into. meanwhile, the cost for brand-new boats...was constant.

The instructor intended to illustrate how the prisoners dilemma plays a role in overfishing, which I don't think is what we actually wound up learning, since our metagame was winner-take-all and not a prisoner's dilemma.

...

Again: you might learn something interesting with winner-take-all, and may even need to use that as the setup to prevent cooperate-bot spam due to the types of the people on this site...but it IS a different metagame. Keep that in mind when interpreting the results.

edit2: By different metagame, I mean that this tournament posesses a nash equilibrium of programs for which each program is considered a move. Finding it is possible in finite steps(It does not run afoul of the halting problem due to the 10 second limitation, as measured by the hardware of a specific machine and environment), but a brute-force approach would probably take more memory than our universe appears to possess.

edit3: see also nshepperd's comments