AI Challenge: Ants - Post Mortem

The problem with programming challenges is that if you win a programming challenge, you probably could've spent the same time doing a very similar thing but produced something of much greater value (money, software that people will actually use, research that will actually be built on). The unfortunate consequence of this is that you aren't likely to see many rationalists entering these competitions without strong motivation, and a little prestige and publicity isn't enough.

Also, building a functioning team of programmers out of loosely-committed geographically-dispersed acquaintances is ridiculously hard, and if you could do it, you wouldn't waste that power on a game.

Just spitballing here:

• Promote the AI challenge as a rationalist meetup topic with the goal of having several working groups

• Instead of trying to get one big group with a leader right from the start, appoint (or whatever) several leaders: assign to each leader a small collection of interested people

• Be clear about what you want the leaders to do: what are the short and medium range goals

• Put up an early post asking people to express interest and (maybe) skill-sets so that teams could be assembled with some balance / hope of accomplishing something

• Keep in contact with the various leaders and see where people are getting stuck (I'm assuming that you are ultimately the person in charge of this project); periodically, have the leaders talk to each other -- but not extremely often; post regular discussion threads focusing on solving specific "We're stuck on this," problems

• Try to reframe the problem or parts of the problem in a way that connects to generic rationality, so that non-programmers can contribute something -- looking over the old thread, it seems that a lot of people were intimidated by the threat of having to code stuff, but the programmers might nonetheless get a good idea or two from what non-programmers have to say about generic rationalist-type problems

• Make some direct suggestions about the "worthwhile things" you mention. For example, apart from the AI project itself, what methods would you suggest site members use to cooperate and why? (Okay, maybe there isn't much more to be said directly about positive publicity and advancing AI ... but then, maybe there is ...)

• Set benchmarks for when things should be done, even if those benchmarks have to be re-set several times along the way

...which I think is about 6 months away...

Outside view suggests it might be more like a year. (Ants was originally planned--at least by one of the contest's officials--for about 6 months after planet wars.)

I'm not too unhappy with my #326 finish, considering that life intervened and I basically didn't work on it at all for the second half of the contest. Percentage-wise, it's an improvement over the last one, anyway. I was also struck by how mundane the winner's strategy was (http://www.xathis.com/posts/ai-challenge-2011-ants.html). It seems he didn't beat it with just a few overarching principles, but rather a lot of little pieces done well.

I have some more postmortem thoughts that I'll try to write down in a bit.

Don't expect people to commit up front to representing LessWrong. The idea of representing the entire community sounds kind of intimidating. In general don't stress benefits like getting positive publicity or learning important things about cooperation too much. I think that people who'll participate in a competition like that will be those who find it intrinsically rewarding. So stress the fun and coolness factor, I guess? Maybe have the people who participate initially write down the accounts of their early experiences in order to capture the imagination and draw in others.

It would be good if people could try their hand at participating in a team without too much commitment. This sounds hard and probably heavily depends on the structure of the particular programming problem and maybe rules of the competition.

Have a ready-to-use platform for cooperation so that people can immediately jump into team activities. Preparing such could amount to researching various websites providing such services, choosing the best combination of them and then writing a short tutorial on their use. In the extreme case, a group of volunteers could give the chosen solution a test run by cooperating on solving a problem from some prior programming contest and then writing down a report of their experience.

As promised, a few more postmortem thoughts.

I made a number of mistakes while writing my bot. (I'm about to compare my approach with that of the winner, xathis, so you might want to read his write up first: http://www.xathis.com/posts/ai-challenge-2011-ants.html)

I assumed I wouldn't have time to do the number of local searches that xathis did, and I never tested this assumption.

For the combat code, I wrote both minimax and probabalistic algorithms; both worked ok, but I never tried to do any branch pruning, so the minimax code only ran when small numbers of ants were fighting. I should have tried some branch pruning. I spent too much time writing the combat code for the benefit it gave my bot.

My biggest mistake, or perhaps the unifying theme of my mistakes, is that I spent way too much time searching for the Deep Underlying Principle; instead of just writing code that directly did what I wanted, I was trying to write code that produced the behavior I wanted automatically as emergent behavior.

I spent too much time trying to fine-tune doomed approaches (of course, that is at least partly hindsight bias). I should have spent more time exploring completely different approaches.

One thing I did do well was an algorithm for listing all local combats, so that my fighting code could run on small(er) numbers of ants. (xathis did something similar.) I'm also not unhappy with my BFS algorithms; they ran quickly and I should have thought to use them more heavily.

As far as working with an "official" less wrong team, I didn't feel sufficiently confident in my abilities to want to represent LW officially (and I'm not updating that belief much because I still think it's quite likely that lots of people here could have done better). I think if there is interest for next time, we should at least figure out a method of working together beforehand.

I'm not sure what else I should say, ask questions if you have them.

There are a lot of standard software development processes that you can follow. (e.g. SCRUM)

What those usually boil down to is:

• Now exactly what tasks need to be done.
• Now exactly who does what task and when it will be completed.
• Check both assumptions regularly.