Comment by yongqli on Demons in Imperfect Search · 2020-02-14T22:40:27.193Z · LW · GW

It seems that there is a fundamental difference between a physical agent that participates in an arrow-of-time versus an algorithm exploring a Platonic realm off-line, for example trying to find the best way to compress a dataset. The algorithm can be tricked by red-herrings in the data into wasting CPU time chasing after mirages, but it can always restore from a checkpoint, do a random restart, spawn multiple threads, etc -- it can always press "undo" and cannot be trapped forever. Most importantly, it can't be stolen from, only tricked into wasting its time. But a physical agent interacting with the world can be have its resources stolen, further fueling its attacker, perhaps starting some sort of Red Queen dynamics.