The Unexpected Clanging

Adversarial simulation by a very powerful entity is so obviously unsolvable that I'm caught off-guard every time it comes up.  

Let Omega be an AI that can perfectly simulate your entire deliberation process ... such as to mess with you

Ok, the result is that you're messed with.  Can we move on to more interesting questions?

Note that it does get a little more interesting (possibly not above 0, though), if you remove the anthropomorphization of Omega, either remove or model fully your time-bounded process (I think remove here, as there's no change during the experiment), and clearly specify whether you know that Omega uses that formula, or if you are predicting without any knowledge.  I assume, but it's nice to specify, that you are indifferent to which direction you're wrong (that is, you don't care whether you guess 1 when it's 2 or guess 2 when it's 1, but you prefer to guess correctly over incorrectly).

I think, as the victim, I will recognize the paradox and give up - I have no clue what that reads as to Omega - either 0.5 or 1-epsilon or 0+epsilon, or "no answer".   If Omega is taking my spoken answer, rather than my belief, and the payout utilities match the odds, I'll say whatever comes to mind, and take. my coinflip.

There is incentive for hidden expectation/cognition that Omega isn't diagonalizing (things like creating new separate agents in the environment). Also, at least you can know how ground truth depends on official "expectation" of ground truth. Truth of knowledge of this dependence wasn't diagonalized away, so there is opportunity for control.

Omega can simulate me perfectly by assumption, therefore, in order to win (at least probabilistically) I need to base my estimate on something external that I have reason to believe Omega cannot simulate. One approach (for example) might be to bring a quantum random number generator (given our current best models of physics such a device cannot be predicted in advance by any simulation), and use that to introduce randomness into my decision making.

Are we supposed to assume that Omega can solve the Halting Problem? Simulating you under some conditions, may implicate it.

(This goes double if you assume "has access to the agent's source code".)

Yes, if Omega accurately simulates me and wants me to be wrong, Omega wins. But why do I need to get the answer exactly "right"? What does it matter if I'm slightly off?

This would be a (very slightly) more interesting problem if Omega was offering a bet or a reward and my goal was to maximize reward or utility or whatever. It sure looks like for this setup, combined with a non-adversarial reward schedule, I can get arbitrarily close to maximizing the reward.

Perhaps it would be fruitful to consider the two participants, who I'll call Alpha and Omega, as finite computer programs, for which Omega has access to Alpha's source code. Maybe Alpha also has access to Omega's source code. Each of them chooses a number (from the natural numbers, the reals, [0,1], {0,1}, or whatever). It is common knowledge between them that Omega's goal is to choose differently from Alpha, and Alpha's goal is to choose the same as Omega.

Given various constraints on the computational or proof-theoretic capabilities of Alpha and Omega, under what circumstances does either player have a winning strategy?

If they each have access to a source of randomness, the game could be generalised to Omega trying to maximise the probability that they differ, and Alpha's being to minimise that probability.

Interesting. This prank seems to be one you could play on a Logical Inductor, I wonder what the outcome would be? One fact that's possibly related is that computable functions are continuous. This would imply that whatever computable function Omega applies to your probability estimate, there exists a fixed point probability you can choose where you'll be correct about the monkey probability. Of course if you're a bounded agent thinking for a finite amount of time, you might as well be outputting rational probability estimates, in which case functions like $f\left(x\right)=\mathrm{mod}(x+\frac{1}{2},1)$ become computable for Omega.

Suppose that I decide that my opinion on the location of the monkey will be left or right dependent on one bit of quantum randomness, which I will sample sufficiently close to the deadline that my doing so is outside Omega's backward lightcone at the time of the deadline, say a few tens of nanoseconds before the deadline if Omega is at least a few tens of feet away from me and the two boxes? By the (currently believed to be correct) laws of quantum mechanics, qbits cannot be cloned, and by locality, useful information cannot propagate faster than light, so unless Omega is capable of breaking very basic principles of (currently hypothesized) physical laws – say, by having access to faster-than-light travel or a functioning time loop not enclosed by an event horizon, or by having root access to a vast quantum-mechanics simulator that our entire universe is in fact running on – then it physically cannot predict this opinion. Obviously we have some remaining Knightian-uncertainty as to whether the true laws of physics (as opposed to our current best guess of them) allow either of these things or our universe is in fact a vast quantum simulation — but it's quite possible that the answer to the physics question is in fact 'No', as all current evidence suggests, in which case no matter how much classical or quantum computational power Omega throws at the problem there are random processes that it simply cannot reliably predict the outcome of.

[Also note that there is some actual observable evidence on the subject of the true laws of physics in this regard: the Fermi paradox, of why no aliens colonized Earth geological ages ago, gets even harder to explain if our universe's physical laws allow those aliens access to FTL and/or time loops.]

Classically, any computation can be simulated given its initial state and enough computational resources. In quantum information theory, that's also true, but a very fundamental law, the no-cloning theorem, implies that the available initial state information has to be classical rather than quantum, which means that the random results of quantum measurements in the real system and any simulation are not correlated. So quantum mechanics means that we do have access to real randomness that no external attacker can predict, regardless of their computational resources. Both quantum mechanical coherence and information not being able to travel faster than light-speed also provide ways for us to keep a secret so that it's physically impossible for it to leak for a short time.

So as long as Omega is causal (rather than being acausal or the sysop of our simulated universe) and we're not badly mistaken about the fundamental nature of physical laws, there are things that it's actually physically impossible for Omega to do, and beating the approach I outlined above is one of them. (As opposed to, say, using nanotech to sabotage my quantum-noise generator, or indeed to sabotage me, which are physically possible.)

So designing ideal decision theories for the correct way to act in a classical universe in the presence of other agents with large computational resources able to predict you perfectly doesn't seem very useful to me. We live in a quantum universe, initial state information will never be perfect, agents are highly non-linear systems, so quantum fluctuations getting blown up to classical scales by non-linear effects will soon cause a predictive model to fail after a few coherence times followed by a sufficient number of Lyapunov times. It's quite easy to build a system whose coherence and Lyapunov times are deliberately made short so that it's impossible to predict over quite short periods, if it wants to be (for example, continuously feed the output from a quantum random noise generator into perturbing the seed of a high-cryptographic-strength pseudo-random-number generator run on well-shielded hardware, ideally quantum hardware).

Of course, in a non-linear system, it's still possible to predict the climate far further ahead than you can predict the weather: if Omega has sufficiently vast quantum computational resources, it can run many full-quantum simulations of the entire system of every fundamental particle in me and my environment as far as I can see (well, apart from incoming photons of starlight, whose initial state it doesn't have access to), and extract statistics from this ensemble of simulations. But that doesn't let Omega predict if I'll actually guess left vs right, just determine that it's 50:50. Also (unless physical law is a great deal weirder than we believe), Omega is not going to be able to run these simulations as fast as real physics can happen — humans are 70% warm water, which contains a vast amount of quantum thermal entropy being shuffled extremely fast, some of it moving at light-speed as infra-red photons, and human metabolism is strongly and non-linearly coupled to this vast quantum-random-number-generator via diffusion and Brownian motion: so because of light-speed limits, the quantum processing units that the simulation was run on would need to be smaller than individual water molecules to be able to run the simulation in real-time. [It just might be possible to build something like that in the outer crust of a neuron star if there's some sufficiently interesting nucelonic chemistry under some combination of pressure and magnetic field strength there, but if so Omega is a long way away, and has many years of light-speed delay on anything they do around here.]

What Omega can do is run approximate simulations of some simplified heuristic of how I work, if one exists. If my brain was a digital computer, this might be very predictive. But a great deal of careful engineering has gone into making digital computers behave (under their operating conditions) in a way that's extremely reliably predictable by a specific simplified heuristic that doesn't require a full atomic-scale simulation. Typical physical or biological systems just don't have this property. Engineering something to ensure that it definitely doesn't have this property is easy, and in any environment containing agents with more computation resources than you, seems like a very obvious precaution.

So, an agent can easily arrange to act unpredictably, by acting randomly based on a suitably engineered randomness source rather than optimizing. Doing so makes its behavior depend on the unclonable quantum details of its initial state, so the probabilities can be predicted but the outcome cannot. In practice, even though they haven't been engineered for it, humans probably also have this property over some sufficiently long timescale (seconds, minutes or hours, perhaps), when they're not attempting to optimize the outcome of their actions.

[Admittedly, humans leak a vast amount of quantum information about their internal state in the form of things like infra-red photons emitted from their skin, but attempting to interpret those to try to keep a vast full-quantum simulation of every electron and nucleus inside a human (and everything in their environment) on-track would clearly require running at least that much quantum calculation (almost certainly many copies of it) in at least real-time, which again due to light-speed limits would require quantum computing elements smaller than water-molecule size. So again, it's not just way outside current technological feasibility, it's actually physically impossible, with the conceivable exception of inside the crust of a neutron star.]

As a more general version of this opinion, while we may have to worry about Omegas whose technology is far beyond ours, as long as they live in the same universe as us, there are some basic features of physical law that we're pretty sure are correct and would thus also apply even to Omegas. If we had managed to solve the alignment problem contingent on basic physical assumptions like information not propagating faster then the speed of light, time loops being impossible outside event horizons, and quanta being unclonable, then personally (as a physicist) I wouldn't be too concerned. Your guess about the Singularity may vary.