Higher than the most high

I came across this problem awhile ago, made some notes and never published. Seems like a good time to write up the highlights.

A world model is a mapping from decisions to probability distributions over outcomes. A preference function is a total ordering over probability distributions over outcomes. Utility functions are the subset of preference functions which is conveniently linear (but we don't need those linearity properties right now). A decision theory maps from (world model, preference function) pairs to decisions, and the goodness of decision theories is a partial order such that for decision theories A and B, A>=B if for all preference functions Pref and world models World, Pref(World(A(World,Pref))) >= Pref(World(B(World,Pref))). This order has a supremum iff for all world models Pref(World(d)) has a supremum.

Where these orderings lack suprema, it is nevertheless possible to define limiting sequences of successively better options and successively better decision theories which generate them, such that for any single decision or decision theory the seqeunce must contain an element which is better. Finding and proving the correctness of such a sequence, for a particular world model, is proving that there is no optimal decision for that world model.

A bounded agent could approximate (but not reach) optimality by finding such a sequence, and traversing some distance along it. Such sequences, and the number of steps taken along them, can each be classified by their improvement rates. The partial order formed by those rates (with an ignored multiplicative constant on the sequence traversal distance) makes a second-order decision problem. If you ignore the constant, this one does have a supremum: the busy beaver function. (All further games you could play with BB can be folded into the one constant.)

A more general case:

For every strategy s and corresponding payoff p(s) there exists a strategy s such that p(s)>p(s).

I have a few clarifying questions about the rules that I am trying to understand.

First Question: Are you allowed to tell Omega an integer that you can't actually calculate but that Omega can, if you know that integer exists and can uniquely define it even if you can't calculate it?

Example: You say "The integer calculated as a result of the maximal strategy that you, Omega can construct without falling into a loop." Does that represent an extremely high integer to Omega, even though you couldn't actually calculate it yourself because you have less room for computations than Omega does, or is it in an invalid response?

Second question: If you and Mrs. X both give the exact same response (for instance, if the above answer is valid and you both say it), then what happens?

Edit: I read your linked post and you said:

It should be noted that God, or any being capable of hypercomputation, has real problems in these situations: they actually have infinite options (not a finite options of choosing their future policy), and so don't have any solution available.

Which means that that particular example is probably bad, since there is no integer calculated as a result of the maximal strategy that Omega can construct without falling into a loop.

However, the question "Can you ask Omega to do you calculations that you yourself can't do?" is still relevant for answers like "The number I am about to say, raised to the power of the number I am about to say, the number I am about to say times. (Long pause as you calculate your Maximum integer.) My Maximum integer."

Although, I suppose if you can run calculations on a hypercomputer, and you live indefinitely long while talking to it, You could potentially end up getting caught in a infinite loop of saying "The number I am about to say, raised to the power of the number I am about to say, the number I am about to say times... And that number can be calculated by taking the number I am about to say, raised to the power of the number I am about to say, the number I am about to say times, and that number can be calculated..."

So if you could have the hypercomputer help you with calculations, and wanted the highest finite number you and the hypercomputer together could generate, you would also have to ask the hyper computer to increment your number in such a way that you didn't get caught in an infinite loop while attempting to increment it.

Eliezer briefly told me about a possible scenario whereby the universe turns into this game, except it's "whoever can name the highest ordinal gets to take everybody's money," but I no longer remember the details.

I'd just like to point out that the construction of this article is an exact ascension into the meta of the construction of the problem within it (except that nobody is paying you the utility you spend on finding the highest-high metastrategy back). This is suggestive of the fact that you need a generalized solution to the strategy-picking problem that picks itself. (If it didn't pick itself, it wouldn't be fully general, or it would be making worse choices than some other strategy, inwhichcase it's insufficiently maximal.) A defined (but not necessarily finite) set of strategies which pick among themselves would also be satisfactory, provided that any one of that set of strategies converges in finite time to the maximal among that set of strategies.

Which if you think about it, is exactly the problem to be solved to produce self-modifying FAI; finding the set of strategy-selecting strategies which select strategies only within their own set, and for which each member of that set converges on the best strategy within that set. (Although FAI doesn't necessarily have to do it in finite time.)