Infra-Bayesianism naturally leads to the monotonicity principle, and I think this is a problem

Some quick comments:

• The monotonicity principle is only a consequence of IB if you assume (as IBP does) Knightian uncertainty that allows for anything to be simulated somewhere. IBP assumes this essentially because it leads to natural mathematical properties of the behavior of hypotheses under ontology refinement, which I conjecture to be important for learning, but we still don't know whether they are truly necessary.
• I reiterate that I am not calling to immediately build a sovereign AI based on the 1st set of philosophical assumptions that came to my mind. I am only pointing directions for investigation, which ultimately might leads to us to a state of such confidence in our philosophical assumptions, that even a sovereign AI becomes reasonable (and if they won't, we will probably still learn much). Right now, we are not close to such a level of confidence.
• There are additional approaches to either explaining or avoiding the monotonicity principle, for example the idea of transcartesian agents I mentioned here [LW · GW].

The sequence on Infra-Bayesianism motivates the min (a.k.a. Murphy) part of its argmax min by wanting to establish lower bounds on utility — that's a valid viewpoint. My own interest in Infra-Bayesianism comes from a different motivation: Murphy's min encodes directly into Infra-Bayesian decision making the generally true, inter-related facts that 1) for an optimizer, uncertainty on the true world model injects noise into your optimization process which almost always makes the outcome worse 2) the optimizer's curse usually results in you exploring outcomes whose true utility you had overestimated, so your regret is generally higher than you had expected 3) most everyday environments and situations are already highly optimized, so random perturbations of their state almost invariably make things worse. All of which justify pessimism and conservatism.

The problem with this argument, is that it's only true when the utility of the current state is higher than the utility of the maximum-entropy equilibrium state of the environment (the state that increasingly randomizing it tends to move it towards due to the law of big numbers). In everyday situations this is almost always true -- making random changes to a human body or in a city will almost invariable make things worse, for example. In most physical environments, randomizing them sufficiently (e.g. by raining meteorites on them, or whatever) will tend to reduce their utility to that of a blasted wasteland (the surface of the moon, for example, has pretty-much reached equilibrium under randomization-by-meteorites, and has a very low utility). However, it's a general feature of human utility functions that there often can be states worse than the maximum-entropy equilibrium. If your environment is a 5-choice multiple-choice test whose utility is the score, the entropic equilibrium is random guessing which will score 20%, and there are chose-wrong-answer policies that score less than that, all the way down to 0% -- and partially randomizing away from one of those policies will make its utility increase towards 20%. Similarly, consider a field of anti-personnel mines left over from a war -- as an array of death and dismemberment waiting to happen, randomizing it with meteorite impacts will clear some mines and make its utility better -- since it starts off actually worse than a blasted wasteland. Or, if a very smart GAI working on alignment research informed you that it had devised an acausal attack that would convert all permanent hellworlds anywhere in the multiverse into blasted wastelands, your initial assumption would probably be that doing that would be a good thing (modulo questions about whether it was pulling your leg, or whether the inhabitants of the hellworld would consider it a hellworld).

In general, hellworlds (or at least local hell-landscapes) like this are rare — they have a negligible probability of arising by chance, so creating one requires work by an unaligned optimizer. So currently, with humans the strongest optimizers on the planet, they usually only arise in adversarial situations such as wars between groups of humans ("War is Hell", as the saying goes). However, Infra-Bayesianism has exactly the wrong intuitions about any hell-environment whose utility is currently lower than that of the entropic equilibrium. If you have an environment that has been carefully optimized by a powerful very-non-aligned optimizer so as to maximize human suffering, then random sabotage such as throwing monkey wrenches in the works or assassinating the diabolical mastermind is actually very likely to improve things (from a human point of view), at least somewhat. Infra-Bayesianism would predict otherwise. I think having your GAI's decision theory based on a system that gives exactly the wrong intuitions about hellworlds is likely to be extremely dangerous.

The solution to this would be what one might call Meso-Bayesianism -- renormalize your utility scores so that of the utility of the maximal entropy state of the environment is by definition zero, and then assume that Murphy minimizes the absolute value of the utility towards the equilibrium utility of zero, not towards a hellworld. (I'm not enough of a pure mathematician to have any idea what this modification does to the network of proofs, other then making the utility renormalization part of a Meso-Bayesian update more complicated.) So then your decision theory understands that any unaligned optimizer trying to create a hellworld is also fighting Murphy, and when fighting them on their home turf Murphy is your ally, since "it's easier to destroy than create" is also true of hellworlds. [Despite the usual formulation of Murphy's law, I actually think the name 'Murphy' suits this particular metaphysical force better -- Infra-Bayesianism's original 'Murphy' might have been better named 'Satan', since it's is wiling to go to any length to create a hellworld, hobbled only by some initially-unknown physical laws.]

Reading your thoughts on sublinearity, I instinctively feel as a CEV utilitarian, instead of feeling that copies matter less and less terminally. It seems to me that caring less about copies amounts to hardcoding curiosity in the utility as a terminal goal, while I expect curiosity to be CEV instrumental (curiosity as a property of the superintelligence; not in the sense of keeping around intrinsically curious entities if that's what we want the superintelligence to value).

I think that in a universe large enough w.r.t. the rate of discount over similarity of computations of your utility, and for certain conditions about how you value size and complexity of the computations, devaluing copies would lead to the garden of God in Unsong: you start tiling the universe with maximally viable satisfactory computations; when you have filled the space of such computations given your "lattice spacing" of what counts as different, you go on creating less satisfactory computations, and so on; if you are not an utilitarian, you stop when you reach your arbitrary level of what counts as "neutral utility"; and, depending on the parameters, you either calculated this to happen at the end of the universe, or you go on repeating your whole construction until the universe is filled with copies of your hierarchy of entities. While instead a utilitarian (that values complexity low enough to not fill the whole universe with a single entity) would tile the universe with the optimal entities, with differences only insofar as its utility says to make these entities value differences and be satisfied.

What do you think about this? Do you see it as a problem? Do you think it's too unlikely to matter due to purely combinatorial reasons? Or else?

Generally, an agent that always assumes the worst unless proven wrong, would have no problem doing all kinds of horrible things, because by the assumption they are still an improvement over what would happen otherwise.