Anthropic answers to logical uncertainties?

You don’t need any complicated relationship between the RH and the number of observers. Just consider the very simple “possibility” that 1+0 = 1E500. Worlds in which this is true have lots more observers than those where 1+0 = 1 :-)

I would argue that anthropic considerations should not move us on such logical facts.

Unless you move fully to UDT (which you don’t seem willing to do, at least for the purposes of this post), such a rule will lead you astray. Consider this thought experiment:

Omega appears and says that a minute ago he generated a physically random number R between 10^9 and 10^10, and if the R-th bit of π is 1, made 100 copies of Earth and scattered them throughout the universe. (He’s appearing to you in all 100 copies if that’s the case.) A minute from now, he will reveal R to you, and then you’ll be given an opportunity to bet $100 on the R-th bit of π being 1 at 1:1 odds.

What do you want your future self to do? If you apply SSA/SIA now, you would conclude that you’re likely to live in a universe where Omega made 100 copies of Earth, in other words a universe where the number R that Omega generated is such that the R-th bit of π is 1. So you have a greater expected utility if your future self were to take the bet.

But once you learn R, and if you follow the proposed rule, you’d become indifferent between taking the bet and not taking it, because whatever R turns out to be, you’d think that the R-th bit of π being 1 has probability .5, since that’s a reasonable prior and you’re not willing to let anthropic considerations move you on that purely logical fact.

If you just use utility function that favors big number of observers getting personal utility, then you'll automatically act as if you are in the many-observers world when you are uncertain. This doesn't require revising the level of belief in the logical fact, and allows to switch to the minority option upon discovering the corresponding solution. The same setting allows to estimate the value of new logical information about the fact. (This has to refer to some standard position on anthropics.)

You seem to be rejecting the "impossible possible worlds" framework, and count what happens in "possible" but not actual worlds more than in "impossible" worlds. Neither world is actual, so why treat them so differently?

Observers who would only exist in logically impossible worlds can't make bets, so the "collective sucker" arguments don't really work.

I'm confused as to how and whether this property of "logical possibility" leaks through to affect the rationality of our decisions. I mean, people in non-actual possible worlds don't actually make bets, either. So if we don't care about bets that can't possibly get made, why would we care about bets that don't actually get made?

It seems to me that I can rationally take favorable bets on far-out digits of pi, even though it's conceivable that it makes all logically possible versions of me worse off. If I bet that some far-out digit is 7, at 1-100 odds, then I should even expect that probably, I will make all possible versions of myself worse off; and yet it's still the rational thing to do. So how is it different when you maximize EU through anthropic reasoning?

You should probably cite Bostrom for the Presumptuous Philosopher thought experiment.

One ominous possibility is that the standard model is such that advanced civilizations always destroy themselves by doing some lethal physics experiment.

If this is true, it requires a lot of revising of our (logical rather than empirical) beliefs, because it sure seems as though we could in principle colonize space without doing further physics experiments, and some civilizations similar to us will.

The Riemann argument seems to differ from the Great Filter argument in this way: the Riemann argument depends only on the sheer number of observers, i.e. the only thing you're taking into account is the fact that you exist. Whereas in the great filter argument you're updating based on what kind of observer you are, i.e. you're intelligent but not a space-travelling, uploaded posthuman.

The first kind of argument doesn't work because somebody exists either way: if the RH or whatever is false then you are one of a small number, if it's true than you are one of a large number, you are in a typical position either way, and the other situation simply isn't possible. But the second kind of argument seems to hold more merit: if the great filter is behind then you are part of the extreme minority of normal humans, but if the great filter is ahead then you are rather typical of intelligent lifeforms. This might count as evidence, and it seems to be the same kind of evidence which suggests that a great filter even exists in the first place: if it doesn't then we are very exceptional not only in being the very first humans but the very first intelligent life as well.

the justification for reasoning anthropically is that the set Ω of observers in your reference class maximizes its combined winnings on bets if all members of Ω reason anthropically

That is a justification for it, yes.

When most of the members of Ω arise from merely non-actual possible worlds, this reasoning is defensible.

Roko, on what do you base that statement? Non-actual observers do not participate in bets.

The SIA is not an example of anthropic reasoning; anthropic implies observers, not "non-actual observers".

See this post for an example of the difference, showing why the SIA is false.

Do we believe this argument?

I believe this argument. One way of thinking about it, that may seem less counter-intuitive, is to imagine that the RH is one of a large group of math proposition each of which (for arguments sake) we give a subjective probability of 50% of being true.

Then if one of the propositions X is selected at random, and we know that X implies 10^500 times more observers. In this situation, we are more in a situation akin to the old fashioned SIA situation; we expect that roughly half the propositions will be true, so standard proability is all we need to state SIA.

But now suppose we find out that X is the RH, and that we don't get any extra information about the likely truth of the RH. By Bayseian rules, if we wish to shift our proability away from standard SIA (towards, for example, making the large universe less likely), then there must be some mathematical proposition that, if we used it instead of the RH would shift the probability the other way (towards making the large universe more likely). Since all we know about these propositions is that they all have a subjective proability of 50% of being true, making them interchangeable, this cannot be the case.

Take home message: SIA for uncertainty over empirical facts works only iff SIA for uncertainty over logical facts does as well.

Can I repeat it again: there are priors for which Katja's argument decreases the probability of a late filter.

You may feel these priors are unintuitive, but please don't base "logical (not empirical) facts" on things that rely on intuition.

Do we believe this argument?

It's not a proof. But it might make RH more plausible.

Mathematics is full of situations where an "empirical logical fact" is held to affect the plausibility of some other mathematical proposition we cannot yet decide. I cannot offhand think of an example of a physical observation being used that way, but it may have occurred. E.g. some quantum system might have equations of motion we don't know how to solve, but empirically we can see that it's stable, and from this we can infer something about solutions to those equations.

Most plausible explanations for a future great filter are logical facts, not empirical ones. The difficulty of surviving a transition through technological singularities, if it convergently causes non-colonization, is some logical fact, derivable by a sufficiently powerful mind. A tendency for advanced civilizations to "realize" that expansionism is pointless is a logical fact. I would argue that anthropic considerations should not move us on such logical facts.

These may be logical facts for a "sufficiently powerful mind", but they are empirical facts to us. For that mind, there is no such thing as an empirical fact about our universe (except for recursive statements about itself), so any distrinction between empirical and logical facts is moot via this kind of argument.

Why do we even need to talk about great filters? We know perfectly well that intelligent life is so extremely unlikely, that we should expect far less than 1 such event per universe.