should the result be BE, then the row player would have known it, and he would have instead picked A rather than B

I think this thing that gets predictably falsified by accepted policy shouldn't be called knowledge, it's merely a meaningful proposition, in this case a false proposition.

So is the idea that when there's only one Pareto-optimal outcome above both players' fallback price, they'll pick it?

In this case it seems like the failure of this concept is when there's more than one such outcome and each player likes a different one (which would normally be indicated by no equilibrium of this type - the actual outcome might still depend on details of the players' source code even after they meet your rationality standards), but by chance there's a barely above-fallback outcome at the intersection of the outcomes the players really prefer, causing this procedure to return something that might not even be Pareto optimal.

Or maybe I'm misunderstanding, since you said you had a proof?

E.g.:

(5,1) (6,2) (4,4)
(9,6) (2,2) (2,6)
(5,5) (6,9) (1,5)

Have you heard about Infra-Bayesianism [LW · GW]?

If I get it correctly, the core idea is that "consider every possible scenario, use a maximin policy while caring about conterfactual branches", which is very similar to the idea presented in the linked post. The "Nirvana trick" in the other post is similar to just eliminating branches/cells, where the agent would take a different action from the predicted policy.

Non-Nashian Game Theory is Pareto optimal, Infra-Bayesianism implements Updateless Decision Theory. If the two are connected, that could mean that UDT and Pareto-optimality are connected too.

Apparently every finite game has at least one Nash equilibrium if you allow mixed strategies. (i.e. players select probability distributions over their moves, not single moves.) Do you happen to know how mixed strategies affect existence and uniqueness of PTEs?

(Aside, it seems that every image appears twice in the article.)

I see the Nash equilibrium as rationally justified in a limit-like sort of way. I see it as what you get if you get arbitrarily close to perfect rationality. Having a good enough model of another's preferences is something you can actually achieve or almost achieve, but you can't really have a good enough grasp of your opponent's source code to acausally coerce him into cooperating with you unless you really have God-like knowledge (or maybe if you are in a very particular situation such as something involving AI and literal source codes). In proportion as a mere mortal becomes more and more smart, he becomes more and more able to get the best deal by having a better grasp on the Nashian game-theoretic intricacies of a given situation, but he won't become any more able to acausally trade. It's all or nothing. I think your whole line of reasoning is a bit like objecting to calculus on the grounds that instantaneous change is an oxymoron (as people did when calculus still rested on less rigorous foundations). Non-Nashian game theory is technically correct, but less useful, just like pointing out to Leibniz that "(x^2+0-x^2)/(x+0-x) = undefined" or whatever

I think this is an important direction of work because, despite lots of concerns on this forum about the interpretability and explainability of ML, I think that in practice we should expect (in the worlds where we survive, at least) AI agents cooperating within systems (or playing against each other in games, or a mixture of these two modes) are going to be more transparent to each other than humans are transparent to each other.

People always think in private; sometimes give off their "real" thoughts and intentions via facial microexpressions and facial flushing, but less so in the era of remote communication.

AIs always learn by receiving and processing data and doing number crunching on the data, should probably expect that we will build infrastructure for logging this data, looking for signs of deception in the weights or activations selectively saved for future security processing. Moreover, if a single integrated stream of thought (like people have in their heads) proves important for general intelligence, we should expect all these streams of thought to be recorded.

I think it's also important to transfer insights from mathematical constructs (including UDT, FDT, superrationality, and games with perfect prediction) onto a physical footing. Here [LW(p) · GW(p)], I argued that FDT should be seen as a group (collective) decision theory, i. e., a piece of collective intelligence. In "An active inference model of collective intelligence", Kaufmann et al. proposed a physics-based explanatory theory of collective intelligence (I don't have an opinion about this theory, just indicate it as one of the proposals out there).

In such theories and game setups, I think it's important to consider bounded rationality (see "Resource-rational analysis: Understanding human cognition as the optimal use of limited computational resources", "Information overload for (bounded) rational agents"), communication costs, communication delays and the delays (and costs) of reaching consensus, and the non-zero cost of updating one's own beliefs (to the point of "Free Will" and the assumption that agents can deflect momentarily, which leads to the idea of pre-commitments in Newcomb problem and Parfit's hitchhiker, without considering that basically, any belief is a micro pre-commitment to this belief). Also, in iterated games, we should ensure that we don't model agents making ergodicity assumptions when it's not "real-world rational".