Proofs Theorems 2,3

post by Diffractor · 2021-12-03T18:48:45.985Z · LW · GW · 0 comments

Contents

No comments

Theorem 2: Acausal Commutative Square: The following diagram commutes for any belief function $\Theta$. Any infradistribution $D$ where $D={\top }_{\Pi }⋉(\lambda \pi .u_{\left\{\pi \right\}}^1(D\left)\right)$ also makes this diagram commute and induces a belief function.

Here's how the proof proceeds. In order to show that the full diagram commutes, we first select properties on the four corners such that for each morphism, if the source fulfills the relevant property, then the target fulfills the relevant property. Obviously, the property for the bottom-right corner must be "is a belief function". There are eight phases for this, some of which are quite easy, but the two most difficult are at the start, corresponding to the two morphisms on the bottom side, because we must exhaustively verify the belief function conditions and infradistribution conditions. As usual, lower-semicontinuity is the trickiest part to show. Then, since we have four sides of the square to show are isomorphisms, we have another eight phases where we verify going forward to a different corner and back is identity. Finally, there's an easy step where we rule out that going around  in a loop produces a nontrivial automorphism, by showing that both paths from first-person static to third-person dynamic are equal.

First, let's define our properties.

The property for third-person static is $D={\top }_{\Pi }⋉(\lambda \pi .p{r}_{(A\times O{)}^{\omega }\ast }(u_{\left\{\pi \right\}}^1\left(D\right)\left)\right)$, and $D$ is an infradistribution.

The property for third-person dynamic is ${\psi }_T={\top }_{\Pi }⋉(\lambda \pi .p{r}_{(A\times O{)}^{\omega }\ast }(u_{\left\{\pi \right\}}^1\left({\psi }_T\right)\left)\right)$, and ${\to }_T(\pi ,aoh)={\delta }_{a,o,({\pi }_{ao},h)}$, and ${\psi }_T$ is an infradistribution.

The property for first-person static is that $\Theta$ is a belief function.

The property for first-person dynamic is ${\psi }_F={\top }_{\Pi }⋉(\lambda \pi .p{r}_{(A\times O{)}^{\omega }\ast }(u_{\left\{\pi \right\}}^1\left({\psi }_F\right)\left)\right)$, and ${\to }_F\left(\right(\pi ,aoh),a^{\prime })={\delta }_{o,({\pi }_{ao},h)}$ when $a^{\prime }=a$, and $\perp$ when $a^{\prime }\ne a$, and ${\psi }_F$ is an infradistribution.

T2.1 Our first step is showing that all 8 morphisms induce the relevant property in the target if the source has the property.

T2.1.1 3-static to 1-static. Our definition is

$\Theta \left(\pi \right):=p{r}_{(A\times O{)}^{\omega }\ast }\left(u_{\left\{\pi \right\}}^1\right(D\left)\right)$

and we must check the five conditions of a belief function for $\Theta$, assuming $D$ is an infradistribution on $PD$ and it has the property $D={\top }_{\Pi }⋉(\lambda \pi .p{r}_{(A\times O{)}^{\omega }\ast }(u_{\left\{\pi \right\}}^1\left(D\right)\left)\right)$

T2.1.1.1 First, bounded Lipschitz constant. Our proof goal is

$\exists {\lambda }^{\odot }<\infty \forall {\pi }^{\prime },U,U^{\prime }:|\Theta \left(\pi \right)\left(U\right)-\Theta \left(\pi \right)\left(U^{\prime }\right)|\le {\lambda }^{\odot }d(U,U^{\prime })$

Let ${\lambda }^{\odot }$ be the Lipschitz constant of $D$, which is finite because $D$ is an infradistribution. Then let ${\pi }^{\prime },U,U^{\prime }$ be arbitrary (they must be bounded-continuous, though). We have

$|\Theta \left({\pi }^{\prime }\right)\left(U\right)-\Theta \left({\pi }^{\prime }\right)\left(U^{\prime }\right)|=|p{r}_{(A\times O{)}^{\omega }\ast }\left(u_{\left\{{\pi }^{\prime }\right\}}^1\right(D\left)\right)\left(U\right)-p{r}_{(A\times O{)}^{\omega }\ast }\left(u_{\left\{{\pi }^{\prime }\right\}}^1\right(D\left)\right)\left(U^{\prime }\right)|$

and then unpack the projection to get

$=|u_{\left\{{\pi }^{\prime }\right\}}^1\left(D\right)\left)\right(\lambda \pi ,h.U\left(h\right))-u_{\left\{{\pi }^{\prime }\right\}}^1(D\left)\right)(\lambda \pi ,h.U^{\prime }(h\left)\right)|$

and unpack the update to get

$=|D(\lambda \pi ,h.{\text{1}}_{\pi ={\pi }^{\prime }}U(h)+{\text{1}}_{\pi \ne {\pi }^{\prime }})-D(\lambda \pi ,h.{\text{1}}_{\pi ={\pi }^{\prime }}{U}^{\prime }(h)+{\text{1}}_{\pi \ne {\pi }^{\prime }})|$

and then by Lipschitzness for $D$,

$\le {\lambda }^{\odot }{sup}_{\pi ,h\sim \pi }|{\text{1}}_{\pi ={\pi }^{\prime }}U\left(h\right)+{\text{1}}_{\pi \ne {\pi }^{\prime }}-\left({\text{1}}_{\pi ={\pi }^{\prime }}{U}^{\prime }\right(h)+{\text{1}}_{\pi \ne {\pi }^{\prime }})|$

Now, when $\pi \ne {\pi }^{\prime }$, the two functions are equal, so we have

$={\lambda }^{\odot }{sup}_{h\sim {\pi }^{\prime }}|U\left(h\right)-U^{\prime }\left(h\right)|\le {\lambda }^{\odot }{sup}_h|U\left(h\right)-U^{\prime }\left(h\right)|={\lambda }^{\odot }d(U,U^{\prime })$

We'll leave lower-semicontinuity for the end, it's hard.

T2.1.1.2 Now, normalization, ie ${inf}_{\pi }\Theta \left(\pi \right)\left(1\right)=1\wedge {inf}_{\pi }\Theta \left(\pi \right)\left(0\right)=0$ To do this, we have

${inf}_{\pi }\Theta \left(\pi \right)\left(1\right)={inf}_{\pi }p{r}_{(A\times O{)}^{\omega }\ast }\left(u_{\left\{\pi \right\}}^1\right(D\left)\right)\left(1\right)$

Then this can be rewritten as

$={\top }_{\Pi }(\lambda \pi .p{r}_{(A\times O{)}^{\omega }\ast }(u_{\left\{\pi \right\}}^1\left(D\right)\left)\right(1\left)\right)$

and then as a semidirect product.

$=({\top }_{\Pi }⋉(\lambda \pi .p{r}_{(A\times O{)}^{\omega }\ast }\left(u_{\left\{\pi \right\}}^1\right(D\left)\right)\left)\right(1)$

and then, from our defining property for $D$, and normalization for $D$ since it's an infradistribution, we have

$=D\left(1\right)=1$

The same proof works for 0 as well.

T2.1.1.3 Now, sensible supports. Pick a ${\pi }^{\prime },U,U^{\prime }$ where $U$ and $U^{\prime }$ are identical on histories compatible with ${\pi }^{\prime }$. We want to show

$\Theta \left({\pi }^{\prime }\right)\left(U\right)=\Theta \left({\pi }^{\prime }\right)\left(U^{\prime }\right)$

To start,

$\Theta \left({\pi }^{\prime }\right)\left(U\right)=p{r}_{(A\times O{)}^{\omega }\ast }\left(u_{\left\{{\pi }^{\prime }\right\}}^1\right(D\left)\right)\left(U\right)$

and undoing the projection and update, we have

$=D(\lambda \pi ,h.{\text{1}}_{\pi ={\pi }^{\prime }}U(h)+{\text{1}}_{\pi \ne {\pi }^{\prime }})$

Remember that actually, $D$ is supported over $PD$, ie, the subset of $\Pi \times (A\times O{)}^{\omega }$ where the destiny is compatible with the policy, so when $\pi ={\pi }^{\prime }$, $h\sim \pi ={\pi }^{\prime }$ and so $U$ and $U^{\prime }$ behave identically. Making this substitution in, we have

$=D(\lambda \pi ,h.{\text{1}}_{\pi ={\pi }^{\prime }}{U}^{\prime }(h)+{\text{1}}_{\pi \ne {\pi }^{\prime }})$

which then packs up as $\Theta \left({\pi }^{\prime }\right)\left(U^{\prime }\right)$ and we're done.

T2.1.1.4 Now for agreement on max value. For the $[0,1]$ type signature on inframeasures, we want to show

$\forall {\pi }^{\prime },{\pi }^{\prime \prime }:\Theta \left({\pi }^{\prime }\right)\left(1\right)=\Theta \left({\pi }^{\prime \prime }\right)\left(1\right)$

Let ${\pi }^{\prime },{\pi }^{\prime \prime }$ be arbitrary. We have

$\Theta \left({\pi }^{\prime }\right)\left(1\right)=p{r}_{(A\times O{)}^{\omega }\ast }\left(u_{\left\{{\pi }^{\prime }\right\}}^1\right(D\left)\right)\left(1\right)$

and then the projection and update unpack to

$D(\lambda \pi ,h.{\text{1}}_{\pi ={\pi }^{\prime }}1+{\text{1}}_{\pi \ne {\pi }^{\prime }})=D\left(1\right)=D(\lambda \pi ,h.{\text{1}}_{\pi ={\pi }^{\prime \prime }}1+{\text{1}}_{\pi \ne {\pi }^{\prime \prime }})$

and then, working in reverse from there, we get $=\Theta \left({\pi }^{\prime \prime }\right)\left(1\right)$ and we're done. For the $R$ type signature, we want to show

$\forall {\pi }^{\prime },{\pi }^{\prime \prime }:\Theta \left({\pi }^{\prime }\right)\left(\infty \right)=\Theta \left({\pi }^{\prime \prime }\right)\left(\infty \right)$

Let ${\pi }^{\prime },{\pi }^{\prime \prime }$ be arbitrary. We have

$\Theta \left({\pi }^{\prime }\right)\left(\infty \right)=p{r}_{(A\times O{)}^{\omega }\ast }\left(u_{\left\{{\pi }^{\prime }\right\}}^{\infty }\right(D\left)\right)\left(\infty \right)$

which then unpacks as

$D(\lambda \pi ,h.{\text{1}}_{\pi ={\pi }^{\prime }}\infty +{\text{1}}_{\pi \ne {\pi }^{\prime }}\infty )=D\left(\infty \right)=D(\lambda \pi ,h.{\text{1}}_{\pi ={\pi }^{\prime \prime }}\infty +{\text{1}}_{\pi \ne {\pi }^{\prime \prime }}\infty )$

and then, working in reverse, we get $=\Theta \left({\pi }^{\prime \prime }\right)\left(\infty \right)$ and we're done.

T2.1.1.5 Time to revisit lower-semicontinuity. We want to show that

${liminf}_{n\to \infty }\Theta \left({\pi }_n\right)\left(U\right)\ge \Theta \left({\pi }_{\infty }\right)\left(U\right)$

This is going to take some rather intricate work with Propositions 2 and 3, and Theorem 1 from Inframesures and Domain Theory [AF · GW]. Namely, the supremum of continuous functions is lower-semicontinuous, any lower-semicontinuous lower-bounded function has an ascending sequence of lower-bounded continuous functions which limits to it pointwise, and for any ascending sequence which limits to a lower-semicontinuous function pointwise, you can shuffle the limit outside of an inframeasure. Fix some continuous bounded $U$, either in $[0,1]$ or in $R$, and sequence of policies ${\pi }_n$ which converge to ${\pi }_{\infty }$.

Our first task is to establish that the subset of $PD$ where $\pi ={\pi }_n$ or ${\pi }_{n+1}$ or ${\pi }_{n+2}$... or ${\pi }_{\infty }$ is closed. This can be viewed as the following subset of $\Pi \times (A\times O{)}^{\omega }$:

$\left(\right\{{\pi }_n{\}}_{n\ge m}\times (A\times O{)}^{\omega })\cap \{\pi ,h|h\sim \pi \}$

$(A\times O{)}^{\omega }$ is compact, and $\{{\pi }_n{\}}_{n\ge m}$ is compact as well, since it limits to a point and the limit point is included. So that product is a compact set, and must be closed. The latter set is $PD$ itself. And so, since this set is the intersection of a closed set with $PD$, it's closed in $PD$ with the subspace topology.

Now, define the functions $u_{\ge m}:PD\to R\bigsqcup \left\{\infty \right\}$ as follows. If $(\pi ,h)$ has $\pi ={\pi }_m$ or ${\pi }_{m+1}$ or ${\pi }_{m+2}$ or... or ${\pi }_{\infty }$, then return $U\left(h\right)$. Otherwise, return 1 (or $\infty$ for the other type signature).

$u_{\infty }$ is the function where, if $\pi ={\pi }_{\infty }$, return $U\left(h\right)$, otherwise return 1 (or $\infty$).

Our first task is to show that all the $u_{\ge m}$, and $u_{\infty }$ are lower-semicontinuous. This is pretty easy to do. They are 1 (or $\infty$) outside of a closed set, and less than that inside the closed set.

Lower-semicontinuity happens because we have one of three cases holding. In the first case, the limit point is outside of the closed set (in the complement open set). Then, at some finite stage and forever afterwards, the sequence is outside closed set of interest, and the sequence becomes just a constant value of 1 (or $\infty$), and so liminf equals the value of the limit.

In the second case, the limit point is inside the closed set, and there's a subsequence of policy-tagged destinies which remains within the closed set of interest. The liminf of the original sequence of policy-tagged destinies evaluated by $u_{\ge m}$ equals the liminf of the subsequence which stays in the "safe" closed set, because all other points in the sequence get maximal value according to $u_{\ge m}$ (because they're outside the closed set), so they're irrelevant for evaluating the liminf. Then, inside the closed set, $u_{\ge m}$ acts like $U$, which is continuous, so the liminf of $u_{\ge m}$ inside the closed set equals the limit of $U$ inside the closed set, which equals $u_{\ge m}$ applied to the limit point (in the closed set)

In the third case, the limit point is inside the closed set, but there's no subsequence of policy-tagged destinies which remains within the closed set of interest. So, after some finite stage, the value of $u_{\ge m}$ is 1 or infinity, so the liminf is high and the value of the limit point is low, and again we get lower-semicontinuity.

Finally, we need to show that $u_{\ge m}$ is an ascending sequence of functions which has $u_{\infty }$ as the pointwise limit. The "ascending sequence" part is easy, because as m increases, $u_{\ge m}$ maps a smaller and smaller set (fewer policies) to the value $U$ reports, and more points to 1 (or infinity).

If $\pi ,h$ has $\pi ={\pi }_{\infty }$, then regardless of n, $u_{\ge n}(\pi ,h)=U\left(h\right)=u_{\infty }(\pi ,h)$, so that works out. If $\pi ,h$ has $\pi \ne {\pi }_{\infty }$, then the sequence ${\pi }_n$ can't hit $\pi$ infinitely often, otherwise $\pi ={\pi }_{\infty }$, so after some finite n, $\pi ,h$ drops out of the closed set associated with $u_{\ge n}$, and we have ${lim}_{n\to \infty }\left(u_{\ge n}\right(\pi ,h\left)\right)=1=u_{\infty }(\pi ,h)$.

Just one more piece of setup. For each $u_{\ge m}$, since it's lower-semicontinuous, we can apply our Proposition 3 from the domain theory post to construct an ascending sequence of bounded continuous functions $f_{m,n}$ where ${lim}_{n\to \infty }{f}_{m,n}=u_{\ge m}$ pointwise.

NOW we can begin showing lower-semicontinuity of $\Theta$.

${liminf}_{n\to \infty }\Theta \left({\pi }_n\right)\left(U\right)={liminf}_{n\to \infty }p{r}_{(A\times O{)}^{\omega }\ast }\left(u_{\left\{{\pi }_n\right\}}^1\right(D\left)\right)\left(U\right)$

by how $\Theta$ was defined. And then we unpack the projection and update to get

$={liminf}_{n\to \infty }D(\lambda \pi ,h.{\text{1}}_{\pi ={\pi }_n}U(h)+{\text{1}}_{\pi \ne {\pi }_n})$

First, we observe that the function inside $D$ is greater than $u_{\ge n}$, because $u_{\ge n}$ copies $U\left(h\right)$ when $\pi ={\pi }_n$ or ${\pi }_{n+1}$ or ${\pi }_{n+2}$... and is maximal otherwise, while the interior function just copies $U\left(h\right)$ when $\pi ={\pi }_n$ and is maximal otherwise. So, by monotonicity, we can go

$\ge {liminf}_{n\to \infty }D\left(u_{\ge n}\right)$

And then, we can use monotonicity again to get

$\ge {liminf}_{n\to \infty }D\left({sup}_{m\le n}{f}_{m,n}\right)$

Wait, what's this? Well, if $m\le n$, then $f_{m,n}\le u_{\ge m}\le u_{\ge n}$, because $f_{m,n}$ was constructed to be an ascending sequence of functions (in n) limiting to $u_{\ge m}$. And $u_{\ge m}$ is an ascending sequence of functions (in m), so it's below $u_{\ge n}$. Thus, the function ${sup}_{m\le n}{f}_{m,n}$ lies below $u_{\ge n}$, we just applied monotonicity.

Our next claim is that ${sup}_{m\le n}{f}_{m,n}$ is an ascending sequence of continuous functions which limits pointwise to $u_{\infty }$. Here's how to show it. The functions are continuous because they're all the supremum of finitely many continuous functions $f_{m,n}$. They ascend because

${sup}_{m\le n+1}{f}_{m,n+1}\ge {sup}_{m\le n}{f}_{m,n+1}\ge {sup}_{m\le n}{f}_{m,n}$

The first inequality was from considering one less function in the sup. the second inequality is because, for all m, $f_{m,n}$ is an ascending sequence of functions in n.

For showing that the sequence ${sup}_{m\le n}{f}_{m,n}$ limits (in n) pointwise to $u_{\infty }$, we fix an arbitrary $m^{\ast }$ and do

${lim}_{n\to \infty }{max}_{m\le n}{f}_{m,n}(\pi ,h)\ge {lim}_{n\to \infty }{f}_{m^{\ast },n}(\pi ,h)=u_{\ge m^{\ast }}(\pi ,h)$

The inequality is because eventually n goes past the fixed number $m^{\ast }$, and then we're selecting a single function out of the supremum of functions. The equality is because $f_{m^{\ast },n}$ limits pointwise in $n$ to $u_{\ge m^{\ast }}$. Now, with this, we can go:

$u_{\infty }(\pi ,h)={lim}_{n\to \infty }{u}_{\ge n}(\pi ,h)={lim}_{n\to \infty }{max}_{m\le n}{u}_{\ge m}(\pi ,h)$

$\ge {lim}_{n\to \infty }{max}_{m\le n}{f}_{m,n}(\pi ,h)\ge {sup}_{m^{\ast }}{u}_{\ge m^{\ast }}(\pi ,h)=u_{\infty }(\pi ,h)$

Let's break that down. The first equality was because we showed that $u_{\infty }$ is the pointwise limit of the $u_{\ge n}$. The second equality is because $u_{\ge m}$ makes an ascending sequence of functions, so the supremum of the functions would just be $u_{\ge n}$ itself. The first inequality is because $f_{m,n}\le u_{\ge m}$, it's one of the approximating continuous functions. The second inequality was the thing we showed earlier, about how the limit exceeds $u_{\ge m^{\ast }}(\pi ,h)$ regardless of $m^{\ast }$. Finally, we use again that $u_{\infty }$ is the pointwise limit of the $u_{\ge m^{\ast }}$.

We just showed a quantity is greater than or equal to itself, so all these inequalities must be equalities, and we have

$u_{\infty }(\pi ,h)={lim}_{n\to \infty }{max}_{m\le n}{f}_{m,n}(\pi ,h)$

So, the ascending sequence of continuous functions ${sup}_{m\le n}{f}_{m,n}$ limits pointwise to $u_{\infty }$. Now that we have that, let's recap what we've got so far.We have

${liminf}_{n\to \infty }\Theta \left({\pi }_n\right)\left(U\right)\ge {liminf}_{n\to \infty }D\left({sup}_{m\le n}{f}_{m,n}\right)$

Since ${sup}_{m\le n}{f}_{m,n}$ is an ascending sequence of continuous functions in $n$, then by monotonicity of $D$, that sequence must be increasing, so we have

$={lim}_{n\to \infty }D\left({sup}_{m\le n}{f}_{m,n}\right)$

and then, we can apply the monotone convergence theorem for infradistributions, that a sequence of continuous functions ascending pointwise to a lower-semicontinuous function has the limit of the infradistribution expectations equaling the expectation of the lower-semicontinuous supremum, so we have

$=D\left(u_{\infty }\right)=D(\lambda \pi ,h.{\text{1}}_{\pi ={\pi }_{\infty }}U(h)+{\text{1}}_{\pi \ne {\pi }_{\infty }})$

and then we pack this back up as an update and projection

$=p{r}_{(A\times O{)}^{\omega }\ast }\left(u_{\left\{{\pi }_{\infty }\right\}}^1\right(D\left)\right)\left(U\right)$

and back up to get

$=\Theta \left({\pi }_{\infty }\right)\left(U\right)$

and lower-semicontinuity of $\Theta$ has been shown. So translating over an infradistribution makes a belief function.

T2.1.2 1-static to 3-static.

$D:={\top }_{\Pi }⋉\Theta$

Now, we'll show the five infradistribution conditions on $D$, along with the sixth condition on $D$ that it equals ${\top }_{\Pi }⋉(\lambda \pi .p{r}_{(A\times O{)}^{\omega }\ast }(u_{\left\{\pi \right\}}^1\left(D\right)\left)\right)$.

First, to show it's even well-defined, we have

$D\left(f\right)=({\top }_{\Pi }⋉\Theta )\left(f\right)={\top }_{\Pi }(\lambda \pi .\Theta (\pi \left)\right(\lambda h.f(\pi ,h)\left)\right)$

That inner function is lower-semicontinuous in $\pi$, because, starting with

${liminf}_{n\to \infty }\Theta \left({\pi }_n\right)(\lambda h.f({\pi }_n,h\left)\right)$

then we have that, because $f$ is continuous and $\Pi \times (A\times O{)}^{\omega }$ is compact, $f$ is uniformly continuous. So, as n increases, the function $\lambda h.f({\pi }_n,h)$ uniformly limits to $\lambda h.f({\pi }_{\infty },h)$. Since all the $\Theta \left({\pi }_n\right)$ have a uniform upper bound on the Lipschitz constant (from $\Theta$ being a belief function) they converge to agreeing that $\lambda h.f({\pi }_n,h)$ has similar value as $\lambda h.f({\pi }_{\infty },h)$, so we get

$={liminf}_{n\to \infty }\Theta \left({\pi }_n\right)(\lambda h.f({\pi }_{\infty },h\left)\right)$

and then, by lower-semicontinuity for $\Theta$,

$\ge \Theta \left({\pi }_{\infty }\right)(\lambda h.f({\pi }_{\infty },h\left)\right)$

So, since the inner function is lower-semicontinuous, it can be evaluated.

T2.1.2.1 Now for the condition that

$D={\top }_{\Pi }⋉(\lambda \pi .p{r}_{(A\times O{)}^{\omega }\ast }(u_{\left\{\pi \right\}}^1\left(D\right)\left)\right)$

We start with $D\left(f\right)$ and unpack the definition of $D$, to get

$=({\top }_{\Pi }⋉\Theta )\left(f\right)$

And unpack the definition of semidirect product and what ${\top }_{\Pi }$ means, to get

$={inf}_{\pi }\Theta \left(\pi \right)(\lambda h.f(\pi ,h\left)\right)$

$={inf}_{\pi }{inf}_{{\pi }^{\prime }}\Theta \left({\pi }^{\prime }\right)(\lambda h.{\text{1}}_{{\pi }^{\prime }=\pi }f(\pi ,h)+{\text{1}}_{{\pi }^{\prime }\ne \pi })$

This step happens because, if ${\pi }^{\prime }\ne \pi$, then $\Theta \left({\pi }^{\prime }\right)$ will be assessing the value of the constant-1 function (or constant-infinity function), and will return a maximal value (by condition 5 on belief functions, that they all agree on the value of 1 (or $\infty$)). So the only way to minimize that line via a choice of ${\pi }^{\prime }$ is to make it equal to $\pi$, as that means that we're evaluating $\Theta \left(\pi \right)(\lambda h.f(\pi ,h\left)\right)$ instead, which may be less than the maximal value.

Now we pack up what ${\top }_{\Pi }$ means, and the semidirect product, to get

$={inf}_{\pi }({\top }_{\Pi }⋉\Theta )(\lambda {\pi }^{\prime },h.{\text{1}}_{{\pi }^{\prime }=\pi }f(\pi ,h)+{\text{1}}_{{\pi }^{\prime }\ne \pi })$

Pack up the definition of $D$ to get

$={inf}_{\pi }D(\lambda {\pi }^{\prime },h.{\text{1}}_{{\pi }^{\prime }=\pi }f(\pi ,h)+{\text{1}}_{{\pi }^{\prime }\ne \pi })$

Pack up the definition of "1-update on $\pi$" to get

$={inf}_{\pi }{u}_{\left\{\pi \right\}}^1\left(D\right)(\lambda {\pi }^{\prime },h.f(\pi ,h\left)\right)$

Pack up how projection mappings work, to get:

$={inf}_{\pi }p{r}_{(A\times O{)}^{\omega }\ast }\left(u_{\left\{\pi \right\}}^1\right(D\left)\right)(\lambda h.f(\pi ,h\left)\right)$

Pack up what ${\top }_{\Pi }$ means and the semidirect product to get

$=({\top }_{\Pi }⋉(\lambda \pi .p{r}_{(A\times O{)}^{\omega }\ast }\left(u_{\left\{\pi \right\}}^1\right(D\left)\right)\left)\right)\left(f\right)$

So we have

$D={\top }_{\Pi }⋉(\lambda \pi .p{r}_{(A\times O{)}^{\omega }\ast }(u_{\left\{\pi \right\}}^1\left(D\right)\left)\right)$

as desired, since $f$ was arbitrary.

T2.1.2.2 Now to verify the infradistribution conditions on $D$. First, monotonicity. Let $f^{\prime }\ge f$

$D\left(f^{\prime }\right)=({\top }_{\Pi }⋉\Theta )\left(f^{\prime }\right)$

And unpack the definition of semidirect product and what ${\top }_{\Pi }$ means, to get

$={inf}_{\pi }\Theta \left(\pi \right)(\lambda h.f^{\prime }(\pi ,h\left)\right)$

Then, by monotonicity for all the $\Theta \left(\pi \right)$, we have

$\ge {inf}_{\pi }\Theta \left(\pi \right)(\lambda h.f(\pi ,h\left)\right)$

which packs back up in reverse as $D\left(f\right)$.

T2.1.2.3 Now for concavity. Start with $D(pf+(1-p\left)f^{\prime }\right)$ and unpack in the usual way to get

$={inf}_{\pi }\Theta \left(\pi \right)(\lambda h.pf(\pi ,h)+(1-p\left)f^{\prime }\right(\pi ,h\left)\right)$

Then, by concavity for the $\Theta \left(\pi \right)$, as they're inframeasures, we have

$\ge {inf}_{\pi }\left(p\Theta \right(\pi \left)\right(\lambda h.f(\pi ,h))+(1-p\left)\Theta \right(\pi \left)\right(\lambda h.f^{\prime }(\pi ,h)\left)\right)$

and distribute the inf

$\ge {inf}_{\pi }\left(p\Theta \right(\pi \left)\right(\lambda h.f(\pi ,h)\left)\right)+{inf}_{\pi }\left(\right(1-p\left)\Theta \right(\pi \left)\right(\lambda h.f^{\prime }(\pi ,h)\left)\right)$

Pulling the constants out, we have

$=p{inf}_{\pi }\left(\Theta \right(\pi \left)\right(\lambda h.f(\pi ,h)\left)\right)+(1-p){inf}_{\pi }\left(\Theta \right(\pi \left)\right(\lambda h.f^{\prime }(\pi ,h)\left)\right)$

and packing back up in the usual way, we have

$=pD\left(f\right)+(1-p)D\left(f^{\prime }\right)$

Concavity is shown.

T2.1.2.4 For Lipschitzness, we start with $|D\left(f\right)-D\left(f^{\prime }\right)|$ and then partially unpack in the usual way to get

$=|{\top }_{\Pi }(\lambda \pi .\Theta (\pi \left)\right(\lambda h.f(\pi ,h)\left)\right)-{\top }_{\Pi }(\lambda \pi .\Theta (\pi \left)\right(\lambda h.f^{\prime }(\pi ,h)\left)\right)|$

And, since ${\top }_{\Pi }$ is a sharp infradistribution, it has a Lipschitz constant of 1, so we have

$\le {sup}_{\pi }|\Theta \left(\pi \right)(\lambda h.f(\pi ,h\left)\right)-\Theta \left(\pi \right)(\lambda h.f^{\prime }(\pi ,h\left)\right)|$

Since all the$\Theta \left(\pi \right)$ have a uniform bound on their Lipschitz constants (one of the belief function conditions), we then have

$\le {sup}_{\pi }{\lambda }^{\odot }{sup}_h|f(\pi ,h)-f^{\prime }(\pi ,h)|$

which rearranges as

$={\lambda }^{\odot }{sup}_{\pi ,h}|f(\pi ,h)-f^{\prime }(\pi ,h)|={\lambda }^{\odot }d(f,f^{\prime })$

and we're done, we got a Lipschitz constant.

T2.1.2.5 The CAS property for $D$ is trival because $PD$ is a closed subset of $\Pi \times (A\times O{)}^{\omega }$ which is a compact space, and so you can just take all of $PD$ as a compact support.

T2.1.2.6 That just leaves normalization. We have

$D\left(1\right)={inf}_{\pi }\Theta \left(\pi \right)\left(1\right)=1$

$D\left(0\right)={inf}_{\pi }\Theta \left(\pi \right)\left(0\right)=0$

by unpacking $D$ and applying normalization for a belief function.

So now, $D$ is an infradistribution.

T2.1.3 For 3-static to 3-dynamic, we have

${\to }_T(\pi ,aoh):={\delta }_{a,o,{\pi }_{ao},h}$ (which fulfills one of the 3-dynamic conditions), and ${\psi }_T:=D$

Since $D$ is an infradistribution, ${\psi }_T$ is too, and for the condition on ${\psi }_T$, we have:

${\psi }_T=D={\top }_{\Pi }⋉(\lambda \pi .p{r}_{(A\times O{)}^{\omega }\ast }(u_{\left\{\pi \right\}}^1\left(D\right)\left)\right)={\top }_{\Pi }⋉(\lambda \pi .p{r}_{(A\times O{)}^{\omega }\ast }(u_{\left\{\pi \right\}}^1\left({\psi }_T\right)\left)\right)$

and we're done, since $D$ had the relevant condition.

T2.1.4 For 3-dynamic to 3-static, we have

$D:=p{r}_{\Pi \times (A\times O{)}^{\omega }\ast }({\psi }_T⋉{\to }_{T:\infty })$

First, we must show that the niceness conditions are fulfilled for the infinite semidirect product to be defined at all. ${\to }_T$ maps $\pi ,aoh$ to $a,o,({\pi }_{ao},h)$. This is clearly a continuous mapping, so in particular, it's lower-semicontinuous. Dirac-delta distributions are 1-Lipschitz when interpreted as inframeasures (all probability distributions have that property, actually). The compact-shared-CAS condition is redundant because $A\times O\times PD$ is already a compact space. And all probability distributions (dirac-delta distributions are probability distributions) map constants to the same constant, so we get the increase-in-constants property and the 1 maps to 1 property. So we can indeed take the infinite semidirect product of the ${\to }_T$.

Now, we'll show that unrolling like this just produces ${\psi }_T$ exactly. We'll use $s_0$ for the initial state (pair of policy and destiny, also written as $(\pi ,h{)}_0$), and $ao{s}_{1:\infty }$ for the successive unrolled history of states, actions, and observations. First, by unpacking $D$'s definition,

$D\left(f\right)=p{r}_{\Pi \times (A\times O{)}^{\omega }\ast }({\psi }_T⋉{\to }_{T:\infty })\left(f\right)$

Then we unpack the projection and semidirect product

$={\psi }_T\left(\lambda \right(\pi ,h{)}_0.{\to }_{T:\infty }\left(\right(\pi ,h{)}_0\left)\right(\lambda ao{s}_{1:\infty }.f(\pi ,a{o}_{1:\infty })\left)\right)$

And pack things up into a projection

$={\psi }_T\left(\lambda \right(\pi ,h{)}_0.p{r}_{(A\times O{)}^{\omega }\ast }\left({\to }_{T:\infty }\right((\pi ,h{)}_0)\left)\right(\lambda a{o}_{1:\infty }.f(\pi ,a{o}_{1:\infty })\left)\right)$

And then we can observe that unrolling the initial state forever via "advance the destiny 1 step, popping actions and observations off, and advance the policy", when you project down to just the actions and observations (not the intermediate states), yields just the point distribution on that destiny, so we get:

$={\psi }_T\left(\lambda \right(\pi ,h{)}_0.{\delta }_h(\lambda h^{\prime }.f(\pi ,h^{\prime }\left)\right))$

And then substitute the value of that dirac-delta in to get

$={\psi }_T(\lambda \pi ,h.f(\pi ,h\left)\right)={\psi }_T\left(f\right)$

So, our $D={\psi }_T$. Now we can go

$D={\psi }_T={\top }_{\Pi }⋉(\lambda \pi .p{r}_{(A\times O{)}^{\omega }\ast }(u_{\left\{\pi \right\}}^1\left({\psi }_T\right)\left)\right)={\top }_{\Pi }⋉(\lambda \pi .p{r}_{(A\times O{)}^{\omega }\ast }(u_{\left\{\pi \right\}}^1\left(D\right)\left)\right)$

and we're done, showing $D$ had the relevant condition from ${\psi }_T$ having it. Since ${\psi }_T$ is an infradistribution, $D$ is too.

T2.1.5 For 3-dynamic to 1-dynamic, we have

${\to }_F(\pi ,h,a):=p{r}_{O\times PD\ast }\left(u_{\left\{a\right\}}^1\right({\to }_T(\pi ,h)\left)\right)$

For the action $\pi \left(\right)$ (which is compatible with the start of the history), we have 

${\to }_F(\pi ,\pi ()oh,\pi (\left)\right)\left(f\right)$

Applying our definition, we have

$=p{r}_{O\times PD\ast }\left(u_{\left\{\pi \right(\left)\right\}}^1\right({\to }_T(\pi ,\pi (\left)oh\right)\left)\right)\left(f\right)$

Unpacking the projection, we have (where $s$ is a state)

$=u_{\left\{\pi \right(\left)\right\}}^1\left({\to }_T\right(\pi ,\pi \left(\right)oh\left)\right)(\lambda a,o,s.f(o,s\left)\right)$

Unpacking the update, we have

$={\to }_T(\pi ,\pi (\left)oh\right)(\lambda a,o,s.{\text{1}}_{a=\pi \left(\right)}f(o,s)+{\text{1}}_{a\ne \pi \left(\right)})$

And then we consider that ${\to }_T(\pi ,\pi (\left)oh\right)$ is just the dirac-delta distribution on $\left(\pi \right(),o,({\pi }_{\pi \left(\right)o},h\left)\right)$, so we have

$={\delta }_{\pi \left(\right),o,({\pi }_{\pi \left(\right)o},h)}(\lambda a,o,s.{\text{1}}_{a=\pi \left(\right)}f(o,s)+{\text{1}}_{a\ne \pi \left(\right)})$

Substitute the values in, and we have

$=f(o,({\pi }_{\pi \left(\right)o},h\left)\right)$

and then pack up the dirac-delta and we get

$={\delta }_{o,({\pi }_{\pi \left(\right)o},h)}\left(f\right)$

So, ${\to }_F(\pi ,\pi ()oh,\pi (\left)\right)$ is just the dirac-delta distribution on $o$ and the pair of $h$ and the policy advanced one step, as it should be. For $a^{\prime }$ that aren't $\pi \left(\right)$ as it should be, we have

${\to }_F(\pi ,\pi ()oh,a^{\prime })\left(f\right)$

Applying our definition, we have

$=p{r}_{O\times PD\ast }\left(u_{\left\{a^{\prime }\right\}}^1\right({\to }_T(\pi ,\pi (\left)oh\right)\left)\right)\left(f\right)$

Unpacking the projection, we have (where $s$ is a state)

$=u_{\left\{a^{\prime }\right\}}^1\left({\to }_T\right(\pi ,\pi \left(\right)oh\left)\right)(\lambda a,o,s.f(o,s\left)\right)$

Unpacking the update, we have

$={\to }_T(\pi ,\pi (\left)oh\right)(\lambda a,o,s.{\text{1}}_{a=a^{\prime }}f(o,s)+{\text{1}}_{a\ne a^{\prime }})$

And then we consider that ${\to }_T(\pi ,\pi (\left)oh\right)$ is just the dirac-delta distribution on $\left(\pi \right(),o,({\pi }_{\pi \left(\right)o},h\left)\right)$, so we have

$={\delta }_{\pi \left(\right),o,({\pi }_{\pi \left(\right)o},h)}(\lambda a,o,s.{\text{1}}_{a=a^{\prime }}f(o,s)+{\text{1}}_{a\ne a^{\prime }})$

Substitute the values in, and remembering that $a^{\prime }$ and $\pi \left(\right)$ are assumed to be different, and we have (regardless of $f$)

$=1$

Or infinity for the other type signature. Therefore, 

${\to }_F(\pi ,\pi ()oh,a^{\prime })=\perp$

So our transition kernel works out. For the other conditions on ${\psi }_F$, just observe that ${\psi }_F:={\psi }_T$, and use the extreme similarity of their defining conditions.

T2.1.6 For 1-dynamic to 3-dynamic, we have ${\psi }_T:={\psi }_F$ so we can use our defining property of ${\psi }_F$ again, clearly show that ${\psi }_T$ has the relevant defining properties, so that just leaves cleaning up the defining property for the ${\to }_T$ infrakernel. We remember that

${\to }_T(\pi ,h):={\top }_A⋉(\lambda a.{\to }_F(\pi ,h,a\left)\right)$

Therefore, we have:

${\to }_T(\pi ,\pi (\left)oh\right)\left(f\right)=({\top }_A⋉(\lambda a.{\to }_F(\pi ,\pi ()oh,a)\left)\right)\left(f\right)$

This occurs because the history must be compatible with the policy. Then, we can unpack as:

$={\top }_A(\lambda a.{\to }_F(\pi ,\pi \left(\right)oh,a\left)\right(\lambda o,s.f(a,o,s)\left)\right)$

We unpack our ${\top }_A$, getting

$={inf}_a\left({\to }_F\right(\pi ,\pi \left(\right)oh,a\left)\right(\lambda o,s.f(a,o,s)\left)\right)$

Then, we observe that for any $a\ne \pi \left(\right)$, ${\to }_F(\pi ,\pi ()oh,a)=\perp$. So, $a$ must be $\pi \left(\right)$, and we have

$={\to }_F(\pi ,\pi ()oh,\pi (\left)\right)(\lambda o,s.f(\pi \left(\right),o,s\left)\right)$

Now, ${\to }_F(\pi ,\pi ()oh,\pi (\left)\right)$ is the dirac-delta distribution on $(o,(h,{\pi }_{\pi \left(\right)o}\left)\right)$, so making that substitution in, we have

$={\delta }_{o,(h,{\pi }_{\pi \left(\right)o})}(\lambda o,s.f(\pi \left(\right),o,s\left)\right)$

Ie

$=f\left(\pi \right(),o,(h,{\pi }_{\pi \left(\right)o}\left)\right)={\delta }_{\pi \left(\right),o,(h,{\pi }_{\pi \left(\right)o})}\left(f\right)$

And this holds for all $f$, so we get our desired result that

${\to }_T(\pi ,\pi (\left)oh\right)={\delta }_{\pi \left(\right),o,(h,{\pi }_{\pi \left(\right)o})}$

T2.1.7 For 1-static to 1-dynamic, we just have

${\psi }_F:={\top }_{\Pi }⋉\Theta$

${\to }_F(\pi ,aoh,a):={\delta }_{o,({\pi }_{ao},h)}$

${\to }_F(\pi ,aoh,a^{\prime }):=\perp$

So this automatically makes the infrakernel have the desired properties. For showing the relevant properties of ${\psi }_F$, we just copy the proof at the start of getting the properties for $D$ from $\Theta$ being a belief function, since $D$ and ${\psi }_F$ are both defined in the same way.

T2.1.8 For 1-dynamic to 1-static, we have

$\Theta \left({\pi }^{\prime }\right):=p{r}_{(A\times O{)}^{\omega }\ast }\left(\right(u_{\left\{{\pi }^{\prime }\right\}}^1\left({\psi }_F\right))⋉{\to }_{F:\infty }^{{\pi }^{\prime }})$

We'll work on getting that latter quantity into a better form, but first we have to verify that
${\to }_{F:\infty }^{\pi }$ is even well-defined at all, w need the ${\to }_{F:n}^{\pi }$ to fulfill the niceness conditions. They're defined as

${\to }_{F:n}^{\pi }(s_0,ao{s}_{1:n}):={\delta }_{\pi \left(a{o}_{1:n}\right)}⋉(\lambda a.{\to }_F(s_n,a\left)\right)$

For lower-semicontinuity, ${\to }_F$ is lower-semicontinuous, because it returns $\perp$ on every action but one (this is a clopen set), and for the action that pairs up with the initial state, the policy-tagged history just has the history and policy advanced, which is continuous. ${\to }_{F:n}^{\pi }$ behaves similarly, continuous variation in input leads to continuous variation in output, because eventually the behavior of the policy settles down to a fixed action as the history of actions and observations stabilizes amongst one of the finitely many options. 

For making inframeasures, ${\to }_{F:n}^{\pi }$ always returns a dirac-delta distribution or $\perp$, so we're good there. Similarly, for 1-Lipschitzness, both $\perp$ and dirac-delta distributions are 1-Lipschitz. For compact-shared CAS, as the target space is $A\times O\times PD$, it's compact and you don't need to worry about that. Finally, both dirac-delta distributions and $\perp$ map constants to either the same constant or higher, and in the $[0,1]$ type signature, both $\perp$ and dirac-delta distributions map 1 to 1. So all niceness conditions are fulfilled.

Let's work on unpacking the value

$p{r}_{(A\times O{)}^{\omega }\ast }\left(\right(u_{\left\{{\pi }^{\prime }\right\}}^1\left({\psi }_F\right))⋉{\to }_{F:\infty }^{\pi })\left(U\right)$

unpacking the projection and semidirect product, we get

$=u_{\left\{{\pi }^{\prime }\right\}}^1\left({\psi }_F\right)(\lambda \pi ,h.{\to }_{F:\infty }^{{\pi }^{\prime }}(\pi ,h\left)\right(\lambda ao{s}_{1:\infty }.U\left(a{o}_{1:\infty }\right)\left)\right)$

And unpacking the update, we have

$={\psi }_F(\lambda \pi ,h.{\text{1}}_{\pi ={\pi }^{\prime }}{\to }_{F:\infty }^{{\pi }^{\prime }}(\pi ,h\left)\right(\lambda ao{s}_{1:\infty }.U\left(a{o}_{1:\infty }\right))+{\text{1}}_{\pi \ne {\pi }^{\prime }})$

We can make a substitution which changes nothing (in the scope of the indicator function where $\pi ={\pi }^{\prime }$),  that

$={\psi }_F(\lambda \pi ,h.{\text{1}}_{\pi ={\pi }^{\prime }}{\to }_{F:\infty }^{{\pi }^{\prime }}({\pi }^{\prime },h\left)\right(\lambda ao{s}_{1:\infty }.U\left(a{o}_{1:\infty }\right))+{\text{1}}_{\pi \ne {\pi }^{\prime }})$

and then we can write this as a projection in the inside

$={\psi }_F(\lambda \pi ,h.{\text{1}}_{\pi ={\pi }^{\prime }}p{r}_{(A\times O{)}^{\omega }\ast }({\to }_{F:\infty }^{{\pi }^{\prime }}({\pi }^{\prime },h)\left)\right(\lambda a{o}_{1:\infty }.U\left(a{o}_{1:\infty }\right))+{\text{1}}_{\pi \ne {\pi }^{\prime }})$

And then we can realize that when we unroll ${\to }_{F:\infty }^{{\pi }^{\prime }}({\pi }^{\prime },h)$, it just always deterministically unrolls the history, along with the policy, since for ${\pi }^{\prime },h$ to be an input state, ${\pi }^{\prime }$ must be compatible with $h$, so picking actions from ${\pi }^{\prime }$ means there's never any action mismatches. Projecting this down to the actions and observations just makes the history $h$ exactly. So we have

$={\psi }_F(\lambda \pi ,h.{\text{1}}_{\pi ={\pi }^{\prime }}{\delta }_h(\lambda a{o}_{1:\infty }.U\left(a{o}_{1:\infty }\right))+{\text{1}}_{\pi \ne {\pi }^{\prime }})$

Substituting the dirac-delta value in, we have

$={\psi }_F(\lambda \pi ,h.{\text{1}}_{\pi ={\pi }^{\prime }}U(h)+{\text{1}}_{\pi \ne {\pi }^{\prime }})$

And we pack the update and projection back up to get

$=p{r}_{(A\times O{)}^{\omega }\ast }\left(u_{\left\{{\pi }^{\prime }\right\}}^1\right({\psi }_F\left)\right)\left(U\right)$

Since $U$ was arbitrary, we have

$p{r}_{(A\times O{)}^{\omega }\ast }\left(\right(u_{\left\{{\pi }^{\prime }\right\}}^1\left({\psi }_F\right))⋉{\to }_{F:\infty }^{\pi })=p{r}_{(A\times O{)}^{\omega }\ast }\left(u_{\left\{{\pi }^{\prime }\right\}}^1\right({\psi }_F\left)\right)$

And so, an alternate way to define $\Theta \left({\pi }^{\prime }\right)$ is as

$=p{r}_{(A\times O{)}^{\omega }\ast }\left(u_{\left\{{\pi }^{\prime }\right\}}^1\right({\psi }_F\left)\right)$

We can run through the same proof that $\Theta$ is indeed a belief function, back from the 3-static to 1-static case, because ${\psi }_F$ fulfills all the same properties that $D$ did.

Alright, now we have, for all four corners, a condition that's basically "came from an acausal belief function" that is preserved under morphisms. Now we need to show that all 8 back-and-forth directions are identity.

T2.2.1 For 1-static to 3-static back to 1-static, we want to show

$\Theta \left({\pi }^{\prime }\right)=p{r}_{(A\times O{)}^{\omega }\ast }\left(u_{\left\{{\pi }^{\prime }\right\}}^1\right({\top }_{\Pi }⋉\Theta \left)\right)$

This is (we assume the hard version first)

$p{r}_{(A\times O{)}^{\omega }\ast }\left(u_{\left\{{\pi }^{\prime }\right\}}^1\right({\top }_{\Pi }⋉\Theta \left)\right)\left(U\right)$

Then unpack the projection and update

$=({\top }_{\Pi }⋉\Theta )(\lambda \pi ,h.{\text{1}}_{\pi ={\pi }^{\prime }}U(h)+{\text{1}}_{\pi \ne {\pi }^{\prime }})$

Then unpack the semidirect product and ${\top }_{\Pi }$ to get

$={inf}_{\pi }\Theta \left(\pi \right)(\lambda h.{\text{1}}_{\pi ={\pi }^{\prime }}U(h)+{\text{1}}_{\pi \ne {\pi }^{\prime }})$

Then realize that the minimizer is picking ${\pi }^{\prime }$ exactly, otherwise you'd just get maximal value and all the $\Theta \left(\pi \right)$ agree on what a maximal input maps to.

$=\Theta \left({\pi }^{\prime }\right)(\lambda h.U(h\left)\right)=\Theta \left({\pi }^{\prime }\right)\left(U\right)$

And we're done.

T.2.2.2 For 3-static to 1-static to 3-static being identity, we want to show

$D={\top }_{\Pi }⋉(\lambda \pi .p{r}_{(A\times O{)}^{\omega }\ast }(u_{\left\{\pi \right\}}^1\left(D\right)\left)\right)$

This is just exactly the condition we're assuming on $D$, so we trivially fulfill this (the nontrivial part was shoved into showing that going belief-function to infradistribution over policy-tagged states produced this property).

T2.2.3 For 3-static to 3-dynamic back to 3-static, we need to show that

$D=p{r}_{\Pi \times (A\times O{)}^{\omega }\ast }(D⋉{\to }_{T:\infty })$

For this, we abbreviate states beyond the first one as $s$, so $(h,\pi {)}_0$ is the initial state, and $ao{s}_{1:\infty }$ is an element of $(A\times O\times PD{)}^{\omega }$. Taking the complicated part, it's

$p{r}_{\Pi \times (A\times O{)}^{\omega }\ast }(D⋉{\to }_{T:\infty })\left(f\right)$

$=(D⋉{\to }_{T:\infty })\left(\lambda \right(\pi ,h{)}_0,ao{s}_{1:\infty }.f(\pi ,a{o}_{1:\infty }))$

And then we unpack the semidirect product a bit, to get

$=D\left(\lambda \right(\pi ,h{)}_0.{\to }_{T:\infty }\left(\right(\pi ,h{)}_0\left)\right(\lambda ao{s}_{1:\infty }.f(\pi ,a{o}_{1:\infty })\left)\right)$

And then we can write this as a projection, to get

$=D\left(\lambda \right(\pi ,h{)}_0.p{r}_{(A\times O{)}^{\omega }\ast }\left({\to }_{T:\infty }\right((\pi ,h{)}_0)\left)\right(\lambda a{o}_{1:\infty }.f(\pi ,a{o}_{1:\infty })\left)\right)$

Now, we observe that because ${\to }_T$, when repeatedly unrolled, is just emitting the actions and observations from the starting destiny/history, this projection is just the point distribution on the action-observation sequence that is h.

$=D\left(\lambda \right(\pi ,h{)}_0.{\delta }_h(\lambda a{o}_{1:\infty }.f(\pi ,a{o}_{1:\infty }\left)\right))$

Then we evaluate the expectation, yielding

$=D(\lambda \pi ,h.f(\pi ,h\left)\right)=D\left(f\right)$

And we're done.

T2.2.4 For going 3-dynamic to 3-static to 3-dynamic, we need that

${\psi }_T=p{r}_{\Pi \times (A\times O{)}^{\omega }\ast }({\psi }_T⋉{\to }_{T:\infty })$

To show this, just reuse the exact same proof from above, just with ${\psi }_T$ instead of $D$. Also, ${\to }_F(\pi ,h)$ works out.

At this point, we've established isomorphism for two sides of the square. We've just got two more sides left to address, then showing that going around the square results in identity instead of a nontrivial automorphism.

T2.2.5,6 For translating back and forth between 3-dynamic and 1-dynamic, we observe that both translations keep the initial infradistribution over policy-tagged destinies the same, and our 3-dynamic to 1-dynamic, and 1-dynamic to 3-dynamic proofs verified that the ${\to }_F$ and ${\to }_T$ behave as they should when they go back and forth, so we don't need to worry about this.

T2.2.7 Next is 1-dynamic to 1-static to 1-dynamic. The infrakernel is guaranteed to have the right form, we just need to show that ${\psi }_F$ is unchanged. So, we must show

${\psi }_F={\top }_{\Pi }⋉(\lambda \pi .p{r}_{(A\times O{)}^{\omega }\ast }(\left(u_{\left\{\pi \right\}}^1\right({\psi }_F\left)\right)⋉{\to }_{F:\infty }^{\pi }\left)\right)$

But, since our condition on 1-dynamic (that we showed is a property of applying the morphism to go from any belief function to first-person dynamic) was

${\psi }_F={\top }_{\Pi }⋉(\lambda \pi .p{r}_{(A\times O{)}^{\omega }\ast }(u_{\left\{\pi \right\}}^1\left({\psi }_F\right)\left)\right)$

The only thing we need to show is that, regardless of $\pi$,

$p{r}_{(A\times O{)}^{\omega }\ast }\left(\right(u_{\left\{\pi \right\}}^1\left({\psi }_F\right))⋉{\to }_{F:\infty }^{\pi })=p{r}_{(A\times O{)}^{\omega }\ast }\left(u_{\left\{\pi \right\}}^1\right({\psi }_F\left)\right)$

For if we could show that, then we could go:

${\top }_{\Pi }⋉(\lambda \pi .p{r}_{(A\times O{)}^{\omega }\ast }(\left(u_{\left\{\pi \right\}}^1\right({\psi }_F\left)\right)⋉{\to }_{F:\infty }^{\pi }\left)\right)={\top }_{\Pi }⋉(\lambda \pi .p{r}_{(A\times O{)}^{\omega }\ast }(u_{\left\{\pi \right\}}^1\left({\psi }_F\right)\left)\right)={\psi }_F$

and we'd be done. So, we'll show that 

$p{r}_{(A\times O{)}^{\omega }\ast }\left(\right(u_{\left\{\pi \right\}}^1\left({\psi }_F\right))⋉{\to }_{F:\infty }^{\pi })=p{r}_{(A\times O{)}^{\omega }\ast }\left(u_{\left\{\pi \right\}}^1\right({\psi }_F\left)\right)$

instead, as that's all we need. Again, we'll massage the more complicated side until we get it into the simple form for the other side.

$p{r}_{(A\times O{)}^{\omega }\ast }\left(\right(u_{\left\{\pi \right\}}^1\left({\psi }_F\right))⋉{\to }_{F:\infty }^{\pi })\left(U\right)$

We undo the projection, and abbreviate states as $s_0$, and action-observation-state sequences as $ao{s}_{1:\infty }$, to yield

$=\left(\right(u_{\left\{\pi \right\}}^1\left({\psi }_F\right))⋉{\to }_{F:\infty }^{\pi })(\lambda s_0,ao{s}_{1:\infty }U(a{o}_{1:\infty }\left)\right)$

Then we reexpress the semidirect product, to yield

$=u_{\left\{\pi \right\}}^1\left({\psi }_F\right)(\lambda s_0.{\to }_{F:\infty }^{\pi }(s_0\left)\right(\lambda ao{s}_{1:\infty }.U\left(a{o}_{1:\infty }\right)\left)\right)$

We unpack the initial state a fuzz to yield

$=u_{\left\{\pi \right\}}^1\left({\psi }_F\right)(\lambda {\pi }^{\prime },h.{\to }_{F:\infty }^{\pi }({\pi }^{\prime },h\left)\right(\lambda ao{s}_{1:\infty }.U\left(a{o}_{1:\infty }\right)\left)\right)$

We apply the update to yield

$={\psi }_F(\lambda {\pi }^{\prime },h.{\text{1}}_{{\pi }^{\prime }=\pi }\cdot ({\to }_{F:\infty }^{\pi }({\pi }^{\prime },h)(\lambda ao{s}_{1:\infty }.U(a{o}_{1:\infty }\left)\right))+{\text{1}}_{{\pi }^{\prime }\ne \pi })$

Then, we realize that if ${\pi }^{\prime }=\pi$, we can swap ${\pi }^{\prime }$ out for $\pi$ in the relevant function associated with it.

$={\psi }_F(\lambda {\pi }^{\prime },h.{\text{1}}_{{\pi }^{\prime }=\pi }\cdot ({\to }_{F:\infty }^{{\pi }^{\prime }}({\pi }^{\prime },h)(\lambda ao{s}_{1:\infty }.U(a{o}_{1:\infty }\left)\right))+{\text{1}}_{{\pi }^{\prime }\ne \pi })$

Now, we can partially pack this up as a projection, to get

$={\psi }_F(\lambda {\pi }^{\prime },h.{\text{1}}_{{\pi }^{\prime }=\pi }\cdot (p{r}_{(A\times O{)}^{\omega }\ast }\left({\to }_{F:\infty }^{{\pi }^{\prime }}\right(\pi ,h\left)\right)\left(U\right))+{\text{1}}_{{\pi }^{\prime }\ne \pi })$

At this point, we can realize something interesting. $p{r}_{(A\times O{)}^{\omega }\ast }\left({\to }_{F:\infty }^{{\pi }^{\prime }}\right({\pi }^{\prime },h\left)\right)$ is "start with an initial policy of ${\pi }^{\prime }$ and h compatible with ${\pi }^{\prime }$, then repeatedly feed in actions as if they were created by ${\pi }^{\prime }$ forever, and forget about the states, leaving just the action-observation sequence". Now, since the action (being produced by ${\pi }^{\prime }$) always lines up with what the ${\pi }^{\prime }$ encoded in the state would do, this process never hits the $\perp$ infradistribution, it keeps going on and on and advancing the history with no issue. In particular, the history unrolls to completion, and the resulting action-observation sequence you get would just be the original destiny packed up in the state. So, this infradistribution is just ${\delta }_h$. Making this substitution in, we get

$={\psi }_F(\lambda {\pi }^{\prime },h.{\text{1}}_{{\pi }^{\prime }=\pi }\cdot {\delta }_h(U)+{\text{1}}_{{\pi }^{\prime }\ne \pi })$

$={\psi }_F(\lambda {\pi }^{\prime },h.{\text{1}}_{{\pi }^{\prime }=\pi }U(h)+{\text{1}}_{{\pi }^{\prime }\ne \pi })$

Now we pack up the update again, to get

$=u_{\left\{\pi \right\}}^1\left({\psi }_F\right)(\lambda {\pi }^{\prime },h.U(h\left)\right)$

And then realize that this is a projection, to get

$=p{r}_{(A\times O{)}^{\omega }\ast }\left(u_{\left\{\pi \right\}}^1\right({\psi }_F\left)\right)\left(U\right)$

And we're done, we showed our desired result to establish that going first-person dynamic to static back to dynamic is identity.

T2.2.8 Time for the last leg of the square, that going first-person static to first-person dynamic and first-person static is identity. We want to show that, regardless of ${\pi }^{\prime }$,

$\Theta \left({\pi }^{\prime }\right)=p{r}_{(A\times O{)}^{\omega }\ast }\left(\right(u_{\left\{{\pi }^{\prime }\right\}}^1({\top }_{\Pi }⋉\Theta ))⋉{\to }_{F:\infty }^{{\pi }^{\prime }})$

Again, like usual, we'll take the complicated thing and repeatedly rewrite it to reach the simpler thing, for an arbitrary function.

$p{r}_{(A\times O{)}^{\omega }\ast }\left(\right(u_{\left\{{\pi }^{\prime }\right\}}^1({\top }_{\Pi }⋉\Theta ))⋉{\to }_{F:\infty }^{{\pi }^{\prime }})\left(U\right)$

Using $s$ for states, we can rewrite the projection, to get

$=\left(\right(u_{\left\{{\pi }^{\prime }\right\}}^1({\top }_{\Pi }⋉\Theta ))⋉{\to }_{F:\infty }^{{\pi }^{\prime }})(\lambda s_0,ao{s}_{1:\infty }.U(a{o}_{1:\infty }\left)\right)$

Then we rewrite the semidirect product, and unpack the initial state into a policy $\pi$ and destiny h compatible with $\pi$, to get

=$u_{\left\{{\pi }^{\prime }\right\}}^1({\top }_{\Pi }⋉\Theta )(\lambda \pi ,h.{\to }_{F:\infty }^{{\pi }^{\prime }}(\pi ,h\left)\right(\lambda ao{s}_{1:\infty }.U\left(a{o}_{1:\infty }\right)\left)\right)$

We can then rewrite the interior as a projection to get

$=u_{\left\{{\pi }^{\prime }\right\}}^1({\top }_{\Pi }⋉\Theta )(\lambda \pi ,h.p{r}_{(A\times O{)}^{\omega }\ast }({\to }_{F:\infty }^{{\pi }^{\prime }}(\pi ,h)\left)\right(U\left)\right)$

Now, it's time to rewrite the update. It rewrites as:

$=({\top }_{\Pi }⋉\Theta )(\lambda \pi ,h.{\text{1}}_{\pi ={\pi }^{\prime }}p{r}_{(A\times O{)}^{\omega }\ast }({\to }_{F:\infty }^{{\pi }^{\prime }}(\pi ,h)\left)\right(U)+{\text{1}}_{\pi \ne {\pi }^{\prime }})$

Now, since $\pi ={\pi }^{\prime }$ inside the scope of the indicator function, we can rewrite as

$=({\top }_{\Pi }⋉\Theta )(\lambda \pi ,h.{\text{1}}_{\pi ={\pi }^{\prime }}p{r}_{(A\times O{)}^{\omega }\ast }({\to }_{F:\infty }^{{\pi }^{\prime }}({\pi }^{\prime },h)\left)\right(U)+{\text{1}}_{\pi \ne {\pi }^{\prime }})$

And use our usual argument from before that $p{r}_{(A\times O{)}^{\omega }\ast }\left({\to }_{F:\infty }^{{\pi }^{\prime }}\right({\pi }^{\prime },h\left)\right)$ is just the dirac-delta distribution on h, to get

$=({\top }_{\Pi }⋉\Theta )(\lambda \pi ,h.{\text{1}}_{\pi ={\pi }^{\prime }}U(h)+{\text{1}}_{\pi \ne {\pi }^{\prime }})$

Now, we can unpack the semidirect product as well as what ${\top }_{\Pi }$ means, to get:

$={inf}_{\pi }\Theta \left(\pi \right)(\lambda h.{\text{1}}_{\pi ={\pi }^{\prime }}U(h)+{\text{1}}_{\pi \ne {\pi }^{\prime }})$

Now, if $\pi \ne {\pi }^{\prime }$, then that inner function turns into 1 (or infinity) which is assigned maximum value by all $\Theta \left(\pi \right)$, so it's minimized when $\pi ={\pi }^{\prime }$, so we get:

$=\Theta \left({\pi }^{\prime }\right)(\lambda h.U(h\left)\right)=\Theta \left({\pi }^{\prime }\right)\left(U\right)$

And we're done with the last "doing this path is identity" result. All that remains in our proof is just showing that taking both paths from one corner to another corner makes the same result, to show that going around in a loop is identity, ruling out nontrivial automorphisms.

T2.3 The easiest pair of corners for this is going from first-person static to third-person dynamic in both ways. Obviously, the transition kernel would be the same no matter which path you took, which just leaves verifying that the starting infradistribution is the same. Down one path, we have

$D:={\top }_{\Pi }⋉\Theta ,{\psi }_T:=D$

Down the other path, we have

${\psi }_F:={\top }_{\Pi }⋉\Theta ,{\psi }_T:={\psi }_F$

Obviously, both these paths produce the same result when you try to define ${\psi }_T$ in both ways from $\Theta$. And we're done!

 

Theorem 3: Pseudocausal Commutative Square: The following diagram commutes for any choice of pseudocausal belief function $\Theta$. Any choice of infradistribution $D$ where $D=p{r}_{(A\times O{)}^{\omega }\ast }({\top }_{\Pi }⋉(\lambda \pi .u_{\sim \pi }^1\left(D\right)\left)\right)$ also makes this diagram commute.

Again, the work we need to put in is getting conditions in the four corners that are "came from a pseudocausal belief function", then, for phase 1, verifying that all 8 morphisms preserve the relevant property in the target if the source had it, then for phase 2, verifying that all 8 ways of going from a corner to 1 away and back result in identity to get 4 isomorphisms, then finally showing that starting in one corner and going to the other corner by two different paths make the same result, to rule out nontrivial automorphisms.

The property for third-person static is $D=p{r}_{(A\times O{)}^{\omega }\ast }({\top }_{\Pi }⋉(\lambda \pi .u_{\sim \pi }^1\left(D\right)\left)\right)$

The property for third-person dynamic is ${\psi }_T=p{r}_{(A\times O{)}^{\omega }\ast }({\top }_{\Pi }⋉(\lambda \pi .u_{\sim \pi }^1\left({\psi }_T\right)\left)\right)$ and ${\to }_T\left(aoh\right)={\delta }_{a,o,h}$

The property for first-person static is pseudocausality, $\forall {\pi }^{\prime },\pi ,U:u_{\sim {\pi }^{\prime }}^1\left(\Theta \right(\pi \left)\right)\left(U\right)\ge \Theta \left({\pi }^{\prime }\right)$

The property for first-person dynamic is ${\psi }_F=p{r}_{(A\times O{)}^{\omega }\ast }({\top }_{\Pi }⋉(\lambda \pi .u_{\sim \pi }^1\left({\psi }_F\right)\left)\right)$ and ${\to }_F(aoh,a^{\prime })={\delta }_{o,h}$ when $a^{\prime }=a$, and $\perp$ when $a^{\prime }\ne a$.

Time to show preservation of these properties by all the morphisms, as well as that the translations make a belief function/infradistribution.

Our first step is showing that all 8 morphisms induce the relevant property in the target if the source has the property.

T3.1.1 3-static to 1-static. Our definition is

$\Theta \left(\pi \right):=u_{\sim \pi }^1\left(D\right)$

and we must check the five conditions of a belief function, as well as pseudocausality.

T3.1.1.1 Checking the five conditions of a belief function proceeds in almost exactly the same way as checking the five conditions of a belief function in the "Acausal Commutative Square" theorem. We leave it as an exercise to the reader, and will just elaborate on the only nontrivial new argument needed.

The only nontrivial modification is in our proof of lower-semicontinuity. In order for it to work, we need that for any m, the subset of $(A\times O{)}^{\omega }$ where $h\sim {\pi }_m$ or $h\sim {\pi }_{m+1}$ or ${\pi }_{m+2}$... or ${\pi }_{\infty }$ is closed. 

We can rephrase the set $\{h|\exists n\ge m:h\sim {\pi }_n\}$ as the projection down to the $(A\times O{)}^{\omega }$ coordinate of the set

$\left(\right\{{\pi }_n{\}}_{n\ge m}\times (A\times O{)}^{\omega })\cap \{\pi ,h|h\sim \pi \}$

$(A\times O{)}^{\omega }$ is compact, and $\{{\pi }_n{\}}_{n\ge m}$ is compact as well, since it limits to a point and the limit point is included. So that product is a compact set. The set $\{\pi ,h|h\sim \pi \}$ is closed. So $\left(\right\{{\pi }_n{\}}_{n\ge m}\times (A\times O{)}^{\omega })\cap \{\pi ,h|h\sim \pi \}$ is the intersection of a compact and a closed set, and so is compact. And projections of compact sets are compact, and compactness implies closure. So, the set of histories $\{h|\exists n\ge m:h\sim {\pi }_n\}$ is closed regardless of m, even if $m=\infty$.

The rest of the proof works out with no issue, so translating over an infradistribution makes a belief function.

T3.1.1.2 We still have to check pseudocausality however. Our translation is

$\Theta \left(\pi \right):=u_{\sim \pi }^1\left(D\right)$ 

and we want to show

$\forall {\pi }^{\prime },\pi ,U:u_{\sim {\pi }^{\prime }}^1\left(\Theta \right(\pi \left)\right)\left(U\right)\ge \Theta \left({\pi }^{\prime }\right)\left(U\right)$

Reexpressing this desired statement in terms of $D$, we have

$\forall {\pi }^{\prime },\pi ,U:u_{\sim {\pi }^{\prime }}^1\left(u_{\sim \pi }^1\right(D\left)\right)\left(U\right)\ge u_{\sim {\pi }^{\prime }}^1\left(D\right)\left(U\right)$

Let ${\pi }^{\prime }$ and $\pi$ be arbitrary. Then the left side is

$u_{\sim {\pi }^{\prime }}^1\left(u_{\sim \pi }^1\right(D\left)\right)\left(U\right)$

We unpack the update

$=u_{\sim \pi }^1\left(D\right)(\lambda h.{\text{1}}_{h\sim {\pi }^{\prime }}U(h)+{\text{1}}_{h\nsim {\pi }^{\prime }})$

And unpack the update again, to get

$=D(\lambda h.{\text{1}}_{h\sim \pi }\cdot ({\text{1}}_{h\sim {\pi }^{\prime }}f\left(h\right)+{\text{1}}_{h\nsim {\pi }^{\prime }})+{\text{1}}_{h\nsim \pi })$

Looking at this, the function is $U$ when h is compatible with both ${\pi }^{\prime }$ and $\pi$, and 1 otherwise. This is a greater function than $U$ when h is compatible with just ${\pi }^{\prime }$, and 1 otherwise, so by monotonicity for infradistributions, we can get:

$\ge D(\lambda h.{\text{1}}_{h\sim {\pi }^{\prime }}U(h)+{\text{1}}_{h\nsim {\pi }^{\prime }})$

and pack up the update to get

$=u_{\sim {\pi }^{\prime }}^1\left(D\right)\left(U\right)$

And we're done, we get pseudocausality.

T3.1.2 1-static to 3-static. Our translation is

$D:=p{r}_{(A\times O{)}^{\omega }\ast }({\top }_{\Pi }⋉\Theta )$

We'll show the infradistribution conditions on $D$. First, to show it's even well-defined, we have

$D\left(U\right)=p{r}_{(A\times O{)}^{\omega }\ast }({\top }_{\Pi }⋉\Theta )\left(U\right)={\top }_{\Pi }(\lambda \pi .\Theta (\pi \left)\right(\lambda h.U\left(h\right)\left)\right)$

That inner function is lower-semicontinuous in $\pi$, because $\Theta$ was assumed to have that property in its first argument. Since the inner function is lower-semicontinuous, it can be evaluated.

Now for the condition that

$D=p{r}_{(A\times O{)}^{\omega }\ast }({\top }_{\Pi }⋉(\lambda \pi .u_{\sim \pi }^1\left(D\right)\left)\right)$

We start with $D\left(U\right)$ and unpack the definition of $D$, to get

$=p{r}_{(A\times O{)}^{\omega }\ast }({\top }_{\Pi }⋉\Theta )\left(U\right)$

And then undo the projection and semidirect product and what ${\top }_{\Pi }$ means, to get

$={inf}_{\pi }\Theta \left(\pi \right)(\lambda h.U(h\left)\right)$

And then, by pseudocausality for $\Theta$ (for all ${\pi }^{\prime }$, $u_{\sim \pi }^1\left(\Theta \right({\pi }^{\prime }\left)\right)\left(U\right)\ge \Theta \left(\pi \right)\left(U\right)$ and also $u_{\sim \pi }^1\left(\Theta \right(\pi \left)\right)\left(U\right)=\Theta \left(\pi \right)\left(U\right)$ because $\Theta \left(\pi \right)$ is supported entirely on histories compatible with $\pi$), we can swap $\Theta \left(\pi \right)(\lambda h.U(h\left)\right)$ out for

$={inf}_{\pi }{inf}_{{\pi }^{\prime }}{u}_{\sim \pi }^1\left(\Theta \right({\pi }^{\prime }\left)\right)(\lambda h.U(h\left)\right)$

And then unpack the update, to get

$={inf}_{\pi }{inf}_{{\pi }^{\prime }}\Theta \left({\pi }^{\prime }\right)(\lambda h.{\text{1}}_{h\sim \pi }U(h)+{\text{1}}_{h\nsim \pi })$

Now we pack up what ${\top }_{\Pi }$ means, and the semidirect product, to get

$={inf}_{\pi }({\top }_{\Pi }⋉\Theta )(\lambda {\pi }^{\prime },h.{\text{1}}_{h\sim \pi }U(h)+{\text{1}}_{h\nsim \pi })$

Write this as a projection

$={inf}_{\pi }p{r}_{(A\times O{)}^{\omega }\ast }({\top }_{\Pi }⋉\Theta )(\lambda h.{\text{1}}_{h\sim \pi }U(h)+{\text{1}}_{h\nsim \pi })$

Pack up the definition of $D$ to get

$={inf}_{\pi }D(\lambda h.{\text{1}}_{h\sim \pi }U(h)+{\text{1}}_{h\nsim \pi })$

Pack up the definition of "1-update on $\pi$" to get

$={inf}_{\pi }{u}_{\sim \pi }^1\left(D\right)(\lambda h.U(h\left)\right)$

Pack up what ${\top }_{\Pi }$ means and the semidirect product and the projection to get

$=p{r}_{(A\times O{)}^{\omega }\ast }({\top }_{\Pi }⋉(\lambda \pi .u_{\sim \pi }^1\left(D\right)\left)\right)\left(U\right)$

So we have

$D=p{r}_{(A\times O{)}^{\omega }\ast }({\top }_{\Pi }⋉(\lambda \pi .u_{\sim \pi }^1\left(D\right)\left)\right)$

as desired, since $U$ was arbitrary.

Now to verify the infradistribution conditions on $D$. This proof is pretty much identical to the proof in the "Acausal Commutative Square" theorem, interested readers can fill it in.

T3.1.3 For 3-static to 3-dynamic, we have ${\to }_T\left(aoh\right):={\delta }_{a,o,h}$ (which fulfills one of the 3-dynamic conditions), and ${\psi }_T:=D$

Since $D$ is an infradistribution, ${\psi }_T$ is too, and for the condition on ${\psi }_T$, we have:

${\psi }_T=D=p{r}_{(A\times O{)}^{\omega }\ast }({\top }_{\Pi }⋉(\lambda \pi .u_{\sim \pi }^1\left(D\right)\left)\right)=p{r}_{(A\times O{)}^{\omega }\ast }({\top }_{\Pi }⋉(\lambda \pi .u_{\sim \pi }^1\left({\psi }_T\right)\left)\right)$

and we're done, since $D$ had the relevant condition.

T3.1.4 For 3-dynamic to 3-static, we have

$D:=p{r}_{(A\times O{)}^{\omega }\ast }({\psi }_T⋉{\to }_{T:\infty })$

First, we must show that the niceness conditions are fulfilled for the infinite semidirect product to be defined. ${\to }_T$ maps $aoh$ to $a,o,h$. This is clearly a continuous mapping, so in particular, it's lower-semicontinuous. Dirac-delta distributions are 1-Lipschitz when interpreted as inframeasures (all probability distributions have that property, actually). The compact-shared-CAS condition is redundant because $A\times O\times (A\times O{)}^{\omega }$ is already a compact space. And all probability distributions (dirac-delta distributions are probability distributions) map constants to the same constant, so we get the increase-in-constants property and the 1 maps to 1 property. So we can indeed take the infinite semidirect product of the ${\to }_T$.

Now, we'll show that unrolling like this just produces ${\psi }_T$ exactly. We'll use $h_0$ for the initial destiny state, and $ao{h}_{1:\infty }$ for the successive unrolled history of states, actions, and observations. First, by unpacking $D$'s definition, 

$D\left(U\right)=p{r}_{(A\times O{)}^{\omega }\ast }({\psi }_T⋉{\to }_{T:\infty })\left(U\right)$

Then we unpack the projection and semidirect product

$={\psi }_T(\lambda h_0.{\to }_{T:\infty }(h_0\left)\right(\lambda ao{h}_{1:\infty }.U\left(a{o}_{1:\infty }\right)\left)\right)$

And pack things up into a projection

$={\psi }_T(\lambda h_0.p{r}_{(A\times O{)}^{\omega }\ast }({\to }_{T:\infty }\left(h_0\right)\left)\right(\lambda a{o}_{1:\infty }.U\left(a{o}_{1:\infty }\right)\left)\right)$

And then we can observe that unrolling the initial destiny forever via "advance the destiny 1 step, popping actions and observations off", when you project down to just the actions and observations, yields just the point distribution on that destiny, so we get:

$={\psi }_T(\lambda h.{\delta }_h(\lambda a{o}_{1:\infty }.U\left(a{o}_{1:\infty }\right)\left)\right)$

And then substitute the value of that dirac-delta in to get 

$={\psi }_T(\lambda h.U(h\left)\right)={\psi }_T\left(U\right)$

So, our $D={\psi }_T$. Now we can go

$D={\psi }_T=p{r}_{(A\times O{)}^{\omega }\ast }({\top }_{\Pi }⋉(\lambda \pi .u_{\sim \pi }^1\left({\psi }_T\right)\left)\right)=p{r}_{(A\times O{)}^{\omega }\ast }({\top }_{\Pi }⋉(\lambda \pi .u_{\sim \pi }^1\left(D\right)\left)\right)$

and we're done, showing $D$ had the relevant condition from ${\psi }_T$ having it. Since ${\psi }_T$ is an infradistribution, $D$ is too.

T3.1.5 For 3-dynamic to 1-dynamic, we have

${\to }_F(h,a^{\prime }):=p{r}_{O\times (A\times O{)}^{\omega }\ast }\left(u_{\left\{a^{\prime }\right\}}^1\right({\to }_T\left(h\right)\left)\right)$

Let's write $h$ as $aoh$ (unpacking the destiny a bit). Now, for the action $a$ (which is compatible with the start of the history), we have 

${\to }_F(aoh,a)\left(f\right)$

Applying our definition, we have

$=p{r}_{O\times (A\times O{)}^{\omega }\ast }\left(u_{\left\{a\right\}}^1\right({\to }_T\left(aoh\right)\left)\right)\left(f\right)$

Unpacking the projection, we have

$=u_{\left\{a\right\}}^1\left({\to }_T\right(aoh\left)\right)(\lambda a^{\prime },o^{\prime },h^{\prime }.f(o^{\prime },h^{\prime }\left)\right)$

Unpacking the update, we have

$={\to }_T\left(aoh\right)(\lambda a^{\prime },o^{\prime },h^{\prime }.{\text{1}}_{a^{\prime }=a}f(o^{\prime },h^{\prime })+{\text{1}}_{a^{\prime }\ne a})$

And then we consider that ${\to }_T\left(aoh\right)$ is just the dirac-delta distribution on $(a,o,h)$, so we have

$={\delta }_{a,o,h}(\lambda a^{\prime },o^{\prime },h^{\prime }.{\text{1}}_{a^{\prime }=a}f(o^{\prime },h^{\prime })+{\text{1}}_{a^{\prime }\ne a})$

Substitute the values in, and we have

$=f(o,h)={\delta }_{o,h}\left(f\right)$

So, ${\to }_F(a,aoh)$ is just the dirac-delta distribution on $o$ and $h$, as it should be. 

For $a^{\prime }$ that aren't $a$ as it should be, we have

${\to }_F(aoh,a^{\prime })\left(f\right)$

Applying our definition, we have

$=p{r}_{O\times (A\times O{)}^{\omega }\ast }\left(u_{\left\{a^{\prime }\right\}}^1\right({\to }_T\left(aoh\right)\left)\right)\left(f\right)$

Unpacking the projection, we have

$=u_{\left\{a^{\prime }\right\}}^1\left({\to }_T\right(aoh\left)\right)(\lambda a^{\prime \prime },o^{\prime },h^{\prime }.f(o^{\prime },h^{\prime }\left)\right)$

Unpacking the update, we have

$={\to }_T\left(aoh\right)(\lambda a^{\prime \prime },o^{\prime },h^{\prime }.{\text{1}}_{a^{\prime \prime }=a^{\prime }}f(o^{\prime },h^{\prime })+{\text{1}}_{a^{\prime \prime }\ne a^{\prime }})$

And then we consider that ${\to }_T\left(aoh\right)$ is just the dirac-delta distribution on $(a,o,h)$, so we have

$={\delta }_{a,o,h}(\lambda a^{\prime \prime },o^{\prime },h^{\prime }.{\text{1}}_{a^{\prime \prime }=a^{\prime }}f(o^{\prime },h^{\prime })+{\text{1}}_{a^{\prime \prime }\ne a^{\prime }})$

and because $a^{\prime }\ne a$, substituting the dirac-delta in produces 1 (or infinity), ie, $\perp \left(f\right)$. Therefore, 

${\to }_F(aoh,a^{\prime })=\perp$

So our transition kernel works out. For the other conditions, just observe that ${\psi }_F:={\psi }_T$, and use the extreme similarity of their defining conditions.

T3.1.6 For 1-dynamic to 3-dynamic, we have ${\psi }_T:={\psi }_F$ so we can use our defining property to again, clearly show that the resulting 3-dynamic starting infradistribution has the relevant property, so that just leaves cleaning up the defining property for the ${\to }_T$ infrakernel. We remember that

${\to }_T\left(h\right):={\top }_A⋉(\lambda a.{\to }_F(h,a\left)\right)$

Therefore, we have:

${\to }_T\left(aoh\right)\left(f\right)=({\top }_A⋉(\lambda a^{\prime }.{\to }_F(aoh,a^{\prime })\left)\right)\left(f\right)$

Then, we can unpack the semidirect product as:

$={\top }_A(\lambda a^{\prime }.{\to }_F(aoh,a^{\prime }\left)\right(\lambda o^{\prime },h^{\prime }.f(a^{\prime },o^{\prime },h^{\prime })\left)\right)$

We unpack our ${\top }_A$, getting

$={inf}_{a^{\prime }}\left({\to }_F\right(aoh,a^{\prime }\left)\right(\lambda o^{\prime },h^{\prime }.f(a^{\prime },o,s)\left)\right)$

Then, we observe that for any $a^{\prime }\ne a$, ${\to }_F(aoh,a^{\prime })=\perp$. So, $a^{\prime }$ must be $a$, and we have

$={\to }_F(aoh,a)(\lambda o^{\prime },h^{\prime }.f(a,o^{\prime },h^{\prime }\left)\right)$

Now, ${\to }_F(aoh,a)$ is the dirac-delta distribution on $(o,h)$, so making that substitution in, we have 

$={\delta }_{o,h}(\lambda o^{\prime },h^{\prime }.f(a,o^{\prime },h^{\prime }\left)\right)=f(a,o,h)={\delta }_{a,o,h}\left(f\right)$

And this holds for all $f$, so we get our desired result that

${\to }_T\left(aoh\right)={\delta }_{a,o,h}$

T3.1.7: For 1-static to 1-dynamic, we just have

${\psi }_F:=p{r}_{(A\times O{)}^{\omega }\ast }({\top }_{\Pi }⋉\Theta )$

${\to }_F(aoh,a):={\delta }_{o,h}$

${\to }_F(aoh,a^{\prime }):=\perp$

So this automatically makes the infrakernel have the desired properties. For showing the relevant property of ${\psi }_F$, we just copy the proof at the start of getting the properties for $D$ from $\Theta$ being a belief function, since $D$ and ${\psi }_F$ are both defined in the same way.

T3.1.8 For 1-dynamic to 1-static, we have

$\Theta \left({\pi }^{\prime }\right):=p{r}_{(A\times O{)}^{\omega }\ast }({\psi }_F⋉{\to }_{F:\infty }^{{\pi }^{\prime }})$

We'll work on getting that latter quantity into a better form, but first we have to verify that
${\to }_{F:\infty }^{\pi }$ has the ${\to }_{F:n}^{\pi }$ fulfilling the niceness conditions. It's defined as

${\to }_{F:n}^{\pi }(h_0,ao{h}_{1:n}):={\delta }_{\pi \left(a{o}_{1:n}\right)}⋉(\lambda a.{\to }_F(h_n,a\left)\right)$

For lower-semicontinuity, ${\to }_F$ is lower-semicontinuous, because it returns $\perp$ on every action but one (this is a clopen set), and for the action that pairs up with the initial destiny, it just advances the destiny, which is continuous. ${\to }_{F:n}^{\pi }$ behaves similarly, continuous variation in input leads to continuous variation in output, because eventually the behavior of the policy settles down to a fixed action as the history of actions and observations stabilizes amongst one of the finitely many options. 

For making inframeasures, ${\to }_{F:n}^{\pi }$ always returns a dirac-delta distribution or $\perp$, so we're good there. Similarly, for 1-Lipschitzness, both $\perp$ and dirac-delta distributions are 1-Lipschitz. For compact-shared CAS, as the target space is $A\times O\times (A\times O{)}^{\omega }$, it's compact and you don't need to worry about that. Finally, both dirac-delta distributions and $\perp$ map constants to either the same constant or higher, and in the $[0,1]$ type sinature, both $\perp$ and dirac-delta distributions map 1 to 1. So all niceness conditions are fulfilled.

Let's work on unpacking the value

$p{r}_{(A\times O{)}^{\omega }\ast }({\psi }_F⋉{\to }_{F:\infty }^{\pi })\left(U\right)$

unpacking the projection and semidirect product, we get

$={\psi }_F(\lambda h_0.{\to }_{F:\infty }^{\pi }(h\left)\right(\lambda ao{h}_{1:\infty }.U\left(a{o}_{1:\infty }\right)\left)\right)$

And then we can write this as a projection in the inside

$={\psi }_F(\lambda h_0.p{r}_{(A\times O{)}^{\omega }\ast }({\to }_{F:\infty }^{\pi }\left(h\right)\left)\right(\lambda a{o}_{1:\infty }.U\left(a{o}_{1:\infty }\right)\left)\right)$

One of two things can happen. In the first case, h is compatible with $\pi$, so playing $\pi$ against it never hits $\perp$, and it unrolls to completion, and projecting down just yields the dirac-delta distribution on h itself. In the second case, h is incompatible with $\pi$, so eventually we hit $\perp$, which maps everything to maximal value (either 1 or infinity). Thus, we can write this as an indicator function

$={\psi }_F(\lambda h.{\text{1}}_{h\sim \pi }{\delta }_h(\lambda a{o}_{1:\infty }.U\left(a{o}_{1:\infty }\right))+{\text{1}}_{h\nsim {\pi }^{\prime }})$

Substituting the dirac-delta value in, we have

$={\psi }_F(\lambda h.{\text{1}}_{h\sim \pi }U(h)+{\text{1}}_{h\nsim \pi })$

And we pack the update back up to get

$=u_{\sim \pi }^1\left({\psi }_F\right)\left(U\right)$

Since $U$ was arbitrary, we have

$p{r}_{(A\times O{)}^{\omega }\ast }({\psi }_F⋉{\to }_{F:\infty }^{\pi })=u_{\sim {\pi }^{\prime }}^1\left({\psi }_F\right)$

And so, an alternate way to define $\Theta \left(\pi \right)$ is as

$=u_{\sim \pi }^1\left({\psi }_F\right)$

We can then run through the same proof that $\Theta$ is indeed a belief function, back from the 3-static to 1-static case, because ${\psi }_F$ fulfills all the same properties that $D$ did.

Alright, now we have, for all four corners, a condition that's basically "came from a pseudocausal belief function" that is preserved under morphisms. Now we need to show that all 8 back-and-forth directions are identity.

T3.2.1 For 1-static to 3-static back to 1-static, we want to show

$\Theta \left({\pi }^{\prime }\right)=u_{\sim {\pi }^{\prime }}^1\left(p{r}_{(A\times O{)}^{\omega }\ast }\right({\top }_{\Pi }⋉\Theta \left)\right)$

This is (we assume the hard version first)

$u_{\sim {\pi }^{\prime }}^1\left(p{r}_{(A\times O{)}^{\omega }\ast }\right({\top }_{\Pi }⋉\Theta \left)\right)\left(U\right)$

Then unpack the update

$=p{r}_{(A\times O{)}^{\omega }\ast }({\top }_{\Pi }⋉\Theta )(\lambda h.{\text{1}}_{h\sim {\pi }^{\prime }}U(h)+{\text{1}}_{h\nsim {\pi }^{\prime }})$

And the projection

$=({\top }_{\Pi }⋉\Theta )(\lambda \pi ,h.{\text{1}}_{h\sim {\pi }^{\prime }}U(h)+{\text{1}}_{h\nsim {\pi }^{\prime }})$

Then unpack the semidirect product and ${\top }_{\Pi }$ to get

$={inf}_{\pi }\Theta \left(\pi \right)(\lambda h.{\text{1}}_{h\sim {\pi }^{\prime }}U(h)+{\text{1}}_{h\nsim {\pi }^{\prime }})$

then rewrite this as an update

$={inf}_{\pi }{u}_{\sim {\pi }^{\prime }}^1\left(\Theta \right(\pi \left)\right)(\lambda h.U(h\left)\right)$

And then we can apply pseudocausality of \BF to get that the minimizer must pick ${\pi }^{\prime }$ exactly because $u_{\sim {\pi }^{\prime }}^1\left(\Theta \right(\pi \left)\right)\left(U\right)\ge \Theta \left({\pi }^{\prime }\right)\left(U\right)=u_{\sim {\pi }^{\prime }}^1\left(\Theta \right({\pi }^{\prime }\left)\right)\left(U\right)$ and so we get

$=\Theta \left({\pi }^{\prime }\right)\left(U\right)$

and we're done.

T3.2.2 For 3-static to 1-static to 3-static, we must show

$D=p{r}_{(A\times O{)}^{\omega }\ast }({\top }_{\Pi }⋉(\lambda \pi .u_{\sim \pi }^1\left(D\right)\left)\right)$

Well, the property for a third-person static $D$ is exactly that.

T3.2.3 For 3-static to 3-dynamic to 3-static, we want that

$D=p{r}_{(A\times O{)}^{\omega }\ast }(D⋉{\to }_{T:\infty })$

Taking the complicated part, it's

$p{r}_{(A\times O{)}^{\omega }\ast }(D⋉{\to }_{T:\infty })\left(U\right)$

We unpack the projection

$=(D⋉{\to }_{T:\infty })(\lambda h_0,ao{h}_{1:\infty }.U(a{o}_{:\infty }\left)\right)$

And then we unpack the semidirect product a bit, to get

$=D(\lambda h_0.{\to }_{T:\infty }(h_0\left)\right(\lambda ao{h}_{1:\infty }.U\left(a{o}_{1:\infty }\right)\left)\right)$

And then we can write this as a projection, to get

$=D(\lambda h_0.p{r}_{(A\times O{)}^{\omega }\ast }({\to }_{T:\infty }\left(h_0\right)\left)\right(\lambda a{o}_{1:\infty }.U\left(a{o}_{1:\infty }\right)\left)\right)$

Now, we observe that because ${\to }_T$, when repeatedly unrolled, is just emitting the actions and observations from the starting destiny, this projection is just the point distribution on the action-observation sequence that is $h_0$.

$=D(\lambda h_0.{\delta }_{h_0}(\lambda a{o}_{1:\infty }.U\left(a{o}_{1:\infty }\right)\left)\right)$

Then we evaluate the expectation, yielding

$=D(\lambda h.U(h\left)\right)=D\left(U\right)$

And we're done.

T3.2.4 For going 3-dynamic to 3-static to 3-dynamic, we need that

${\psi }_T=p{r}_{(A\times O{)}^{\omega }\ast }({\psi }_T⋉{\to }_{T:\infty })$

To show this, just reuse the exact same proof from above, just with ${\psi }_T$ instead of $D$. Also, ${\to }_F\left(h\right)$ works out as it should.

At this point, we've established isomorphism for two sides of the square. We've just got two more sides left to address, then showing that going around the square results in identity instead of a nontrivial automorphism.

T3.2.5,6 For translating back and forth between 3-dynamic and 1-dynamic, we observe that both translations keep the initial infradistribution over destinies the same, and our 3-dynamic to 1-dynamic, and 1-dynamic to 3-dynamic proofs verified that the ${\to }_F$ and ${\to }_T$ behave as they should when they go back and forth, so we don't need to worry about this.

T3.2.7 Next is 1-dynamic to 1-static to 1-dynamic. The infrakernel is guaranteed to have the right form, we just need to show that ${\psi }_F$ is unchanged. So, we must show

${\psi }_F=p{r}_{(A\times O{)}^{\omega }\ast }({\top }_{\Pi }⋉(\lambda \pi .p{r}_{(A\times O{)}^{\omega }\ast }({\psi }_F⋉{\to }_{F:\infty }^{\pi })\left)\right)$

But, since our condition on 1-dynamic (that we showed is a property of applying the morphism to go from any belief function to first-person dynamic) was

${\psi }_F=p{r}_{(A\times O{)}^{\omega }\ast }({\top }_{\Pi }⋉(\lambda \pi .u_{\sim \pi }^1\left({\psi }_F\right)\left)\right)$

The only thing we need to show is that, regardless of $\pi$,

$u_{\sim \pi }^1\left({\psi }_F\right)=p{r}_{(A\times O{)}^{\omega }\ast }({\psi }_F⋉{\to }_{F:\infty }^{\pi })$

For if we could show that, then we could go:

$p{r}_{(A\times O{)}^{\omega }\ast }({\top }_{\Pi }⋉(\lambda \pi .p{r}_{(A\times O{)}^{\omega }\ast }({\psi }_F⋉{\to }_{F:\infty }^{\pi })\left)\right)=p{r}_{(A\times O{)}^{\omega }\ast }({\top }_{\Pi }⋉(\lambda \pi .u_{\sim \pi }^1\left({\psi }_F\right)\left)\right)={\psi }_F$

and we'd be done. However, from the 1-dynamic to 1-static proof of property preservation, we already showed that

$p{r}_{(A\times O{)}^{\omega }\ast }({\psi }_F⋉{\to }_{F:\infty }^{\pi })\left(U\right)=u_{\sim \pi }^1\left({\psi }_F\right)\left(U\right)$

back then, so we're just done.

T3.2.8 Time for the last leg of the square, that going first-person static to dynamic and back to static is identity. We want to show that, regardless of ${\pi }^{\prime }$,

$\Theta \left({\pi }^{\prime }\right)=p{r}_{(A\times O{)}^{\omega }\ast }\left(\right(p{r}_{(A\times O{)}^{\omega }\ast }({\top }_{\Pi }⋉\Theta ))⋉{\to }_{F:\infty }^{{\pi }^{\prime }})$

To do this, we use the result from the 1-dynamic to 1-static proof of property preservation, that

$p{r}_{(A\times O{)}^{\omega }\ast }({\psi }_F⋉{\to }_{F:\infty }^{\pi })=u_{\sim \pi }^1\left({\psi }_F\right)$

Applying this fact with ${\psi }_F$ as an abbreviation for $p{r}_{(A\times O{)}^{\omega }\ast }({\top }_{\Pi }⋉\Theta )$, we can rewrite our proof goal equivalently with this fact as

$\Theta \left({\pi }^{\prime }\right)=p{r}_{(A\times O{)}^{\omega }\ast }\left(u_{\sim {\pi }^{\prime }}^1\right(p{r}_{(A\times O{)}^{\omega }\ast }({\top }_{\Pi }⋉\Theta )\left)\right)$

But we showed this exact result when we showed that 1-static to 3-static to 1-static was identity, so we're done.

And we're done with the last "doing this path is identity" result. All that remains in our proof is just showing that taking both paths from one corner to another corner makes the same result, to show that going around in a loop is identity, ruling out nontrivial automorphisms.

T3.3 The easiest pair of corners for this is going from first-person static to third-person dynamic in both ways. Obviously, the transition kernel would be the same no matter which path you took, which just leaves verifying that the starting infradistribution is the same. Down one path, we have

$D:=p{r}_{(A\times O{)}^{\omega }\ast }({\top }_{\Pi }⋉\Theta ),{\psi }_T:=D$

Down the other path, we have

${\psi }_F:=p{r}_{(A\times O{)}^{\omega }\ast }({\top }_{\Pi }⋉\Theta ),{\psi }_T:={\psi }_F$

Obviously, both these paths produce the same result when you try to define ${\psi }_T$ in both ways from $\Theta$. And we're done!

0 comments

Comments sorted by top scores.