$r\left(q\right):=\underset{(p_1,\dots ,p_k)\in {\varphi }_t^{-1}\left(q\right)}{max}\stackrel{n}{\sum _{k=1}}{q}_k\cdot r_k^{\prime }\left(p_k\right)$.

What is $q_k$? Also, we should allow adding some valid reward function of $\tilde{t}$.

$P\left(x_i\right)$ is a polytope with $P\left(x_i\right)\subseteq \Delta A$, corresponding to allowed action distributions at that state. 

I think it's mathematically cleaner to get rid of A and have those be abstract polytopes.