• Pr π∈ξ [U(⌈G⌉,π) ≥ U(⌈G⌉,G*)] is the probability that, for a random policy π∈ξ, that policy has worse utility than the policy G* its program dictates; in essence, how good G's policies are compared to random policy selection

What prior over policies?

given g(G|U), we can infer the probability that an agent G has a given utility function U, as Pr[U] ∝ 2^-K(U) / Pr π∈ξ [U(⌈G⌉,π) ≥ U(⌈G⌉,G*)]) where ∝ means "is proportional to" and K(U) is the kolmogorov complexity of utility function U.

Suppose the prior over policies is max-entropy (uniform over all action sequences).  If the number of "actions" is greater than the number of bits it takes to specify my brain^[1], it seems like it would conclude that my utility function is something like "1 if {acts exactly like [insert exact copy of my brain] would}, else 0".

1. ^{^}

   Idk if this is plausible

In Pr[U] ≈ 2^-K(U) / Pr π∈ξ [U(⌈G⌉,π) ≤ U(⌈G⌉,G*)]): Shouldn't the inequality sign be the other way around? I am assuming that we want to maximize $U$ and not minimize $U$. As currently written, a good agent $G$ with utility function $U$ would be better than most random policies, and therefore $P{r}_{\pi \in \xi }\left[U\right(\lceil G\rceil ,\pi )\le U(\lceil G\rceil ,G\ast \left)\right]$ would be close to $1$ and $Pr\left[U\right]$ therefore be rather small.

If the sign should indeed be the other way around, then a similar problem might be present in the definition of $g\left(G|U\right)$, if you want $g$ to be high for more agenty programs $G$.

(Copied partially from here [LW(p) · GW(p)])

My intuition is that preDCa falls short on the "extrapolated" part in "Coherent extrapolated volition". PreDCA would extract a utility function from the flawed algorithm implemented by a human brain. This utility function would be coherent, but might not be extrapolated: The extrapolated utility function (ie what humans would value if they would be much smarter) is probably more complicated to formulate than the un-extrapolated utility function.

For example, the policy implemented by an average human brain probably contributes more to total human happiness than most other policies. Lets say $U_1$ is an utility function that values human happiness as measured by certain chemical states in the brain, and $U_2$ is "extrapolated happiness" (where "putting all humans brains in vat to make it feel happy" would not be good for $U_2$). Then it is plausible that $K\left(U_1\right)<K\left(U_2\right)$. But the policy implemented by an average human brain would do approximately equally well on both utility functions. Thus, $Pr\left[U_1\right]>Pr\left[U_2\right]$.

I am starting to learn theoretical stuff about AI alignment and have a question. Some of the quantities in your post contain the Kolmogorov complexity of U. Since it is not possible to compute the Kolmogorov complexity of a given function or to write down a list of all function whose complexity is below a certain bound, I wonder how it would be possible to implement the PreDCA protocol on a physical computer.