harfe's Shortform

A potentially impactful thing: someone competent runs as a candidate for the 2028 election on an AI notkilleveryoneism^[1] platform. Maybe even two people should run, one for the democratic primary, and one in the republican primary. While getting the nomination is rather unlikely, there could be lots of benefits even if you fail to gain the nomination (like other presidential candidates becoming sympathetic to AI notkilleveryoneism, or more popularity of AI notkilleveryoneism in the population, etc.)

On the other hand, attempting a presidential run can easily backfire.

A relevant previous example to this kind of approach is the 2020 campaign by Andrew Yang, which focussed on universal basic income (and downsides of automation). While the campaign attracted some attention, it seems like it didn't succeed in making UBI a popular policy among democrats.

PreDCA might not lead to CEV.

My summarized understanding of preDCA: preDCA has a bunch of hypotheses how the universe might be like. For each hypothesis, it detects which computations are running in the universe, then figures out which of these computations is the "user", then figures out likely utility functions of the user. Then it takes actions to increase a combination of these utility functions (possibly using something like maximal lotteries, rather than averaging utility functions). There are also steps to ignore certain hypotheses which might be malign, but I will ignore this issue here.

Lets add more detail to the "figuring out the utility function of the user" step. The probability $Pr\left[U\right]$ that an agent $G$ has utility function $U$ is proportional to $$\frac{2^{-}K\left(U\right)}{P{r}_{\pi \in \xi }\left(\right[U(\lceil G\rceil ,\pi )\le U(\lceil G\rceil ,G\ast )\left]\right)},$$ where $K\left(U\right)$ is the complexity of the utility function $U$, and $P{r}_{\pi \in \xi }\left[U\right(\lceil G\rceil ,\pi )\ge U(\lceil G\rceil ,G\ast \left)\right]$ is the probability that a random policy $\pi$ (according to a distribution $\xi$ over possible policies) is better than the policy $G\ast$ of the agent $G$.

So, a utility function is more likely if it is less complex, and if the agent's policy is better at satisfying $U$ than a random policy.

How would a human's utility function according to preDCA compare with CEV?

My intuition is that preDCa falls short on the "extrapolated" part in "Coherent extrapolated volition". PreDCA would extract a utility function from the flawed algorithm implemented by a human brain. This utility function would be coherent, but might not be extrapolated: The extrapolated utility function (ie what humans would value if they would be much smarter) is probably more complicated to formulate than the un-extrapolated utility function.

For example, the policy implemented by an average human brain probably contributes more to total human happiness than most other policies. Lets say $U_1$ is an utility function that values human happiness as measured by certain chemical states in the brain, and $U_2$ is "extrapolated happiness" (where "putting all humans brains in vat to make it feel happy" would not be good for $U_2$). Then it is plausible that $K\left(U_1\right)<K\left(U_2\right)$. But the policy implemented by an average human brain would do approximately equally well on both utility functions. Thus, $Pr\left[U_1\right]>Pr\left[U_2\right]$.

The concerns might also apply to a similar proposal [LW · GW].