The Big Picture Of Alignment (Talk Part 2)

This is great. One question it raises for me is: Why is there a common assumption in AI safety that values are a sort of existent (i.e., they exist) and latent (i.e., they are not directly observable) phenomena? I don't think those are unreasonable partial definitions of "values," but they're far from the only ones and not at all obvious that they're the values with which we want to align AI. Philosophers Iason Gabriel (2020) and Patrick Butlin (2021) have pointed out some of the many definitions of "values" that we could use for AI safety.

I understand that just picking an operationalization and sticking to it may be necessary for some technical research, but I worry that the gloss reifies these particular criteria and may even reify semantic issues (e.g., Which latent phenomena do we want to describe as "values"?; a sort of verbal dispute a la Chalmers) incorrectly as substantive issues (e.g., How do we align an AI with the true values?).

Some thoughts on this question that you mention briefly in the talk.

What decision theory does evolution select for?

I think that evolution selects for functional decision theory [? · GW] (FDT). More specifically, it selects the best policy over a life time, and not the best action in a given situation. I don't mean that we actually cognitively calculate FDT, but that there is an evolutionary pressure to act as if we follow FDT 

Example: Revenge
By revenge I mean burning some of your utility just to get back at someone who hurt you. 

Revenge is equivalent to transparent Newcomb's problem. You can see that Omega has predicted that you will two-box, i.e. the box that could have lots of money is empty. What do you do? You can one-box anyway, and counterfactually make this situation less likely, but giving up the smaller reward too (take revenge) or you can accept that you can't change the past, cut your losses, and just take the smaller reward (no revenge).

The way this is evolutionally encoded in humans is not a tendency to think about counterfactual situations. Instead we get mad at Omega for withholding the larger reward, and we one-box out of spite, to make Omegas prediction wrong, forcing a loose-loose outcome. But it is still FDT in practice.

Taking revenge can also be justified causally, if it it's about upholding your reputation so no one crosses you again. Humans defiantly do this calculation too. But it seems like most humans have a revenge drive that is stronger than what CDT would recommend, which is why I think this example backs up my claim that evolution selects for FDT.

My claim is of a similar type as Caspar's Doing what has worked well in the past leads to evidential decision theory. It's a statement about the resulting policy, not the reasoning steps of the agent. Caspar describes an agent that does the action that has worked well in the past. Evolution is a process that select the policy that has worked well in the past, which should give you effectively some version of FDT-son-of-EDT. 

There are situations where FDT behave differently depending on when and why it was created (e.g.). I think I could figure out how this would playout out in the context of evolution, but it would take some more thinking. If you think I'm on the right track, and convince me that this is useful, I'll give it a try. 

Is there a transcript of this anywhere? Does this have new content on top of the links, or is it "merely" a presentation containing the content in the links?

One thing I can't help but think is:

So obviously as you mention, the whole thing about taking the infinite limits etc. is meant to be a hypothetical stand-in for doing things at a large scale. And similarly, obviously we don't have finite observations, so it's also an idealization there.

But this makes me think that perhaps some useful insights about the range of applicability for abstractions could be derived by thinking about convergence rates. E.g. if the information in an abstraction is spread among $n$ variables, then each "layer" (e.g. Markov blanket, resampling) could be expected to introduce noise on the scale of $n^{-1/2}$, so that seems to suggest that the abstraction is only valid up to a distance of around $\frac{1}{n^{-1/2}}=n^{1/2}$.

I'm not really sure how to translate this into practical use, because it seems like it would require some conversion factor between variable count and distance. I guess maybe one could translate it into a comparative rule, like "Getting $X$ times more observations of a system should allow you to understand it in a $\sqrt{X}$ times broader setting", but this is probably a mixture of being too well-known, too abstract or too wrong to be useful.

But regardless of whether I'll come up with any uses for this, I'd be curious if you or anyone else has any ideas here.

Thought: at one point, you talk about taking a general (not necessarily causal, infinite, or anything like that) distribution and applying the resampling process to this, leading to capturing the redundant information.

But does that necessarily work? I'd think that if e.g. your distribution was a multivariate Gaussian with nondeterministic correlations, you'd get regression to the mean, such that the limit of the resampling process just makes you end up with the mean. But this means that there's no information left in the limiting variable.

I think what goes wrong is that if you start resampling the multivariate Gaussian, you end up combining two effects: blurring it (which is what you want, to abstract out nonlocal stuff), and dissipating it to the mean (which is what you don't want). As long as you haven't removed all the variance in the dissipation, the blur will still capture the information you want. But as you take the limit, the variance goes to zero and that prevents it from carrying any information.

In the Gaussian case specifically, you can probably solve that by just continually rescaling as you take the limit to keep the variance high, but I don't know if there is a solution for e.g. discrete variables.