Thoughts on the good regulator theorem

post by JonasMoss · 2022-08-11T12:08:28.897Z · LW · GW · 0 comments

Contents

  Setup of the paper
  The theorem
    Proposition
    Proof
    Rough explanation
None
No comments

I just became aware of the good regulator theorem through John Wentworth's post [LW · GW]. Then I tried to read the paper he talked about, Every Good Regulator Of A System Must Be A Model Of That System. This is among the worst papers I've ever skimmed. It's barely comprehensible. Reading this paper is not like reading a regular text. I had to try to read their intentions instead.

This post is my attempt to make sense of the setup and the results of the paper. Summed up, my conclusions are:

• Their main theorem is correct but trivial and not very interesting. (It can be strengthened too, see the proposition below.) Their proof is a mess I haven't bothered looking at, it's way too complex.
• Their interesting conclusion, that an optimal action can only be made if you have a model of the system (i.e., everything outside your control), is false.
• Some thoughts about terminology:
  • It's definition of "most simple" probably doesn't make sense.
  • The definition of "isomorphisms" at least makes sense, but shouldn't have been called "isomorphism".

Setup of the paper

I don't follow the notation or terminology in the paper as it can be replaced with more familiar notation from decision theory.

Define the sets (the name in the paper in parenthesis) $$\begin{array}{ccc}\Omega \left(D\right)& :& \text{the domain of inputs, can be thought of as a probability space},\\ X\left(S\right)& :& \text{everything outside our control,}\\ A\left(R\right)& :& \text{the set of actions we can take}.\\ Y\left(Z\right)& :& \text{the final output, combining}X\text{and}a\text{, defined below}.\end{array}$$ Define the functions $$\begin{array}{ccc}X\left(\varphi \right)& :& \Omega \to X,\text{a random variable we don't control},\\ a\left(\rho \right)& :& \Omega \to A,\text{our action, choosen by us,}\\ \psi \left(\psi \right)& :& X\times A\to Y,\text{the outcome space.}\end{array}$$ For each uncontrollable outcome $x$ we have a non-empty set $G\left(x\right)$ describing the good outcomes. Two examples are:

• Goal sets. Let $G\subseteq Y$ be a goal set, a set of outcomes that are good, independently of the value of $x$. Assume that for all $x$ there is an action $a$ so that $\psi (x,a)\in G$. Then $G\left(x\right)=\left\{\right(x,a)\mid \psi (x,a)\in G\}$ is non-empty for all $x$. This is what they do in the beginning of their paper.
• Utility maximization. Let $u$ be a utility function and assume that ${min}_{a\in A}u\left(\psi \right(a,x\left)\right)$ exists for all $x$. Define $G_u\left(x\right)=\left\{\right(x,a)\mid a\in {max}_{a\in A}u(\psi (x,a)\left)\right\}$. In the paper they talk about minimizing entropy specifically, i.e., $G\left(x\right)=\left\{\right(x,a)\mid a\in {max}_{a\in A}-\text{Entropy}(\psi (x,a)\left)\right\}$.

Note: The authors don't use $G\left(x\right)$; they only use the entropy.

Our goal is to find an optimal action function $a$ that satisfies $$\psi \left(X\right(\omega ),a(\omega \left)\right)\in G\left(X\right(\omega \left)\right),\text{for all}\omega .$$ In other words, we want to make sure that every outcome is good. This is the regulation problem.

The regulation problem is equivalent to a utility-maximization problem with no randomness. Let $u:Y\to R^{+}$ be an arbitrary utility function. An optimal action $a\left(\omega \right)$ according to the regulator problem satisfies

$$\psi \left(X\right(\omega ),a(\omega \left)\right)\in G_u\left(X\right(\omega \left)\right)=\left\{\psi \right(X\left(w\right),a)\mid a\in \underset{a\in A}{min}u(\psi \left(X\right(\omega ),a)\left)\right\},$$ but that is equivalent to the optimal action as defined by ordinary decision theory, $$a\left(\omega \right)\in \underset{a\in A}{min}u\left(\psi \right(X\left(\omega \right),a\left)\right).$$The other way around is similar.

The theorem

The point of the paper is to prove that any simple action function $a$ has to be isomorphic to the mapping $X:\Omega \to X$. Their main theorem states

Theorem: The simplest optimal regulator $R$ of a reguland $S$ produces events $R$ which are related to the events $S$ by a mapping $h:S\to R$.

In my notation this is

Theorem: The "simplest" optimal action function $a$ can be written as $a\left(\omega \right)=h\left(X\right(\omega \left)\right)$ for some function $h$.

In the abstract they claim that "making a model is necessary," where "making a model" presumably means that you have to reconstruct $X\left(\omega \right)$ in order to calculate the optimal action $a\left(\omega \right)$. This conclusion is false. I guess they made the mistake due to their misapplication of the word isomorphism.

Their notion of isomorphism (used elsewhere in the paper, including in the abstract, but not in the main theorem) is quite straight-forward, but not similar to any other use of word that I've seen. They say (equation 8) that two functions $f,g$ are isomorphic to one another if there is any $h$ so that $f\left(x\right)=h\left(g\right(x\left)\right)$. This is not even close to the spirit of isomorphism, as the fundamental property of isomorphisms is to preserve structure perfectly. For instance, isomorphisms should be transitive, i.e., if $A\sim B$ and $B\sim C,$then $A\sim B$. This is obviously not the case in this example, as it's easy to see that the constant function is "isomorphic" to any function. I could agree to $f$ and $g$ being homomorphic, as homomorphism do not require invertibility, but isomorphism is just simply the wrong word. (The point here is not to argue over definitions, just that you shouldn't use words in a way that confuses yourself and other people.)

That said, their notion of isomorphism isn't useless. It allows you to calculate the value of $f\left(x\right)$ from the value of $g\left(x\right)$. And that is a nice property. But it doesn't tell you that you need $g\left(x\right)$ to calculate $f\left(x\right)$, as claimed in the abstract. If we were talking about real isomorphisms on the other hand, you would've always been able to at least recover $g\left(x\right)$ from $f\left(x\right)$ by the inverse mapping $g\left(x\right)=h^{-1}\left(f\right(x\left)\right)$ -- and I think that's closer to what they actually wanted to show.

I haven't made much of an effort to understand their usage of the word "simple". But it turns out it doesn't matter, because their result holds for all action functions $a$, not only the "simple" ones.

What is the spirit of their claims? I think the point is that you should expect $X\left(\omega \right)$ to be either

• (a) directly used in the computation of $a$, or
• (b) recoverable from $a$, at least provided we know $\psi$.

Both claims are really easy to disprove, you only need to have $x$ independent of the outcome $\psi (x,a)$. Then we don't need to use $X\left(\omega \right)$ and we can't recover $X\left(\omega \right)$ from $a$ if the space $X$ contains more than one element. I care about claim (b) because it's one of the few ways to formalize that $a$ "uses" or "knows" $X$. Recall that $a$ is just a function, a collection of ordered pairs $(\omega ,a(\omega \left)\right)$. This function doesn't know anything. (I think this might be relevant for the ELK problem too. A function doesn't have any latent knowledge. It's just a function.)

The following proposition is, as far as I understand their paper, an extension of their main theorem. There they claim there is a mapping $h:X\to Y$ that produces the optimal outcome in the sense that $a\left(\omega \right)=h\left(X\right(\omega \left)\right)$. This is true, and in fact an even stronger statement is true. But it doesn't mean what they think it means.

Proposition

Let $f_x,x\in X$ be a parameterized family of choice functions on $A$. Define $h_{\left\{f_x\right\}}:X\to A$ by $$h_{\left\{f_x\right\}}\left(x\right)=f_x\left(\right\{a\mid \psi (x,a)\in G\left(x\right)\left\}\right).$$ Then

• (a) The function $a\left(\omega \right)=h_{\left\{f_x\right\}}\left(X\right(\omega \left)\right)$ is an optimal action function (a solution to the regulation problem),
• (b) If $a$ is an optimal action function (solution to the regulation problem) there is a family of choice functions so that $a\left(\omega \right)=h_{\left\{f_x\right\}}\left(X\right(\omega \left)\right)$,
• (c) Even if we know $\psi$ and one of the families of choice functions $f_x,x\in X$ so that $a\left(\omega \right)=h_{\left\{f_x\right\}}\left(X\right(\omega \left)\right)$, we will not always be able to reconstruct $X\left(\omega \right)$.

Proof

The first two claims are trivial. The third was explained above. The point of the parameterized choice functions is to make it possible to construct functions that choose any element from $\{a\mid \psi (x,a)\in G(x\left)\right\}$. There are probably more elegant ways to formulate it.

Rough explanation

The regulation problem is equivalent to a utility-maximization problem as already explained. A function is an optimal action function iff it satisfies $$a\left(\omega \right)\in \underset{a\in A}{min}u\left(\psi \right(X\left(\omega \right),a\left)\right).$$ Here ${min}_{a\in A}u\left(\psi \right(X\left(\omega \right),a\left)\right)$ can be a set, that's why I use the $\in$ symbol. There are potentially many optimal action functions. For instance, if $|\Omega |=10$ and each set ${min}_{a\in A}u\left(\psi \right(X\left(\omega \right),a\left)\right)$ contains two elements, there are $2^{10}$ action functions. This should explain the rôle of the choice functions. You can see from the defining property of $a\left(w\right)$ that it only depends on $\omega$ through $X\left(\omega \right)$; this explains the existence of the $h$ functions.

0 comments

Comments sorted by top scores.