This is a cool result. If I'm understanding correctly, M- increases its loss the more that M+ is represented in the mixture, thereby encouraging SGD to make M- more prominent.

Is there a way to extend this to cases where M- doesn't have access to the weights? I think that probably requires an RL environment, but that's entirely based on "I thought about it for a few minutes and couldn't find a way to do it without RL" so I could be way off here.

Given an RL environment I suspect M- could steer the model into scenarios that make it look better than M+...

When you say "which yields a solution of the form $f\left(w\right)=c_1/(1-w)+c_2$", are you saying that $f^{\prime }\left(w\right)/f\left(w\right)=1/(1-w)$ yields that, or are you saying that $(1-w)f^{\prime }\left(w\right)-f\left(w\right)>0$ yields that? Because, for the former, that seems wrong? Specifically, the former should yield only things of the form $f\left(w\right)=c_1/(1-w)$ .

But, if the latter, then, I would think that the solutions would be more solutions than that?

Like, what about $g\left(w\right):=c_1/(1-w)+c_2\cdot (1-\frac{1}{{10}^6}+\frac{1}{{10}^6}\cos (w\left)\right)$ ? (where, say, $c_1=\epsilon +\delta$ and $c_2=-\delta$
$g^{\prime }\left(w\right)=\frac{c_1}{(1-w{)}^2}+c_2\cdot (-\frac{1}{{10}^6}\sin (w\left)\right)$ . so $(1-w)g^{\prime }\left(w\right)-g\left(w\right)=c_2\cdot (-\frac{1}{{10}^6}(1-w\left)\sin \right(w\left)\right)-c_2\cdot (1-\frac{1}{{10}^6}+\frac{1}{{10}^6}\cos (w\left)\right)$
$=-c_2\cdot (1-{10}^{-6}\cdot (1-\cos \left(w\right)-(1-w)\sin \left(w\right)\left)\right)$
which, for $c_2<0$, and $w\in [0,1]$, is positive, and so g should also be a solution to $(1-w)f^{\prime }\left(w\right)-f\left(w\right)>0$ , yes?

Or, were you just giving a subset of the solutions, of the differential inequality, which were all the ones you needed to make the point?

Separately (btw, what's the convention if one has unrelated comments? to put them in one comment or separate comments?) :
As for how the network could detect something about its current weights, if it is a recurrent network, this seems like something that some form of this should be learnable.

Suppose you set up the architecture for a recurrent neural network, and pick some arbitrary linear combination (but where the coefficients aren't especially big) of the weights in the network, and then for the loss function, take the main part of the network's output as doing some normal task, and then another part of the output be judged for how well it approximates the current value of the linear combination of all the network's weights, with the initially chosen coefficients. It seems plausible to me that this would do well. Of course, one way it might fail to work properly is if this linear combination ends up becoming roughly constant, or just if some weights stop changing during the training, leading to this second output not getting the signal needed to learn that part. Maybe if you used dropout and took that which weights were dropped out when evaluating the weighted sum of the weights (counting them as zero) in the loss, it could fix that issue. (dropout being done the same in each recurrence, not separately.)

Why "a linear combination of the weights" instead of the particular weights? Well, I imagine getting all the individual weights would be not particularly feasible unless the architecture was specifically designed to do that (like in the architectures for those quine-ing neural nets), but, if we are looking at a mix between "how the net should be", and "a net that is doing gradient hacking", then it seems like "some particular weighted sum of the weights", might be all that would be needed. Or perhaps two weighted sums with different coefficients if having the $w_1$ and $w_2$ separate.

On the other hand, if the network isn't recurrent, or if all the recurrence is done in a vector-quantization-y way, I would imagine it would be more difficult for the network to estimate a given linear combination of its current weights. (Though, if the linear combination of weights is only of weights before a certain layer, then layers after that layer might be able to approximate it without any recurrence, but, the layers that do and use this approximation could be modified without this being detected, and therefore, I'd think it couldn't protect these later parts from being changed by gradient descent, or at least, it would be harder.)
 

(I'm imagining, if people are trying to make this sort of thing work, that these networks are using just RELU, not like a smoothed version of RELU, so that the values of the weights are reflected more piecewise-linearly in the activations of one layer, given activations in the previous layer, which I think would facilitate estimating other linear combinations of the weights between the previous layers, and as such also for estimating this for a linear combination of weights in any layer.)

Epistemic status: Somewhat confused by the scenario described here, possible noob questions and/or commentary.

I am not seeing how this toy example of “gradient hacking” could actually happen, as it doesn’t map on to my understanding of how gradient descent is supposed to work in any realistic case.

Suppose, we have a mixture consisting of a good model $M^{+}$ which gets 0 loss in the limit (because it’s aligned with our training procedure) and a gradient hacker $M^{-}$ which gets $ϵ$loss in the limit (because its actual objective is paperclips).

The mixture is roughly a linear combination of the two models, weighted by $w_1$ for the good model and $w_2$ for the bad model. Suppose we’re doing SGD on the vector $(w_1,w_2)$, and the total loss for the mixture is $L_t(w_1,w_2)=w_{1}L\left(M^{+}\right)+w_{2}L\left(M^{-}\right)$

Can you flesh this out more by giving an example of what a training data point would be in this scenario?  What are the input features of a training case and what output target variable is the model trying to predict?

Also, it is not clear from this example if the outputs of the whole composite model are:

1. A weighted average of the outputs of the 2 component models OR
2. The weighted sum of $w_1\left(M^{+}Output\right)+w_2\left(M^{-}Output\right)$
    

But either way, I don't see why the loss for the composite model would necessarily be equal to a weighted sum of the losses of each component model, as stated in the example above.

In principle, it seems to me the loss on such a composite model could (for example) be 0 on some data set, even if both components had a positive loss function value for that same data.

As a simplified example of this, suppose you have a 2-component weighted-average model that is trying to predict the Y value of points in a 2-dimensional (X,Y) coordinate plane given the X value.  And suppose all the data points are on the line:

 $Y=X+1$ 

If the 2 components of the model are:

1. $Y=X$ AND
2. $Y=X+2$

I would expect the weighted average composite model would have a lower loss then either component model (and 0 loss if the weights are equal).

Also suppose $M^{-}$ can see $(w_1,w_2)$ perfectly and its loss is a function of its relative weight in the mixture. I.e. $L\left(M^{-}\right)=f(w_1,w_2)$, which I'll refer to as $M^{-}$'s loss modulating function.

I am disinclined to suppose this - unless someone explains how it could make sense for the machine learning engineers to do it that way.

My understanding is that the loss function of a model should track how close the model’s answers are to the correct or optimal answers for some data. But in this example, $W_1$ and $W_2$ are not part of the training data, and can vary independently from the $M^{-}$ outputs on the training cases.

$M^{-}$ has a calculable loss on the training data, which means $M^{-}$ could be a stand-alone model for this dataset in its own right even if $M^{+}$,  $W_1$, and $W_2$ didn’t exist.

(Which is to say, $M^{-}$ takes inputs in a form that would represent the features of a training data point and deterministically produces outputs of a data type that matches the target variable. This seems to me to be implied by the description, otherwise I don’t understand what it would mean to say that the loss of $M^{-}$ is ϵ.)

Furthermore, this example supposes that the gradient descent process is only optimizing $W_1$ and $W_2$, and not optimizing, or altering the internal parameters of, $M^{+}$ or $M^{-}$.

So it seems to me, based on this description, that the loss for $M^{-}$on a given set of training data should *not* vary with $W_1$ or $W_2$ - if they are doing gradient descent in any kind of normal way (unless I am misunderstanding some big part of how gradient descent works). Rather, you should be able to give $M^{-}$ the same training data batch $1000$ times in a row, while varying $W_1$ and $W_2$, and you should get the same outputs and the same loss (if the parameters for the stand-alone $M^{-}$ are the same each time).

So I don't see how "gradient hacking" could occur in this scenario if the composite model is using any reasonable loss function.

If the composite model is a weighted average I would expect gradient descent to reduce  $W_2$ to $0$ or nearly $0$, since if $M^{+}$ is matching the correct output exactly, and $M^{-}$ is not, then the composite model can always get a closer answers by giving more relative weight to $M^{+}$.

If the composite model is a weighted sum of the outputs, I would expect that (for most possible training data sets and versions of $M^{-}$) $W_1$ would tend to gravitate towards $1$ and $W_2$ would tend to gravitate towards $0$. There might be exceptions to this if $M^{-}$'s outputs have a strong correlation with $M^{+}$'s outputs on the training data, such that the model could achieve low loss with some other weighted sum, but I would expect that to be unusual.