Challenge: construct a Gradient Hacker

Seems like the easiest way to satisfy that definition would be to:

• Set up a network and dataset with at least one local minimum which is not a global minimum
• ... Then add an intermediate layer which estimates the gradient, and doesn't connect to the output at all.

↑ comment by Thomas Larsen (thomas-larsen) · 2023-03-10T00:51:57.018Z · LW(p) · GW(p)

This is a plausible internal computation that the network could be doing, but the problem is that the gradients flow back through from the output to the computation of the gradient to the true value y, and so GD will use that to set the output to be the appropriate true value. 

Replies from: thomas-larsen

↑ comment by Thomas Larsen (thomas-larsen) · 2023-03-20T21:10:53.354Z · LW(p) · GW(p)

Following up to clarify this: the point is that this attempt fails 2a because if you perturb the weights along the connection ${\nabla }_{\theta }L\left(\theta \right)\underset{ϵ\cdot Id}{\overset{}{\to }}output$, there is now a connection from the internal representation of $y$ to the output, and so training will send this thing to the function $f(D,\theta )\approx y$. 

This is very interesting and I'll have a crack. I have an intuition that two restrictions are too strong, and a proof in either direction will exploit them without telling you much about real gradient hackers:

1. Having to take your weights as inputs: 
   If I understand this correctly the weight restrictions needed mean $\theta$ actually lives on a d dimensional subspace of $R^w$.  My intuition is that in real life the scenarios we expect gradient hackers to get clues to their parameters but not necessarily a full summary. 
   
   In a reality maybe the information would be clues that the model can gleam from it's own behaviour about how it's being changed. Perhaps this is some low dimensional summary of it's functional behaviour on a variety of inputs? 
2. Having to move toward a single point ${\theta }_g$:
   I think the phenomena should not be viewed in parameter space, but function space although I do anticipate that makes this problem harder.
   Instead of expecting the gradient hacker to move toward the single point ${\theta }_g$, it is more appropriate to expect it to move towards the set of parameters that correspond to the same function (if such redundancy exists). 
   Secondly, I'm unsure but If the model is exhibiting gradient hacking behaviour, does that necessarily mean it tends toward a single point in function space. Perhaps the selections pressures would focus on some capabilities (such as the capacity to gradient hack) and not preserve others. 

Epistemics: Off the cuff. Second point about preserving capabilities seems broadly correct but a bit too wide for what you're aiming to actually achieve here. 

[Deleted]

My troll example is a fully connected network with all zero weights and biases, no skip connections.

This isn't something that you'd reach in regular training, since networks are initialized away from zero to avoid this. But it does exhibit a basic ingredient in controlling the gradient flow.

To look for a true hacker I'd try to reconfigure the way the downstream computation works (by modifying attention weights, saturating relus, or similar) based on some model of the inputs, in a way that pushes around where the gradients go.