LessWrong 2.0 Reader

You can't always use liability to internalise all the externality because e.g. you can't effectively sue companies for more than they have, and for some companies that stay afloat by regular fundraising rounds, that may not even be that much? like, if they're considering an action that is a coinflip between "we cause some huge liability" and "we triple the value of our company" then it's usually going to be correct from a shareholder perspective to take it, no matter the size of the liability, right?

Criminal law has the ability to increase the deterrent somewhat – probably many people will not accept any amount of money for a significant enough chance of prison – though obviously it's not perfect either

Declarative and procedural knowledge are two different memory systems. Spaced repetition is good for declarative knowledge, but for procedural (like playing music) you need lots of practice. Other examples include math and programming - you can learn lots of declarative knowledge about the concepts involved, but you still need to practice solving problems or writing code.

Edit: as for why practice every day - the procedural system requires a lot more practice than the declarative system does.

I think this little mistake doesn't affect the gist of your summary post, I wouldn't worry about it too much.

The mistake

The mistaken argument was an attempt to explain why $\alpha \ge 2/d$. Believe it or not, the paper never actually argued $\alpha \ge 2/d$, it argued $\alpha \ge 1/d$. That's because the paper used the L2 loss, and $\alpha \ge 2/d\text{(L2)}⟺\alpha \ge 1/d\text{(L1)}$.

I feel that was the main misunderstanding.

If you read closely, it only says $\alpha \ge 4/d\text{(L2)}$ for piecewise linear approximations:

If we assume that $F$ and $f$ are analytic functions on $M_d$ and that the loss function $L(f,F)$ is analytic in $f-F$ and minimized at $f=F$, then the loss at a given test input, $x_{\text{test}}$, can be expanded around the nearest training point, ${\hat{x}}_{\text{train}}$, $L\left(x_{\text{test}}\right)={\sum }_{m=n\ge 2}^{\infty }{a}_m\left({\hat{x}}_{\text{train}}\right)(x_{\text{test}}-{\hat{x}}_{\text{train}}{)}^m$,2 where the first term is of finite order $n\ge 2$ because the loss vanishes at the training point. As the typical distance between nearest neighbor points scales as $D^{-1/d}$ on a $d$-dimensional manifold (an observation also made in [18]), the loss will be dominated by the leading term, $L\propto D^{-n/d}$, at large $D$.  Note that if the model provides an accurate piecewise linear approximation, we will generically find $n\ge 4$.

Figure 2b in the paper shows $\alpha =4/d\text{(L2)}$ for most models but $\alpha =2/d\text{(L2)}$ for some models.

Some people like the L2 loss (variance) because you can use simple addition to combine independent sources of variance. The variance increase from each source doesn't depend on the variance increase from the other sources.

The paper never mentioned L2 loss, just that the loss function is "analytic" and minimized at $f=F$. Such a loss function converges to L2 when the loss is small enough. This important sentence is hard to read because it's cut in half by graph.

Nearest neighbors regression/classification has $\alpha =1/d\text{(L1)}$

If you use the nearest neighbors regression to fit the function $f\left(x\right)=sin\left(x/\pi \right)$, it creates a piecewise constant function that looks like a staircase trying to approximate a curve. The L1 loss is proportional to D^(-1) because adding data makes the staircase smaller. A better attempt to connect the dots of the training data (e.g. a piecewise linear function) would have L1 loss proportional to D^(-2) or better because adding data makes the "staircase" both smaller and smoother. Both the distances and the loss gradient decrease.

My guess is that nearest neighbors classification also has L1 loss proportional to D^(-1/d), because the volume that is misclassified is roughly proportional to distance between the decision boundary and the nearest point. Again, I think it creates a decision boundary that is rough (analogous to the staircase), and the roughness gets smaller with additional data but never gets any smoother.

What a rough decision boundary looks like:
https://upload.wikimedia.org/wikipedia/commons/5/52/Map1NN.png

Suggestions

If you want a quick fix (still using the L1 loss), you might change the following:

Under the assumption that our test loss is sufficiently “nice”, we can do a Taylor expansion of the test loss around this nearest training data point and take just the first non-zero term. Since we have perfectly fit the training data, at the training data point, the loss is zero; and since the loss is minimized, the gradient is also zero. Thus, the first non-zero term is the second-order term, which is proportional to the square of the distance. So, we expect that our scaling law will look like kD^(-2/d), that is, α = 2/d. (EDIT: I no longer endorse the above argument, see the comments.)

The above case assumes that our model learns a piecewise constant function. However, neural nets with Relu activations learn piecewise linear functions. For this case, we can argue that since the neural network is interpolating linearly between the training points, any deviation of the distance between the true value and the actual value should scale as D^(-2/d) instead of D^(-1/d), since the linear term is being approximated by the neural network. In this case, for loss functions like the L2 loss, which are quadratic in the distance, we get that α = 4/d.

Note that it is possible that scaling could be even faster, e.g. because the underlying manifold is simple or has some nice structure that the neural network can quickly capture. So in general, we might expect α >= 2/d and for L2 loss α >= 4/d.

becomes

Under the assumption that our test loss is sufficiently “nice”, we can do a Taylor expansion of the test loss around this nearest training data point and take just the first non-zero term. Since we have perfectly fit the training data, the loss is zero at the training data point. With a constant term of zero, we use the linear term (gradient $\cdot$ displacement), which is proportional to the distance. So, we expect that our scaling law will look like kD^(-1/d), that is, α = 1/d.

The above case assumes that our model learns a piecewise constant function. However, neural nets with Relu activations learn piecewise linear functions. For this case, we can argue that since the neural network is interpolating linearly between the training points, any deviation of the distance between the true value and the actual value should scale as D^(-2/d) instead of D^(-1/d), since the linear term is being approximated by the neural network (as the distance decreases the linear term also decreases). In this case, for loss functions like the L2 loss, which are quadratic in the distance, we get that α = 4/d.

Note that it is possible that scaling could be even faster, e.g. because the underlying manifold is simple or has some nice structure that the neural network can quickly capture. So in general, we might expect α >= 1/d and for L2 loss α >= 2/d.

Also change

α >= 2/d (and α >= 4/d for the case of L2 loss)

checking whether α >= 4/d

they find that α is significantly larger than 4/d

to

α >= 1/d (and α >= 2/d for the case of L2 loss)

checking whether α >= 2/d

they find that α is significantly larger than 2/d

Another option is to switch to the L2 loss.

Conclusion

Once AI scientists make a false conclusion like $\alpha \ge 2/d\text{(L1)}$ or $\nabla L\left(x_{\text{train}}\right)=0$, they may hallucinate arguments which justify the conclusion. A future research direction is to investigate whether large language models learned this behavior from the AI scientists who trained them.

I'm so sorry I'm so immature but I can't help it.

Overall, it's not a big mistake because it doesn't invalidate the gist of the summary.

 

Some excellent points (and I enjoyed the neat self-referentialism). 

Headline take is I agree with you that CoT unfaithfulness - as Turpin and Lanham have operationalised it - is unlikely to pose a problem for the alignment of LLM-based systems.

I think this for the same reasons you state:

1. Unfaithfulness is primarily a function of the training distribution, only appears in particular contexts, and might potentially be avoided by simply asking the LLM to explain its rationale;

and 

2. The o1 paradigm of RLing the CoT seems likely to remove unfaithfulness from o1's CoTs. 
The way that these CoTs work seems make the model systematically hyper-attentive to different facets of its context window in a way that is verbalised in detail (and is typically enabled by a literal "Hmmmm..." that draws its attention to an unconsidered detail). It strikes me that this would totally rule out the chance that the model still pays attention to bias without saying it: "But wait... The user implied that option A was probably correct". This is partially an empirical question - since we can't see the o1 CoTs, I'd pipedream love to see OpenAI do and publish research on whether this is true.  

This suggests to me that o1's training might already have succeeded at giving us what we'd want: an LLM that does, in fact, just say how it made its decision. (It remains an open question whether simply prompting normal LLMs to explain their rationale would also work). 

The only part of the Turpin paper that remains potentially worrying to me is the (actually unsurprising) demonstrated capacity of an LLM to fabricate spurious reasoning ("shooting outside the eighteen is not a common phrase in soccer") in order to support a particular decision. 

You can imagine all sorts of innocuous contexts that might incentivise an LLM to do this kind of thing. This might present some difficulties for oversight regimes - this is why I'd be interested in seeing whether something like o1 is capable of front-to-back undertaking an entire complex and malicious action whilst managing to obscure the actual nature of that malicious action (such that an automated LLM judge fails to detect that the action is in fact malicious).      

then we can all go home, right?

Doesn't this just shift what we worry about? If control of roughly human level and slightly superhuman systems is easy, that still leaves:

• Human institutions using AI to centralize power
• Conflict between human-controlled AI systems
• Going out with a whimper [LW · GW] scenarios (or other multi-agent problems)
• Not understanding the reasoning of vastly superhuman AI (even with COT)

What feels underexplored to me is: If we can control roughly human-level AI systems, what do we DO with them? 

Being able to vote for 'liking' or 'disliking' the quality of a Digital Twin response. Enough dislikes might trigger a model update and regeneration of the response.

@Nathan Helm-Burger [LW · GW] As a first step, moderators can now regenerate AI responses. I have also added you as a moderator, so feel free to use this feature.

Once we have more users (and have more time/resources to do it properly), we may introduce a rating system as you suggest. For now, any user can report a comment, and a moderator can regenerate it if deemed reasonable. Does it make sense?

Thanks!

I've been thinking about non AI catastrophic risks.

One that I've not seen talked about is the idea of cancerous ideas. That is ideas that spread throughout a population and crowd out other ideas for attention and resources.

This could lead to civilisational collapse due to basic functions not being performed.

Safeguards for this are partitioning the idea space and some form of immune system that targets ideas that spread uncontrollably.

Thanks! We'll take a closer look at these when we decide to extend our results for more models.

Let me make sure I understand your idea correctly:

1. We use a separate single-layer model (analogous to the SAE encoder) to predict the SAE feature activations
2. We train this model on the SAE activations of the finetuned model (assuming that the SAE wasn't finetuned on the finetuned model activations?)
3. We then use this model to determine "what direction most closely maps to the activation pattern across input sequences, and how well it maps".

I'm most unsure about the 2nd step - how we train this feature-activation model. If we train it on the base SAE activations in the finetuned model, I'm afraid we'll just train it on extremely noisy data, because feature activations essentially do not mean the same thing, unless your SAE has been finetuned to appropriately reconstruct the finetuned model activations. (And if we finetune it, we might just as well use the SAE and feature-universality techniques I outlined without needing a separate model).

If spaced repetition is the most efficient way of remembering information, why do people who learn a music instrument practice every day instead of adhering to a spaced repetition schedule?