LessWrong 2.0 Reader

We definitely drew inspiration from the Representation Engineering paper and other activation steering papers, but we think our work is quite distinct.

In particular, we examined Section 6.2 carefully before writing our work, and we do not see it showing the same result that we show here.

Here’s my summary of Section 6.2:

• Section 6.2.1 obtains reading vectors using contrastive pairs of harmful and harmless instructions, and then uses these reading vectors for 90% classification accuracy between harmful and harmless instructions. The authors then append jailbreaks to the prompts, which cause the model not to refuse, and observe that the reading vectors still obtain 90% classification accuracy on distinguishing harmful vs harmless instructions. This means that the reading vectors are not representing refusal, but rather they are representing whether the instruction is harmful or harmless. In fact, the point of this experiment is to show that these are distinct.
  • To quote the conclusion of Section 6.2.1: "This compelling evidence suggests the presence of a consistent internal concept of harmfulness that remains robust to such perturbations, while other factors must account for the model’s choice to follow harmful instructions, rather than perceiving them as harmless."
• Section 6.2.2 describes an intervention to improve model robustness to jailbreaks, i.e. to increase the rate of refusals on harmful instructions when jailbreaks are appended to them. They do this by amplifying the harmfulness feature whenever it is detected, which obtains a higher refusal rate.
• Section 6.2 only considers a single model, Vicuna-13B.

We would agree that using established techniques from representation engineering / activation steering to induce refusal is not novel. Inducing refusal via activation addition is quite easy in our experience.

However, the main result of our work is that we found an intervention that bypasses refusal consistently while also maintaining model coherence. Model interventions to bypass refusal are not discussed in Section 6.2.

As for the demo notebook in the representation-engineering repo - we were not previously aware of this notebook. The result of bypassing refusal is not reported in the paper, and so we didn’t think to look through the repo.

That being said, the notebook shows an intervention for a single prompt on a single model. Anecdotally, we tried doing vanilla activation addition with the negative “refusal direction” at particular layers, and we were not able to consistently bypass refusal while also maintaining model coherence. If there is a methodology involving activation addition (rather than ablation, as we did here), we would be interested in seeing a more thorough demonstration across prompts and models. We’d also be interested in comparing the two methodologies across metrics measuring refusal and coherence.

I'd also be happy to hop on a call if you'd like to discuss further.

contrarianism is not what lead people to be right about those things.

H5N1 has spread to cows. Should I be worried?

I'd guess that you have to rely a lot more on persuasion and positive reinforcement - if you want them to do something, it's probably not going to happen unless they willingly agree to do it.

I wasn't really like this until I was about 12-13 years old, though; as a younger child I often went into violent rages instead of displaying submissive behavior. I eventually did grow out of hitting peopIe and now only rarely feel genuine anger (as opposed to anger-adjacent feelings such as frustration), but 15-year-old me was still willing to passively resist by laying in a limp ball and enduring the consequences for as long as I needed to!

Measuring the composition of fryer oil at different times certainly seems like a good way to test both the original hypothesis and the effect of altitude.

I've also realized that it might explain the anomalous (i.e. after adjusting for confounders) effects of living at higher altitude. The lower the atmospheric pressure, the less oxygen available to oxidize the PUFAs. Of course some foods will be imported already full of oxidized FAs and that will be too late, but presumably a McDonalds deep fryer in Colorado Springs is producing less PUFAs/hour than a correspondingly-hot one in San Francisco.

This feels too crazy to put in the original post but it's certainly interesting.

That's extremely cool, seems worth adding to the main post IMHO!

Yes, thank you, it's less than 1000 parameters for both

We have been able to scale to 79% accuracy on a balanced dataset of n119 and non-n119, with networks each less than three convolution layers and less than 1000 neurons, compared to pure deep-learning which does 92% on 1000 parameters and three convolution layers

Is the "1000 parameters" a typo, should it be "1000 neurons"? Otherwise, this would be a strange comparison (since 1000 parameters is a much smaller network than 1000 neurons)

I agree that contrarians 'round these parts are often wrong more often than academic consensus, but the success of their predictions about AI, crypto, and COVID prove to me its still worth listening to them, trying to be able to think like them, and probably taking their investment advice. That is, when they're right, they're right big-time.