Comment by intern on Hooray for stepping out of the limelight · 2025-01-31T05:46:46.576Z · LW · GW

I don't know if this is just hindsight, but tracr has in no way turned out to be safety relevant. Was it considered to be so at the time of commenting?

Comment by intern on Sheikh Abdur Raheem Ali's Shortform · 2025-01-10T07:11:59.597Z · LW · GW

(I think a lot of unlearning research is bullshit, but besides that, is anyone deploying large models doing unlearning?)

Why do you think this? Is there specific research you have in mind? Some kind of reference would be nice. In the general case, it seems to me that unlearning matters because knowing how to effectively remove something from a model is just the flip-side of understanding how to instill values. Although not the primary goal of unlearning, work into how to 'remove' should also equally benefit attempts to 'instill' robust values into the model. If fine-tuning for value alignment just patches over 'bad facts' with 'good facts' any 'aligned' model will be less robust than one with harmful knowledge properly removed. If the alignment faking paper and peripheral alignment research are important at a meta level, then perhaps unlearning will be important because it can tell us something about 'how deep' our value installation really is, at an atomic scale. Lack of current practical use isn't really important, we should be able to develop theory that will tell us something important about model internals. I think there is a lot of very interesting mech-interp of unlearning work waiting to be done that can help us here.

Comment by intern on EIS IX: Interpretability and Adversaries · 2024-12-06T00:29:50.321Z · LW · GW

Networks that have to learn more features may become more adversary-prone simply because the adversary can leverage more features which are represented more densely. 

 

Also, in the top figure the loss is 'relative to the non-superposition model', but if I'm not mistaken the non-superposition model should basically be perfectly robust. Because it's just one layer, its Jacobian would be the identity, and because the loss is MSE, any perturbation to the input would be perfectly reflected only in the correct output feature, meaning no change in loss whatsoever. It's only when you introduce superposition that any change to the input can change the loss (as features actually 'interact').