LessWrong 2.0 Reader

The trouble is that (unless I'm misreading you?) that's a fully general argument against measuring what models can and can't do. If we're going to continue to build stronger AI (and I'm not advocating that we should), it's very hard for me to see a world where we manage to keep it safe without a solid understanding of its capabilities.

Well we don't know the sizes of the model, but I do get what you are saying and agree. Distil usually means big to small. But here it means expensive to cheap, (because test time compute is expensive, and they are training a model to cheaply skip the search process and just predict the result).

In RL, iirc, they call it "Policy distillation". And similarly "Imitation learning" or "behavioral cloning" in some problem setups. Perhaps those would be more accurate.

I suppose that means it might be worth writing an additional post that more directly responds to the idea that AGI will end material scarcity. I agree that thesis probably deserves a specific refutation.

Re. making this more efficient, I can think of a few options.

1. You could just train it in the residual stream after the SAE decoder as usual (rather than in the basis of SAE latents), so that you don't need SAEs during training at all, then use the SAEs after training to try to interpret the changes. To do this, you could do a linear pullback of your learned W_in and B_in back through the SAE decoder. That is, interpret (SAE_decoder)@(W_in), etc. Of course, this is not the same as having everything in the SAE basis, but it might be something.

2. Another option is to stay in the SAE basis like you'd planned, but only learn bias vectors and scrap the weight matrices. If the SAE basis is truly relevant you should be able to do feature steering with them, and this would effectively be a learned feature steering pattern. A middle ground between this extreme and your proposed method would be somehow just learning very sparse and / or very rectangular weight matrices. Preferably both.

Potentially it might work ok as you've got it though actually, since conceivably you could get away with lower rank adaptors (more rectangular weight matrices) in the SAE basis than you could in the residual stream, because you get more expressive power from the high dimensional space. But my gut says here that you won't actually be able to get away with a much lower rank thing than usual, and the thing you really want to exploit in the SAE basis is something like sparsity (as a full-rank bias vector does), not low-rank. In fact sparsity is often strongly in tension with low-rank (it's difficult to make a matrix which is simultaneously sparse and low rank).

I recommend most readers skip this subsection on a first read; it’s not very central to explaining the alignment problem.

Suggest either putting this kind of aside in a footnote, or giving the reader a handy link to the next section for convenience?

Actually, I have a little more to say:

Another way to think about higher-rank density matrices is as probability distributions over pure states; I think this is what Charlie Steiner's comment is alluding to. 

So, the rank-2 matrix from my previous comment, $\rho =\frac{1}{2}{I}_2$  can be thought of as$\frac{1}{2}|0⟩⟨0|+\frac{1}{2}|1⟩⟨1|$

, i.e., an equal probability of observing each of $|0⟩,|1⟩$. And, because $I_2=|x⟩⟨x|+|y⟩⟨y|$ for any orthonormal vectors $|x⟩,|y⟩$, again there's nothing special about using the standard basis here (this is mathematically equivalent to the argument I made in the above comment about why you can use any basis for your measurement). 

I always hated this point of view; it felt really hacky, and I always found it ugly and unmotivated to go from states $|\Psi ⟩$ to projections $|\Psi ⟩⟨\Psi |$ just for the sake of taking probability distributions. 

The thing above about entanglement and decoherence, IMO, is a more elegant and natural way to see why you'd come up with this formalism. To be explicit, suppose you have the state $|0⟩$, and there is an environment state that you don't have access to, say it also begins in state $|0⟩$, and initially everything is unentangled, so we begin in the state $|00⟩$. Then some unitary evolution happens that entangles us, say it takes $|00⟩$ to the Bell state $\frac{|00⟩+|11⟩}{\sqrt{2}}$. 

As we've seen, you should think of your state as being $\frac{1}{2}{I}_2$, and now it's clear why this is the right framework for probabilistic mixtures of quantum states: it's entirely natural to think of your part of the now-entangled system to be "an equal chance of $|0⟩$ and $|1⟩$", and this indeed gives us the right density matrix. It also immediately implies that you are forced to also allow that it could be represented as "an equal chance of  $|+⟩$ and $|-⟩$" where $|+⟩,|-⟩=\frac{|0⟩\pm |1⟩}{\sqrt{2}}$, and etc. 

But it makes it clear why we have this non-uniqueness of representation, or where the missing information went: we don't just "have a probabilistic mixture of quantum states", we have a small part of a big quantum system that we can't see all of, so the best we can do is represent it (non-uniquely) as a probabilistic mixture of quantum states. 

Now, you aren't obliged to take this view, that the only reason we have any uncertainty about our quantum state is because of this sort of decoherence process, but it's definitely a powerful idea. 

I assume that phenomenal consciousness is a sub-component of the mind.

I'm not sure what is meant by this; would you mind explaining?

Also, the in-post link to the appendix is broken; it's currently linking to a private draft.

Wow. I need to learn how to search for papers.  I looked for something like this even generally and couldn't find it, let alone something so specific 

Not on sci-hub or Anna's Archive, so I'm just going off the abstract and summary here; would love a PDF if anyone has one.

If you email the authors they will probably send you the full article.

Thanks for the support. I'll try and work a bit more on my first post in the coming days and I hope it will be up soon.