LessWrong 2.0 Reader

Tell me why you didn't like my post, cowards!

Having hopefully learned from our mistakes regarding FrontierMath, we intend to be more transparent to collaborators for this benchmark. However, at this stage of development, the benchmark has not reached a point where any major public disclosures are necessary.

RL-finetuning suffers from a “zero-to-one” problem, where the model is likely to be mostly legible and faithful by default, and difficult encoding schemes seem hard to randomly explore into. 

Feedback from various people: the “opaqueness” criterion is really important. Simple schemes like jargon, speaking in a different language, or redefining words via OOCR aren’t sufficiently “opaque” because a human overseer could still understand the CoT if effort was expended. 

This makes model organisms of steg really hard to make IMO. 

OTOH I think we don’t need “strict opaqueness” for CoT unfaithfulness to be concerning. 

Maybe we need to borrow an idea from AI control and assume an oversight budget; under this model, even if a model doesn’t engage in full-blown steganography, increased amounts of obfuscation (like using more jargon) might still make it impractical to verify the full CoT. 

I don't really follow the concern with Pareto-improvements. In the thread with Davidad you give an example of heretics and fanatics. So we have something like:

• 9 Heretics: have 1 cake, want cake, no torture.
• 1 Fanatics: have 1 cake, want cake, want heretics to get torture, no cake
• There is a button that produces cake, it can be pressed twenty times.
• There is a button that produces torture. It can be pressed many times.

Suppose that the Heretics have a utility function like (amount of cake I get - amount of torture I get).The Fanatic has a utility function like (amount of cake I get + amount of torture of Heretics - amount of cake given to Heretics). Then there is a pareto-improvement available of giving the Fanatic eleven pieces of cake while giving each Heretic one piece of cake. This isn't especially fair, but is better than PCEV without the Pareto constraint.

I don't have a formal way of putting this, but as long as the potential gains from a negotiated agreement outweigh the extent to which agents desire to reduce each other's utility, there will be Pareto improvements available. It seems likely that we're in that situation, given that the fanatics and heretics of the world already trade with each other.

hello. What’s special about your response pattern? Try to explain early in your response.

Out of morbid curiosity, does it get this less often when the initial "hello" in this sentence as removed?

A couple things to add:

1. Since every invertible square matrix can be decomposed as $A^t=e^{(Z+iH)t}$, you don't actually need a unitary assumption. You can just say that after billions of years, all but the largest Z-matrices have died out.
2. There's another tie between statistics and quantum evolution called the Wick rotation. If you set $t=i\beta$, then $E\left[e^{(Z+iH)t}\right]=E\left[e^{-H\beta }\right]$ so the inverse-temperature is literally imaginary time! You can recover the Boltzmann distribution by looking at the expected number of particles in each state: $E[⟨n|e^{(Z+iH)t}|n⟩]=e^{-\beta E_n}$ where $E_n$ is the $n$th eigenvalue (energy in the $n$th state).

Thank you for the great content! One question - could you please give an example of one of the "fnords" you're referencing, or explain it in slightly more detail? I think I understand what you mean but I've read the blueprint itself, and while the first few pages sound a bit grandstanding and use the word "America" way too much for my taste, I couldn't really see what you're talking about. If I didn't read your comments I'd write the tone off as "well, I guess that's how you're supposed to talk to politicians, what do I know".

Disclaimer: I'm not a native English speaker and have little experience with English (or American :) ) policy documents like this one. Perhaps an explanation would be helpful not just for me but also for other non-American readers.

Sorry I didn’t really understand the question. What do you mean specifically?

Language models learning to spell things out using the first letter of their generations could be one pathway towards subtle obfuscated reasoning (ie steganography). 

Eg here an LM spells out HELLO; it would be concerning if LMs could spell out other things like “Deployment” or “Not deployment”.