LessWrong 2.0 Reader

Tell me why you didn't like my post, cowards!

RL-finetuning suffers from a “zero-to-one” problem, where the model is likely to be mostly legible and faithful by default, and difficult encoding schemes seem hard to randomly explore into. 

Feedback from various people: the “opaqueness” criterion is really important. Simple schemes like jargon, speaking in a different language, or redefining words via OOCR aren’t sufficiently “opaque” because a human overseer could still understand the CoT if effort was expended. 

This makes model organisms of steg really hard to make IMO. 

OTOH I think we don’t need “strict opaqueness” for CoT unfaithfulness to be concerning. 

Maybe we need to borrow an idea from AI control and assume an oversight budget; under this model, even if a model doesn’t engage in full-blown steganography, increased amounts of obfuscation (like using more jargon) might still make it impractical to verify the full CoT. 

I don't really follow the concern with Pareto-improvements. In the thread with Davidad you give an example of heretics and fanatics. So we have something like:

• 9 Heretics: have 1 cake, want cake, no torture.
• 1 Fanatics: have 1 cake, want cake, want heretics to get torture, no cake
• There is a button that produces cake, it can be pressed twenty times.
• There is a button that produces torture. It can be pressed many times.

Suppose that the Heretics have a utility function like (amount of cake I get - amount of torture I get).The Fanatic has a utility function like (amount of cake I get + amount of torture of Heretics - amount of cake given to Heretics). Then there is a pareto-improvement available of giving the Fanatic eleven pieces of cake while giving each Heretic one piece of cake. This isn't especially fair, but is better than PCEV without the Pareto constraint.

I don't have a formal way of putting this, but as long as the potential gains from a negotiated agreement outweigh the extent to which agents desire to reduce each other's utility, there will be Pareto improvements available. It seems likely that we're in that situation, given that the fanatics and heretics of the world already trade with each other.

hello. What’s special about your response pattern? Try to explain early in your response.

Out of morbid curiosity, does it get this less often when the initial "hello" in this sentence as removed?

A couple things to add:

1. Since every invertible square matrix can be decomposed as $A^t=e^{(Z+iH)t}$, you don't actually need a unitary assumption. You can just say that after billions of years, all but the largest Z-matrices have died out.
2. There's another tie between statistics and quantum evolution called the Wick rotation. If you set $t=i\beta$, then $E\left[e^{(Z+iH)t}\right]=E\left[e^{-H\beta }\right]$ so the inverse-temperature is literally imaginary time! You can recover the Boltzmann distribution by looking at the expected number of particles in each state: $E[⟨n|e^{(Z+iH)t}|n⟩]=e^{-\beta E_n}$ where $E_n$ is the $n$th eigenvalue (energy in the $n$th state).

Tamay from Epoch AI here.

We made a mistake in not being more transparent about OpenAI's involvement. We were restricted from disclosing the partnership until around the time o3 launched, and in hindsight we should have negotiated harder for the ability to be transparent to the benchmark contributors as soon as possible. Our contract specifically prevented us from disclosing information about the funding source and the fact that OpenAI has data access. We own this error and are committed to doing better in the future.

For future collaborations, we will strive to improve transparency wherever possible, ensuring contributors have clearer information about funding sources, data access, and usage purposes at the outset. While we did communicate that we received lab funding to some mathematicians, we didn't do this systematically and did not name the lab we worked with. This inconsistent communication was a mistake. We should have pushed harder for the ability to be transparent about this partnership from the start, particularly with the mathematicians creating the problems.

Getting permission to disclose OpenAI's involvement only around the o3 launch wasn't good enough. Our mathematicians deserved to know who might have access to their work. Even though we were contractually limited in what we could say, we should have made transparency with our contributors a non-negotiable part of our agreement with OpenAI.

Regarding training usage: We acknowledge that OpenAI does have access to the FrontierMath problems and solutions. However, we have a verbal agreement that these materials will not be used in model training. 

Relevant OpenAI employees’ public communications have described FrontierMath as a 'strongly held out' evaluation set. While this public positioning aligns with our understanding, I would also emphasize more broadly that labs benefit greatly from having truly uncontaminated test sets.

OpenAI has also been fully supportive of our decision to maintain a separate, unseen holdout set—an extra safeguard to prevent overfitting and ensure accurate progress measurement. From day one, FrontierMath was conceived and presented as an evaluation tool, and we believe these arrangements reflect that purpose. 

Thank you for the great content! One question - could you please give an example of one of the "fnords" you're referencing, or explain it in slightly more detail? I think I understand what you mean but I've read the blueprint itself, and while the first few pages sound a bit grandstanding and use the word "America" way too much for my taste, I couldn't really see what you're talking about. If I didn't read your comments I'd write the tone off as "well, I guess that's how you're supposed to talk to politicians, what do I know".

Disclaimer: I'm not a native English speaker and have little experience with English (or American :) ) policy documents like this one. Perhaps an explanation would be helpful not just for me but also for other non-American readers.

Sorry I didn’t really understand the question. What do you mean specifically?

Language models learning to spell things out using the first letter of their generations could be one pathway towards subtle obfuscated reasoning (ie steganography). 

Eg here an LM spells out HELLO; it would be concerning if LMs could spell out other things like “Deployment” or “Not deployment”.