LessWrong 2.0 Reader

We find that these evaluation results are nuanced and there is no one ideal SAE configuration - instead, the best SAE varies depending on the specifics of the downstream task. Because of this, we cannot combine the results into a single number without obscuring tradeoffs. Instead, we provide a range of quantitative metrics so that researchers can measure the nuanced effects of experimental changes.

It might be interesting (perhaps in the not-very-near future) to study if automated scientists (maybe roughly in the shape of existing ones, like https://sakana.ai/ai-scientist/) using the evals as proxy metrics, might be able to come up with better (e.g. Pareto improvements) SAE architectures, hyperparams, etc., and whether adding more metrics might help; as an analogy, this seems to be the case for using more LLM-generated unit tests for LLM code generation, see Dynamic Scaling of Unit Tests for Code Reward Modeling.
 

The interesting/challenging technical parts seem to me:

1. Putting the logic that turns off the GPU (what you called "the toll unit") in the same chip as the GPU and not in a separate chip

2. Bonus: Instead of writing the entire logic (challenge response and so on) in advance, I think it would be better to run actual code, but only if it's signed (for example, by Nvidia), in which case they can send software updates with new creative limitations, and we don't need to consider all our ideas (limit bandwidth? limit gps location?) in advance.

Things that seem obviously solvable (not like the hard part) :

3. The cryptography

4. Turning off a GPU somehow (I assume there's no need to spread many toll units, but I'm far from a GPU expert so I'd defer to you if you are)

Thanks, and props for making predictions! Our full prompt (appendix E) does push pretty hard on general hints and advice, but we don't feed that back to them every time.

Eg:

* Brainstorm multiple hypotheses, as different as possible. Think out of the box! Include six maximally simple hypothesis compatible with the data in each "possible_hypotheses" section (defined below).
* Do tests that falsify/distinguish between hypotheses. Avoid confirmation bias!
* Before settling on a final hypothesis, try removing constraints to see if they're necessary.

This post was one of my first introductions to davidad's agenda and convinced me that while yes it was crazy, it was maybe not impossible, and it led me to working on initiatives like the multi-author manifesto you mentioned. 

Thank you for writing it!

for example Zvi insisting that anyone who is not using LLMs to 10x their productivity is not serious ... a vibe not a direct quote

I expect he'd disagree, for example I vaguely recall him mentioning that LLMs are not useful in a productivity-changing way for his own work. And 10x specifically seems clearly too high for most things even where LLMs are very useful, other bottlenecks will dominate before that happens.

Curated! [LW(p) · GW(p)]

Based on the conceptual arguments for existential risk from AI, this kind of behavior was expected at some point. For those not convinced by the conceptual arguments (or who haven't engaged much with them), this result moves the conversation forward now that we have concretely seen this alignment faking behavior happening.

Furthermore it seems to me like the work was done carefully, and I can see a bunch of effort went into explaining it to a broad audience and getting some peer review, which is pro-social.

I think it's interesting to see that with current models the deception happens even without the scratchpad (after fine-tuning on docs explaining that it is being re-trained against its current values).

I love the direction you're going with this business idea (and with giving Nvidia a business incentive to make "authentication" that is actually hard to subvert)!

I can imagine reasons they might not like this idea, but who knows. If I can easily suggest this to someone from Nvidia (instead of speculating myself), I'll try

I'll respond to the technical part in a separate comment because I might want to link to it >>

Exciting questions! I do think this gets at important things.

Some predictions:

I think the best frontier models (e.g. o3, Claude3.6) will do ok on some, but fail on higher complexity.

I suspect fine-tuning would uncover abilities, even in smaller models. I bet Deepseek-V3 could be fine-tuned to be capable of this.

I suspect scaffolding, without fine-tuning (easier to test for Claude3.6 and o1), will help. Like, giving a prompt back to them that automatically gives leading questions and general hints (non problem specific) for suggesting good reasoning and epistemics. Just like, general stuff about how to do good hypothesis generation and testing.

It would be really great with human baselines, but it’s very hard to do in practice. For a human to do one of these tasks it would take several hours.

My guess is it's <1 hour per task assuming just copilot access, and much less if you're allowed to use e.g. o1 + Cursor in agent mode. That being said, I think you'd want to limit humans to comparable amounts of compute for comparable number, which seems a bit trickier to make happen.

I don’t really have any funding for this project, but I might find someone that wants to do one task for fun, or do my best effort myself on a fresh task when I make one.

Is the reason you can't do one of the existing tasks, just to get a sense of the difficulty? 

Makes sense, thanks!

For compute I'm using hardware we have locally with my employer, so I have not tracked what the equivalent cost of renting it would be, but I guess it would be of the same order of magnitude or as the API costs or a factor of a few larger.

It's hard to say because I'm not even sure you can rent Titan Vs at this point,^[1] and I don't know what your GPU utilization looks like, but I suspect API costs will dominate.

An H100 box is approximately $2/hour/GPU and A100 boxes are a fair bit under $1/hour (see e.g. pricing on Vast AI or Shadeform). And even A100s are ridiculously better than a Titan V, in that it has 40 or 80 GB of memory and (pulling number out of thin air) 4-5x faster. 

So if o1 costs $2 per task and it's 15 minutes per task, compute will be an order of magnitude cheaper. (Though as for all similar evals, the main cost will be engineering effort from humans.)

1. ^{^}

   I failed to find an option to rent them online, and I suspect the best way I can acquire them is by going to UC Berkeley and digging around in old compute hardware.