LessWrong 2.0 Reader

share good access with third-party evaluators

Glad to see my personal hobbyhorse got a mention!

Overall this is great, and I have just one concern about a possible what-if case that isn't covered.

Model development secrets being disproportionately important.

I know, it's a really tricky thing to deal with if this becomes the case. But I think it's worth having a 'what if' plan in place in case this suddenly happens. Imagine a lab is working on their research and some researcher stumbles across an algorithmic innovation which makes a million-fold improvement in training efficiency. The peak capability level of the model that can now be trained for a few thousand dollars is now on par with their leading multi-billion dollar frontier model. Furthermore, this peak capability level is sufficient for the 3x speedup in AI R&D that you specified as being a threshold of high concern. Under such a circumstance, this secret becomes even more dangerous and valuable than the multi-billion-dollar-training-cost frontier model weights. Seems like having a plan in place for what if this happens would be wise.

Something like: don't tell your co-workers, report to so-and-so specific person whose job it is to handle evaluating and reporting up-the-chain about possible dangerous algorithmic developments. Probably you'd need a witness protection / isolation setup for the researchers who'd been exposed to the secret. You'd need to start planning around how soon others might stumble onto the same discovery, what other similar discoveries might be out there to be found, what government actions should be taken, etc.

I know this sounds like an implausible scenario to most people currently, but I think it's not something ruled out as physically impossible, and is worth having a what-if plan for. 

Not sure of the title. The tagline was "almost no one is evil; almost everything is broken." The address was http://blog.jaibot.com. Some specific essays originating there were "500 million, but not a single one more," "Foes Without Faces", and "The Copenhagen Interpretation of Ethics".

For the purposes of the original question of whether people are overinvesting in interp due to it being useful for capabilities and therefore being incentivized, I think there's a pretty important distinction between direct usefulness and this sort of diffuse public good that is very hard to attribute. Things with large but diffuse impact are much more often underincentivized and often mostly done as a labor of love. In general, the more you think an organization is shaped by incentives that are hard to fight against, the more you should expect diffusely impactful things to be relatively neglected.

Separately, it's also not clear to me that the diffuse intuitions from interpretability have actually helped people a lot with capabilities. Obviously this is very hard to attribute, and I can't speak too much about details, but it feels to me like the most important intuitions come from elsewhere. What's an example of an interpretability work that you feel has affected capabilities intuitions a lot?

Thanks Naci, that's helpful clarification. An active call to change the odds, rather than a passive hoping that things will go well, does seem like a more robust plan.

SAE steering doesn't seem like it obviously beats other steering techniques in terms of usefulness. I haven't looked closely into Hyena but my prior is that subquadratic attention papers probably suck unless proven otherwise.

Interpretability is certainly vastly more appealing to lab leadership than weird philosophy, but it's vastly less appealing than RLHF. But there are many many ML flavored directions and only a few of them are any good, so it's not surprising that most directions don't get a lot of attention.

Probably as interp gets better it will start to be helpful for capabilities. I'm uncertain whether it will be more or less useful for capabilities than just working on capabilities directly; on the one hand, mechanistic understanding has historically underperformed as a research strategy, on the other hand it could be that this will change once we have a sufficiently good mechanistic understanding.

Tyler Cowen often has really good takes (even some good stuff against AI as an x-risk!), but this was not one of them: https://marginalrevolution.com/marginalrevolution/2024/10/a-funny-feature-of-the-ai-doomster-argument.html

Title: A funny feature of the AI doomster argument

If you ask them whether they are short the market, many will say there is no way to short the apocalypse.  But of course you can benefit from pending signs of deterioration in advance.  At the very least, you can short some markets, or go long volatility, and then send those profits to Somalia to mitigate suffering for a few years before the whole world ends.

Still, in a recent informal debate at the wonderful Roots of Progress conference in Berkeley, many of the doomsters insisted to me that “the end” will come as a complete surprise, given the (supposed) deceptive abilities of AGI.

But note what they are saying.  If markets will not fall at least partially in advance, they are saying the passage of time, and the events along the way, will not persuade anyone.  They are saying that further contemplation of their arguments will not persuade any marginal investors, whether directly or indirectly.  They are predicting that their own ideas will not spread any further.

I take those as signs of a pretty weak argument.  “It will never get more persuasive than it is right now!”  “There’s only so much evidence for my argument, and never any more!”  Of course, by now most intelligent North Americans with an interest in these issues have heard these arguments and they are most decidedly not persuaded.

There is also a funny epistemic angle here.  If the next say twenty years of evidence and argumentation are not going to persuade anyone else at the margin, why should you be holding this view right now?  What is it that you know, that is so resistant to spread and persuasion over the course of the next twenty years?

I would say that to ask such questions is to answer them.

Hi Evan, I published this paper on arxiv recently and it also got accepted at the SafeGenAI workshop at Neurips in December this year. Thanks for adding the link, I will probably work on the paper again and put an updated version on arxiv as I am not quite happy with the current version.

I think that using the base model without instruction fine-tuning would prove bothersome for multiple reasons:

1. In the paper I use the new 3.1 models which are fine-tuned for tool using, these base models where never fine-tuned to use tools through function calling.

2. Base models are highly random and hard to control, they are not really steerable. They require very careful prompting/conditioning to do anything useful.

3. I think current post-training basically improves all benchmarks

I am also working on using such agents and directly evaluating how good they are on humans at spear phishing: https://openreview.net/forum?id=VRD8Km1I4x

I basically agree with everything @Lukas Finnveden [LW · GW] and @ryan_greenblatt [LW · GW] said above.  Another relevant "failure" state is that we might only be able to come up with expensive or capability-reducing mitigations, meaning everyone will have an incentive to disable or work around them.

Yeah, I meant medical/covid masks imply the wearer is diseased. I would have also believed the cat mask is a medical/covid mask if you hadn't give a different reason for wearing it, so it has that going against it in terms of coolness. It also has a lack of plausible deniability going against it too. If you're wearing sunglasses there's actually a utilitarian reason behind wearing them outside of just creating information asymmetry. If you're just trying to obscure half your face, there's no such plausible deniability. You're just trying to obscure your face, so it becomes far less cool.

My impression is that's a little simplistic, but I also don't have the best knowledge of the market outside WGS/WES and related tools. That particular market is a bloodbath. Maybe there's better scope in proteomics/metabolomics/stuff I know nothing about.