LessWrong 2.0 Reader

Multiple entries are very welcome!

[With some kind of anti-munchkin caveat. Submitting your analyses of several different disjoint questions seems great; submitting two versions of largely the same basic content in different styles not so great. I'm not sure exactly how we'd handle it if someone did the latter, but we'd aim for something sensible that didn't incentivise people to have been silly about it.]

I'm white.

This is great, bookmarked for future warm and fuzzies. I've just had my second, a son, on February 8th. My first, a daughter, is six next week.

Let it be known to all and sundry that kids are fantastic and fatherhood is wondrous. It is much work and a high cost in money and sleep, in exchange for which you are endowed with glorious purpose and wireheaded to the future.

Also there is the love. Strongly recommended.

I guess I'm more willing to treat Anthropic's marketing as not-representing-Anthropic.

Like, when OpenAI marketing says GPT-4 is our most aligned model yet! you could say this shows that OpenAI deeply misunderstands what we mean by alignment but I tend to ignore it. Even mostly when Sam Altman says it himself.

I'm excited to see how Paul performs in the new role. He's obviously very qualified on a technical level, and I suspect he's one of the best people for the job of designing and conducting evals.

I'm more uncertain about the kind of influence he'll have on various AI policy and AI national security discussions. And I mean uncertain in the genuine "this could go so many different ways" kind of way. 

Like, it wouldn't be particularly surprising to me if any of the following occurred:

• Paul focuses nearly all of his efforts on technical evals and doesn't get very involved in broader policy conversations
• Paul is regularly asked to contribute to broader policy discussions, and he advocates for RSPs and other forms of voluntary commitments.
• Paul is regularly asked to contribute to broader policy discussions, and he advocates for requirements that go beyond voluntary commitments and are much more ambitious than what he advocated for when he was at ARC.
• Paul is regularly asked to contribute to broader policy discussions, and he's not very good at communicating his beliefs in ways that are clear/concise/policymaker-friendly, so his influence on policy discussions is rather limited.
• Paul [is/isn't] able to work well with others who have very different worldviews and priorities.

Personally, I see this as a very exciting opportunity for Paul to form an identity as a leader in AI policy. I'm guessing the technical work will be his priority (and indeed, it's what he's being explicitly hired to do), but I hope he also finds ways to just generally improve the US government's understanding of AI risk and the likelihood of implementing reasonable policies. On the flipside, I hope he doesn't settle for voluntary commitments (especially as the Overton Window shifts) & I hope he's clear/open about the limitations of RSPs.

More specifically, I hope he's able to help policymakers reason about a critical question: what do we do after we've identified models with (certain kinds of) dangerous capabilities? I think the underlying logic behind RSPs could actually be somewhat meaningfully applied to USG policy. Like, I think we would be in a safer world if the USG had an internal understanding of ASL levels, took seriously the possibility of various dangerous capabilities thresholds being crossed, took seriously the idea that AGI/ASI could be developed soon, and had preparedness plans in place that allowed them to react quickly in the event of a sudden risk. 

Anyways, a big congratulations to Paul, and definitely some evidence that the USAISI is capable of hiring some technical powerhouses. 

As you know from our conversations, I'm largely in the same camp as you on this point.  

But one point I'd make incrementally is this: USG folks are also concerned about warning shots of the nature, "The President's Daily Brief ran an article 6 months ago saying warning signs for dangerous thing X would be events W, Y, and Z, and today the PDB had an article saying our intelligence agencies assess that Y and Z have happened due to super secret stuff".

If rationalists want rationalist warning shots to be included, they need to convince relevant government analytic stakeholders of their relevance. 

That was when I took a week off work to do side projects, glad it at least shows up in this graph 😌

Thanks for the tip... deep down a part of me knows there are ways to get around Freedom, but they're non-obvious enough that I haven't cheated yet

I'll admit I have only been loosely following the control stuff, but FWIW I would be excited about a potential @peterbarnett [LW · GW] & @ryan_greenblatt [LW · GW] dialogue in which you two to try to identify & analyze any potential disagreements. Example questions:

• What is the most capable system that you think we are likely to be able to control?
• What kind of value do you think we could get out of such a system?
• To what extent do you expect that system to be able to produce insights that help us escape the acute risk period (i.e., get out of a scenario where someone else can come along and build a catastrophe-capable system without implementing control procedures or someone else comes along and scales to the point where the control procedures are no longer sufficient)

Whoohoo!