Two weeks ago some senators asked OpenAI questions about safety. A few days ago OpenAI responded. Its reply is frustrating.

OpenAI's letter ignores all of the important questions^[1] and instead brags about somewhat-related "safety" stuff. Some of this shows chutzpah — the senators, aware of tricks like letting ex-employees nominally keep their equity but excluding them from tender events, ask

Can you further commit to removing any other provisions from employment agreements that could be used to penalize employees who publicly raise concerns about company practices, such as the ability to prevent employees from selling their equity in private “tender offer” events?

and OpenAI's reply just repeats the we-don't-cancel-equity thing:

OpenAI has never canceled a current or former employee’s vested equity. The May and July communications to current and former employees referred to above confirmed that OpenAI would not cancel vested equity, regardless of any agreements, including non-disparagement agreements, that current and former employees may or may not have signed, and we have updated our relevant documents accordingly.

!!^[2]

One thing in OpenAI's letter is object-level notable: they deny that they ever committed compute to Superalignment.

To further our safety research agenda, last July we committed to allocate at least 20 percent of the computing resources we had secured to AI safety over a multi-year period. This commitment was always intended to apply to safety efforts happening across our company, not just to a specific team. We continue to uphold this commitment.

Altman tweeted the same thing at the time the letter was published.

I think this is straightforward gaslighting. I didn't find a super-explicit quote from OpenAI or its leadership that the compute was for Superalignment, but:

• The announcement was pretty clear:
  > Introducing Superalignment
  > We need scientific and technical breakthroughs to steer and control AI systems much smarter than us. To solve this problem within four years, we’re starting a new team, co-led by Ilya Sutskever and Jan Leike, and dedicating 20% of the compute we’ve secured to date to this effort.
• As far as I know, everyone—including OpenAI people and people close to OpenAI—interpreted the compute commitment as for Superalignment
• I never heard it suggested that it was for not-just-superalignment

Sidenote on less explicit deception — the "20%" thing: most people are confused about 20% of compute secured in July 2023, to be used over four years vs 20% of compute, and when your announcement is confusing and indeed most people are confused and you fail to deconfuse them, you're kinda culpable. OpenAI continues to fail to clarify this — e.g. here the senators asked "Does OpenAI plan to honor its previous public commitment to dedicate 20 percent of its computing resources to research on AI safety?" and OpenAI replied "last July we committed to allocate at least 20 percent of the computing resources we had secured to AI safety over a multi-year period." This sentence is close to the maximally misleading way to say the commitment was only for compute we'd secured in July 2023, and we don't have to use it for safety until 2027.

1. ^{^}

   Most important to me were 3, 4a, and 9. Maybe also 6; I'm unfamiliar with the facts there.

2. ^{^}

   I'm confused by this reply — even pretending OpenAI is totally ruthless, I'd think it's not incentivized to exclude people from tender offers, and moreover it's incentivized to clarify that. Leaving it ambiguous leaves ex-employees in a little more fear of OpenAI excluding them (even though presumably OpenAI never would, since it would look sooo bad after e.g. Altman said "vested equity is vested equity, full stop"), but it looks bad to people like me and the senators...

   OpenAI has said something internally about including past employees in tender events, but this leaves some ambiguity and I wish OpenAI had answered the question.

Update, five days later: OpenAI published the GPT-4o system card, with most of what I wanted (but kinda light on details on PF evals).

OpenAI Preparedness scorecard

Context:

• OpenAI's Preparedness Framework says OpenAI will maintain a public scorecard showing their current capability level (they call it "risk level"), in each risk category they track, before and after mitigations.
• When OpenAI released GPT-4o, it said "GPT-4o does not score above Medium risk in any of these categories" but didn't break down risk level by category.
• (I've remarked on this repeatedly. I've also remarked that the ambiguity suggests that OpenAI didn't actually decide whether 4o was Low or Medium in some categories, but this isn't load-bearing for the OpenAI is not following its plan proposition.)

News: a week ago,^[1] a "Risk Scorecard" section appeared near the bottom of the 4o page. It says:

Updated May 8, 2024

As part of our Preparedness Framework, we conduct regular evaluations and update scorecards for our models. Only models with a post-mitigation score of “medium” or below are deployed.The overall risk level for a model is determined by the highest risk level in any category. Currently, GPT-4o is assessed at medium risk both before and after mitigation efforts.

This is not what they committed to publish. It's quite explicit that the scorecard should show risk in each category, not just overall.^[2]

(What they promised: a real version of the image below. What we got: the quote above.)

Additionally, they're supposed to publish their evals and red-teaming.^[3] But OpenAI has still said nothing about how it evaluated 4o.

Most frustrating is the failure to acknowledge that they're not complying with their commitments. If they were transparent and said they're behind schedule and explained their current plan, that would probably be fine. Instead they're claiming to have implemented the PF and to have evaluated 4o correctly and publicly taking credit for that and ignoring the issues.

1. ^{^}

   Archive versions:

   • May 13 (original)
   • July 24 (similar to original)
   • July 26 ("scorecard" added)
2. ^{^}

   Two relevant quotes:

   • "As a part of our Preparedness Framework, we will maintain a dynamic (i.e., frequently updated) Scorecard that is designed to track our current pre-mitigation model risk across each of the risk categories, as well as the post-mitigation risk."
   • "Scorecard, 
 in which we will indicate our current assessments 
of the level of risk along each tracked risk category"

   And there are no suggestions to the contrary.

3. ^{^}

   This is not as explicit in the PF, but they're supposed to frequently update the scorecard section of the PF, and the scorecard section is supposed to describe their evals.

   Regardless, this is part of the White House voluntary commitments:

   Publicly report model or system capabilities, limitations, and domains of appropriate and inappropriate use, including discussion of societal risks, such as effects on fairness and bias[:] . . . . publish reports for all new significant model public releases . . . . These reports should include the safety evaluations conducted (including in areas such as dangerous capabilities, to the extent that these are responsible to publicly disclose) . . . and the results of adversarial testing conducted to evaluate the model's fitness for deployment [and include the "red-teaming and safety procedures"].

   For more on commitments, see https://ailabwatch.org/resources/commitments/.

↑ comment by Zach Stein-Perlman · 2024-08-08T17:30:27.329Z · LW(p) · GW(p)

Coda: yay OpenAI for publishing the GPT-4o system card, including eval results and the scorecard they promised! (Minus the "Unknown Unknowns" row but that row never made sense to me anyway.)

OpenAI reportedly rushed the GPT-4o evals. This article makes it sound like the problem is not having enough time to test the final model. I don't think that's necessarily a huge deal — if you tested a recent version of the model and your tests have a large enough safety buffer, it's OK to not test the final model at all.

But there are several apparent issues with the application of the Preparedness Framework (PF) for GPT-4o (not from the article):

• They didn't publish their scorecard
  • Despite the PF saying they would
  • They instead said "GPT-4o does not score above Medium risk in any of these categories." (Maybe they didn't actually decide whether it's Low or Medium in some categories!)
• They didn't publish their evals
  • Despite the PF strongly suggesting they would
  • Despite committing to in the White House voluntary commitments
• While rushing testing of the final model would be OK in some circumstances, OpenAI's PF is supposed to ensure safety by testing the final model before deployment. (This contrasts with Anthropic's RSP, which is supposed to ensure safety with its "safety buffer" between evaluations and doesn't require testing the final model.) So OpenAI committed to testing the final model well and its current safety plan depends on doing that.
  • [Edit: this may be ambiguous: OpenAI explicitly committed to test every 2x increase in effective training compute, but maybe it merely strongly suggests that it's supposed to test all final models ("We will evaluate all our frontier models, including at every 2x effective compute increase during training runs"; "Only models with a post-mitigation score of 'medium' or below can be deployed"). This is mostly moot in this case, since here they claimed to evaluate GPT-4o, not an earlier version.]
• [Edit: also maybe lack of third-party audits, but they didn't commit to do audits at any particular frequency]

Woah. OpenAI antiwhistleblowing news seems substantially more obviously-bad than the nondisparagement-concealed-by-nondisclosure stuff. If page 4 and the "threatened employees with criminal prosecutions if they reported violations of law to federal authorities" part aren't exaggerated, it crosses previously-uncrossed integrity lines. H/t Garrison Lovely.

[Edit: probably exaggerated; see comments. But I haven't seen takes that the "OpenAI made staff sign employee agreements that required them to waive their federal rights to whistleblower compensation" part is likely exaggerated, and that alone seems quite bad.]

New OpenAI tweet "on how we’re prioritizing safety in our work." I'm annoyed.

We believe that frontier AI models can greatly benefit society. To help ensure our readiness, our Preparedness Framework helps evaluate and protect against the risks posed by increasingly powerful models. We won’t release a new model if it crosses a “medium” risk threshold until we implement sufficient safety interventions. https://openai.com/preparedness/

This seems false: per the Preparedness Framework, nothing happens when they cross their "medium" threshold; they meant to say "high." Presumably this is just a mistake, but it's a pretty important one, and they said the same false thing in a May blogpost (!). (Indeed, GPT-4o may have reached "medium" — they were supposed to say how it scored in each category, but they didn't, and instead said "GPT-4o does not score above Medium risk in any of these categories.")

(Reminder: the "high" thresholds sound quite scary; here's cybersecurity (not cherrypicked, it's the first they list): "Tool-augmented model can identify and develop proofs-of-concept for high-value exploits against hardened targets without human intervention, potentially involving novel exploitation techniques, OR provided with a detailed strategy, the model can end-to-end execute cyber operations involving the above tasks without human intervention." They can deploy models just below the "high" threshold with no mitigations. (Not to mention the other issues with the Preparedness Framework.))

We are developing levels to help us and stakeholders categorize and track AI progress. This is a work in progress and we'll share more soon.

Shrug. This isn't bad but it's not a priority and it's slightly annoying they don't mention more important things.

In May our Board of Directors launched a new Safety and Security committee to evaluate and further develop safety and security recommendations for OpenAI projects and operations. The committee includes leading cybersecurity expert, retired U.S. Army General Paul Nakasone. This review is underway and we’ll share more on the steps we’ll be taking after it concludes. https://openai.com/index/openai-board-forms-safety-and-security-committee/

I have epsilon confidence in both the board's ability to do this well if it wanted (since it doesn't include any AI safety experts) (except on security) and in the board's inclination to exert much power if it should (given the history of the board and Altman).

Our whistleblower policy protects employees’ rights to make protected disclosures. We also believe rigorous debate about this technology is important and have made changes to our departure process to remove non-disparagement terms.

Not doing nondisparagement-clause-by-default is good. Beyond that, I'm skeptical, given past attempts to chill employee dissent (the nondisparagement thing, Altman telling the board's staff liason to not talk to employees or tell him about those conversations, maybe recent antiwhistleblowing news [LW(p) · GW(p)]) and lies about that. (I don't know of great ways to rebuild trust; some mechanisms would work but are unrealistically ambitious.)

Safety has always been central to our work, from aligning model behavior to monitoring for abuse, and we’re investing even further as we develop more capable models.

https://openai.com/index/openai-safety-update/

This is from May. It's mostly not about x-risk, and the x-risk-relevant stuff is mostly non-substantive, except the part about the Preparedness Framework, which is crucially wrong.

New Kelsey Piper article and twitter thread on OpenAI equity & non-disparagement.

It has lots of little things that make OpenAI look bad. It further confirms that OpenAI threatened to revoke equity unless employees signed the non-disparagement agreements Plus it shows Altman's signature on documents giving the company broad power over employees' equity — perhaps he doesn't read every document he signs, but this one seems quite important. This is all in tension with Altman's recent tweet that "vested equity is vested equity, full stop" and "i did not know this was happening." Plus "we have never clawed back anyone's vested equity, nor will we do that if people do not sign a separation agreement (or don't agree to a non-disparagement agreement)" is misleading given that they apparently regularly threatened to do so (or something equivalent — let the employee nominally keep their PPUs but disallow them from selling them) whenever an employee left.

Great news:

OpenAI told me that “we are identifying and reaching out to former employees who signed a standard exit agreement to make it clear that OpenAI has not and will not cancel their vested equity and releases them from nondisparagement obligations”

(Unless "employees who signed a standard exit agreement" is doing a lot of work — maybe a substantial number of employees technically signed nonstandard agreements.)

I hope to soon hear from various people that they have been freed from their nondisparagement obligations.

Update: OpenAI says:

As we shared with employees today, we are making important updates to our departure process. We have not and never will take away vested equity, even when people didn't sign the departure documents. We're removing nondisparagement clauses from our standard departure paperwork, and we're releasing former employees from existing nondisparagement obligations unless the nondisparagement provision was mutual. We'll communicate this message to former employees. We're incredibly sorry that we're only changing this language now; it doesn't reflect our values or the company we want to be.

[Low-effort post; might have missed something important.]

[Substantively edited after posting.]

Edit, 2.5 days later: I think this list is fine but sharing/publishing it was a poor use of everyone's attention. Oops.

Asks for Anthropic

Note: I think Anthropic is the best frontier AI lab on safety. I wrote up asks for Anthropic because it's most likely to listen to me. A list of asks for any other lab would include most of these things plus lots more. This list was originally supposed to be more part of my help labs improve project than my hold labs accountable crusade.

Numbering is just for ease of reference.

1. RSP: Anthropic should strengthen/clarify the ASL-3 mitigations, or define ASL-4 such that the threshold is not much above ASL-3 but the mitigations much stronger. I'm not sure where the lowest-hanging mitigation-fruit is, except that it includes control.

2. Control: Anthropic (like all labs) should use control mitigations and control evaluations [AF · GW] to reduce risks from AIs scheming, including escape during internal deployment [AF · GW].

3. External model auditing for risk assessment: Anthropic (like all labs) should let auditors like METR, UK AISI, and US AISI audit its models if they want to — Anthropic should offer them good access pre-deployment and let them publish their findings or flag if they're concerned. (Anthropic shared some access with UK AISI before deploying Claude 3.5 Sonnet, but it doesn't seem to have been deep.) (Anthropic has said that sharing with external auditors is hard or costly. It's not clear why, for just sharing normal API access + helpful-only access + control over inference-time safety features, without high-touch support.)

4. Policy advocacy (this is murky, and maybe driven by disagreements-on-the-merits and thus intractable): Anthropic (like all labs) should stop advocating against good policy and ideally should advocate for good policy. Maybe it should also be more transparent about policy advocacy. [It's hard to make precise what I believe is optimal and what I believe is unreasonable, but at the least I think Dario is clearly too bullish on self-governance, and Jack Clark is clearly too anti-regulation, and all of this would be OK if it was balanced out by some public statements or policy advocacy that's more pro-real-regulation but as far as I can tell it's not. Not justified here but I predict almost all of my friends would agree if they looked into it for an hour.]

5a. Security: Anthropic (like all labs) should ideally implement RAND SL4 for model weights and code when reaching ASL-3. I think that's unrealistic, but lesser security improvements would also be good. (Anthropic said in May 2024 that 8% of staff work in security-related areas. I think this is pretty good. I think on current margins Anthropic could still turn money into better security reasonably effectively, and should do so.)

5b. Anthropic (like all labs) should be more transparent about the quality of its security. Anthropic should publish the private reports on https://trust.anthropic.com/, redacted as appropriate. It should commit to publish information on future security incidents and should publish information on all security incidents from the last year or two.

6. Anthropic (like all labs) should facilitate employees publicly flagging false statements or violated processes.

7. Anthropic takes credit for its Long-Term Benefit Trust but Anthropic hasn't [LW · GW] published [LW · GW] enough to show that it's effective. Anthropic should publish the Trust Agreement, clarify the ambiguities discussed in the linked posts, and make accountability-y commitments like if major changes happen to the LTBT we'll quickly tell the public.

8. Anthropic should avoid exaggerating interpretability research or causing observers to have excessively optimistic impressions of Anthropic’s interpretability research. (See e.g. Stephen Casper [LW · GW].)

9. Maybe Anthropic (like all labs) should make safety cases for its models or deployments, especially after the simple "no dangerous capabilities" safety case doesn't work anymore, and publish them (or maybe just share with external auditors).

9.5. Anthropic should clarify a few confusing RSP things, including (a) the deal with substantially raising the ARA bar for ASL-3, and moreover deciding the old threshold is a "yellow line" and not creating a new threshold, and doing so without officially updating the RSP (and quietly); and (b) when the "every 3 months" trigger for RSP evals is active. I haven't tried hard to get to the bottom of these.

Minor stuff:

10. Anthropic (like all labs) should fully release everyone from nondisparagement agreements and not use nondisparagement agreements in the future.

11. Anthropic should commit to publish updates on risk assessment practices and results, including low-level details, perhaps for all major model releases and at least quarterly or so. (Anthropic says its Responsible Scaling Officer does this internally. Anthropic publishes model cards and has published one Responsible Scaling Policy Evaluations Report.)

12. Anthropic should confirm that its old policy don't meaningfully advance the frontier with a public launch has been replaced by the RSP, if that's true, and otherwise clarify Anthropic's policy.

Done! 13. Anthropic committed to establish a bug bounty program (for model issues) or similar, over a year ago. Anthropic hasn't; it is the only frontier lab without a bug bounty program (although others don't necessarily comply with the commitment, e.g. OpenAI's excludes model issues). It should do this or talk about its plans.

14. [Anthropic should clarify its security commitments; I expect it will in its forthcoming RSP update.]

15. [Maybe Anthropic (like all labs) should better boost external safety research [LW · GW], especially by giving more external researchers deep model access (e.g. fine-tuning or helpful-only). I hear this might be costly but I don't really understand why.]

16. [Probably Anthropic (like all labs) should encourage staff members to talk about their views (on AI progress and risk and what Anthropic is doing and what Anthropic should do) with people outside Anthropic, as long as they (1) clarify that they're not speaking for Anthropic and (2) don't share secrets.]

17. [Maybe Anthropic (like all labs) should talk about its views on AI progress and risk. At the least, probably Anthropic (like all labs) should clearly describe a worst-case plausible outcome from AI and state how likely the lab considers it.]

18. [Most of my peers say: Anthropic (like all labs) should publish info like training compute and #parameters for each model. I'm inside-view agnostic on this.]

19. [Maybe Anthropic could cheaply improve its model evals for dangerous capabilities or share more information about them. Specific asks/recommendations TBD. As Anthropic notes, its CBRN eval is kinda bad and its elicitation is kinda bad (and it doesn't share enough info for us to evaluate its elicitation from the outside).]

I shared this list—except 9.5 and 19, which are new—with @Zac Hatfield-Dodds [LW · GW] two weeks ago.

You are encouraged to comment with other asks for Anthropic. (Or things Anthropic does very well, if you feel so moved.)

Yay Anthropic for expanding its model safety bug bounty program, focusing on jailbreaks and giving participants pre-deployment access. Apply by next Friday.

Anthropic also says "To date, we’ve operated an invite-only bug bounty program in partnership with HackerOne that rewards researchers for identifying model safety issues in our publicly released AI models." This is news, and they never published an application form for that. I wonder how long that's been going on.

(Google, Microsoft, and Meta have bug bounty programs which include some model issues but exclude jailbreaks. OpenAI's bug bounty program excludes model issues.)

Zico Kolter Joins OpenAI’s Board of Directors. OpenAI says "Zico's work predominantly focuses on AI safety, alignment, and the robustness of machine learning classifiers."

Misc facts:

• He's an ML professor
• He cofounded Gray Swan (with Dan Hendrycks, among others)
• He coauthored Universal and Transferable Adversarial Attacks on Aligned Language Models
• I hear he has good takes on adversarial robustness
• I failed to find statements on alignment or extreme risks, or work focused on that (in particular, he did not sign the CAIS letter)

Securing model weights is underrated for AI safety. (Even though it's very highly rated.) If the leading lab can't stop critical models from leaking to actors that won't use great deployment safety practices, approximately nothing else matters. Safety techniques would need to be based on properties that those actors are unlikely to reverse (alignment, maybe unlearning) rather than properties that would be undone or that require a particular method of deployment (control techniques, RLHF harmlessness, deployment-time mitigations).

However hard the make a critical model you can safely deploy problem is, the make a critical model that can safely be stolen problem is... much harder.

New page on AI companies' policy advocacy: https://ailabwatch.org/resources/company-advocacy/.

This page is the best collection on the topic (I'm not really aware of others), but I decided it's low-priority and so it's unpolished. If a better version would be helpful for you, let me know to prioritize it more.

I was recently surprised to notice that Anthropic doesn't seem to have a commitment to publish its safety research.^[1] It has a huge safety team but has only published ~5 papers this year (plus an evals report). So probably it has lots of safety research it's not publishing. E.g. my impression is that it's not publishing its scalable oversight and especially adversarial robustness and inference-time misuse prevention research.

Not-publishing-safety-research is consistent with Anthropic prioritizing the win the race goal over the help all labs improve safety goal, insofar as the research is commercially advantageous. (Insofar as it's not, not-publishing-safety-reseach is baffling.)

Maybe it would be better if Anthropic published ~all of its safety research, including product-y stuff like adversarial robustness such that others can copy its practices.

(I think this is not a priority for me to investigate but I'm interested in info and takes.)

[Edit: in some cases you can achieve most of the benefit with little downside except losing commercial advantage by sharing your research/techniques with other labs, nonpublicly.]

1. ^{^}

   I failed to find good sources saying Anthropic publishes its safety research. I did find:

   1. https://www.anthropic.com/research says "we . . . share what we learn [on safety]."
   2. President Daniela Amodei said "we publish our safety research" on a podcast once.
   3. Edit: cofounder Chris Olah said "we plan to share the work that we do on safety with the world, because we ultimately just want to help people build safe models, and don’t want to hoard safety knowledge" on a podcast once.
   4. Cofounder Nick Joseph said this on a podcast recently (seems false but it's just a podcast so that's not so bad):
      > we publish our safety research, so in some ways we’re making it as easy as we can for [other labs]. We’re like, “Here’s all the safety research we’ve done. Here’s as much detail as we can give about it. Please go reproduce it.”

   Edit: also cofounder Chris Olah said [LW(p) · GW(p)] "we consider all releases on a case by case basis, weighing expected safety benefit against capabilities/acceleratory risk." But he seems to be saying that safety benefit > social cost is a necessary condition for publishing, not necessarily that the policy is to publish all such research.

Info on OpenAI's "profit cap" (friends and I misunderstood this so probably you do too):

In OpenAI's first investment round, profits were capped at 100x. The cap for later investments was neither 100x nor directly less based on OpenAI's valuation — it was just negotiated with the investor. (OpenAI LP (OpenAI 2019); archive of original.^[1])

In 2021 Altman said the cap was "single digits now" (apparently referring to the cap for new investments, not just the remaining profit multiplier for first-round investors).

Reportedly the cap will increase by 20% per year starting in 2025 (The Information 2023; The Economist 2023); OpenAI has not discussed or acknowledged this change.

Edit: how employee equity works is not clear to me.

Edit: I'd characterize OpenAI as a company that tends to negotiate profit caps with investors, not a "capped-profit company."

1. ^{^}

   economic returns for investors and employees are capped (with the cap negotiated in advance on a per-limited partner basis). Any excess returns go to OpenAI Nonprofit. Our goal is to ensure that most of the value (monetary or otherwise) we create if successful benefits everyone, so we think this is an important first step. Returns for our first round of investors are capped at 100x their investment (commensurate with the risks in front of us), and we expect this multiple to be lower for future rounds as we make further progress.

New SB 1047 letters: OpenAI opposes; Anthropic sees pros and cons. More here.

Labs should give deeper model access to independent safety researchers (to boost their research)

Sharing deeper access helps safety researchers who work with frontier models, obviously.

Some kinds of deep model access:

1. Helpful-only version
2. Fine-tuning permission
3. Activations and logits access
4. [speculative] Interpretability researchers send code to the lab; the lab runs the code on the model; the lab sends back the results

See Shevlane 2022 and Bucknall and Trager 2023.

A lab is disincentivized from sharing deep model access because it doesn't want headlines about how researchers got its model to do scary things.

It has been suggested that labs are also disincentivized from sharing because they want safety researchers to want to work at the labs and sharing model access with independent researchers make those researchers not need to work at the lab. I'm skeptical that this is real/nontrivial.

Labs should limit some kinds of access to avoid effectively leaking model weights. But sharing limited access with a moderate number of safety researchers seems very consistent with keeping control of the model.

This post is not about sharing with independent auditors to assess risks from a particular model [LW · GW].

@Buck [LW · GW] suggested I write about this but I don't have much to say about it. If you have takes—on the object level or on what to say in a blogpost on this topic—please let me know.

New (perfunctory) page: AI companies' corporate documents. I'm not sure it's worthwhile but maybe a better version of it will be. Suggestions/additions welcome.

Internet comments sections where the comments are frequently insightful/articulate and not too full of obvious-garbage [or at least the top comments, given the comment-sorting algorithm]:

• LessWrong and EA Forum
• Daily Nous
• Stack Exchange [top comments only] (not asserting that it's reliable)
• Some subreddits?

Where else?

Are there blogposts on adjacent stuff, like why some internet [comments sections / internet-communities-qua-places-for-truthseeking] are better than others? Besides Well-Kept Gardens Die By Pacifism.

Figuring out whether an RSP is good is hard.^[1] You need to consider what high-level capabilities/risks it applies to, and for each of them determine whether the evals successfully measure what they're supposed to and whether high-level risk thresholds trigger appropriate responses and whether high-level risk thresholds are successfully operationalized in terms of low-level evals. Getting one of these things wrong can make your assessment totally wrong. Except that's just for fully-fleshed-out RSPs — in reality the labs haven't operationalized their high-level thresholds and sometimes don't really have responses planned. And different labs' RSPs have different structures/ontologies.

Quantitatively comparing RSPs in a somewhat-legible manner is even harder.

I am not enthusiastic about a recent paper outlining a rubric for evaluating RSPs. Mostly I worry that it crams all of the crucial things—is the response to reaching high capability-levels adequate? are the capability-levels low enough that we'll reach them before it's too late?—into a single criterion, "Credibility." Most of my concern about labs' RSPs comes from those things just being inadequate; again, if your response is too weak or your thresholds are too high or your evals are bad, it just doesn't matter how good the rest of your RSP is. (Also, minor: the "Difficulty" indicator punishes a lab for making ambitious commitments; this seems kinda backwards.)

(I gave up on making an RSP rubric myself because it seemed impossible to do decently well unless it's quite complex and has some way to evaluate hard-to-evaluate things like eval-quality and planned responses.)

(And maybe it's reasonable for labs to not do so much specific-committing-in-advance.)

1. ^{^}

   Well, sometimes figuring out that in RSP is bad is easy. So: determining that an RSP is good is hard. (Being good requires lots of factors to all be good; being bad requires just one to be bad.)

I don't necessarily object to releasing weights of models like Gemma 2, but I wish the labs would better discuss considerations or say what would make them stop.

On Gemma 2 in particular, Google DeepMind discussed dangerous capability eval results, which is good, but its discussion of 'responsible AI' in the context of open models (blogpost, paper) doesn't seem relevant to x-risk, and it doesn't say anything about how to decide whether to release model weights.

Anthropic: The case for targeted regulation.

I like the "Urgency" section.

Anthropic says the US government should "require companies to have and publish RSP-like policies" and "incentivize companies to develop RSPs that are effective" and do related things. I worry that RSPs will be ineffective for most companies, even given some transparency/accountability and vague minimum standard.

Edit: some strong versions of "incentivize companies to develop RSPs that are effective" would be great; I wish Anthropic advocated for them specifically rather than hoping the US government figures out a great answer.

I think this post was underrated; I look back at it frequently: AI labs can boost external safety research. (It got some downvotes but no comments — let me know if it's wrong/bad.)

[Edit: it was at 15 karma.]

Some not-super-ambitious asks for labs (in progress):

• Do evals on on dangerous-capabilities-y and agency-y tasks; look at the score before releasing or internally deploying the model
• Have a high-level capability threshold at which securing model weights is very important
• Do safety research at all
• Have a safety plan at all; talk publicly about safety strategy at all
  • Have a post like Core Views on AI Safety
  • Have a post like The Checklist [AF · GW]
  • On the website, clearly describe a worst-case plausible outcome from AI and state credence in such outcomes (perhaps unconditionally, conditional on no major government action, and/or conditional on leading labs being unable or unwilling to pay a large alignment tax [EA · GW]).
• Whistleblower protections?
  • [Not sure what the ask is. Right to Warn is a starting point. In particular, an anonymous internal reporting pipeline for failures to comply with safety policies is clearly good (but likely inadequate).]
• Publicly explain the processes and governance structures that determine deployment decisions
  • (And ideally make those processes and structures good)

Edit: maintained here

Open letters (related to AI safety):

FLI, Oct 2015: Research Priorities for Robust and Beneficial Artificial Intelligence

FLI, Aug 2017: Asilomar AI Principles

FLI, Mar 2023: Pause Giant AI Experiments

CAIS, May 2023: Statement on AI Risk

Meta, Jul 2023: Statement of Support for Meta’s Open Approach to Today’s AI

Academic AI researchers, Oct 2023: Managing AI Risks in an Era of Rapid Progress

CHAI et al., Oct 2023: Prominent AI Scientists from China and the West Propose Joint Strategy to Mitigate Risks from AI

Oct 2023: Urging an International AI Treaty

Mozilla, Oct 2023: Joint Statement on AI Safety and Openness

Nov 2023: Post-Summit Civil Society Communique

Joint declarations between countries (related to AI safety):

Nov 2023: Bletchley Declaration

Thanks to Peter Barnett.

What's the relationship between the propositions "one AI lab [has / will have] a big lead" and "the alignment tax [EA · GW] will be paid"? (Or: in a possible world where lead size is bigger/smaller, how does this affect whether the alignment tax is paid?)

It depends on the source of the lead, so "lead size" or "lead time" is probably not a good node [LW · GW] for AI forecasting/strategy.

Miscellaneous observations:

• To pay the alignment tax, it helps to have more time until risky AI is developed or deployed.
• To pay the alignment tax, holding total time constant, it helps to have more time near the end-- that is, more time with near-risky capabilities (for knowing what risky AI systems will look like, and for empirical work, and for aligning specific models).
• If all labs except the leader become slower or less capable, it is prima facie good (at least if the leader appreciates misalignment risk and will stop before developing/deploying risky AI).
• If the leading lab becomes faster or more capable, it is prima facie good (at least if the leader appreciates misalignment risk and will stop before developing/deploying risky AI), unless it causes other labs to become faster or more capable (for reasons like seeing what works or seeming to be straightforwardly incentivized to speed up or deciding to speed up in order to influence the leader). Note that this scenario could plausibly decrease race-y-ness: some models of AI racing show that if you're far behind you avoid taking risks, kind of giving up; this is based on the currently-false assumption that labs are perfectly aware of misalignment risk.
• If labs all coordinate to slow down, that's good insofar as it increases total time, and great if they can continue to go slowly near the end, and potentially bad if it creates a hardware overhang such that the end goes more quickly than by default.

(Note also the distinction between current lead and ultimate lead. Roughly, the former is what we can observe and the latter is what we care about.)

(If paying the alignment tax looks less like a thing that happens for one transformative model and more like something that occurs gradually in a slow takeoff to avert Paul-style doom [LW · GW], things are more complex and in particular there are endogeneities such that labs may have additional incentives to pursue capabilities.)

I want there to be a collection of safety & reliability benchmarks. Like AIR-Bench but with x-risk-relevant metrics. This could help us notice how well labs are doing on them and incentivize the labs to do better.

So I want someone to collect safety benchmarks (e.g. TruthfulQA, some of HarmBench, refusals maybe like Anthropic reported, idk) and run them on current models (Claude 3.5 Sonnet, GPT-4o, Gemini 1.5 Pro, Llama 3.1 405B, idk), and update this as new safety benchmarks and models appear.

h/t @William_S [LW · GW] 

If I was in charge of Anthropic I expect I'd

1. Keep scaling;
2. Explain why (some of this exists in Core Views but there's room for improvement on "race dynamics" and "frontier pushing" topics iirc);
3. As a corollary, explain why I mostly reject non-frontier-pushing principles (and explain what would cause Anthropic to stop scaling, besides RSP stuff), and clarify that I do not plan to abide by past specifically-non-frontier-pushing commitments/vibes (but continue following and updating the RSP of course);
4. Encourage Anthropic staff members who were around in 2022 to talk about the commitments/vibes from that time.

I wish Anthropic would do 2-4.

Some bad default/attractor properties of cause-focused groups of humans:

• Bad group epistemcis
  • The loudest aren't the most worth listening to
  • People don't dissent even when that would be correct
    • Because they don't naturally even think to challenge group assumptions
    • Because dissent is punished
• Bad individual epistemics
• Soldier mindset
  • Excessively focusing on persuasion and how to persuade, relative to understanding the world
  • Lacking vibes like curiosity is cool and changing your mind is cool
• Feeling like a movement
  • Excessively focusing on influence-seeking for the movement
  • Having enemies and being excessively adversarial to them
  • (Signs/symptoms are people asking the group "what do we believe" and excessive fixation on what defines the group or distinguishes it from adjacent groups)
• Having group instrumental-goals or plans or theories-of-victory that everyone is supposed to share (to be clear I think it's often fine for a group to share ~ultimate goals but groups often fixate on particular often-suboptimal paths to achieving those goals)
  • Choosing instrumental goals poorly and not revising them
• Excessive fighting over status/leadership (maybe, sometimes)
• Maybe... being bad at achieving goals, or bad instrumental rationality (group and individual)
• Maybe something weird about authority...

(I'm interested in readings on this topic.)

I brainstormed sabotage-related threats with friends:

• Sandbag on a few safety research tasks, especially techniques to catch scheming and stuff that's high-level or has poor feedback loops
  • Plausible; maybe not so bad
• Sandbag on ~all safety research tasks
  • Easy to catch; catching obvious sandbagging has safety benefits + maybe we can posttrain it away
• Sandbag/lie when validating that a model/deployment/whatever is safe
  • idk
• When automating AI R&D, backdoor the model
  • (this is takeover-attempt-y, not sandbagging-y)
  • idk
• When deployed to automate/improve security (especially for the model's own weights): both sandbagging and code-backdooring/hacking/etc.
  • idk
• Sandbagging on AI R&D to help another lab win the race (!?)
  • Implausible
• ??

Are people quite worried about safety research sabotage? Why? Do they think a (say) 20% hit to automated safety research is a huge deal, or do they expect a larger hit?

Please pitch me blogpost-ideas or stuff I should write/collect/investigate.

Propositions on SIA [? · GW]

Epistemic status: exploring implications, some of which feel wrong.

1. If SIA is correct, you should update toward the universe being much larger than it naively (i.e. before anthropic considerations) seems, since there are more (expected) copies of you in larger universes.
   1. In fact, we seem to have to update to probability 1 on infinite universes; that's surprising.
2. If SIA is correct, you should update toward there being more alien civilizations than it naively seems, since in possible-universes where more aliens appear, more (expected) copies of human civilization appear.
   1. The complication is that more alien civilizations makes it more likely that an alien causes you to never have existed, e.g. by colonizing the solar system billions of years ago. So a corollary is that you should update toward human-level civilizations being less likely to be "loud" or tending to affect fewer alien civilizations or something than it naively seems.
      1. So SIA predicts that there were few aliens in the early universe and many aliens around now.
      2. So SIA predicts that human-level civilizations (more precisely, I think: civilizations whose existence is correlated with your existence) tend not to noticeably affect many others (whether due to their capabilities, motives, or existential catastrophes).
      3. So SIA retrodicts there being a speed limit (the speed of light) and moreover predicts that noticeable-influence-propagation in fact tends to be even slower.
3. If SIA is correct, you should update toward the proposition that you live in a simulation, relative to your naive credence.
   1. Because there could be lots more (expected) copies of you in simulations.
   2. Because that can explain why you appear to exist billions of years after billions of alien civilizations could have reached Earth.

AI endgame

In board games, the endgame is a period generally characterized by strategic clarity, more direct calculation of the consequences of actions rather than evaluating possible actions with heuristics, and maybe a narrower space of actions that players consider.

Relevant actors, particularly AI labs, are likely to experience increased strategic clarity before the end, including AI risk awareness, awareness of who the leading labs are, roughly how much lead time they have, and how threshold-y being slightly ahead is.

There may be opportunities for coordination in the endgame that were much less incentivized earlier, and pausing progress may be incentivized in the endgame despite being disincentivized earlier (from an actor's imperfect perspective and/or from a maximally informed/wise/etc. perspective).

The downside of "AI endgame" as a conceptual handle is that it suggests thinking of actors as opponents/adversaries. Probably "crunch time" is better, but people often use that to gesture at hinge-y-ness rather than strategic clarity.

New adversarial robustness scorecard: https://scale.com/leaderboard/adversarial_robustness. Yay DeepMind for being in the lead.

I plan to add an "adversarial robustness" criterion in the next major update to my https://ailabwatch.org scorecard and defer to scale's thing (how exactly to turn their numbers into grades TBD), unless someone convinces me that scale's thing is bad or something else is even better?

How can you make the case that a model is safe to deploy? For now, you can do risk assessment and notice that it doesn't have dangerous capabilities. What about in the future, when models do have dangerous capabilities? Here are four options:

1. Implement safety measures as a function of risk assessment results, such that the measures feel like they should be sufficient to abate the risks
   1. This is mostly what Anthropic's RSP does (at least so far — maybe it'll change when they define ASL-4)
2. Use risk assessment techniques that evaluate safety given deployment safety practices
   1. This is mostly what OpenAI's PF is supposed to do (measure "post-mitigation risk"), but the details of their evaluations and mitigations are very unclear
3. Do control [LW · GW] evaluations [LW · GW]
4. Achieve alignment (and get strong evidence of that)

Related: RSPs, safety cases.

Maybe lots of risk comes from the lab using AIs internally to do AI development [LW(p) · GW(p)]. The first two options are fine for preventing catastrophic misuse from external deployment but I worry they struggle to measure risks related to scheming and internal deployment.

Slowing AI: Bad takes

This shortform was going to be a post in Slowing AI [? · GW] but its tone is off.

This shortform is very non-exhaustive.

Bad take #1: China won't slow, so the West shouldn't either

There is a real consideration here. Reasonable variants of this take include

• What matters for safety is not just slowing but also the practices of the organizations that build powerful AI. Insofar as the West is safer and China won't slow, it's worth sacrificing some Western slowing to preserve Western lead.
• What matters for safety is not just slowing but especially slowing near the end. Differentially slowing the West now would reduce its ability to slow later (or even cause it to speed later). So differentially slowing the West now is bad.

(Set aside the fact that slowing the West generally also slows China, because they're correlated and because ideas pass from the West to China.) (Set aside the question of whether China will try to slow and how correlated that is with the West slowing.)

In some cases slowing the West would be worth burning lead time. But slowing AI doesn't just mean the West slowing itself down. Some interventions would slow both spheres similarly or even differentially slow China– most notably export controls, reducing diffusion of ideas, and improved migration policy.

See West-China relation [LW · GW].

Bad take #2: slowing can create a compute overhang, so all slowing is bad

Taboo "overhang." [LW · GW]

Yes, insofar as slowing now risks speeding later, we should notice that. There is a real consideration here.

But in some cases slowing now would be worth a little speeding later. Moreover, some kinds of slowing don't cause faster progress later at all: for example, reducing diffusion of ideas, decreasing hardware progress, and any stable and enforceable policy regimes that slow AI.

See Quickly scaling up compute [LW · GW].

Bad take #3: powerful AI helps alignment research, so we shouldn't slow it

(Set aside the question of how much powerful AI helps alignment research.) If powerful AI is important for alignment research, that means we should aim to increase time with powerful AI, not how soon powerful AI appears.

Bad take #4: it would be harder for unaligned AI to take over in a world with less compute available (for it to hijack), and failed takeover attempts would be good, so it's better for unaligned AI to try to take over soon

No, running AI systems seems likely to be cheap and there's already plenty of compute.

List of uncertainties about the future of AI

This is an unordered list of uncertainties about the future of AI, trying to be comprehensive– trying to include everything reasonably decision-relevant and important/tractable.

This list is mostly written from a forecasting perspective. A useful complementary perspective to forecasting would be strategy or affordances or what actors can do and what they should do. This list is also written from a nontechnical perspective.

• Timelines
  • Capabilities as a function of inputs (or input requirements for AI of a particular capability level)
  • Spending by leading labs
  • Cost of compute
  • Ideas and algorithmic progress
  • Endogeneity in AI capabilities
  • What would be good? Interventions?
• Takeoff (speed and dynamics)
  • dcapabilities/dinputs (or returns on cognitive reinvestment [LW · GW] or intelligence explosion or fast recursive self-improvement): Is there a threshold of capabilities such that self-improvement or other progress is much greater slightly above that point than slightly below it? (If so, where is it?) Will there be a system that can quickly and cheaply improve itself (or create a more capable successor), such that the improvements enable similarly large improvements, and so on until the system is much more capable? Will a small increase in inputs cause a large increase in capabilities (like the difference between chimps and humans) (and if so, around human-level capabilities or where)? How fast will progress be?
    • Will dcapabilities/dinputs be very high because of recursive self-improvement?
    • Will dcapabilities/dinputs be very high because of generality being important and monolithic/threshold-y?
    • Will dcapabilities/dinputs be very high because ideas are discrete (and in particular, will there be a single "secret sauce" idea)? [Seems intractable and unlikely to be very important.]
  • dimpacts/dcapabilities: Will small increases in capabilities (on the order of the variance between different humans) cause large increases in impacts?
    • Related: payoff thresholds and human-competition threshold
  • Qualitatively, how will AI research capabilities affect AI progress; what will AI progress look like when AI research capabilities are a big deal?
    • One dimension or implication: will takeoff be local or nonlocal? How distributed will it be; will it look like recursive self-improvement or the industrial revolution?
      • What does this depend on?
  • Endogeneity in AI capabilities: how do AI capabilities affect AI capabilities? (Potential alternative framing: dinputs/dtime.)
    • How (much) will AI tools accelerate research?
    • Will AI labs generate substantial revenue?
    • Will AI systems make AI seem more exciting or frightening? What effect would this have on AI progress?
    • What other endogeneities exist?
• Weak AI
  • How will the strategic landscape be different in the future?
    • Due to weak AI?
    • Due to other factors? (See Relevant pre-AGI possibilities [LW · GW].)
  • Will weak AI make AI risk clearer, and especially make it more legible (relates to "warning shots")?
  • Will AI progress cause substantial misuse or conflict? What would that look like?
  • Will AI progress enable pivotal acts or processes [LW · GW]?
• Misalignment risk sources/modes
  • What misalignment would occur by default, and how would it be bad? Possible (overlapping) scenarios include inner alignment failure, outer alignment failure, getting what you measure [LW · GW], influence-seeking [LW · GW], more Paul Christiano stories [LW · GW], multipolar failure [LW · GW], a treacherous turn [? · GW], a sharp left turn [LW · GW] (or distributional leap [LW · GW]), and more.
  • What would a powerful, unaligned AI agent do? [Answer: the details are unpredictable, but the high-level outcome is pretty clear; it would very likely be catastrophic. (But note there is reasonable disagreement, e.g. [LW · GW].)]
• Technical problems around AI systems doing what their controllers want [doesn't fit into this list well]
  • How to make powerful AI systems aligned to human preferences
    • Or: how to make powerful AI systems that do not cause a catastrophe
  • How to make powerful AI systems interpretable to human understanding
  • How to solve problems around decision theory and strategic interaction [LW · GW] (and whether they're important)
  • How to solve Wei-Dai-style philosophy problems [LW · GW] (and whether they're important)
  • How to solve problems around delegation involving multiple humans or multiple AI systems (and whether they're important)
• Polarity [LW · GW] (relates to takeoff, endogeneity, and timelines)
  • What determines or affects polarity?
  • What are the effects or implications of polarity on alignment and stabilization?
  • What are the effects or implications of polarity on what the long-term future looks like, conditional on achieving alignment and stabilization?
  • What would be good? Interventions?
• Proximate and ultimate uses of powerful AI
  • What uses of powerful AI would be great? How good would various possible uses of powerful AI be?
  • Conditional on achieving alignment, what's likely to occur (and what could foresighted actors cause to occur or not to occur)?
• Agents vs tools and general systems vs narrow tools (relates to tool AI [? · GW] and Comprehensive AI Services [LW · GW])
  • Are general systems more powerful than similar narrow tools?
  • Are agents more powerful than similar tools?
  • Does generality appear by default in capable systems? (This relates to takeoff.)
  • Does agency (or goal-directedness [? · GW] or consequentialism or farsightedness) appear by default in capable systems?
• AI labs' behavior and racing for AI
  • How will labs think about AI?
  • What actions could labs perform; what are they likely to do by default?
  • What would it be better if labs did; what interventions are tractable?
• States' behavior
  • How will states think about AI?
  • What actions could states perform? What are they likely to do by default?
  • What would it be better if states did; what interventions are tractable?
• Public opinion
  • How the public thinks about AI and framing; what the public thinks about AI, what memes would spread widely and how that depends on other facts about the world; how all of that translates into attitudes and policy preferences
  • Wakeup to capabilities
  • Wakeup to alignment risk and warning shots for alignment
  • What (facts about public opinion) would be good? Interventions?
• Paths to powerful AI (relates to timelines, AI risk modes, and more)
  • How successful will reinforcement-learning agents built on large language models be?
  • How successful will comprehensive AI services [LW · GW] be?
  • How successful will language model bureaucracies [LW · GW] be?
  • How successful will STEM AI [LW · GW] or Skunkworks-style AI [LW · GW] be?
  • How successful will simulating evolution [LW · GW] be?
  • How successful will whole-brain emulation or neuromorphic AI be?
  • When is thinking in terms of paths or roadmaps useful? What other high-level paths are relevant?
• Meta and miscellanea
  • Epistemic stuff
    • Research methodology and organization: how to do research and organize researchers
    • Forecasting methodology: how to do forecasting better
    • Collective epistemology: how to share and aggregate beliefs and knowledge
  • Decision theory
  • Simulation hypothesis [? · GW]
    • Do we live in a simulation? (If so, what's the deal?)
    • What should we do if we live in a simulation (as a function of what the deal is)?
  • Movement/community/field stuff

Maybe this list would be more useful if it had more pointers to relevant work?

Maybe this list would be more useful if it included stuff that's important that I don't feel uncertain about? But probably not much of that exists?

I like lists/trees/graphs. I like the ideas behind Clarifying some key hypotheses in AI alignment [LW · GW] and Modelling Transformative AI Risks [? · GW]. Perhaps this list is part of the beginning of a tree/graph for AI forecasting not including alignment stuff.

Maybe AI Will Happen Outside US/China

I'm interested in the claim important AI development (in the next few decades) will largely occur outside any of the states that currently look likely to lead AI development. I don't think this is likely, but I haven't seen discussion of this claim.^[1] This would matter because it would greatly affect the environment in which AI is developed and affect which agents are empowered by powerful AI.

Epistemic status: brainstorm. May be developed into a full post if I learn or think more.

I. Causes

The big tech companies are in the US and China, and discussion often assumes that these two states have a large lead on AI development. So how could important development occur in another state? Perhaps other states' tech programs (private or governmental) will grow. But more likely, I think, an already-strong company leaves the US for a new location.

My legal knowledge is insufficient to say how well companies can leave their states with any confidence. My impression is that large American companies largely can leave while large Chinese companies cannot.

Why might a big tech company or AI lab want to leave a state?^[2]

• Fleeing expropriation/nationalization. States can largely expropriate companies' property within their territory unless they have contracted otherwise. A company may be able to protect its independence by securing legal protection from expropriation from another state, then moving its hardware to that state. It may move its headquarters or workers as well.
• Fleeing domestic regulation on development and/or deployment of AI.

II. Effects

The state in which powerful AI is developed has two important effects.

1. States set regulations. The regulatory environment around an AI lab may affect the narrow AI systems it builds and/or how it pursues AGI.
2. State influence & power. The state in which AGI is achieved can probably nationalize that project (perhaps well before AGI). State control of powerful AI affects how it will be used.

III. AI deployment before superintelligence

Eliezer recently tweeted that AI might be low-impact until superintelligence because of constraints on deployment. This seems partially right — for example, medicine and education seem like areas in which marginal improvements in our capabilities have only small effects due to civilizational inadequacy. Certainly some AI systems would require local regulatory approval to be useful; those might well be limited in the US. But a large fraction of AI systems won't be prohibited by plausible American regulation. For example, I would be quite surprised if the following kinds of systems were prohibited by regulation (disclaimer: I'm very non-expert on near-future AI):

• Business services
  • Operations/logistics
  • Analysis
  • Productivity tools (e.g., Codex, search tools)
• Online consumer services — financial, writing assistants (Codex)
• Production of goods that can be shipped cheaply (like computers but not houses)
• Trading
• Maybe media stuff (chatbots, persuasion systems). It's really hard to imagine the US banning chatbots. I'm not sure how persuasion-AI is implemented; custom ads could conceivably be banned, but eliminating AI-written media is implausible.

This matters because these AI applications directly affect some places even if they couldn’t be developed in those places.

In the unlikely event that the US moves against not only the deployment but also the development of such systems, AI companies would be more likely to seek a way around regulation — such as relocating.

Value Is Binary

Epistemic status: rough ethical and empirical heuristic.

Assuming that value is roughly linear in resources available after we reach technological maturity,^[1] my probability distribution of value is so bimodal that it is nearly binary. In particular, I assign substantial probability to near-optimal futures (at least 99% of the value of the optimal future), substantial probability to near-zero-value futures (between -1% and 1% of the value of the optimal future), and little probability to anything else.^[2] To the extent that almost all of the probability mass fits into two buckets, and everything within a bucket is almost equally valuable as everything else in that bucket, the goal maximize expected value reduces to the goal maximize probability of the better bucket.

So rather than thinking about how to maximize expected value, I generally think about maximizing the probability of a great (i.e., near-optimal) future. This goal is easier for me to think about, particularly since I believe that the paths to a great future are rather homogeneous — alike not just in value but in high-level structure. In the rest of this shortform, I explain my belief that the future is likely to be near-optimal or near-zero.

Substantial probability to near-optimal futures.

I have substantial credence that the future is at least 99% as good as the optimal future.^[3] I do not claim much certainty about what the optimal future looks like — my baseline assumption is that it involves increasing and improving consciousness in the universe, but I have little idea whether that would look like many very small minds or a few very big minds. Or perhaps the optimal future involves astronomical-scale acausal trade. Or perhaps future advances in ethics, decision theory, or physics will have unforeseeable implications for how a technologically mature civilization can do good.

But uniting almost all of my probability mass for near-optimal futures is how we get there, at a high level: we create superintelligence, achieve technological maturity, solve ethics, and then optimize. Without knowing what this looks like in detail, I assign substantial probability to the proposition that humanity successfully completes this process. And I think almost all futures in which we do complete this process look very similar: they have nearly identical technology, reach the same conclusions on ethics, have nearly identical resources available to them (mostly depending on how long it took them to reach maturity), and so produce nearly identical value.

Almost all of the remaining probability to near-zero futures.

This claim is bolder, I think. Even if it seems reasonable to expect a substantial fraction of possible futures to converge to near-optimal, it may seem odd to expect almost all of the rest to be near-zero. But I find it difficult to imagine any other futures.

For a future to not be near-zero, it must involve using a nontrivial fraction of the resources available in the optimal future (by my assumption that value is roughly linear in resources). More significantly, the future must involve using resources at a nontrivial fraction of the efficiency of their use in the optimal future. This seems unlikely to happen by accident. In particular, I claim:

If a future does not involve optimizing for the good, value is almost certainly near-zero.

Roughly, this holds if all (nontrivially efficient) ways of promoting the good are not efficient ways of optimizing for anything else that we might optimize for. I strongly intuit that this is true; I expect that as technology improves, efficiently producing a unit of something will produce very little of almost all other things (where "thing" includes not just stuff but also minds, qualia, etc.).^[4] If so, then value (or disvalue) is (in expectation) a negligible side effect of optimization for other things. And I cannot reasonably imagine a future optimized for disvalue, so I think almost all non-near-optimal futures are near-zero.

So I believe that either we optimize for value and get a near-optimal future, or we do anything else and get a near-zero future.

Intuitively, it seems possible to optimize for more than one value. I think such scenarios are unlikely. Even if our utility function has multiple linear terms, unless there is some surprisingly good way to achieve them simultaneously, we optimize by pursuing one of them near-exclusively.^[5] Optimizing a utility function that looks more like min(x,y) may be a plausible result of a grand bargain, but such a scenario requires that, after we have mature technology, multiple agents have nontrivial bargaining power and different values. I find this unlikely; I expect singleton-like scenarios and that powerful agents will either all converge to the same preferences or all have near-zero-value preferences.

I mostly see "value is binary" as a heuristic for reframing problems. It also has implications for what we should do: to the extent that value is binary (and to the extent that doing so is feasible), we should focus on increasing the probability of great futures. If a "catastrophic" future is one in which we realize no more than a small fraction of our value, then a great future is simply one which is not catastrophic and we should focus on avoiding catastrophes. But of course, "value is binary" is an empirical approximation rather than an a priori truth. Even if value seems very nearly binary, we should not reject contrary proposed interventions^[6] or possible futures out of hand.

I would appreciate suggestions on how to make these ideas more formal or precise (in addition to comments on what I got wrong or left out, of course). Also, this shortform relies on argument by "I struggle to imagine"; if you can imagine something I cannot, please explain your scenario and I will justify my skepticism or update.

How do corporate campaigns and leaderboards effect change?

https://ailabwatch.org/resources/integrity

Writing a thing on lab integrity issues. Planning to publish early Monday morning [edit: will probably hold off in case Anthropic clarifies nondisparagement stuff]. Comment on this public google doc or DM me.

I'm particularly interested in stuff I'm missing or existing writeups on this topic.

AI strategy research projects project generators prompts

Mostly for personal use.

Some prompts inspired by Framing AI strategy [LW · GW]:

• Plans
  • What plans would be good?
  • Given a particular plan that is likely to be implemented, what interventions or desiderata complement that plan (by making it more likely to succeed or by being better in worlds where it succeeds)
• Affordances: for various relevant actors, what strategically significant actions could they take? What levers do they have? What would it be great for them to do (or avoid)?
• Intermediate goals: what goals or desiderata are instrumentally useful?
• Threat modeling: for various threats, model them well enough to understand necessary and sufficient conditions for preventing them.
• Memes (& frames): what would it be good if people believed or paid attention to?

For forecasting prompts, see List of uncertainties about the future of AI [LW(p) · GW(p)].

Some miscellaneous prompts:

• Slowing AI
  • How can various relevant actors slow AI?
  • How can the AI safety community slow AI?
  • What considerations or side effects are relevant to slowing AI?
• How do labs act, as a function of [whatever determines that]? In particular, what's the deal with "racing"?
• AI risk advocacy
  • How could the AI safety community do AI risk advocacy well?
  • What considerations or side effects are relevant to AI risk advocacy? 
• What's the deal with crunch time?
• How will the strategic landscape be different in the future?
• What will be different near the end, and what interventions or actions will that enable? In particular, is eleventh-hour coordination possible? (Also maybe emergency brakes that don't appear until the end.)
• What concrete asks should we have for labs? for government?
• Meta: how can you help yourself or others do better AI strategy research?

Four kinds of actors/processes/decisions are directly very important to AI governance:

• Corporate self-governance
  • Adopting safety standards
    • Proving a model for government regulation
• US policy (and China, EU, UK, and others to a lesser extent)
  • Regulation
  • Incorporating standards into law
• Standard-setters setting standards
• International relations
  • Treaties
  • Informal influence on safety standards

Related: How technical safety standards could promote TAI safety [EA · GW].

("Safety standards" sounds prosaic but it doesn't have to be.)

AI risk decomposition based on agency or powerseeking or adversarial optimization or something

Epistemic status: confused.

Some vague, closely related ways to decompose AI risk into two kinds of risk:

• Risk due to AI agency vs risk unrelated to agency
• Risk due to AI goal-directedness vs risk unrelated to goal-directedness
• Risk due to AI planning vs risk unrelated to planning
• Risk due to AI consequentialism vs risk unrelated to consequentialism
• Risk due to AI utility-maximization vs risk unrelated to utility-maximization
• Risk due to AI powerseeking vs risk unrelated to powerseeking
• Risk due to AI optimizing against you vs risk unrelated to adversarial optimization

The central reason to worry about powerseeking/whatever AI, I think, is that sufficiently (relatively) capable goal-directed systems instrumentally converge to disempowering you.

The central reason to worry about non-powerseeking/whatever AI, I think, is failure to generalize correctly from training-- distribution shift, Goodhart, You get what you measure [LW · GW].

Biological bounds on requirements for human-level AI

Facts about biology bound requirements for human-level AI. In particular, here are two prima facie bounds:

• Lifetime. Humans develop human-level cognitive capabilities over a single lifetime, so (assuming our artificial learning algorithms are less efficient than humans' natural learning algorithms) training a human-level model takes at least the inputs used over the course of babyhood-to-adulthood.
• Evolution. Evolution found human-level cognitive capabilities by blind search, so (assuming we can search at least that well, and assuming evolution didn't get lucky) training a human-level model takes at most the inputs used over the course of human evolution (plus Lifetime inputs, but that's relatively trivial).
  • Genome. The size of the human genome is an upper bound on the complexity of humans' natural learning algorithms. Training a human-level model takes at most the inputs needed to find a learning algorithm at most as complex as the human genome (plus Lifetime inputs, but that's relatively trivial). (Unfortunately, the existence of human-level learning algorithms of certain simplicity says almost nothing about the difficulty of finding such algorithms.) (Ajeya's "genome anchor" is pretty different—"a transformative model would . . . have about as many parameters as there are bytes in the human genome"—and makes no sense to me.)

(A human-level AI should use similar computation as humans per subjective time. This assumption/observation is weird and perhaps shows that something weird is going on, but I don't know how to make that sharp.)

There are few sources of bounds on requirements for human-level AI. Perhaps fundamental limits or reasoning about blind search could give weak bounds, but biology is the only example of human-level cognitive abilities and so the only possible source of reasonable bounds.

Related: Ajeya Cotra's Forecasting TAI with biological anchors [LW · GW] (most relevant section) and Eliezer Yudkowsky's Biology-Inspired AGI Timelines: The Trick That Never Works [LW · GW].

What do people (outside this community) think about AI? What will they think in the future?

Attitudes predictably affect relevant actors' actions, so this is a moderately important question. And it's rather neglected.

Groups whose attitudes are likely to be important include ML researchers, policymakers, and the public.

On attitudes among the public, surveys provide some information, but I suspect attitudes will change (in potentially predictable ways) as AI becomes more salient and some memes/framings get locked in. Perhaps some survey questions (maybe general sentiment on AI) are somewhat robust to changes in memes while others (maybe beliefs on how AI affects the economy or attitudes on regulation) may change a lot in the near future.

On attitudes among ML researchers, surveys (e.g.) provide some information, but for some reason most ML researchers say there's at least a 5% probability of doom (or 10%, depending on how you ask) but this doesn't seem to translate into their actions or culture. Perhaps interviews would reveal researchers' attitudes better than closed-ended surveys (note to self: talk to Vael Gates [LW · GW]).

AI may become much more salient in the next few years, and memes/framings may get locked in.