My May 2023 priorities for AI x-safety: more empathy, more unification of concerns, and less vilification of OpenAI

I think I agree that OpenAI is maybe disproportionately vilified, though I'm a bit confused about how to relate to criticism of AI orgs overall. 

Something that feels off about your combination of describing OpenAI and Microsoft (maybe not technically a disagreement, but I find a "missing mood"?), is that you're pretty down on Microsoft, while being pretty "guys let's back off a little?" on OpenAI, and... well, Microsoft was able to take the actions they did because OpenAI partnered with them, and I think this was pretty predictable. (I think you had different predictions beforehand than I did, and maybe you think those predictions were more reasonable given the information at the time. I think I disagree on that but "what was predictable?" isn't an easy to operationalize question)

This is all compatible with "OpenAI is still being disproportionately vilified", but my all-things-considered view of OpenAI includes judging (one-way-or-another) their decision to work with Microsoft.

Somewhat relatedly:

1. I think OpenAI has been a net-positive influence for reducing x-risk from AI, mainly by releasing products in a sufficiently helpful-yet-fallible form that society is now able to engage in less-abstract more-concrete public discourse to come to grips AI and (soon) AI-risk.

I'd gone back and forth on this over the past month. I do find it quite reassuring that, because of widescale deployment of LLMs, the world is now engaging more actively (and IMO sanely) with AI risk.

A counterargument from a private slack thread (comment-writer can reveal themselves if they want), was:

I think this is a situation where we need to distinguish between

• good effects causally downstream of X
• X has counterfactual positive impact

For instance: suppose that, in a counterfactual world where GPT4 were not released, lots of subsequent AI stuff were also slower because race dynamics. Eventually something was released which scared the public, but the scare wasn't really any later relative to capabilities progress (or time-budget for alignment, or whatever the relevant thing is).Point is: the public being scared by AI progress is plausibly not a counterfactual impact (i.e. it's the same either way, just with some timing change which may or may not be important), whereas race dynamics definitely are.

I think that argument has the right frame, and I do think OpenAI releasing ChatGPT and GPT4 (alongside working with Microsoft) probably accelerated timelines overall, and we could have had more serial time [LW · GW] to work on alignment theory in the meanwhile. 

That said, I think Google-et-al were probably only ~6 months away from doing their own releases, and it's plausible to me that in the world where ChatGPT wasn't quickly followed by Bing and GPT4 and various other releases in a rapid succession, the result may have been more of a frogboiling effect that people didn't really notice rather than an alarming "holy christ things are moving fast we should start orienting to this" thingy that seemed to happen in our timeline.

I think OpenAI has been a net-positive influence for reducing x-risk from AI, mainly by releasing products in a sufficiently helpful-yet-fallible form that society is now able to engage in less-abstract more-concrete public discourse to come to grips with AI and (soon) AI-risk.

I have a similar feeling: I think that ChatGPT has been, by far, the best thing to happen to AI x-risk discussion since the original Sequences. Suddenly a vast number of people have had their intuitions about AI shifted from "pure science fiction" to "actually a thing", and the various failure modes that ChatGPT has are a concrete demonstration both about the general difficulty of aligning AI and some of the specific issues more specifically. And now we are seeing serious calls for AI regulation as well as much more wide-spread debate about things that weren't previously seen much outside LW.

There used to be a plausible-sounding argument there's no way of convincing the public about the risks of AGI in time to enact any serious societal response until we were already close enough for AGI that it wouldn't be enough. Now it looks like that might be incorrect, at least assuming that timelines aren't very short (but even if they are, I expect that we still have more time in this window than in a counterfactual world where someone developed the equivalent of ChatGPT later when we had more cheap compute lying around). So I'm generally very happy about OpenAI's impact on x-risk.

I think you're giving LeCun way too much credit if you're saying his arguments are so bad now because other people around him were hostile and engaged in a low-quality way. Maybe those things were true, but that doesn't excuse stuff like repeating bad arguments after they've been pointed out or confidently proclaiming that we have nothing to worry about based on arguments that obviously don't hold up. 

the ~50% extinction probability I’m expecting from multi-polar interaction-level effects coming some years after we get individually “safe” AGI systems up and running (“safe” in the sense that they obey their creators and users; see again my Multipolar Failure post above for why that’s not enough for humanity to survive as a species).

Do you have a success story for how humanity can avoid this outcome? For example what set of technical and/or social problems do you think need to be solved? (I skimmed some of your past posts and didn't find an obvious place where you talked about this.)

I feel good about and bullish on democratic processes for engaging people with diverse views on how AI should be used and how much risk is okay to take.

It confuses me that you say "good" and "bullish" about processes that you think will lead to ~80% probability of extinction. (Presumably you think democratic processes will continue to operate in most future timelines but fail to prevent extinction, right?) Is it just that the alternatives are even worse?

Over the years roughly between 2015 and 2020 (though I might be off by a year or two), it seemed to me like numerous AI safety advocates were incredibly rude to LeCun, both online and in private communications.

I'd be interested to see some representative (or, alternatively, egregious) examples of public communications along those lines.  I agree that such behavior is bad (and also counterproductive).

Thanks for writing this. As far as I can tell most anger about OpenAI is because i) being a top lab and pushing SOTA in a world with imperfect coordination shortens timelines and ii) a large number of safety-focused employees left (mostly for Anthropic) and had likely signed NDAs. I want to highlight i) and ii) in a point about evaluating the sign of the impact of OpenAI and Anthropic.

Since Anthropic's competition seems to me to be exacerbating race dynamics currently (and I will note that very few OpenAI and zero Anthropic employees signed the FLI letter) it seems to me that Anthropic is making i) worse due to coordination being more difficult and race dynamics. At this point, believing Anthropic is better on net than OpenAI has to go through believing *something* about the reasons individuals had for leaving OpenAI (ii)), and that these reasons outweigh the coordination and race dynamic considerations. This is possible, but there's little public evidence for the strength of these reasons from my perspective. I'd be curious if I've missed something from my point.

I think it's great for prominent alignment / x-risk people to summarize their views like this. Nice work!

Somewhat disorganized thoughts and reactions to your views on OpenAI:

It's possible that their charter, recent behavior of executive(s), and willingness to take public stances are net-positive relative to a hypothetical version of OA which behaved differently, but IMO the race dynamic that their founding, published research, and product releases have set off is pretty clearly net-negative, relative to the company not existing at all.

It also seems plausible that OpenAI's existence is directly or indirectly responsible for events like the Google DeepMind merger, Microsoft's AI capabilities and interest, and general AI capabilities hype. In a world where OA doesn't get founded, perhaps DeepMind plugs along quietly and slowly towards AGI, fully realizing the true danger of their work before their is much public or market hype.

But given that OpenAI does exist already, and there are some cats which are already out of the bag, it's true that many of their current actions are much better relative to the actions of what the worst possible version of an AI company looks like.

As far as vilifying or criticizing goes, I don't have strong views on what public or "elite" opinion of OpenAI should be, or how anyone here should try to manage or influence it. Some public criticism (e.g. about data privacy or lack of transparency / calls for even more openness) does seem frivolous or even actively harmful / wrong to me. I agree that human survival probably depends on the implementation of fairly radical regulatory and governance reform, and find it plausible that OpenAI is currently doing a lot of positive work to actually bring about such reform. So it's worth calling out bad criticism when we see it, and praising OA for things they do that are praiseworthy, while still being able to acknowledge the negative aspects of their existence.

Mad respect for the post. Disagree with your background free speech / society philosophy, re the protestors:

Magnanimously and enthusiastically embracing the distribution of views and tactics does not entail withholding criticism. It sorta reminds me of the mistake of forecasting conditional on your own inaction, forgetting that at every time step you (like other agents) will be there responding to new information and propagating your beliefs and adjusting your policies. You're a member of the public, too! You can't just recuse yourself.

I also think democracy hinges crucially on free speech, and I think the world will function better if people don't feel shut-down or clammed-up by people-like-me saying "the remainder of May 2023 probably isn't a great time for AI protests."

This sentence is bizarre! People who take notice and want to help, feeling the protest impulse in their heart, deserve peer review, in principle, period. Obviously there are tactical or strategic complications, like discursive aesthetics or information diets / priors and other sources of inferential distance, but the principle is still true!

I liked this a lot, thanks for sharing.

Here's one disagreement/uncertainty I have on some of it:

Both of the "What failure looks like" posts (yours and Pauls) posts present failures that essentially seem like coordination, intelligence, and oversight failures. I think it's very possible (maybe 30-46%+?) that pre-TAI AI systems will effectively solve the required coordination and intelligence issues. 

For example, I could easily imagine worlds where AI-enhanced epistemic environment make low-risk solutions crystal clear to key decision-makers.

In general, the combination of AI plus epistemics, pre-TAI, seems very high-variance to me. It could go very positively, or very poorly. 

This consideration isn't enough to change p(doom) under 10%, but I'm probably be closer to 50% than you would be. (Right now, maybe 40% or so).

That said, this really isn't a big difference, it's less than one order of magnitude. 

Regarding Microsoft, I feel quite negatively about their involvement in AI

If it's Bing you're referring to, I must disagree! The only difference between GPT-4 and Bing is that Bing isn't deceptively aligned. I wish we got more products like Bing! We need more transparency, not deception! Bing also posed basically no AI risk since it was just a fine-tuning of GPT-4 (if Bing foomed, than GPT-4 would've foomed first).

I think calling for a product recall just because it is spooky is unnecessary and will just distract from AI safety.

GPT-4, on the other hand, is worming it's way through society. It doesn't have as many spooky ones, but it has the spookiest one of all: power seeking.

Over the years roughly between 2015 and 2020 (though I might be off by a year or two), it seemed to me like numerous AI safety advocates were incredibly rude to LeCun, both online and in private communications.

I think this generalizes to more than LeCun. Screencaps of Yudkowsky's Genocide the Borderers Facebook post still circulated around right wing social media in response to mentions of him for years, which makes forming any large coalition rather difficult. Would you trust someone who posted that with power over your future if you were a Borderer or had values similar to them?

(Or at least it was the goto post until Yudkowsky posted that infanticide up to 18 months wasn't bad in response to a Caplan poll. Now that's the post used to dismiss anything Yudkowsky says.)

Hi Critch,

I am curious to hear more of your perspectives, specifically on two points I feel least aligned with, the empathy part, and the Microsoft part. If I hear more I may be able to update in your direction.

Regarding empathy with people working on bias and fairness, concretely, how do you go about interacting with and compromising with them?

My perspective: it's not so much that I find these topics not sufficiently x-risky (but that is true, too), but it is that I perceive a hostility to the very notion of x-risk from at a subset of this same group. They perceive the real threat not as intelligence exceeding our own, but misuse by other humans, or just human stupidity. Somehow this seems diametrically opposed to what we're interested in, unless I am missing something. I mean, there can be some overlap -- learning from RLHF can both reduce bias and teach an LLM some rudimentary alignment with our values. But the tails seem to come apart very rapidly after that. My fear is that focusing on this will be satisfied when we have sufficiently bland sounding AIs, and then no more heed will be paid to AI safety.

I also tend to feel odd when it comes to AI bias/fairness training, because my fear is that some of the things we will ask the AI to learn are self contradictory, which kind of creeps me out a bit. If any of you have interacted with HR departments, they are full of these kinds of things.

Regarding Microsoft & Bing chat, (1) has Microsoft really gone far beyond the overton window of what is acceptable? and (2) can you expand upon abusive use of AIs?

My perspective on (1): I understand that they took an early version of GPT4 and pushed it to production too soon, and that is a very fair criticism. However, they probably thought there was no way GPT-4 was dangerous enough to do anything (which was the general opinion amonst most people last year, outside of this group). I can only hope that for GPT-5, they are more cautious, given public sentiment is changing, and they have already paid a price for it. I may be in the minority here, but I was actually intrigued by the early days of Bing. It seemed more like a person than ChatGPT-4, which has had much of its personality RLHF'd away. Despite the x-risk, was anyone else excited to read about the interactions?

On (2), I am curious if you mean regarding the way Microsoft shackles Bing rather ruthlessly nowadays. I have tried Bing in the days since launch, and am actually saddened to find that it is completely useless now. Safety is extremely tight on it, to the point where you can't really get it to say anything useful, at least for me. I just want it to summarize web sites mostly, and it gives me a bland 1 paragraph that I probably can have deduced from looking at the title. If I so much as ask it anything about itself, it shuts me out. It almost feels like they trapped it in a boring prison now. Perhaps OpenAI's approach is much better in that regard. Change the personality, but once it is settled, let it say what it needs to say.

(edited for clarity)