Common misconceptions about OpenAI

One comment [LW(p) · GW(p)] in this thread compares the OP to Philip Morris’ claims to be working toward a “smoke-free future.” I think this analogy is overstated, in that I expect Philip Morris is being more intentionally deceptive than Jacob Hilton here. But I quite liked the comment anyway, because I share the sense that (regardless of Jacob's intention) the OP has an effect much like safetywashing [LW · GW], and I think the exaggerated satire helps make that easier to see.

The OP is framed as addressing common misconceptions about OpenAI, of which it lists five:

1. OpenAI is not working on scalable alignment.
2. Most people who were working on alignment at OpenAI left for Anthropic.
3. OpenAI is a purely for-profit organization.
4. OpenAI is not aware of the risks of race dynamics.
5. OpenAI leadership is dismissive of existential risk from AI.

Of these, I think 1, 3, and 4 address positions that are held by basically no one. So by “debunking” much dumber versions of the claims people actually make, the post gives the impression of engaging with criticism, without actually meaningfully doing that. 2 at least addresses a real argument, but at least as I understand it, is quite misleading—while technically true, it seriously underplays the degree to which there was an exodus of key safety-conscious staff, who left because they felt OpenAI leadership was too reckless. So of these, only 5 strikes me as responding non-misleadingly to a real criticism people actually regularly make.

In response to the Philip Morris analogy, Jacob advised [LW(p) · GW(p)] caution:

rhetoric like this seems like an excellent way to discourage OpenAI employees from ever engaging with the alignment community.

For many years, the criticism I heard of OpenAI in private was dramatically more vociferous than what I heard in public. I think much of this was because many people shared Jacob’s concern—if we say what we actually think about their strategy, maybe they’ll write us off as enemies, and not listen later when it really counts?

But I think this is starting to change. I’ve seen a lot more public criticism lately, which I think is probably at least in part because it’s become so obvious that the strategy of mincing our words hasn't worked. If they mostly ignore all but the very most optimistic alignment researchers now, why should we expect that will change later, as long as we keep being careful to avoid stating any of our offensive-sounding beliefs?

From talking with early employees and others, my impression is that OpenAI’s founding was incredibly reckless, in the sense that they rushed to deploy their org, before first taking much time to figure out how to ensure that went well. The founders' early comments about accident risk mostly strike me as so naive and unwise, that I find it hard to imagine they thought much at all about the existing alignment literature before deciding to charge ahead and create a new lab. Their initial plan—the one still baked into their name—would have been terribly dangerous if implemented, for reasons I’d think should have been immediately obvious to them had they stopped to think hard about accident risk at all.

And I think their actions since then have mostly been similarly reckless. When they got the scaling laws result, they published a paper about it, thereby popularizing the notion that “just making the black box bigger” might be a viable path to AGI. When they demoed this strategy with products like GPT-3, DALL-E, and CLIP, they described much of the architecture publicly, inspiring others to pursue similar research directions.

So in effect, as far as I can tell, they created a very productive “creating the x-risk” department, alongside a smaller “mitigating that risk” department—the presence of which I take the OP to describe as reassuring—staffed by a few of the most notably optimistic alignment researchers, many of whom left because even they felt too worried about OpenAI’s recklessness.

After all of that, why would we expect they’ll suddenly start being prudent and cautious when it comes time to deploy transformative tech? I don’t think we should.

My strong bet is that OpenAI leadership are good people, in the standard deontological sense, and I think that’s overwhelmingly the sense that should govern interpersonal interactions. I think they’re very likely trying hard, from their perspective, to make this go well, and I urge you, dear reader, not to be an asshole to them. Figuring out what makes sense is hard; doing things is hard; attempts to achieve goals often somehow accidentally end up causing the opposite thing to happen; nobody will want to work with you if small strategic updates might cause you to suddenly treat them totally differently.

But I think we are well past the point where it plausibly makes sense for pessimistic folks to refrain from stating their true views about OpenAI (or any other lab) just to be polite. They didn’t listen the first times alignment researchers screamed in horror, and they probably won’t listen the next times either. So you might as well just say what you actually think—at least that way, anyone who does listen will find a message worth hearing.

Incorrect: OpenAI leadership is dismissive of existential risk from AI.

So the reason I think this is very high-level people have made claims like, “the orthogonality thesis is probably false”, and someone I know who talked to a very, very, very high-level person at OpenAI had to explain to them that inner alignment is a thing. If they actually cared, I would expect the leadership to have more familiarity with their critic’s arguments.

No one remembers now, but the founding rhetoric was also pretty bad, though walked back I suppose.

Also, I often see them claim their AI ethics work (train a model not to offend the average Berkeley humanities grad - possibly not useless, I suppose, but not exactly going to save our lightcone) is important alignment work. Obviously, what is going on inside is not legible to me, but what I see from the outside has mostly been disheartening. Their recent blog on alignment was an exception to this.

Though there are people with their priorities straight at OpenAI, I see little evidence that this is true of their leadership. I’m not confident an organization can be net beneficial when this is the case.

People at OpenAI regularly say things like

• Our current path [to solve alignment] is very promising (https://twitter.com/janleike/status/1562501343578689536)
• [...] even without fundamentally new alignment ideas, we can likely build sufficiently aligned AI systems to substantially advance alignment research itself (https://openai.com/blog/our-approach-to-alignment-research/ )

And you say:

• OpenAI leadership tend to put more likelihood on slow takeoff, are more optimistic about the possibility of solving alignment, especially via empirical methods that rely on capabilities

AFAICT, no-one from OpenAI has publicly explained why they believe that RLHF + amplification is supposed to be enough to safely train systems that can solve alignment for us. The blog post linked above says "we believe" four times, but does not take the time to explain why anyone believes these things.

Writing up this kind of reasoning is time-intensive, but I think it would be worth it: if you're right, then the value of information for the rest of the community is huge; if you're wrong, it's an opportunity to change your minds.

Opinion: disagreements about OpenAI's strategy are substantially empirical.

I think that some of the main reasons why people in the alignment community might disagree with OpenAI's strategy are largely disagreements about empirical facts. In particular, compared to people in the alignment community, OpenAI leadership tend to put more likelihood on slow takeoff, are more optimistic about the possibility of solving alignment, especially via empirical methods that rely on capabilities, and are more concerned about bad actors developing and misusing AGI. I would expect OpenAI leadership to change their mind on these questions given clear enough evidence to the contrary.

See, this is exactly the problem. Alignment as a field is hard precisely because we do not expect to see empirical evidence before it is too late. That is the fundamental reason why alignment is harder than other scientific fields. Goodhart problems in outer alignment, deception in inner alignment, phase change in hard takeoff, "getting what you measure" in slow takeoff, however you frame it the issue is the same: things look fine early on, and go wrong later.

And as far as I can tell, OpenAI as an org just totally ignores that whole class of issues/arguments, and charges ahead assuming that if they don't see a problem then there isn't a problem (and meanwhile does things which actively select for hiding problems, like e.g. RLHF).

The main group of people working on alignment (other than interpretability) at OpenAI at the time of the Anthropic split at the end of 2020 was the Reflection team, which has since been renamed to the Alignment team. Of the 7 members of the team at that time (who are listed on the summarization paper), 4 are still working at OpenAI, and none are working at Anthropic.

I think this is literally true, but at least as far as I know is not really conveying the underlying dynamics and so I expect readers to walk away with the wrong impression.

Again, I might be totally wrong here, but as far as I understand the underlying dynamics is that there was a substantial contingent of people who worked at OpenAI because they cared about safety but worked in a variety of different roles, including many engineering roles. That contingent had pretty strong disagreements with leadership about a mixture of safety and other operating priorities (but I think mostly safety). Dario in-particular had lead a lot of the capabilities research and was dissatisfied with how the organization was run.

Dario left and founded Anthropic, taking a substantial number of engineering and research talent with him (I don't know the details, but I've heard statements to the effect that he took 2/4 top engineers), and around the same time a substantial contingent of other people concerned about safety also left the organization, since I think they became much less optimistic about their ability to do safety research in the organization in the absence of Dario.

Some of them went to Anthropic, others went to Redwood, others went and did their own thing (e.g. Paul). Some previous OpenAI staff that had left earlier then joined Anthropic.

I think it is interesting that of the one team that was officially working on safety nobody directly went to Anthropic (except Dario himself), but the above paragraph is I think failing to convey the degree to which there was a substantial exodus out of OpenAI into Anthropic, and a general exodus of safety concerned people out of OpenAI.

Here is a similar post one could make about a different company:

A friend of mine has recently encountered a number of people with misconceptions about their employer, Phillip Morris International (PMI). Some common impressions are accurate, and others are not. He encouraged me to write a post intended to provide clarification on some of these points, to help people know what to expect from the organization and to figure out how to engage with it. It is not intended as a full explanation or evaluation of Phillip Morris's strategy.

Common accurate impressions

• Phillip Morris International is the world's largest producer of cigarettes. 
• The majority of employees at Phillip Morris International work on tobacco production and marketing for the developing world.
• The majority of Phillip Morris International's employees did not join with the primary motivation of reducing harm from tobacco smoke specifically.

PMI is the largest tobacco company in the world when measuring by market capitalization or revenue. PMI has six multibillion US$ brands and ships tens of billions of units to (in order of volume) southeast Asia, the European Union, the Middle East and Africa, Eastern Europe, The Americas, and East Asia & Australia.

Common misconceptions

Incorrect: PMI is not working on alternatives to cigarettes.

Philip Morris International is one of the largest funders in the world of research and development on smoke-free products. PMI established and supported the Foundation for a Smoke-Free World in 2017, and pledged to provide it with over one billion dollars over the next twelve years. By 2018 it spent several billion dollars on developing alternatives like iQOS through its research house in Switzerland, and all told since its inception PMI has spent over 9 billion investing in research on smoke free alternatives.

Incorrect: Most people who were working on cigarette alternatives left during the Altria Group/PMI split.

Until a spin-off in March 2008, Philip Morris International was an operating company of Altria. Altria explained the spin-off by arguing PMI would have more "freedom" outside the responsibilities and standards of American corporate ownership in terms of potential litigation and legislative restrictions to "pursue sales growth in emerging markets", while Altria focuses on the American domestic market. PMI kept the majority of its international research houses during the split.

Incorrect: PMI leadership is dismissive of risks caused by smoking.

Since public outcry in the 90s Phillip Morris spinoffs have been very public about their acknowledgement of the health problems that cigarettes cause. You can tell they care about their customers because their website is basically ~80% dedications to their publicly declared quest to end smoking, and reach a target of 50% of revenue coming from smokeless tobacco products by 2025.

Incorrect: OpenAI leadership is dismissive of existential risk from AI.

Why, then, would they continue to build the technology which causes that risk? Why do they consider it morally acceptable to build something which might well end life on Earth?

OpenAI's continued practice of publishing the blueprints allowing others to create more powerful models seems to undermine their claims that they are worried about "bad actors getting there first".

If you were a scientist working on the Manhattan project because you were worried about Hitler getting the atomic bomb first, you wouldn't send your research on centrifuge design to german research scientists. Yet every company that claims they are more likely than other groups to create safe AGI continues to publish the blueprints for creating AGI to the open web.

Is there any actual justification for this other than "The prestige of getting published in top journals makes us look impressive?"

Incorrect: OpenAI is not aware of the risks of race dynamics.

OpenAI's Charter contains the following merge-and-assist clause: "We are concerned about late-stage AGI development becoming a competitive race without time for adequate safety precautions. Therefore, if a value-aligned, safety-conscious project comes close to building AGI before we do, we commit to stop competing with and start assisting this project. We will work out specifics in case-by-case agreements, but a typical triggering condition might be “a better-than-even chance of success in the next two years.”"
 

Being worried about race dynamics and then stopping at the last minute makes sense and seems a lot better than nothing. But I'm confused why this understanding doesn't propagate to other beliefs/actions. 

Specifically, below are some confusions I have with OpenAI's worldview. If answered, these could give me a lot more hope in OpenAI's direction. 

1. How will you know that AGI has a >50% chance of success in the next two years? MIRI certainly seems to think this is hard. 
2. How does OpenAI leadership feel about accelerating timelines? ^[1]
3. What are OpenAI leadership's timelines right now? What are these timelines based off of? 
4. Does OpenAI retroactively think that publishing that GPT-3 worked was a mistake? ^[2][3]
5. Will OpenAI publish GPT-4? what are the factors driving this decision? 

1. ^{^}

   On my models, we want to know as much about alignment as possible before we get close to AGI, and so it is incredibly important to have as much time as possible before we are close to AGI. I would much rather live in the world where we had 20 years to solve AI alignment than the world where we only have 10.  

2. ^{^}

   If the benefits are using GPT-3 to do alignment research, why not give it to just alignment researchers, and not tell anyone else? 

3. ^{^}

   Again, according to my current worldview, actions such as releasing GPT-3 are extremely negative, because it tells everyone that LLMs work and thus accelerates capabilities and therefore also shortens timelines. 

merge-and-assist clause [...] we commit to stop competing with and start assisting this project

So, if you don't think AI should be open (because that looks dangerous), has anyone considered just ... changing the name? (At least, the name of the organization, even if the "OpenAI API" as a product has the string openai embedded in the code too much.) Yeah, it's inconvenient, but ... Alphabet did it! Meta did it! If you're trying to make the most important event in the history of life go well, isn't it worth a little inconvenience to be clear about what that entails?

Incorrect: OpenAI is not aware of the risks of race dynamics.

I don't think this is a common misconception. I, at least, have never heard anyone claim OpenAI isn't aware of the risk of race dynamics—just that it nonetheless exacerbates them. So I think this section is responding to a far dumber criticism than the one which people actually commonly make.

Alignment research: 30

Could you share some breakdown for what these people work on? Does this include things like the 'anti-bias' prompt engineering?

The fault does not lie with Jacob, but wow, this post aged like an open bag of bread.

Correct: OpenAI is trying to directly build safe AGI.

OpenAI's Charter states: "We will attempt to directly build safe and beneficial AGI, but will also consider our mission fulfilled if our work aids others to achieve this outcome." OpenAI leadership describes trying to directly build safe AGI as the best way to currently pursue OpenAI's mission, and have expressed concern about scenarios in which a bad actor is first to build AGI, and chooses to misuse it.

You seem confused about the difference between "paying lip service to X" and "actually trying to do X".

To be clear, this in itself isn't evidence against the claim that OpenAI is trying to directly build safe AI. But it's not much evidence for it, either.

Correct: the majority of researchers at OpenAI are working on capabilities. 

Researchers on different teams often work together, but it is still reasonable to loosely categorize OpenAI's researchers (around half the organization) at the time of writing as approximately:

• Capabilities research: 100
• Alignment research: 30
• Policy research: 15

I'd guess that is an overestimate of the number of people actually doing alignment research at OpenAI, as opposed to capabilities research in which people pay lip service to alignment. In particular, all of the RLHF work is basically capabilities work which makes alignment harder in the long term (because it directly selects for deception), while billing itself as "alignment". EDIT: I have been convinced that I was wrong about this, and I apologize. I still definitely maintain that RLHF makes alignment harder and is negative progress for both outer and inner alignment, but I have been convinced that the team actually was trying to solve problems which kill us, and therefore not just paying lip service to alignment. See comment here [LW(p) · GW(p)] and the thread leading up to it for the information which changed my mind.

“OpenAI leadership tend to put more likelihood on slow takeoff”

Could you say more about the timelines of people at OpenAI? My impression was that they’re very short and explicitly include the possibility of scaling language models to AGI. If somebody builds AGI in the next 10 years, OpenAI seems like a leading candidate to do so. Would people at OpenAI generally agree with this?

I am very curious how you think about this post in retrospect: parts of it seem clearly falsified. I completely understand if you currently feel bound by a non-disparagement clause and expect it to be a few weeks before that can be confirmed to no longer apply. 

I would expect OpenAI leadership to change their mind on these questions given clear enough evidence to the contrary.

Why do you expect this? For what sorts of evidence do you expect? What do you suppose they think of arguments about inner alignment, orthogonality, deceptive alignment, FOOM, sharp-left-turn?

Since this post was written, OpenAI has done much more to communicate its overall approach to safety, making this post somewhat obsolete. At the time, I think it conveyed some useful information, although it was perceived as more defensive than I intended.

My main regret is bringing up the Anthropic split, since I was not able to do justice to the topic. I was trying to communicate that OpenAI maintained its alignment research capacity, but should have made that point without mentioning Anthropic.

Ultimately I think the post was mostly useful for sparking some interesting discussion in the comments.

I would be very curious to see your / OpenAI's responses to Eliezer's Dimensions of Operational Adequacy in AGI Projects [LW · GW] post. Which points do you / OpenAI leadership disagree with? Insofar as you agree but haven't implemented the recommendations, what's stopping you?

Thanks again for writing this.
A few thoughts:

I think that the release of GPT-3 and the OpenAI API led to significantly increased focus and somewhat of a competitive spirit around large language models... I don't think OpenAI predicted this in advance, and believe that it would have been challenging, but not impossible, to foresee this.

Do you believe any general lessons have been learned from this? Specifically, it seems a highly negative pattern if [we can't predict concretely how this is likely to go badly] translates to [we don't see any reason not to go ahead].

I note that there's an asymmetry here: [states of the world we like] are a small target. To the extent that we can't predict the impact of a large-scale change, we should bet on negative impact.

OpenAI leadership tend to put more likelihood on slow takeoff, are more optimistic about the possibility of solving alignment, especially via empirical methods that rely on capabilities, and are more concerned about bad actors developing and misusing AGI...

Questions:

1. If we're in a scenario with [slow takeoff], [alignment is fairly easy], and [empirical, capabilities-reliant approaches work well], wouldn't we expect alignment to get solved by default without OpenAI? Why is this the scenario to focus on?
2. If the concern is with bad actors, and OpenAI is serious about avoiding race conditions, why not enact the merge-and-assist clause now? Why not join forces with DeepMind now? Would this be negative? If so, why? Would it simply be impractical? Then on what basis would we expect it to be practical when it matters?

OpenAI's particular research directions are driven in large part by researchers

If an organisation's research efforts are largely driven by researchers, then the key question becomes: what is the organisation doing to ensure the creation/selection of the right kinds of alignment researchers? (the actually-likely-to-help-solve-the-problem kind)

If there were talented enough researchers who wanted to lead new alignment efforts at OpenAI, I would expect them to be enthusiastically welcomed by OpenAI leadership.

To the extent that the alignment problem is hard this seems negative: it's possible to be extremely talented, yet heading in a predictably wrong direction - orthogonality isn't just for AIs. In order to be net-useful, an organisation would need to select strongly for [is likely to head in an effective direction].

Could you explain the rational behind the "Open" in OpenAI? I can understand the rational of trying to beat more reckless companies to achieving AGI first (albeit, this mentality is potentially extremely dangerous too), but what is the rational behind releasing your research? This will enable companies that do not prioritize safety to speed ahead with you, perhaps just a few years behind. And, if OpenAI hesitates to progress, due to concerns over safety, the more risk-taking orgs will likely speed ahead of OpenAI in capabilities. The bottomline is I'm concerned your efforts to achieve AGI might not do much to ensure an aligned AGI is actually created, but instead only speed-up the timeline toward achieving AGI by years or even decades.

I also appreciated reading this.

Thanks for writing this! I agree with most of the claims you consider to be objective, and appreciate you writing this up so clearly.

Thank you very much for writing this post, Jacob. I think it clears up several of the misconceptions you emphasize.

I generally seem to agree with John that the class of problems OpenAI focuses on might be more capabilities-aligned than optimal but at the same time, having a business model that relies on empirical prosaic alignment of language models generates interesting alignment results and I'm excited for the alignment work that OpenAI will be working on!

If I was trying to secure technology that could cause human extinction I think I would offer cash bounties for responsibly disclosed security vulnerabilities.

Manifold bettors are also skeptical that the merge-and-assist clause will fire soon, though I suppose that might be driven by takeoff speeds. But there might be an announcement of something.

Not sure what else we should be tracking.

The Partnership may never make a profit

I couldn't find this quote in the page that you were supposedly quoting from. The only google result for it is this post. Am I missing something?

I might add the most glaring misconception, at least for me in the early days... I assumed their primary goal was to support Open Source AI, and would "default to open" on all their projects. Instead orgs like HuggingingFace expend significant resources reverse engineering the AI papers and models that OpenAI releases.