I think the cool thing that could come out of this post is "a bit more consensus about the social value of working various places." For example, if it turns out that everyone thinks "probably you shouldn't work at a tobacco company", then maybe fewer people will go there accidentally, and the people who are at those companies might think harder about their choices.

But I worry that this post is set up in a way that leads to polarization instead of consensus; if it is cheap to say "working at company X is probably bad/evil/etc." and it is expensive to say the opposite (so people don't respond or get into the discussion), then we would end up with a false consensus among LW readers that various companies are bad, and a false consensus among people at various companies that LW isn't worth engaging with (and multiple disagreeing false consensuses seem like the typical form of polarization to me).

So I think it'd be worth putting some meta thought into the question of how we could make this post a place where people who are optimistic about any of the mentioned companies feel hopeful about sharing their opinions and reaching consensus.

[There is, of course, a background fact that it is generally harder to argue for incorrect positions than correct positions, and someone's fear about getting into an argument about something may really be their fear that they're incorrect. I think it is a mistake to assume that lack of willingness to engage is generally of that form, rather than first checking that in fact you haven't make any mistakes that could lead to disconnection of information flow or a sensible lack of hope in the process.]

IMO most of the jobs where the employer says the job is about AI alignment or AI safety, e.g., the alignment team at OpenAI, is actually contributing to AI danger.

More generally, I humbly suggest erring on the side of safety: avoid contributing to not just AI --except for MIRI and maybe 1 or 2 other outfits -- but also anything AI needs, such as hardware, operating systems, compilers, version-control tools, issue-trackers and development methodologies -- but only if doing so does not put you at a significant personal disadvantage relative to people with your skills who do not care about AI safety.

If for example, you have invested 3 years of your working life in getting good a compiler development specifically, and you are not yet bored with compiler development, then the right move is to continue to work on compiler development if that is the most efficient way for you to earn money because having a higher income makes a person more effective and influential and we want the people concerned enough about about AI safety to read this thread of conversation to be effective and influential.

215 years ago in France, if your father was not part of the nobility, there were few opportunities for a young man to achieve status and material security. For most young men, the best way to achieve those things was to join Napoleon's Army, but of course Napoleon's Army was spreading death and destruction across Europe. If you know or suspect that AI research (and research and development of any computing infrastructure that supports it) is today's version of Napoleon's Army and you care about the global consequences of your actions, the right move is not to join Napoleon's Army in the hopes that you will be able to use your insider position to limit the death and destruction. The right move is to choose not to join Napoleon's Army (and to avoid spending any of your time and energy acquiring skills and knowledge whose main market is Napoleon's Army -- and to consider joining the movement to get Napoleon's Army banned by the governments of the developed world if you think that movement has a chance of succeeding).

In contrast to 215 years ago in France, choosing not to join today's version of Napoleon's Army is not even much of a personal sacrifice because of the diversity and general abundance of today's economy.

I suspect my advice is the exact opposite of they Less Wrong/EY consensus, so here goes:

Choose to work at whatever company will allow you personally to get as good at AI/Machine learning as possible.

This is a restatement of my advice at the end of my essay on AI Alignment [LW(p) · GW(p)].  Specifically, the two strategies I am the most optimistic about, Game Theory and The Plan [LW · GW] both depend on very smart people becoming as wise as possible before the Singularity comes.

From a game-theory point of view, advancing AI knowledge in general is a tragedy of the commons.  It would require coordination from everyone all at once in order to stop advancing AI beyond the danger level (whatever that might be).  And it isn't even possible to know if a particular field (compilers, formal mathematical methods, hardware improvement, AI art) will be the one that puts us over the top.  That means there is very little benefit for you not to work on advancing AI (and it comes at a huge cost, since you basically have to give up on any career even tangentially related to technology).

On the other hand AI Alignment is likely to be solved by a "small group of thoughtful individuals".   Increasing your skills proportionally increases your chance of being a member of this category (since it seems like you already care about the topic).

One way to think about this advice is: every day Google, Open AI, Hugging Face, and 1000 other companies are hiring someone and that someone will likely work to advance AI.  If we imagine the marginal case where a company is deciding between hiring you and someone slightly less concerned about AI alignment.  Wouldn't you rather they hire you?  

Note that this advice does not mean you get to leave your ethics at the door.  Quite the opposite, if you are working somewhere and it turns out they are doing something egregious stupid (like deploying a non-airgapped AI), it is your duty to do everything in your power to stop them.  Complain to your boss, leak information to the press, chain yourself to the server.  Whatever you do, do not become the engineer who warned about disaster but then quietly shrugged when pressured by management.  But if you refuse to take any jobs related to AI, you won't even be in the room when the disaster is about to happen.  And on the margin, you should assume that somebody worse will be.

Mod note: I activated two-axis voting on this post, since I feel like that might help with this discussion going better.

I think decision-making relies on counterfactuals, and it also seems good to have a sense of what counterfactuals we're considering here, or what the underlying model of AGI production is.

I think the main thing that I'm hoping for is a separation out of 'purity ethics' and 'expected value maximization', or something, or the purity ethics is explicitly grounded in a FDT-style "if people running good decision theory simply decline to coordinate with evil systems, those evil systems will be disadvantaged" rather than just a "this seems icky." For example, you might imagine researchers or engineers signing something like an "adequacy pledge" based off of Eliezer's guidelines [LW · GW], where they commit to only working to orgs that are adequate along all six dimensions (or to the top current org if no orgs are adequate, tho that version seems worse), and now orgs can see how much talent cares about that sort of thing.

I wrote some basic thoughts [LW(p) · GW(p)] about this a while ago in response to a similar question [LW(p) · GW(p)]; roughly speaking, I think the 'total effects' of interventions are not obvious even if the 'direct effects' are obvious. I think there's value in trying to figure out why people are motivated and what sorts of 'gains from trade' are possible.

As a specific example, DeepMind has made public claims of the form "once we get close enough to AGI, we'll slow down and move carefully." But imagine being the person at DeepMind in charge of making the lever such that DeepMind leadership can pull it to slow down, without immediately shedding all of their employees that want to keep moving at full speed, or other sorts of evaporative cooling [LW · GW]. What work can be done now to set that up? What work can be done now to figure out the benefits and costs of pulling that lever at various times? [And, as a "checking if people's words line up with their behaviors", at a company DeepMind's size there really needs to be at least one FTE who is preparing for a project of that size. Does such a person exist? Can we help DeepMind hire them if not so?]

Relevant. [LW · GW]

Deep mind in general: wdyt?

The worst thing you could possibly do is work for the capabilities section of existing AGI enterprises like Google Brain, DeepMind or OpenAI. This includes, obviously, the "AI alignment" companies that really just do capabilities research, and does not include the sections within these companies that do genuine alignment research. Dan Heydricks has an excellent sequence here [? · GW] on how not to fuck this up. Use your critical thinking and and ask simple questions to find out which position is which. 

The second worst thing in terms of expected impact would be to work at or support pioneering ML  research at a general company like Facebook's division, that isn't necessarily explicitly trying to engineer AGI but effectively involves a dayjob of burning the capabilities commons.

Below that would be to work on straightforward ML tooling that has generalist applications; things like working on frameworks (PyTorch, wandb.ai, etc.), computer hardware designed explicitly for ML, or for companies like Scale.

Somewhere deep below that is making money for or investing in the parent companies that pioneer these things (Facebook, Microsoft, Google). Depending on specifics you can lump in certain more general types of computer engineering work here.

After that though, I think if you just donate a reasonable fraction of your income to charity, or AI alignment enterprises, you're probably net positive. It's really not that complicated: if you're making or contributing to research that pushes the boundary of artificial intelligence, then... stop doing that.

Subscribe to this comment to get notified about questions about other companies.

Do not reply to this comment please

Next Silicon: They make chips for super computers that are not optimized for neural networks