The Alignment Community Is Culturally Broken

I am (was) an X% researcher, where X<Y. I wish I had given up on AI safety earlier. I suspect it would've been better for me if AI safety resources explicitly said things like "if you're less than Y, don't even try", although I'm not sure if I would've believed them. Now, I'm glad that I'm not trying to do AI safety anymore and instead I just work at a well paying relaxed job doing practical machine learning. So, I think pushing too many EAs into AI safety will lead to those EAs suffering much more, which happened to me, so I don't want that to happen and I don't want the AI Alignment community to stop saying "You should stay if and only if you're better than Y".

Actually, I wish there were more selfish-oriented resources for AI Alignment. Like, with normal universities and jobs, people analyze how to get into them, have a fulfilling career, earn good money, not burn out, etc. As a result, people can read this and properly analyze if it makes sense for them to try to get into jobs or universities for their own food. But with a career in AI safety, this is not the case. All the resources look out not only for the reader, but also for the whole EA project. I think this can easily burn people.

(Disclaimer: The Law of Equal and Opposite Advice probably applies to some people here.)

I think that by default, people should spend more time doing things, and less time worrying about if they're smart enough to do things. Then, if they don't enjoy working they should 100% feel free to do something else!

That is, try lots of things, fail fast, then try something else. 

Here's some reasons why:

• As you say, people can improve at things. I think this is less of a factor than you do, but it's probably still underrated on the margin. Taking time to study math or practice coding will make you better at coding; while effort is no guarantee of success, it's still generally related. 
• People can be pretty bad at judging aptitude. I think this is the main factor. I've definitely misjudged people's aptitude in the past, where people I've thought were pretty competent turned out to not so and vice versa. In general, the best predictor of whether someone can or cannot do a thing is seeing if they can do the thing. 
• Aptitude is often context-dependent. People are often more-or-less competent depending on the context. PhD programs and depressing jobs are two common factors that make people seem significantly less competent. Antidepressants also exist, and seem to be a contributing factor in helping people be more functional.
• It's healthier to try than to worry. I think people often end up sad and unmotivated because they feel incompetent before they've put in a solid effort to do good things. Empirically, focusing too much on feelings of inferiority is really bad for productivity and momentum; I've often found that people can do many more things when they put in some effort than when they're beating themselves up. 

Overall, I think people should lean more toward work trials/work trial tasks (and being more okay with not being hired after work trials). I also think we should have more just do the thing energy and more let's move on to the next thing energy, and less being stuck worrying about if you can do the thing energy. 

==========

I do think the AI alignment community should try to be slightly more careful when dismissing people, since it can be quite off-putting or discouraging to new people. It also causes people to worry more about "does someone high status think I can do it?" instead of "can I actually do it?". This is the case especially when the judgments might be revised later. 

For what it's worth, "you're not good enough" is sometimes stated outright, instead of implied. To share a concrete example: in my first CFAR workshop in 2015, I was told by a senior person in the community that I was probably not smart enough to work on AI Alignment (in particular, it was suggested that I was not good enough at math, as I avoided math in high school and never did any contest math). This was despite me having studied math and CS for ~2 hours a day for almost 7 months at that point, and was incredibly disheartening.Thankfully, other senior people in the community encouraged me to keep trying, which is why I stuck around at all. (In 2017, said senior person told me that they were wrong. Though, like, I haven't actually done that much impressive alignment research yet, so we'll see!)

Insofar as we should be cautious about implying that people are not good enough, we should be even more cautious about directly stating this to people. 
 

There's something deeply discouraging about being told "you're an X% researcher, and if X>Y, then you should stay in alignment. Otherwise, do a different intervention." No other effective/productive community does this. (Emphasis added.) I don't know how to put this, but the vibes are deeply off. 

I think this is common, actually.

We apply the logic of only taking top people to other areas. Take medicine. The cost of doing medicine badly is significant, so tons of filters exist. Don't do well in organic chemistry? You can't be a doctor. Low GPA? Nope. Can't get a pretty good score on the MCAT? No again. Get into med school but can't get an internship? Not gonna be able to practice.

It's similar for many other high-stakes fields. The US military has a multi-decade long weeding out process to end up as a general. Most corporations effectively do the same. Academic research is brutal in similar ways. All of these systems are broken, but not because they have a filter, more because the filter works poorly, and moral mazes, etc.

Alignment work that's not good can be costly, and can easily be very net negative. But it's currently mostly happening outside of institutions with well-defined filters. So I agree that people should probably try to improve their skills if they want to help, but they should also self filter to some extent.

Our community seems to love treating people like mass-produced automatons with a fixed and easily assessable "ability" attribute.

Have you considered the implied contradiction between the "culturally broken community" you describe and the beliefs - derived from that same community - which you espouse below?

I was crying the other night because our light cone is about to get ripped to shreds. I'm gonna do everything I can to do battle against the forces that threaten to destroy us.

Your doom beliefs derive from this "culturally broken community" - you probably did not derive them from first principles yourself. Is it broken in just the right ways to reach the correct conclusion that the world is doomed - even when experts knowledgeable in the relevant subject matters that would actually lead to doom find this laughable?

Consider a counterfactual version of yourself that never read HMPOR, the sequences, and LW during the critical periods of your upper cortical plasticity window, and instead spent all that time reading/studying deep learning, systems neuroscience, etc. Do you think that alternate version of yourself - after then reviewing the arguments that lead you to the conclusion that the "light cone is about to get ripped to shreds" - would reach that same conclusion? Do you think you would you win a debate against that version of yourself? Have you gone out of your way to read the best arguments of those who do not believe "our light cone is about to get ripped to shreds"? The sequences are built on a series of implicit assumptions about reality, some of which are likely incorrect [LW(p) · GW(p)] based on recent evidence.

One distinction I want to make here is between people who are really excited to work on AI Alignment (or, any particular high-impact-career), and who are motivated to stick with it for years (but who don't seem sufficiently competent), vs people who are doing it out of a vague sense of obligation, don't feel excited (and don't seem sufficiently competent).

For the first group, a) I can imagine them improving over time, b) if they're excited about it and find it fulfilling, like, great! It's the second group I feel most worried about, and I really worry about the vague existential angst driving people to throw themselves into careers they aren't actually well suited for. (and for creative research I suspect you do need a degree of enthusiasm in order to make it work)

A quote from Eliezer's short fanfic Trust in God, or, The Riddle of Kyon that you may find interesting:

Sometimes, even my sense of normality shatters, and I start to think about things that you shouldn't think about. It doesn't help, but sometimes you think about these things anyway.

I stared out the window at the fragile sky and delicate ground and flimsy buildings full of irreplaceable people, and in my imagination, there was a grey curtain sweeping across the world. People saw it coming, and screamed; mothers clutched their children and children clutched at their mothers; and then the grey washed across them and they just weren't there any more. The grey curtain swept over my house, my mother and my father and my little sister -

Koizumi's hand rested on my shoulder and I jerked. Sweat had soaked the back of my shirt.

"Kyon," he said firmly. "Trying to visualize the full reality of the situation is not a good technique when dealing with Suzumiya-san."

How do you handle it, Koizumi!

"I'm not sure I can put it in words," said Koizumi. "From the first day I understood my situation, I instinctively knew that to think 'I am responsible for the whole world' is only self-indulgence even if it's true. Trying to self-consciously maintain an air of abnormality will only reduce my mind's ability to cope."

Also: I agree that people who want to do alignment research should just go ahead and do alignment research, without worrying about credentials or whether or not they're smart enough. On a problem as wickedly difficult as alignment, it's more important to be able to think of even a single actually-promising new approach than to be very intelligent and know lots of math. (Though even people who don't feel they're suited for thinking up new approaches can still work on the problem by joining an existing approach.)

The linked post talks about the large value of buying even six months of time, but six months ago it was May 2022. What has been accomplished in AI alignment since then? I think we urgently need to figure out how to make real progress on this problem, and it would be a tragedy if we turned away people who were genuinely enthusiastic about doing that for reasons of "efficiency". Allocating people to the tasks for which they have the most enthusiasm is efficient.

I don't really see the why for your assertions in your post here. For example:

It's not nice to be in a community that constantly hints that you might just not be good enough and that you can't get good enough.

Ok, it's not nice. Its understandable that many people don't want to think they're not good enough. But if they truly are not good enough then the effort they spend towards solving alignment in a way they can't contribute towards doesn't help solving alignment. The niceness of the situation has little bearing on how effective the protocols are.

If we want to actually accomplish anything, we need to encourage people to make bigger bets, and to stop stacking up credentials so that fellow EAs think they have a chance

Ok, this is a notion, maybe it's right, maybe it's not, I'm just not getting much from this post telling me why it's right.

Are you sure you want to drop out of Stanford?

You will have signifantly more prestige, capability to choose your own future, and leverage with that degree. And depending on your degree subject, some very useful skills.

I don't know your situation, but I recommend getting viewpoints from off this forum. Relatives, uni student support services, etc. This may be a symptom, not a cause. Harsh, but potentially helpful.

I want to address the central flaw of Akash+Olivia+Thomas's argument in the Buying Time post, which is that actually, people can improve at things.

Why do you think the Buying Time post suggested otherwise? Note that the authors did not make a blanket statement telling "bad" alignment researchers to stop working on alignment. One recommendation is for alignment researchers to consider working on technical research that concretely presents important alignment concepts [LW · GW] or empirically demonstrates concerning alignment failures [LW · GW].

I was crying the other night because our light cone is about to get ripped to shreds. I'm gonna do everything I can to do battle against the forces that threaten to destroy us.

If you find yourself in a stereotypically cultish situation like that, consider that you might have been brainwashed, like countless others before you, and countless others after you. "This time is different" is the worst argument in the world, because it is always wrong. "But really, this time the AI/Comet/Nibiru/... will kill us all!" Yeah. Sure. Every time.

Upvoted. 

Also a thing about that post which struck me as sort of crazy, and maybe I should have commented, was that it seemed to model things as like, there's different types of people; different types of people work on different things; you should figure out which type you are using thresholds and such. In particular this idea is silly because someone can do more than one thing in their life. On the margin, "buy time people" should be spending more time thinking about object-level technical alignment, more so than "technical alignment people" should be spending more time thinking about buying time.

[fixed link to Habryka's comment]

I wasn't aware of the "you're < Y%" vibe. At least not explicitly. I felt that my edge wasn't with the advanced math early AI Safety work MIRI was doing. Maybe I'm not sensitive enough for the burn-out dynamics, but more likely, I wasn't exposed much to it, me being from Germany. I'm also pretty resilient/anti-fragile. 

Anyway, I wanted to chime in to say that even if you are objectively below Y% on hard alignment math skill, there are so many areas where you can support AI Alignment, even as a researcher. This may not have been the case in the early days but is now: AI Alignment teams need organizers, writers, managers, cyber security professionals, data scientists, data engineers, software engineers, system administrators, frontend developers, designers, neuroscientists, psychologists, educators, communicators, and more. We no longer have these small, closely-knit AI teams. With a growing field, specialization increases. And all of these skills are valuable and likely needed for alignment to succeed and should get recognition. 

Why do I say so? I have been in organizations growing from a handful to a few hundred people all my life. I'm in one such company right now and in parallel, I'm running the operations of a small AI Alignment project [LW · GW]. I like being a researcher, but out of need, I take care of funding, outreach, and infrastructure. In bigger teams or the community at large, I see a need for the following: 

• Organizers for running events or small team daily work, as simple as coordinating meeting invites. Without this, teams often fall apart. Shared community helps with Common Good Commitment.
• Writers who write summaries, essays, grant applications, and blog posts, whether technical, social, or organizational. And who promote Alignment Mindset. Where do you think LW posts come from?
• Managers, no, not moral maze middle managers, but people who find the right people to work together and make sure work goes smoothly. Also, who establish Trustworthy Command and Research Closure. Not sure what else you call that role. 
• Cybersecurity professionals who secure the sensitive core of a project matching Strong Opsec as the project grows
• Data scientists and data engineers who, well, deal with the amount of data that inevitably goes into and out of projects that have to deal with the real world earlier or later. Also, who maintain Research Closure.
• Software engineers to build all the glue stuff between websites, build processes, data exchange, SaaS parts, and whatnot. 
• System administrators, dito, provisioning and administering the above while maintaining Strong Opsec. 
• Frontend developers - you want to see the results - graphs, data, videos, right? 
• Designers - and it should be easy to use and understand.
• Neuroscientists, cognitive scientists, psychologists, psychiatrists, therapists, and pedagogues who provide hard evidence about the biological, psychological, or cognitive plausibility of models or behaviors of hypothetical or simulated agents or who can suggest directions to explore or validate.    
• Educators and communicators who reach out to interested parties and who promote Alignment Mindset and Common Good Commitment.
• Accountants, finance professionals, financial advisors, and of course, sponsors who help establish, maintain and monitor Requisite Resource Levels.
• Lawyers who set up a legal entity that aspires to Trustworthy Command, Research Closure, and Common Good Commitment.
• Everybody else who can help by affirming the Common Good Commitment thus creating common knowledge that this is a worthy cause.

In the above, Trustworthy Command, Research Closure, Strong Opsec, Common Good Commitment, Alignment Mindset, Requisite Resource Levels refer to the Six Dimensions of Operational Adequacy in AGI Projects [LW · GW].

Hi, thanks for writing this. Sorry to hear that things are hard. I would really like if you can help me to understand these points:

A few days later, I saw this post [LW · GW]. And it reminded me of everything that bothers me about the EA community. Habryka covered [LW(p) · GW(p)] the object level problems pretty well, but I need to communicate something a little more... delicate.

 What bothers you about the EA community specifically? At times, I am not sure if you are talking about the EA community, the AIS technical research community, the rationalist community or the Berkeley AIS community? I think of them all as being very different.

I want to address the central flaw of Akash+Olivia+Thomas's argument in the Buying Time post, which is that actually, people can improve at things.

I feel I don't properly understand what you think of this argument and why you think it is flawed. 

If a fantastic programmer who could prove her skills in a coding interview doesn't have a degree from an elite college, could she get a job in alignment?

As someone who's been feeling a similar portfolio cocktail of emotions about alignment work under short timelines [LW · GW]: thank you.