The Seeker’s Game – Vignettes from the Bay

post by Yulia · 2023-07-09T19:32:58.717Z · LW · GW · 19 comments

Contents

  Introduction
  Personal Insecurities and Alienation
    Dumb Questions
    The Wrong Kind of Weird
  Attitudes towards Doom
    An Oasis of Sanity
    Flinching Away
    Clear-Eyed Despair
  Attitudes towards the EA Sphere
    Hard to Own-Up, Hard to Criticise
    Is EA bad, actually?
    The Dangers of EA
  Status Dynamics and Social Signalling
    Value and Proxies
    Distrust
    The Seeker's Game
    Belonging vs Competition
  Self-Censorship in Social Contexts
    Social Justice Conversations
    Delicate Topics
    To Gossip or Not to Gossip
    I won’t start a Culture War
    High Variance
  Self-Censorship in Professional Contexts
    “Declaring Loyalty”
    The Temptation of Legibility
    A Game of Diplomacy
    Tedious and Tiring 
    Team Player
    Preformal Ideas
None
19 comments

Introduction

Last year, one conversation left a lasting impression on me. A friend remarked on the challenges of navigating "corrupting forces" in the Bay Area. Intrigued by this statement, I decided to investigate the state of affairs in the Bay if I had the chance. So when I got the opportunity to visit Berkeley in February 2023, I prepared a set of interview questions. Can you share an experience where you had difficulty voicing your opinion? What topics are hard to clearly think about due to social pressures and factors related to your EA community or EA in general? Is there anything about your EA community that makes you feel alienated? What is your attitude towards dominant narratives in Berkeley? ^[1] In the end, I formally interviewed fewer than ten people and had more casual conversations about these topics with around 30 people. Most people were involved in AI alignment to some extent. The content for this collection of vignettes draws from the experience of around ten people. ^[2] I chose the content for the vignettes for one of two reasons – potential representativity and potential extraordinariness. I hypothesized that some experiences represent the wider EA Berkeley community accurately. Others, I included because they surprised me, and I wanted to find out how common they are. All individuals gave me their consent to post the vignettes in their current form. 

How did I arrive at these vignettes? It was a four-step process. First, I conducted the interviews while jotting down notes. For the more casual conversations, I took notes afterwards. The second step involved transcribing these notes into write-ups. After that, I obscured any identifying details to ensure the anonymity of the interviewees. Lastly, I converted the write-ups into vignettes by condensing them into narratives and honing in on key points while trying to retain the essence of what was said.

I tried to reduce artistic liberties by asking participants to give feedback on how close the vignettes were to the spirit of what they meant (or think they meant at the time). It is worth noting that I bridged some gaps with my own interpretations of the conversations, relying on the participants to point out inaccuracies. By doing that, I might have anchored their responses. Moreover, people provided different levels of feedback. Some shared thorough, detailed reviews pointing out many imprecisions and misconceptions. Sometimes, that process spanned multiple feedback cycles. Other participants gave minimal commentary.

Because I am publishing the vignettes months after the conversations and interviews, I want to include how attitudes have changed in the intervening period. I generalised the attitudes into the following categories:

• Withdrawn endorsement (Status: The interviewee endorsed the following content during the interview but no longer endorses it at the time of publication.)
• Weakened endorsement (Status: The interviewee has weakened their endorsement of the following content since the interview.)
• Unchanged endorsement (Status: The interviewee maintains their endorsement of the following content, which has remained unchanged since the interview.)
• Strengthened endorsement (Status: The interviewee has strengthened their endorsement of the following content since the interview.)

I clustered the vignettes according to themes so it's easier to navigate them. Classifying them was difficult because many vignettes addressed overlapping topics. Especially, Self-Censorship in Social Contexts and Self-Censorship in Professional Contexts seem to intersect in intricate ways. I might reclassify the vignettes in the future.

What remains uncertain is how representative these vignettes are. I am keen to uncover this. So if you found this work valuable and wish to support it further, consider participating in the accompanying poll (particularly if you’re involved in the Berkeley community!). If you are interested in participating in potential future blog posts, please feel free to complete this form. Feel free to send me a direct message on LessWrong if you have suggestions for other relevant spaces to investigate!

I thank all participants who have shared their stories and experiences with me! Your willingness to engage in open dialogue, provide reviews, and answer all my questions has been instrumental in creating this vignette collection. I am particularly grateful for your patience throughout the feedback cycles. Thank you for your trust and collaboration!

Personal Insecurities and Alienation

Dumb Questions

Status: The interviewee has weakened their endorsement of the following content since the interview.

Hamlet is a young AI safety researcher. He vigilantly avoids asking stupid questions. His reason? The rampant elitism he perceives in the Berkeley community. Of course, that’s just the flip side of a great thing. A space like this, with tons of driven and talented people, allows for advanced intellectual conversations and remarkable collaborations. The filters maintaining the quality will exclude some who want to be there. Others, even when invited in, will be intimidated. Hamlet is one of those who get intimidated sometimes. Being associated with stupid questions is very uncomfortable for him. His primary concern lies in the second-order consequences. If others perceived him as dumb, they would dismiss him. He came to Berkeley to find mentors and collaborators, not to be ignored! So better lie low, occasionally drop sophisticated remarks – and don't be caught asking dumb questions.

The Wrong Kind of Weird

Status: The interviewee maintains their endorsement of the following content, which remains unchanged since the interview.

Bay Area rationalists have a penchant for a specific brand of weird ideas – at least in Adrian's experience. Sadly, he is often intrigued by topics whose weirdness rationalists do not appreciate. Adrian enjoys thinking about moral realism and consciousness, for example. What kind of structures could be conscious? Is there something like a global consciousness? After collecting weird looks at parties and being dismissed for bringing those topics up, Adrian has become more reserved. Now he has less trouble fitting in. Another person like him would probably have difficulty gaining acceptance and entry into the community. And navigating the differences in thinking style still poses a challenge.

Adrian develops his ideas through free association and creative, open exploration. Stream-of-consciousness writing works well for him. He likes using parables, metaphors, examples and anecdotes to express meaning. Because his approach does not conform to established epistemic norms on LessWrong, Adrian feels pressure to cloak and obscure how he develops his ideas. One way in which this manifests is his two-step writing process. When Adrian works on LessWrong posts, he first develops ideas through his free-form approach. After that, he heavily edits the structure of the text, adding citations, rationalisations and legible arguments before posting it. If he doesn’t "translate" his writing, rationalists might simply dismiss what he has to say.

Attitudes towards Doom

An Oasis of Sanity

Status: The interviewee maintains their endorsement of the following content, which has remained unchanged since the interview.

Romeo is a talented graduate excited to contribute to AI alignment. He comes to Berkeley to participate in an AI safety program. His first week in town feels like finally coming home. For years, Romeo had to deal with people confused by intellectualized tribalism or absorbed in trivial philosophical naval-gazing – individuals who didn’t seem to care whether humanity thrives or dies. Now, finally, Romeo is surrounded by people who actually give a damn. People who speak his language. People who take the possibility of the world ending seriously. In a world of irrationality and madness, Romeo has found an oasis of sanity!

Flinching Away

Status: The interviewee maintains their endorsement of the following content, which remains unchanged since the interview.

Antony's mind becomes clouded when contemplating scenarios with very high p(doom). He returns to them over and over – but again and again, he flinches away. Imagining concrete scenarios of that kind is oppressive, after all. But that's not all. A fear is lingering in the back of his mind. What kind of mindset would set in if he thought through the scenarios? Would he even want to live with such a mentality?

Clear-Eyed Despair

Status: The interviewee maintains their endorsement of the following content, which has remained unchanged since the interview.

Ajax wants to have well-calibrated ideas and intuitions around AI alignment. He wants to work hard – and he doesn't believe he can do that if he lets wishful thinking cloud his mind. So he tries to viscerally understand, no, feel what AI doom would be like. How would he feel if all the people he loved were killed? If all of civilization were destroyed? Ajax wants to let go of all delusional positivity. He wants to tap into the depths of despair so his mind can stay clear and focused.

Attitudes towards the EA Sphere

Hard to Own-Up, Hard to Criticise

Status: The interviewee endorsed the following content during the interview but no longer endorses it at the time of publication.

“The EA community building program is overrated – and quite possibly net-negative” – is what Aaron would like to say when the topic comes up. He doesn’t, though. He did not always see it that way. Aaron used to trust CEA's approach to community building. Before launching his career in AI alignment, he worked on community building himself, together with his friend Timon.

Aaron’s sense of his job was informed by reading between the lines of CEA material. Discussion templates came with the conclusions panel filled in? That probably meant he should set up engaging, seemingly open-ended conversations – while making sure that the group reached the predetermined conclusions. After all, new members should not be trusted with thinking for themselves. CEA had figured things out, and other EAs should be kept in line.

Aaron understood and agreed with many of the EA takes. Others, he tribally supported – even though he didn't fully understand them. He put subtle pressure on members of his local group to support them as well. As time went by, Aaron became more sceptical. Isn’t it much more critical to cultivate truth-seeking in people who want to do good rather than force-feeding them “correct” ideas? Now Aaron regrets manipulating conversations to reach "sanctioned" conclusions. He regrets pressuring others to fall in line.

Even though Aaron thinks that CEA's approach nudges community organisers into some poor behaviours, he feels bad about criticising a system he used to be part of. That would mean coming clean about his mistakes to his friends from the local group. But mostly, Aaron keeps quiet because he doesn’t want to trash his former collaborator’s work. Timon is a good person, and he tries hard.

Is EA bad, actually?

Status: The interviewee maintains their endorsement of the following content, which remains unchanged since the interview.

Fang would like to unflinchingly face the question: “Is EA bad, actually?” Of course, this broad concern drags many children behind it: Has AI alignment been net-negative so far? How should we integrate (sceptical) outside perspectives? What are sketchy assumptions that people make about AI? Thinking about whether the movement is going in the right direction is difficult for Fang. His map was built on writing that, to some extent, has fossilised into AI alignment orthodoxy. It’s hard for him to look at developments in alignment and the community with fresh eyes.

And then there’s a further dimension to these questions. Fang has bought into the movement, socially and professionally. Most of his friends and colleagues are part of EA. If he concluded that he was in the right spot doing his best, it would mean that nothing would change. But what would happen if he understood that his calling lied elsewhere, in a pursuit unrelated to the EA community? Fang fears that his current social circle would only stay superficially involved with him.

Support among EAs only goes so far. They’ll be there for you, stand by you, and some of them will be close friends – as long as you’re part of the community. Or work for a big player in that space. At least, that’s what Fang fears. The idea of doing something else is equivalent to starting over. Fang doesn’t like the possibility of such a conclusion. And he doesn't like that he flinches away from such a possibility. He wants to be more resilient, so he works on finding friends outside EA. Until then, it will probably be hard to think about the question closely.

The Dangers of EA

Status: The interviewee has strengthened their endorsement of the following content since the interview.

Is EA a dangerous movement? Maybe it isn’t. Malcome feels strongly, however, that he should at least ask that question. People around him seem to assume that EA is, by default, built to be altruistic. But should we really expect a group of people with firm ideologies to change the world positively? What are the consequences of a bunch of young, inexperienced people trying to gain influence and power? Should they be the ones who shape the future? These questions are critical, yet they only stir up confusion in his mind. 

The potential implications are disturbing. Still, it’s hard to investigate these questions – they might be insurmountably difficult to answer. And it probably doesn't help that Malcome is invested in the movement. He tries not to think of EA as part of his identity. But at the end of the day, he must acknowledge that his identity is, to some extent, attached to EA. If Malcome gets to the bottom of this, he might want to distance himself from the ideological side of the movement.

Status Dynamics and Social Signalling

Value and Proxies

Status: The interviewee has weakened their endorsement of the content since the interview. At the time of publication, they don't consider the incident described below as representative of the EA Berkeley community.

Ulysses believes that status hierarchies are not inherently problematic. Basing them on people's usefulness for a shared mission can benefit coordination. Problems can arise, however, when people try to exploit those systems. 

Ulysses has witnessed such manipulative tendencies firsthand. Once, while at a party, he was immersed in a discussion with an acquaintance named Kent. Amid their conversation, King Lear, a high-status member of the Berkeley EA community, approached to greet Kent. The two started chatting. Soon, their conversation revolved around acquiring status. That exchange struck Ulysses. It was incredible how casually those influential people talked about applying social tricks to gain power. They underscored the disparity between proxies for doing good and the actual act of doing good. They discussed how to inflate and manipulate these proxies to enhance the appearance of social contribution – without actually increasing it.

Ulysses felt increasingly uneasy as he listened to this exchange. He would have expected people familiar with prisoner's dilemmas and Goodhart's law to be less drawn to exploiting proxies. It felt like they were defecting in a multi-person prisoner’s dilemma! Despite his discomfort, he refrained from interjecting. Speaking up would have felt awkward and out of place since he wasn't directly involved in the discussion. He was but an incidental observer, after all. And what good would it have done to criticise? People like King Lear probably assume that applying such tricks is justified, believing that, as EAs, they will use that underhandedly harvested power for good.

Distrust

Status: The interviewee maintains their endorsement of the following content, which remains unchanged since the interview.

Richard worries that professional decisions and personal relations are highly coupled in EA. The social sphere seems to intersect with object-level conversations and decisions. That observation affects how he interprets interactions with fellow community members. For example, Richard doesn’t always know if he should take people’s word on their life decisions at face value. How much of what they say is social signalling? When he asks about their decisions, people bring up object-level arguments and seem to undervalue or downplay social reasons. That makes it confusing and difficult to navigate social interactions!

The Seeker's Game

Status: The interviewee maintains their endorsement of the following content, which has remained unchanged since the interview.

By chance, Valentine runs into Rosaline, an established professional in Berkeley. They chat for a bit. Rosaline seems to have a favourable first impression of Valentine and invites him to an after-hours office party. Valentine wasn’t even aware that these kinds of parties existed! He decides to attend – more out of curiosity than anything else. He doesn't have high expectations because he doesn't know who will be there. Surprisingly, there are a few high-profile AI alignment researchers in attendance. Valentine gets to talk to some of them. The conversations prove to be engaging and enlightening! They provide perspectives that might help Valentine with his research. Would he have heard of the party if he hadn’t randomly talked to Rosaline? In hindsight, that interaction reminds him of a computer game. It's similar to how interactions with other player characters can unlock new game events and help with game progress. Maybe he should talk to people like Rosaline more often.

Belonging vs Competition

Status: The interviewee maintains their endorsement of the following content, which remains unchanged since the interview.

Brutus is a young AI alignment researcher. He struggles with resolving the tension between social belonging and competition in the Berkeley community. Brutus is firmly rooted in this social circle. At the same time, he competes with fellow community members for limited resources and opportunities. This inner conflict manifests in Brutus being sensitive about not being invited to parties. The sense of competition makes it tempting to build up his social prestige. But that is not something he wants to focus on! He wants to do good alignment work. Brutus feels paranoid about unwittingly steering towards status and prestige – even if he doesn’t necessarily endorse it. Adjacent to that is a nebulous, ill-defined fear of being left behind. What will happen if he opts out while others keep working on their prestige? Will that give them an edge that Brutus can’t compensate for?

Self-Censorship in Social Contexts

Social Justice Conversations

Status: The interviewee has weakened their endorsement of the content since the interview.

Charles hesitates to participate in social justice conversations. Even when he knows specifics about incidents on the scale of sexual assault cases, he feels uncomfortable bringing them up. Determining in advance who will be reasonable in a conversation and who will not is just too difficult. There are some fanatics on the social justice side, after all. What if Charles expressed himself clumsily? Which is not unlikely since he doesn’t know much about the topic, hasn’t thought deeply about it and lacks the subtle tact required for such conversations. If he said something silly once, would people forgive him? Maybe not. Maybe they would cancel him – or at least try to do that – if he said the wrong thing. The concern is not particularly intense. Most likely, nothing would happen. However, the potential downsides are still not worth it.

Delicate Topics

Status: The interviewee maintains their endorsement of the following content, which has remained unchanged since the interview.

Some topics simply ooze bad vibes. Jupiter doesn’t bring them up. Sure, there might be reasons for disturbing the peace. Discussing EA's role in producing SBF-type people could be significant. It could be crucial to talk about the links between social, romantic and professional relationships in EA and how they could give rise to nepotistic behaviour. Is it worth the price, though? Why should Jupiter be the one to ruin the mood? These conversations are downers and simply not fun – not even for him. And how awkward would it be if he accidentally brought this up to someone involved in these issues? Not to mention that Jupiter might simply be overlooking some apparent arguments and end up looking foolish.

He could raise these controversial topics publically and risk awkward, uncomfortable conversations – and heck, maybe even end up accidentally accusing someone! Or he could stick to safe and non-polemical subjects. From Jupiter’s perspective, the choice is obvious.

To Gossip or Not to Gossip

Status: The interviewee maintains their endorsement of the following content, which remains unchanged since the interview.

Hastings is confused. He unmistakably recognizes the need to let people know about bad actors. It can be an integral part of keeping the community healthy. However, there is never a good time for it. Conversations like that just do not come up naturally! Hastings doesn't feel comfortable gossiping about random people to friends. And he doesn't feel comfortable gossiping with people he doesn’t consider a friend. It also feels inappropriate – don’t people deserve some privacy? Still, Hastings tried. Post-FTX, he unpromptedly pointed out individuals he perceived as bad actors. That was a tiring, awkward and uncomfortable ordeal. In the end, he wasn't sure if he did any good.

It’s just as bad when unfamiliar people approach Hastings to gossip. A remote acquaintance once warned him about dating a person he adored. They are of bad character and a poor match for you, the acquaintance said. Then they went ahead and presented evidence supporting their claim. That did nothing but leave Hastings bewildered, confused and vaguely annoyed. No more indulging in gossip for Hastings, that's for sure!

I won’t start a Culture War

Status: The interviewee has strengthened their endorsement of the following content since the interview.

With some regularity, discussions about visions for a future to strive for come up in Berkeley. People seem to agree that it’s a transhumanist future. Well, William rejects this idea. He opposes the implicit plans of what humanity should do after solving X-risks. It’s irritating how this cultural marinade soaks the discourse, slightly tinting the taste of beliefs and decisions. William firmly believes that Berkeley’s shared futuristic tech mythology is not an appropriate foundation for these conversations!

One of the reasons that he doesn’t challenge these ideas loudly is that he can’t offer alternatives on how to ground these discussions. Moreover, he has a lingering frustration about not being taken seriously. But most importantly, he fears starting a culture war about what the future should look like. William is not sure if it’s likely that challenging that shared vision would lead to that. Nevertheless, the significance of these questions pales in comparison to the imminent threats humanity faces. William doesn’t want to risk distracting the community from the most urgent problems.

High Variance

Status: The interviewee has strengthened their endorsement of the content since the interview.

Othello has a friend, Cassio, who works at Fancy AI Lab (FAIL). Othello suspects that the potential outcome of his friend's work has a very high variance – from insignificant to civilization ending bad. He does not press the matter, however. Cassio probably believes that his work is helping. It would be unbearably awkward to tell someone you’re friendly with, "Hey, I think you’re destroying the world!" Othello does not want to go that far when he is uncertain that his assessment is correct! Who knows, maybe it would even endanger the friendship.

Self-Censorship in Professional Contexts

“Declaring Loyalty”

Status: The interviewee endorsed the following content during the interview but no longer endorses it at the time of publication.

Brabantio is a young, not yet established AI alignment researcher. He has a hard time navigating conversations around AI policy. Many people have strong opinions on AI policy that they think are obvious. Brabantio feels that sharing his opinion on the topic would be akin to declaring loyalty to a whole cluster of ideas, part of which he doesn’t agree with or isn’t even aware of. He already has enough to think about – he doesn’t want to add navigating community politics to that! The clever thing is to keep quiet.

The Temptation of Legibility

Status: The interviewee endorsed the following content during the interview but no longer endorses it at the time of publication.

Cleon suspects that legible alignment research is over-prioritised. Of course, legible work has its perks. It can improve research quality, and it feels more motivating to work on a problem when you can track your progress. But there are costs to that. Focusing on legibility might sap the curiosity of the researcher. What Cleon is truly afraid of, however, is legible research diverting attention from more crucial, less tangible work. Cleon observes that there is additional pressure to make your research legible. With growing distrust in the community, it becomes vital that other people can easily evaluate the quality of the research. Additionally, there is more ease in communicating with outside groups like people in finance, academia and capability labs.

Cleon is confused and unsure about the extent of the problem. Is legible research truly overly prioritised? He might be wrong about this. He would like to think more publicly, openly and collaboratively about it. 

He doesn’t bring the topic up, though. Not often, anyway. A vision in his mind holds him back. He imagines bringing up the topic – and seeing his conversation partner's smirk. “Very interesting that you want to do less legible research”, they would say as if to imply that Cleon wants to hide something. What would he do if someone wanted to discredit him like that? Cleon doesn't want to open himself up to such political attacks! So he keeps his confusion to himself.

A Game of Diplomacy

Status: The interviewee has weakened their endorsement of the content since the interview.

Thurio used to work as an alignment researcher for an AI organisation. When people inquire about his former employer, he hesitates to share his genuine opinion. Before he speaks, he usually activates a diplomatic filter. That way, “management was terrible” transforms into mild commentary on subpar leadership.

Why is Thurio holding back? For one, he doesn’t want people asking for advice to walk away with the impression that his former employer is worse than is the case. He doesn’t want to exaggerate the badness. In addition, Thurio doesn’t want people who still work there to learn about him “bad-mouthing” the organisation. He is sceptical that he could find high-impact opportunities outside of EA. AI alignment is a small world. Maintaining civil relations is crucial, even if not everything went splendidly in the past. Thurio doesn’t want to burn bridges – especially in a tiny village.

Tedious and Tiring 

Status: The interviewee has weakened their endorsement of the content since the interview.

Seyton is strongly averse to bringing up opinions about the quality of other AI alignment research. It would take up too much bandwidth and energy, and the resulting fights would likely go nowhere. People would just get defensive and drag him into long, tedious debates – especially because the critique is often not anonymous or double-blinded. That’s why he didn't participate in the 2021 LW year-end review. Of course, Seyton considered using a pseudonym to share his critiques. But it was never high enough on his priority list. After all, it’s also not clear if people would even read that. And it takes a while to build up credibility and traction for a pseudonym.

Team Player

Status: The interviewee has weakened their endorsement of the content since the interview, particularly concerning friends. They maintain their endorsement with respect to more distant acquaintances.

When Anthropic received cloud compute from Google, Taurus was sceptical. Was that investment a good thing for the world? What kind of promises did Anthropic make to the investors? However, when talking with his friends who work at the lab, Taurus refrains from voicing his concerns. They might start doubting if he is on the same team otherwise. In addition, he doesn't want his friends to feel like he thinks that they are working for a bad company. If he was sure that was the case, he wouldn’t hesitate to tell them! And who knows, Taurus might apply for a job at Anthropic in the coming years. He doesn’t want to express concerns until he has a foot in the door. Why jeopardise his chances with a potential employer?

Preformal Ideas

Status: The interviewee maintains their endorsement of the following content, which remains unchanged since the interview.

Cato feels that people in rationalist and EA communities discourage voicing preformal ideas. They are strict, imposing a high bar on expression. Some budding ideas undergo destructive scrutiny – the first instinct seems to be to tear down half-baked concepts before they can mature. Finding someone who wants to collaborate on building up a not-yet-fleshed-out idea can be difficult. Sometimes Cato is lucky to encounter a person to think with. Often, he refrains from expressing thoughts that he has not fully developed yet. He keeps them to himself, brooding over them, until he thinks it would be challenging to dismiss them immediately.

1. ^{^}

   I also asked questions about what would make it easier to bring up opinions that are hard to voice. I concluded that I couldn't easily represent the content in my chosen format. Maybe I'll publish those results in another post.

2. ^{^}

   The appropriate choice of pronouns initially left me in a state of confusion. Most of my participants identify as male, while a minority identify as female. At first, I exclusively used gender-neutral pronouns to prevent inadvertently outing the women. Later, I decided on 'he/him' pronouns in all vignettes where gender dynamics seemed insignificant. I didn't want to obfuscate the skewed gender ratio, after all. The entire matter of pronoun selection still leaves me somewhat confused.

19 comments

Comments sorted by top scores.