Occupational Infohazards

Experimental Two-Axis Voting: "Overall" & "Agreement"

The LW team has spent the last few weeks developing alternative voting systems. We've enabled two-axis voting on this post. The two dimensions are:

• Overall: what is your overall feeling about the comment? Does it contribute positively to the conversation? Do you want to see more comments like this?
• Agreement: do you agree with the position of this comment?

Separating these out allows for you to express more nuanced reactions to comments such as "I still disagree with what you're arguing for, but you've raised some interesting and helpful points" and "although I agree with what you're saying, I think this is a low-quality comment".

Edited to Add: I checked with Jessica first whether she was happy for us to try this experiment with her post.

↑ comment by Yoav Ravid · 2021-12-19T09:32:07.546Z · LW(p) · GW(p)

The agreement box got a bit excited :)

Replies from: Yoav Ravid, jkaufman

↑ comment by Yoav Ravid · 2021-12-19T16:05:29.912Z · LW(p) · GW(p)

Recording my reaction to the new system: I looked back at the comment and saw it got a downvote on agreement and got that small twinge of negative affect that I sometimes get from seeing a downvote on my own comment, then realized it probably just means that person doesn't have the same bug, and it passed. It would be interesting to see if this instinct changes after some time getting used to this system.

Replies from: Viliam

↑ comment by Viliam · 2021-12-20T18:24:30.663Z · LW(p) · GW(p)

I try to imagine the two numbers as parts of a complex number, like "37+8i".

From that perspective, both "37+8i" and "37-8i" feel positive.

↑ comment by jefftk (jkaufman) · 2021-12-19T15:34:47.619Z · LW(p) · GW(p)

I also see the CSS as a bit wonky, at least on mobile, though not as wonky as that. I see the agreement box as about one pixel higher than the overall box.

function author(meta) {
  return meta.children[1].innerText;
}
function votes(meta) {
  return meta.children[3].innerText.split("\n");
}
const metas = document.getElementsByClassName("CommentsItem-meta");
const output = {};
for (let i = 0; i < metas.length; i++) {
  const meta = metas[i];
  output[author(meta)] = output[author(meta)] || {
    comments: 0,
    upvotes: 0,
    agreement: 0,
  };
  output[author(meta)].comments++;
  output[author(meta)].upvotes += Number(votes(meta)[0]);
  output[author(meta)].agreement += Number(votes(meta)[1]);
}

-13:7    Alastair JL
7:-15    Liam Donovan
-5:35    Martin Randall
0:18     Quintin Pope
0:3      countingtoten
0:4      Pattern
0:6      bn22
1:15     Yoav Ravid
2:27     Benquo
2:26     Scott Alexander
32:211   jessicata
6:35     TekhneMakre
21:95    So8res
8:32     Aella
7:27     Raemon
23:76    ChristianKl
20:56    Ruby
8:22     romeostevensit
19:51    cata
9:24     tailcalled
65:160   Duncan_Sabien
40:93    Eliezer Yudkowsky
7:16     Bjartur Tómas
43:97    T3t
10:22    Viliam
38:64    habryka
85:143   jimrandomh
23:35    Davis_Kingsley
2:3      Joe_Collman
11:14    Veedrac
2:2      [comment deleted]
43:32    adamzerner
76:56    jefftk
10:5     Joe Rocca
4:2      Thomas Kehrenberg

There's a model-fragment that I think is pretty important to understanding what's happened around Michael Vassar, and Scott Alexander's criticism.

Helping someone who is having a mental break is hard. It's difficult for someone to do for a friend. It's difficult for professionals to do in an institutional setting, and I have tons of anecdotes from friends and acquaintances, both inside and outside the rationality community, of professionals in institutions fucking up in ways that were traumatizing or even abusive. Friends have some natural advantages over institutions: they can provide support in a familiar environment instead of a prison-like environment, and make use of context they have with the person.

When you encounter someone who's having a mental break or is giving off signs that they're highly stressed and at risk of a mental break, the incentivized action is to get out of the radius of blame (see Copenhagen Interpretation of Ethics). I think most people do this instinctively. Attempting to help someone through a break is a risky and thankless job; many more people will hear about it if it goes badly than if it goes well. Anyone who does it repeatedly will probably find name attached to a disaster and a mistake they made that sounds easier to avoid than it really was. Nevertheless, I think people should try to help their friends (and sometimes their acquaintances) in those circumstances, and that when we hear how it went, we should adjust our interpretation accordingly.

I've seen Michael get involved in a fair number of analogous situations that didn't become disasters and that no one heard about, and that significantly affects my interpretation, when I hear that he's been in the blast-radius of situations that did.

I think Scott Alexander looked at some stories (possibly with some rumor-mill distortions added on), and took a "this should be left to professionals" stance. And I think the "this should be left to professionals" stance looks better to him, as a professional who's worked only in above-average institutions and who can fix problems when he sees them, than it does to people collecting anecdotes from others who've been involuntarily committed.

So in general I'm noticing a pattern where you make claims about things that happened, but it turns out those things didn't happen, or there's no evidence that they happened and no reason one would believe they did a priori, and you're actually just making an inference and presenting as the state of reality.  These seem to universally be inferences which cast other's motives or actions in a negative light.  They seem to be broadly unjustified by the provided evidence and surrounding context, or rely on models of reality (both physical and social) which I think are very likely in conflict with the models held by the people those inferences are about.  Sometimes you draw correspondences between the beliefs and/or behaviors of different people or groups, in what seems like an attempt to justify the belief/behavior of the first, or to frame the second as a hypocrite for complaining about the first (though you don't usually say why these comparisons are relevant).  These correspondences turn out to only be superficial similarities while lacking any of the mechanistic similarities that would make them useful for comparisons, or actually conceal the fact that the two parties in question have opposite beliefs on a more relevant axis.  For lack of a better framing, it seems like you're failing to disentangle your social reality from the things that actually happened.  I am deeply sympathetic to the issues you experienced, but the way this post (and the previous post) was written makes it extremely difficult to engage with productively, since so many of the relevant claims turn out to not to be claims about things that actually occurred, despite their appearance as such.

I went through the post and picked out some examples (stopping only because it was taking too much of my evening, not because I ran out), with these being the most salient:

> 1. Many others have worked to conceal the circumstances of their deaths

As far as I can tell, this is almost entirely unsubstantiated, with the possible exception of Maia, and in that case it would have been Ziz's circle doing the concealment, not any of the individuals you express specific concerns about.

> 2. My psychotic break in which I imagined myself creating hell was a natural extension of this line of thought.

The way this is written makes it sound like you think that it ought to have been a (relatively) predictable consequence.

> 3. By the law of excluded middle, the only possible alternative hypothesis is that the problems I experienced at MIRI and CFAR were unique or at least unusually severe, significantly worse than companies like Google for employees' mental well-being.

In theory, the problems you experienced could have come from sources other than your professional environment.  That is a heck of a missing middle.

> 4. This view is rapidly becoming mainstream, validated by research performed by MAPS and at Johns Hopkins, and FDA approval for psychedelic psychotherapy is widely anticipated in the field.

This seems to imply that Michael's view on the subject corresponds in most relevant ways to the views taken by MAPS/etc.  I don't know what Michael's views on the subject actually are, but on priors I'm extremely skeptical that the correspondence is sufficient to make this a useful comparison (which, as an appeal to authority, is already on moderately shaky grounds).

> 5. including a report from a friend along the lines of "CFAR can't legally recommend that you try [a specific psychedelic], but..."

Can you clarify what relationship this friend had with CFAR?  This could be concerning if they were a CFAR employee at the time.  If they were not a CFAR employee, were they quoting someone who was?  If neither, I'm not sure why it's evidence of CFAR's views on the subject.

> 7. MIRI leaders were already privately encouraging me to adopt a kind of conflict theory in which many AI organizations were trying to destroy the world

This is not supported by your later descriptions of those interactions.

First, at no point do you describe any encouragement to adopt a conflict-theoretic view.  I assume this section is the relevant one: "MIRI leaders including Eliezer Yudkowsky and Nate Soares told me that this was overly naive, that DeepMind would not stop dangerous research even if good reasons for this could be given.  Therefore (they said) it was reasonable to develop precursors to AGI in-house to compete with organizations such as DeepMind in terms of developing AGI first.  So I was being told to consider people at other AI organizations to be intractably wrong, people who it makes more sense to compete with than to treat as participants in a discourse."  This does not describe encouragement to adopt a conflict-theoretic view.  It describes encouragement to adopt some specific beliefs (e.g. about DeepMind's lack of willingness to integrate information about AI safety into their models and then behave appropriately, and possible ways to mitigate the implied risks), but these are object-level claims, not ontological claims.

Second, this does not describe a stated belief that "AI organizations were trying to destroy the world".  Approximately nobody believes that AI researchers at e.g. DeepMind are actively trying to destroy the world.  A more accurate representation of the prevailing belief would be something like "they are doing something that may end up destroying the world, which, from their perspective, would be a totally unintentional and unforeseen consequence of their actions".  This distinction is important, I'm not just nitpicking.

> 8.I was given ridiculous statements and assignments including the claim that MIRI already knew about a working AGI design and that it would not be that hard for me to come up with a working AGI design on short notice just by thinking about it, without being given hints.

I would be pretty surprised if this turned out to be an accurate summary of those interactions.  In particular, that:
1) MIRI (Nate) believed, as of 2017, that would it be possible to develop AGI given known techniques & technology in 2017, with effectively no new research or breakthroughs required, just implementation, and
2) You were told that you should be able to come up with such a design yourself on short notice without any help or collaborative effort.

Indeed, the anecdote you later relay about your interaction with Nate does not support either of those claims, though it carries its own confusions (why would he encourage you to think about how to develop a working AGI using existing techniques if it would be dangerous to tell you outright?  The fact that there's an extremely obvious contradiction here makes me think that there was a severe miscommunication on at least one side of this conversation).

> 10. His belief that mental states somewhat in the direction of psychosis, such as those had by family members of schizophrenics, are helpful for some forms of intellectual productivity is also shared by Scott Alexander and many academics.

This seems to be (again) drawing a strong correspondence between Michael's beliefs and actions taken on the basis of those beliefs, and Scott's beliefs.  Scott's citation of research "showing greater mental modeling and verbal intelligence in relatives of schizophrenics" does not imply that Scott thinks it is a good idea to attempt to induce sub-clinical schizotypal states in people - in fact I would bet a lot of money that Scott thinks doing so is an extremely bad idea, which is a more relevant basis on which to compare his belief's with Michael's.

> 11. Scott asserts that Michael Vassar discourages people from seeking mental health treatment.  Some mutual friends tried treating me at home for a week as I was losing sleep and becoming increasingly mentally disorganized before deciding to send me to a psychiatric institution, which was a reasonable decision in retrospect.

Were any of those people Michael Vassar?  If not, I'm not sure how it's intended to rebut Scott's claim (though in general I agree that Scott's claim about Michael's discouragement could stand to be substantiated in some way).  If so, retracted, but then why is that not specified here, given how clearly it rebuts one of the arguments?

> 13. This is inappropriately enforcing the norms of a minority ideological community as if they were widely accepted professional standards.

This does not read like a charitable interpretation of Scott's concerns.  To wit, if I was friends with someone who I knew was a materialist atheist rationalist, and then one day they came to me and started talking about auras and demons in a way which made it sound like they believed they existed (the way materialist phenomena exist, rather than as "frames", "analogies", or "I don't have a better word for this purely internal mental phenomenon, and this word, despite all the red flags, has connotations that are sufficiently useful that I'm going to use it as a placeholder anyways"), I would update very sharply on them having had a psychotic break (or similar).  The relevant reference class for how worrying it is that someone believes something is not what the general public believes, it's how sharp a departure it is from that person's previous beliefs (and in what direction - a new, sudden, literal belief in auras and demons is a very common symptom of a certain cluster of mental illnesses!).  Note: I am not making any endorsement about the appropriateness of involuntary commitment on the basis of someone suddenly expressing these beliefs.  I'm not well-calibrated on the likely distribution of outcomes from doing so.

Moving on from the summary:

> I notice that I have encountered little discussion, public or private, of the conditions of Maia Pasek's death. To a naive perspective this lack of interest in a dramatic and mysterious death would seem deeply unnatural and extremely surprising, which makes it strong evidence that people are indeed participating in this cover-up.

I do not agree that this is the naive perspective.  People, in general, do not enjoy discussing suicide.  I have no specific reason to believe that people in the community enjoy this more than average, or expect to get enough value out of it to outweigh the unpleasantness.  Unless there is something specifically surprising about a specific suicide that seems relevant to the community more broadly, my default expectation would be that people largely don't talk about it (the same way they largely don't talk about most things not relevant to their interests).  As far as I can tell, Maia was not a public figure in a way that would, by itself, be sufficient temptation to override people's generalized dispreference for gossip on the subject.  Until today I had not seen any explanation of the specific chain of events preceding Maia's suicide; a reasonable prior would have been "mentally ill person commits suicide, possibly related to experimental brain hacking they were attempting to do to themselves (as partially detailed on their own blog)".  This seems like fairly strong supporting evidence.  I suppose the relevant community interest here is "consider not doing novel neuropsychological research on yourself, especially if you're already not in a great place".  I agree with that as a useful default heuristic, but it's one that seems "too obvious to say out loud".  Where do you think a good place for a PSA is?

In general it's not clear what kind of cover-up you're imagining.  I have not seen any explicit or implicit discouragement of such discussion, except in the (previously mentioned) banal sense that people don't like discussing suicide and hardly need additional reasons to avoid it.

> While there is a post about Jay's death on LessWrong, it contains almost no details about Jay's mental state leading up to their death, and does not link to Jay's recent blog post.  It seems that people other than Jay are also treating the circumstances of Jay's death as an infohazard.

Jay, like Maia, does not strike me as a public figure.  The post you linked is strongly upvoted (+124 at time of writing).  It seems to be written as a tribute, which is not the place where I would link to Jay's most recent blog post if I were trying to analyze causal factors upstream of Jay's suicide.  Again, my prior is that nothing about the lack of discussion needs an explanation in the form of an active conspiracy to suppress such discussion.

> There is a very disturbing possibility (with some evidence for it) here, that people may be picked off one by one (sometimes in ways they cooperate with, e.g. through suicide), with most everyone being too scared to investigate the circumstances.

Please be specific, what evidence?  This is an extremely serious claim.  For what it's worth, I don't agree that people can be picked off in "ways they cooperate with, e.g. though suicide", unless you mean that Person A would make a concerted effort to convince Person B that Person B ought to commit suicide.  But, to the extent that you believe this, the two examples of suicides you listed seem to be causally downstream of "self-hacking with friends(?) in similarly precarious mental states", not "someone was engaged in a cover-up (of what?) and decided it would be a good idea to try to get these people to commit suicide (why?) despite the incredible risks and unclear benefits".

> These considerations were widely regarded within MIRI as an important part of AI strategy. I was explicitly expected to think about AI strategy as part of my job.   So it isn't a stretch to say that thinking about extreme AI torture scenarios was part of my job.

S-risk concerns being potentially relevant factors in research does not imply that thinking about specific details of AI torture scenarios would be part of your job.  Did someone at MIRI make a claim that imagining S-risk outcomes in graphic detail was necessary or helpful to doing research that factored in S-risks as part of the landscape?  Roll to disbelieve.

> Another part of my job was to imagine myself in the role of someone who is going to be creating the AI that could make everything literally the worst it could possibly be, in order to avoid doing that, and prevent others from doing so.

Similarly, roll to disbelieve that someone else at MIRI suggested that you imagine this without significant missing context which would substantially alter the naïve interpretation of that claim.

Skipping the anecdote with Nate & AGI, as I addressed it above, but following that:

> Nate and others who claimed or implied that they had such information did not use it to win bets or make persuasive arguments against people who disagreed with them, but instead used the shared impression or vibe of superior knowledge to invalidate people who disagreed with them.

Putting aside the question of whether Nate or others actually claimed or implied that they had a working model of how to create an AGI with 2017-technology, if they had made such a claim, I am not sure why you would expect them to try to use that model to win bets or make persuasive arguments.  I would in fact expect them to never say anything about it to the outside world, because why on earth would you do that given, uh, the entire enterprise of MIRI?

> But I was systematically discouraged from talking with people who doubted that MIRI was for real or publicly revealing evidence that MIRI was not for real, which made it harder for me to seriously entertain that hypothesis.

Can you concretize this?  When I read this sentence, an example interaction I can imagine fitting this description would be someone at MIRI advising you in a conversation to avoid talking to Person A, because Person A doubted that MIRI was for real (rather than for other reasons, like "people who spend a lot of time interacting with Person A seem to have a curious habit of undergoing psychotic breaks").

> In retrospect, I was correct that Nate Soares did not know of a workable AGI design.

I am not sure how either of the excerpts support the claim that "Nate Soares did not know of a workable AGI design" (which, to be clear, I agree with, but for totally unrelated reasons described earlier).  Neither of them make any explicit or implicit claims about knowledge (or lack thereof) of AGI design.

> In a recent post, Eliezer Yudkowsky explicitly says that voicing "AGI timelines" is "not great for one's mental health", a new additional consideration for suppressing information about timelines.

This is not what Eliezer says.  Quoting directly: "What feelings I do have, I worry may be unwise to voice; AGI timelines, in my own experience, are not great for one's mental health, and I worry that other people seem to have weaker immune systems than even my own."

This is making a claim that AI timelines themselves are poor for one's mental health (presumably, the consideration of AI timelines), not the voicing of them.

> Researchers were told not to talk to each other about research, on the basis that some people were working on secret projects and would have to say so if they were asked what they were working on.  Instead, we were to talk to Nate Soares, who would connect people who were working on similar projects.  I mentioned this to a friend later who considered it a standard cult abuse tactic, of making sure one's victims don't talk to each other.

The reason cults attempt to limit communication between their victims is to prevent the formation of common knowledge of specific abusive behaviors that the cult is engaging in, and similar information-theoretic concerns.  Taking for granted the description of MIRI's policy (and application of it) on internal communication about research, this is not a valid correspondence; they were not asking you to avoid discussing your interactions with MIRI (or individuals within MIRI) with other MIRI employees, which could indeed be worrying if it were a sufficiently general ask (rather than e.g. MIRI asking someone in HR not to discuss confidential details of various employees with other employees, which would technically fit the description above but is obviously not what we're talking about).

> It should be noted that, as I was nominally Nate's employee, it is consistent with standard business practices for him to prevent me from talking with people who might distract me from my work; this goes to show the continuity between "cults" and "normal corporations".

I can't parse this in a way which makes it seem remotely like "standard business practices".  I disagree that it is a standard business practice to actively discourage employees from talking to people who might distract them from their work, largely because employees do not generally have a problem with being distracted from their work because they are talking to specific people.  I have worked at number of different companies, each very different from the last in terms of size, domain, organizational culture, etc, and there was not a single occasion where I felt the slightest hint that anyone above me on the org ladder thought I ought to not talk to certain people to avoid distractions, nor did I ever feel like that was part of the organization's expectations of me.

> MIRI researchers were being very generally denied information (e.g. told not to talk to each other) in a way that makes more sense under a "bad motives" hypothesis than a "good motives" hypothesis.  Alternative explanations offered were not persuasive.

Really?  This seems to totally disregard explanations unrelated to whether someone has "good motives" or "bad motives", which are not reasons that I would expect MIRI to have at the top of their list justifying whatever their info-sec policy was.

> By contrast, Michael Vassar thinks that it is common in institutions for people to play zero-sum games in a fractal manner, which makes it unlikely that they could coordinate well enough to cause such large harms.

This is a bit of a sidenote but I'm not sure why this claim is interesting w.r.t. AI alignment, since the problem space, almost by definition, does not require coordination with intent to cause large harms, in order to in fact cause large harms.  If the claim is that "institutions are sufficiently dysfunctional that they'll never be able to build an AGI at all", that seems like a fully-general argument against institutions ever achieving any goals that require any sort of internal and/or external coordination (trivially invalidated by looking out the window).

> made a medication suggestion (for my sleep issues) that turned out to intensify the psychosis in a way that he might have been able to predict had he thought more carefully

I want to note that while this does not provide much evidence in the way of the claim that Michael Vassar actively seeks to induce psychotic states in people, it is in fact a claim that Michael Vassar was directly, causally upstream of your psychosis worsening, which is worth considering in light of what this entire post seems to be arguing against.

Status: writing-while-frustrated. As with the last post, many of Jessica's claims seem to me to be rooted in truth, but weirdly distorted. (Ever since the end of Jessica's tenure at MIRI, I have perceived a communication barrier between us that has the weird-distortion nature.)

Meta: I continue to be somewhat hesitant to post stuff like this, on the grounds that it sucks to air dirty laundry about your old employer and then have your old employer drop by and criticize everything you said. I’ve asked Jessica whether she objects to me giving a critical reply, and she said she has no objections, so at least we have that. I remain open to suggestions for better ways to navigate these sorts of situations.

Jessica, I continue to be sad about the tough times you had during the end of your tenure at MIRI, and in the times following. I continue to appreciate your research contributions, and to wish you well.

My own recollections follow. Note that these are limited to cases where Jessica cites me personally, in the interest of time. Note also that I'm not entirely sure I've correctly identified the conversations she's referring to, due to the blurring effects of the perceived distortion, and of time. And it goes almost-without-saying that my own recollections are fallible.

As a MIRI employee I was coerced into a frame where I was extremely powerful and likely to by-default cause immense damage with this power, and therefore potentially responsible for astronomical amounts of harm. I was discouraged from engaging with people who had criticisms of this frame, and had reason to fear for my life if I published some criticisms of it.

My own frame indeed says that the present is the hinge of history, and that humans alive today have extreme ability to affect the course of the future, and that this is especially true of humans working in the AI space, broadly construed. I wouldn't personally call this "power" -- I don't think anyone in the space has all that much power in the present. I think a bunch of people in the AI-o-sphere have a lowish likelihood of a large amount of future power, and thus high future expected power, which is kinda like power. From my perspective, MIRI researchers have less of this than researchers from current top AI labs, but do have a decent amount. My own model does not predict that MIRI researchers are likely to cause astronomical harm by default. I do not personally adhere to the copenhagen interpretation of ethics, and in the event that humanity destroys itself, I would not be assigning extra blame to alignment researchers on the grounds that they were closer to the action. I'm also not personally very interested in the game of pre-assigning blame, favoring object-level alignment research instead.

Insofar as I was influencing Jessica with my own frame, my best guess is that she misunderstood my frame, as evidenced by these differences between the frame she describes feeling coerced into, and my own picture.

I don't recall ever discouraging Jessica from engaging with people who had criticisms of my frame. I readily admit that she was talking to folks I had little intellectual respect for, and I vaguely remember some of these people coming up in conversation and me noting that I lacked intellectual respect for them. To the best of my recollection, in all such instances, I added caveats of the form "but, just because I wouldn't doesn't mean you shouldn't". I readily admit that my openness about my lack of intellectual respect may have been taken as discouragement, especially given my position as her employer. The aforementioned caveats were intended to counteract such forces, at least insofar as I recall.

I was aware at the time that Jessica and I didn't see eye-to-eye on various issues. I remember at least two occasions where I attempted to explicitly convey that I knew we didn't see eye-to-eye, that it was OK for her to have views that didn't match mine, and that I encouraged her to think independently and develop her own views.

Jessica said she felt coerced into a frame she found uncomfortable, and I believe her. My notes here are not intended to cast doubt on the honesty of her reports. My intent in saying all this is merely to express that (1) the frame she reports feeling coerced into, is not one that I recognize, nevermind one that I intentionally coerced her into; and (2) I was aware of the pressures and actively tried to counteract them. Clearly, I failed at this. (And I have a decent chunk of probability mass that Jessica would clarify that she's not accusing me of intentional coercion.) From my own perspective, she was misreading my own frame and feeling pressured into it despite significant efforts on my part to ameliorate the pressure. I happily solicit advice for what to do better next time, but do not consider my comport to have been a mistake.

talked about hellscapes

I don't recall ever "talking about hellscapes" per se. I recall mentioning them in passing, rarely. In my recollection, that mainly happened in response to someone else broaching the topic of fates worse than death. (Maybe there were other occasional throwaway references? But I don't recall them.) My cached reply to others raising the idea of fates worse than death went something like:

"Goal-space is high dimensional, and almost all directions of optimization seem likely to be comparably bad to death from our perspective. To get something that is even vaguely recognizable to human values you have to be hitting a very narrow target in this high-dimensional space. Now, most of that target is plausibly dystopias as opposed to eutopias, because once you're in the neighborhood, there are a lot of nearby things that are bad rather than good, and value is fragile. As such, it's reasonable in principle to worry about civilization getting good enough at aiming AIs that they can hit the target but not the bullseye, and so you might worry that that civilization is more likely to create a hellscape than a eutopia. I personally don't worry about this myself, because it seems to me that the space is so freaking high dimensional and the target so freaking small, that I find it implausible that a civilization successfully able to point an AI in a human-relevant direction, isn't also able to hit the bullseye. Like, if you're already hitting a quarter with an arrowhead on the backside of the moon, I expect you can also hit a dime."

Two reasons I'd defend mentioning hellscapes in such situations: firstly, to demonstrate that I at least plausibly understood the concern my conversation partner had raised (as a matter of course before making a counterpoint), and secondly, so as to not undermine our friends working on S-risk reduction (a program I support).

My reason for not hesitating to use terms like "hellscapes" rather than more banal and less evocative terms was (to the best of my current recollection) out of a desire to speak freely and frankly, at least behind closed doors (eg, in the privacy of the MIRI research space). At the time, there was a bunch of social pressure around to stop thought experiments that end with the AI escaping and eating the galaxy, and instead use thought experiments about AIs that are trying to vacuum the house and accidentally break a lamp or whatever, and this used to rub me the wrong way. The motivations as I previously understood them were that, if you talk about star-eating rather than lamp-breaking, then none of the old guard AI researchers are ever going to take your field seriously. I thought (and still think) this is basically a bad reason. However, I have since learned a new reason, which is that mentioning large-scale disasters freely and frankly, might trigger psychotic episodes in people predisposed to them. I find this a much more compelling reason to elide the high-valence examples.

(Also, the more banal term "S-risk" hadn't propagated yet, IIRC.)

Regardless, I have never thought in detail about fates worse than death, never mind discussed fates worse than death in any depth. I have no reason to, and I recommend against it. Me occasionally mentioning something in passing, and Jessica glossing it as "Nate talked about it" (with an implication of depth and regularity), is a fine example of the "weird distortion" I perceive in Jessica's accounts.

I was told, by Nate Soares, that the pieces to make AGI are likely already out there and someone just has to put them together.

I contest this. According to my best recollection of the conversation that I think Jessica is referring to, she was arguing that AGI will not arrive in our own lifetimes, and seemed unresponsive to my attempts to argue that a confident claim of long timelines requires positive knowledge, at which point I exasperatedly remarked that for all we knew, the allegedly missing AGI insights had already been not only had, but published in the literature, and all that remains is someone figuring out how to assemble them. (cf no one knows what science doesn't know [LW · GW].) I do not assign particularly high credence to this claim myself, and (IIRC) I was using it rhetorically to test for acknowledgement of the idea that confident long timelines require positive knowledge that we seem to lack.

(This seems like another central example of my throwaway lines becoming weirdly distorted and heavily highlighted in Jessica's recounting.)

He did not tell me anything about how to make such an AGI, on the basis that this would be dangerous.

Here Jessica seems to be implying that, not only did I positively claim that the pieces of AGI were already out there in the literature, but also that I had personally identified them? I deny that, and I'm not sure what claim I made that Jessica misunderstood in that way. Given the surrounding context where Jessica made this claim, my guess is that it was in the same conversation as the exasperated remark described above, and that the conversation past that point became so desynched that Jessica's recounting is no longer recognizable to me.

To be clear, I have claimed that AI alignment work is sometimes intertwined with AI capabilities work, and I have claimed that capabilities insights shouldn't be publicized (as a strong default) on account of the negative externalities. Perhaps I said something along those lines that got distorted into Jessica's claim?

Instead, he encouraged me to figure it out for myself, saying it was within my abilities to do so.

I occasionally recommend that our researchers periodically (every 6mo or so) open a text file and see if they can write pseudocode for an FAI (ignoring computational constraints, at least for now), to help focus their attention on exactly where they're confused and ground out their alignment research in things that are literally blocking them from actually writing a flippin’ FAI. I don't recall ever telling Jessica that I thought she could figure out how to build an AGI herself. I do recall telling her I expected she could benefit from the exercise of attempting to write the pseudocode for an FAI.

If memory serves, this is an exercise I'd been advocating for a couple years before the time period that Jessica's discussing (and IIRC, I've seen Jessica advocate it, or more general variants like "what could you do with a hypercomputer on a thumb drive", as an exercise to potential hires). One guess as to what's going on is that I tried to advocate the exercise of pseudocoding an FAI as I had many times before, but used some shorthand for it that I thought would be transparent, in some new conversational context (eg, shortly after MIRI switched to non-disclosure by default), and while Jessica was in some new mental state, and Jessica misunderstood me as advocating figuring out how to build an AGI all on her own while insinuating that I thought she could?

[From the comments, in answer to the query "How did you conclude from Nate Soares saying that that the tools to create agi likely already exist that he wanted people to believe he knew how to construct one?"] Because he asked me to figure it out in a way that implied he already had a solution; the assignment wouldn't make sense if it were to locate a non-workable AGI design (as many AI researchers have done throughout the history of the field); that wouldn't at all prove that the pieces to make AGI are already out there. Also, there wouldn't be much reason to think that his sharing a non-workable AGI design with me would be dangerous.

In light of this, my guess is that Jessica flatly misread my implications here.

To be very explicit: Jessica, I never believed you capable of creating a workable AGI design (using, say, your 2017 mind, unaugmented, in any reasonable amount of time). I also don't assign particularly high credence to the claim that the insights are already out in the literature waiting to be found (or that they were in 2017). Furthermore, I never intentionally implied that I have myself succeeded at the "pseudocode an FAI" exercise so hard as to have an AGI design. Sorry for the miscommunication.

Researchers were told not to talk to each other about research, on the basis that some people were working on secret projects and would have to say so if they were asked what they were working on.

This suggests a picture of MIRI’s nondisclosure-by-default policies that’s much more top-down than reality, similar to a correction I made on a post by Evan Hubinger [LW(p) · GW(p)] a few years ago.

The sequence of events as I recall them was: Various researchers wanted to do some closed research. There was much discussion about how much information was private: Research results? Yes, if the project lead wants privacy. Research directions? Yes, if the project lead wants privacy. What about the participant list for each project? Can each project determine their own secrecy bounds individually, or is revealing who's working with you defecting against (possibly-hypothetical) projects that don't want to disclose who they're working with? etc. etc. I recall at least one convo with a bunch of researchers where, in efforts to get everyone to stop circling privacy questions like moths to a flame and get back to the object level research, I said something to the effect of "come to me if you're having trouble".

I separately recall Jessica coming to me afterwards and asking a bunch more questions about who she can ask about what. I recall trying to convey something like "just work on what you want to work on, with whatever privacy level you want, and if someone working on something closed wants you working with them they'll let you know (perhaps through me, if they want to), and you can bang out details with them as need be".

The fact that people shouldn't have to reveal whether they are in fact working on closed research if they don't want to sounds like the sort of thing that came up in one or both of those conversations, and my guess is that that's what Jessica's referring to here. From my perspective, that wasn't a particularly central point, and the point I recall attempting to drive home was more like "let's just work on the object-level research and not get all wound up around privacy (especially because all that we've changed are the defaults, and you're still completely welcome to publicize your own research, with my full support, as much as you'd like)".

Nate Soares also wrote a post discouraging people from talking about the ways they believe others to be acting in bad faith.

According to me, I was not trying to say "you shouldn't talk about ways you believe others to be acting in bad faith". I was trying to say "I think y'all are usually mistaken when you're accusing certain types of other people of acting in bad faith", plus "accusing people of acting in bad faith [in confrontational and adversarial ways, instead of gently clarifying and confirming first] runs a risk of being self-fulfilling, and also burns a commons, and I'm annoyed by the burned commons". I think those people are wrong and having negative externalities, not that they're bad for reporting what they believe.

Note that the sorts of talking the post explicitly pushes back against is arguments of the form "person X is gaining [status|power|prestige] through their actions, therefore they are untrustworthy and have bad intentions", which I believe to be invalid. Had I predicted Jessica's particular misread in advance, I would have explicitly noted that I'm completely ok with arguments of the form "given observations X and Y, I have non-trivial probability on the hypothesis that you're acting in bad faith, which I know is a serious allegation. Are you acting in bad faith? If not, how do you explain observations X and Y?".

In other words, the thing I object to is not the flat statement of credence on the hypothesis "thus-and-such is acting in bad faith", it's the part where the author socially rallies people to war on flimsy pretenses.

In other other words, I both believe that human wonkiness makes many people particularly bad at calculating P(bad-faith|the-evidence) in particular, and recommend being extra charitable and cautious when it feels like that probability is spiking. Separately but relatedly, in the moment that you move from stating your own credence that someone is acting in bad faith, to socially accusing someone of acting in bad faith, my preferred norms require a high degree of explicit justification.

And, to be explicit: I think that most of the people who are acting in bad faith will either say "yes I'm acting in bad faith" when you ask them, or will sneer at you or laugh at you or make fun of you instead, which is just as good. I think a handful of other people are harmful to have around regardless of their intentions, and my guess is that most community decisions about harmful people should revolve around harm rather than intent.

(Indeed, I suspect the community needs to lower, rather than raise, the costs of shunning someone who's doing a lot of harm. But I reiterate that I think such decisions should center on the harm, not the intent, and as such I continue to support the norm that combative/adversarial accusations of ill intent require a high degree of justification.)

Nate Soares expressed discontent that Michael Vassar was talking with "his" employees, distracting them from work.

I don't actually know what conversation this is referring to. I recall a separate instance, not involving Jessica, of a non-researcher spending lots of time in the office hanging out and talking with one of our researchers, and me pulling the researcher aside and asking whether they reflectively endorsed having those conversations or whether they kept getting dragged into them and then found themselves unable to politely leave. (In that case, the researcher said they reflectively endorsed them, and thereafter I left them alone.)

There might have been a time when Michael Arc (nee Vassar) was spending a lot of time talking to Jessica and one other employee, and I said something about how I don't have much intellectual respect for Michael? I doubt I said this unsolicited, but I definitely would have said it if anyone asked, and I at least vaguely remember something like that happening once or twice. It's also possible that towards the end of Jessica's tenure we were trying to have scheduled meetings to see if we could bridge the communications barrier, and it came up naturally in the conversation? But I'm not sure, as (unlike most of the other claims) I don't concretely recognize this reference.

It should be noted that, as I was nominally Nate's employee, it is consistent with standard business practices for him to prevent me from talking with people who might distract me from my work during office hours.

I'm confident I did not prevent anyone from talking to anyone. I occasionally pulled people aside and asked them if they felt trapped in a given conversation when someone was loitering in the office having lots of conversations, so that I could rescue them if need be. I occasionally answered honestly, when asked, what I thought about people's conversation partners. I leave almost all researchers to their own devices (conversational or otherwise) almost all of the time.

In Jessica's particular case, she was having a lot of difficulty at the workplace, and so I stepped deeper into the management role than I usually do and we spent more time together seeing whether we could iron out our difficulties or whether we would need to part ways. It's quite plausible that, during one of those conversations, I noted of my own accord that she was spending lots of her office-time deep in conversation with Michael, and that I didn't personally expect this to help Jessica get back to producing alignment research that passed my research quality bar. But I am confident that, insofar as I did express my concerns, it was limited to an expression of skepticism. I… might have asked Michael to stop frequenting the offices quite so much? But I doubt it, and I have no recollection of such a thing.

I am confident I didn't ever tell anyone not to talk to someone else, that feels way out-of-line to me. I may well have said things along the lines of "I predict that that conversation will prove fruitless", which Jessica interpreted as a guess-culture style command? I tend to couch against that interpretation by adding hedges of the form "but I'm not you" or whatever, but perhaps I neglected to, or perhaps it fell on deaf ears?

Or perhaps Jessica's just saying something along the lines of "I feared that if I kept talking to Michael all day, I'd be fired, and Nate expressing that he didn't expect those conversations to be productive was tantamount to him saying that if I continued he'd fire me, which was tantamount to him saying that I can't talk to Michael"? In which case, my prediction is indeed that if she hadn't left MIRI of her own accord, and her research performance didn’t rebound, at some point I would have fired her on the grounds of poor performance. And in worlds where Jessica kept talking to Michael all the time, I would have guessed that a rebound was somewhat less likely, because I didn’t expect him to provide useful meta-level or object-level insights that lead to downstream alignment progress. But I'm an empiricist, and I would have happily tested my "talking to Michael doesn't result in Nate-legible research output" hypothesis, after noting my skepticism in advance.

(Also, for the record, "Nate-legible research output" does not mean "research that is useful according to Nate's own models". Plenty of MIRI researchers disagree with me and my frames about all sorts of stuff, and I’m happy to have them at MIRI regardless, given that they’ve demonstrated the ability to seriously engage with the problem. I'm looking for something more like a cohesive vision that the researcher themself believes in, not research that necessarily strikes me personally as directly useful.)

MIRI certainly had a substantially conflict-theoretic view of the broad situation, even if not the local situation. I brought up the possibility of convincing DeepMind people to care about AI alignment. MIRI leaders including Eliezer Yudkowsky and Nate Soares told me that this was overly naive, that DeepMind would not stop dangerous research even if good reasons for this could be given.

I contest this. I endorse talking with leadership at leading AI labs, and have done so in the past, and expect to continue doing so in the future.

It’s true that I don't expect any of the leading labs to slow down or stop soon enough, and it’s true that I think converging beliefs takes a huge time investment. On the mainline I predict that the required investment won't in fact be paid in the relevant cases. But, as I told Jessica at the time (IIRC), I expect folks at leading AGI labs to be much more sensitive to solutions to the alignment problem, despite the fact that I don't think you can talk them into giving up public capabilities research in practice. (This might be what she misunderstood as me saying we'd have better luck "competing"? I don't recall saying any such thing, but I do recall saying that we'd have better luck solving alignment first and persuading second.)

(And for the record, while I think these big labs are making a mistake, it's a very easy mistake to make: knowing that you're in the bad Nash equilibrium doesn't let you teleport to a better one, and it's at least understandable that each individual capabilities lab thinks that they're better than the next guy, or that they can't get the actual top researchers if they implement privacy protocols right out the gate. It’s an important mistake, but not a weird one that requires positing unusual levels of bad faith.)

In case it's not clear from the above, I don't have much sympathy for conflict theory in this, and I definitely don't think in broadly us-vs-them terms about the AGI landscape. And (as I think I said at the time) I endorse learning how to rapidly converge with people. I recommend figuring out how to more rapidly converge with friends before burning the commons of time-spent-converging-with-busy-people-who-have-limited-attention-for-you, but I still endorse figuring it out. I don't expect it to work, and I think solving the dang alignment problem on the object-level is probably a better way to convince people to do things differently, but also I will cheer on the sidelines as people try to figure out how to get better and faster at converging their beliefs.

There's no law saying that, when someone's making a mistake, there's some way to explain it to them such that suddenly it's fixed. I think existing capabilities orgs are making mistakes (at the very least, in publishing capabilities advances (though credit where credit is due, various labs are doing better at keeping their cutting-edge results private, at least until somebody else replicates or nearly-replicates them, than they used to be (though to be clear I think we have a long way to go before I stop saying that I believe I see a big mistake))), and deny the implicit inference from "you can't quickly convince someone with words that they're making a mistake" to "you must be using conflict theory".

I was concerned about the linked misleading statement in 2017 and told Nate Soares and others about it, although Nate Soares insisted that it was not a lie, because technically the word "excited" could indicate the magnitude of a feeling rather than the positiveness of it.

That doesn't seem to me like a good characterization of my views.

My recollection is that, in my conversation about this topic with Jessica, I was trying to convey something more like "Yeah, I'm pretty worried that they're going to screw lots of things up. And the overt plan to give AGI to everyone is dumb. But also there are a bunch of sane people trying to redirect OpenAI in a saner direction, and I don't want to immediately sic our entire community on OpenAI and thereby ruin their chances. This whole thing looks real high-variance, and at the very least this is "exciting" in the sense that watching an adventure movie is exciting, even in the parts where the plot is probably about to take a downturn. That said, there's definitely a sense in which I'm saying things with more positive connotations than I actually feel -- like, I do feel some real positive hope here, but I'm writing lopsidedly from those hopes. This is because the blog post is an official MIRI statement about a new AI org on the block, and my sense of politics says that if a new org appears on the block and you think they're doing some things wrong, then the politic thing to do initially is talk about their good attributes out loud, while trying to help redirect them in private."

For the record, I think I was not completely crazy to have some hope about OpenAI at the time. As things played out, they wound up pretty friendly to folks from our community, and their new charter is much saner than their original plan. That doesn’t undo the damage of adding a new capabilities shop at that particular moment in that particular way; but there were people trying behind the scenes, that did in real life manage to do something, and so having some advance hope before they played their hands out was a plausible mistake to make, before seeing the actual underwhelming history unfold.

All that said, I do now consider this a mistake, both in terms of my “don’t rock the boat” communications strategy and in terms of how well I thought things might realistically go at OpenAI if things went well there. I have since updated, and appreciate Jessica for being early in pointing out that mistake. I specifically think I was mistaken in making public MIRI blog posts with anything less than full candor.

While someone bullshitting on the public Internet doesn't automatically imply they lie to their coworkers in-person, I did not and still don't know where Nate is drawing the line here.

As I said to Jessica at the time (IIRC), one reason I felt (at the time) that the blog post was fine, is that it was an official MIRI-organization announcement. When speaking as an organization, I was (at the time) significantly more Polite and significantly more Politically Correct and significantly less Dour (and less opinionated and more uncertain, etc).

Furthermore, I (wrongly) expected that my post would not be misleading, because I (wrongly) expected my statements made as MIRI-the-org to be transparently statements made as MIRI-the-org, and for such statements to be transparently Polite and Politically Correct, and thus not very informative one way or another. (In case it wasn't clear, I now think this was a mistake.)

That said, as I told Jessica at the time (IIRC), you can always just ask me whether I'm speaking as MIRI-the-organization or whether I'm speaking as Nate. Similarly, when I'm speaking as Nate-the-person, you can always just ask me about my honesty protocols [LW · GW].

I have since updated against the idea that I should ever speak as MIRI-the-organization, and towards speaking uniformly with full candor as Nate-the-person. I'm not sure I'll follow this perfectly (I'd at least slip back into politic-speak if I found myself cornered by a journalist), but again, you can always just ask.

Extracting and signal boosting this part from the final blog post linked by Winterford:

One time when I was being sexually assaulted after having explicitly said no, a person with significant martial arts training pinned me to the floor. ... name is Storm.

I had not heard this accusation before, and do not know whether it was ever investigated. I don't think I've met Storm, but I'm pretty sure I could match this nickname to the legal name of someone in the East Bay by asking around. Being named as a rapist in the last blog post of someone who later committed suicide is very incriminating, and if this hasn't been followed up it seems important to do so.

Disclaimer: I currently work for MIRI in a non-technical capacity, mostly surrounding low-level ops and communications (e.g. I spent much of the COVID times disinfecting mail for MIRI employees).  I did not overlap with Jessica and am not speaking on behalf of MIRI.

I'm having a very hard time with the first few thousand words here, for epistemic reasons.  It's fuzzy and vague in ways that leave me feeling confused and sleight-of-handed and motte-bailey'd and 1984'd.  I only have the spoons to work through the top 13-point summary at the moment.  

I acknowledge here, and will re-acknowledge at the end of this comment, that there is an obvious problem with addressing only a summary; it is quite possible that much of what I have to say about the summary is resolved within the larger text.

But as Jessica notes, many people will only read the summary and it was written with those people in mind.  This makes it something of a standalone document, and in my culture would mean that it's held to a somewhat higher standard of care; there's a difference between points that are just loosely meant to gesture at longer sections, and points which are known to be [the whole story] for a substantial chunk of the audience.  Additionally, it sets the tone for the ten-thousand-or-so words to follow.  I sort of fear the impact of the following ten thousand words on people who were just fine with the first three thousand; people whose epistemic immune systems did not boot up right at the start.

"Claim 0"

As a MIRI employee I was coerced into a frame where I was extremely powerful and likely to by-default cause immense damage with this power, and therefore potentially responsible for astronomical amounts of harm. I was discouraged from engaging with people who had criticisms of this frame, and had reason to fear for my life if I published some criticisms of it. Because of this and other important contributing factors, I took this frame more seriously than I ought to have and eventually developed psychotic delusions of, among other things, starting World War 3 and creating hell. Later, I discovered that others in similar situations killed themselves and that there were distributed attempts to cover up the causes of their deaths.

The passive voice throws me.  "I was coerced" reads as a pretty strong statement of fact about the universe; "this simply is."  I would have liked to hear, even in a brief summary paragraph, something more like "I was coerced by A, B, and C, via methods that included X, Y, and Z, into a frame where etc," because I do not yet know whether I should trust Jessica's assessment of what constitutes coercion.

Ditto "I had reason to fear for my life."  From whom?  To what extent?  What constitutes [sufficient] reason?  Is this a report of fear of being actually murdered?  Is this a report of fear of one's own risk of suicide?  To some extent this might be clarified by the followup in Claim 1, but it's not clear whether Claim 1 covers the whole of that phrase or whether it's just part of it.  In all, the whole intro feels ... clickbaity?  Maximum-attention-grabbing while being minimum-substantive?

"Claim 1"

Multiple people in the communities I am describing have died of suicide in the past few years. Many others have worked to conceal the circumstances of their deaths due to infohazard concerns [LW · GW]. I am concerned that in my case as well, people will not really investigate the circumstances that made my death more likely, and will discourage others from investigating, but will continue to make strong moral judgments about the situation anyway.

The phrase "many others have worked to conceal the circumstances of their deaths due to infohazard concerns" unambiguously implies something like a conspiracy, or at least a deliberate deceptiveness on the part of five-plus (?) people.  This is quite a strong claim.  It seems to me that it could in fact be true.  In worlds where it is true, I think that (even in the summary) it should be straightforward about who it is accusing, and of what.  "Many others" should be replaced with a list of names, or at least a magnitude; is it four?  Fourteen?  Forty? "Worked to conceal" should tell me whether there were lies told, or whether evidence was destroyed, or whether people were bribed or threatened, or whether people simply played things close to the vest.

(In general, all the motte-and-baileys in this should have been crafted to be much less motte-and-bailey-ish, in my opinion.  It seems genuinely irresponsible to leave them as vague and as fill-in-the-gaps-with-your-own-preconceptions; virtually-all-interpretations-are-nonzero-defensible as they are.  Claims like these should claim things, and they should claim them clearly so that they can later be unambiguously judged true or false.)

"Claim 2"

My official job responsibilities as a researcher at MIRI included thinking seriously about hypothetical scenarios, including the possibility that someone might cause a future artificial intelligence to torture astronomical numbers of people. While we considered such a scenario unlikely, it was considered bad enough if it happened to be relevant to our decision-making framework. My psychotic break in which I imagined myself creating hell was a natural extension of this line of thought.

I don't know what it means to be "a natural extension of this line of thought."  I don't think it is the case that >5% of people who are aware of the concept of an AI hellscape become concerned that their actions might directly result in the creation of hell.  Some dots need connecting.  This is not a criticism I would ordinarily level, were it not for the things I've already said above; at some point in the first ten sentences, I smelled something like an attempt to ensnare my mind and my shields went up and now I'm noticing all the things, whereas if I were in a less defensive mood I might not mention this one at all.

"Claim 3"

Scott asserts that Michael Vassar thinks "regular society is infinitely corrupt and conformist and traumatizing". This is hyperbolic (infinite corruption would leave nothing to steal) but Michael and I do believe that the problems I experienced at MIRI and CFAR were not unique or unusually severe for people in the professional-managerial class. By the law of excluded middle, the only possible alternative hypothesis is that the problems I experienced at MIRI and CFAR were unique or at least unusually severe, significantly worse than companies like Google for employees' mental well-being.

This section encourages thinking of MIRI and CFAR as a single unit, which I argued against at length on the original post [LW · GW].  It's somewhat like describing "my experiences in Berkeley and Oakland" in a single breath.  I do not believe that there are very many of Jessica's experiences that can't be clearly separated into "descended from experiences at MIRI, and the responsibility of MIRI and its culture" and "descended from experiences at CFAR, and the responsibility of CFAR and its culture."  I think that distinction is pretty important.

Separately, I feel encouraged to adopt a false dichotomy, in which either [some unspecified combination of experiences, some of which involved MIRI and others of which involved CFAR] were fairly mundane and within the range of normal workplace experiences or they were unique or severely bad.  I feel like this dichotomy leaves out all sorts of nuance that's pretty important, such as how things might be good on one day and bad on another, or interactions with one staff member might be positive while interactions with another are harmful, or experience X might have bothered person A quite a lot while having no impact on person B whatsoever.  Invoking the law of the excluded middle does the same thing as using the passive voice does up above—it makes things feel floaty and absolute and unchallengeable, rather than grounded in the mundane realm of "X did Y and it resulted in Z."

I don't think there exists a version of Claim 3 which is not fundamentally misleading (whereas Claims 1 and 2 feel like they could have been written in a non-misleading fashion and still communicated what they were trying to communicate).

"Claim 4"

Scott asserts that Michael Vassar thinks people need to "jailbreak" themselves using psychedelics and tough conversations. Michael does not often use the word "jailbreak" but he believes that psychedelics and tough conversations can promote psychological growth. This view is rapidly becoming mainstream, validated by research performed by MAPS and at Johns Hopkins, and FDA approval for psychedelic psychotherapy is widely anticipated in the field.

"Because some instances of [the thing referred to as jailbreaking] are good and growing in popularity, criticisms of [the thing referred to as jailbreaking] are invalid or should at least be treated with suspicion"?

That's what it seems to me that Claim 4 wants me to believe.  In Claim 4's defense, it may just be responding-in-kind to a similar kind of rounding-off in Scott's original comment.  But if so, it's the second wrong that doesn't make right.

It is entirely possible for Scott to have been correctly critical of a thing, and for psychedelics and tough conversations to finally be coming out from under an unfair cloud of suspicion.  The whole challenge is figuring out whether X is [the bad thing it looks like] or [the good thing that bears an unfortunate resemblance to the bad thing].  The text of Claim 4 tries to make me forget this fact.  It nudges me toward a bucket error [LW · GW] in which I have to treat Michael Arc (née Vassar) and MAPS and Johns Hopkins and the FDA all the same.  Either they're all right, and Scott is wrong, or they're all wrong, and Scott is right.  This does not make it easier for me to see and think clearly around whatever-happened-here.

"Claim 5"

I was taking psychedelics before talking extensively with Michael Vassar. From the evidence available to me, including a report from a friend along the lines of "CFAR can't legally recommend that you try [a specific psychedelic], but...", I infer that psychedelic use was common in that social circle whether or not there was an endorsement from CFAR. I don't regret having tried psychedelics. Devi Borg reports that Michael encouraged her to take fewer, not more, drugs; Zack Davis reports that Michael recommended psychedelics to him but he refused.

I do not see why this claim does not simply say: "I was taking psychedelics before talking extensively with Michael Vassar.  I don't regret having tried psychedelics. Devi Borg reports that Michael encouraged her to take fewer, not more, drugs; Zack Davis reports that Michael recommended psychedelics to him but he refused."

I think it does not simply say that because it wants me to believe (without having to actually demonstrate) that psychedelic use was common in [not clearly defined "social circle" that I suppose is intended to include CFAR leadership].  

If there is a defense of "Michael Vassar wasn't pushing psychedelics," it seems to me that it can and should be lodged separately from an accusation that "CFAR and its social circle (?) tacitly encouraged psychedelics, or at least included a lot of psychedelic use."

"Claim 6"

Scott asserts that Michael made people including me paranoid about MIRI/CFAR and that this contributes to psychosis. Before talking with Michael, I had already had a sense that people around me were acting harmfully towards me and/or the organization's mission. Michael and others talked with me about these problems, and I found this a relief.

Nothing wrong with this claim.  Did not cause me to feel confused or mentally yanked around.

"Claim 7"

If I hadn't noticed such harmful behavior, I would not have been fit for my nominal job. MIRI leaders were already privately encouraging me to adopt a kind of conflict theory in which many AI organizations were trying to destroy the world on <20-year timescales and could not be reasoned with about the alignment problem, such that aligned AGI projects including MIRI would have to compete with them.

I think this is the same sort of confusion as in Claim 4.  All members of a category are not the same.  I do not see how "MIRI leadership encouraged me to view other AI orgs through an adversarial lens" (which is a claim I'm much more suspicious of in light of the previous few hundred words, and how they seem to be trying to hypnotize me) necessarily implies "I would have been doing a bad job if I hadn't been viewing my colleagues with suspicion and scanning their actions for potential threats to me personally and to the org as a whole."

Obviously these two are compatible; it is absolutely possible for there to be a single mindset which was both useful for thinking about other AI orgs and also appropriate to turn on one's own experiences within MIRI, and it's possible for that very mindset to have been a necessary prerequisite for the type of work one was doing at MIRI.

But there are many, many more worlds in which two separate things are going on, and I do not like that Claim 7 tried to handwave me into not noticing this fact, and into thinking that two very-likely-separate things must obviously be linked (so obviously that the link does not need to be described).

"Claim 8"

MIRI's information security policies and other forms of local information suppression thus contributed to my psychosis. I was given ridiculous statements and assignments including the claim that MIRI already knew about a working AGI design and that it would not be that hard for me to come up with a working AGI design on short notice just by thinking about it, without being given hints. The information required to judge the necessity of the information security practices was itself hidden by these practices. While psychotic, I was extremely distressed about there being a universal cover-up of things-in-general.

I do not see the "thus."  

On the first layer, it is not clear to me that the infosec policies and information suppression are being accurately described, since I am not at all sure that Jessica and I would use the word "ridiculous" to describe similar things.  In my culture, I would have wanted this to say "ridiculous-to-me" rather than the more authoritative, statement-of-fact-sounding "ridiculous [period]."  

On the second layer, it is not clear to me that, even if accurately described, this can be said to "contribute to psychosis" by any mechanism other than "they happened around someone who was psychotic, and thus became a part of the psychosis."  I grant that, if one is becoming psychotic, and one is in a professionally-paranoid setting, this will not help.  But the claim leaves me with a tinge of "this is MIRI's fault" and I don't like having to swallow that tinge without knowing where it came from and whether I can trust the process that created it.

(Here I pause to reiterate two things: first, I am self-awaredly only diving into the summary, and the text likely contains much of the detail I'm seeking.  But these summaries are really strong, and really want me to draw certain conclusions, and they don't "own" their persuasive/assertive nature.  They keep leaning toward "and that's the way it was" in a way that is not demanded by the constraint of being-a-summary.  They could be much less epistemically weaponized and still be short and digestible.

Second, I'm pointing at everything that bothers me in a way that might seem churlish or unreasonable, but the reason I'm doing that is because I started to feel like I was being manipulated, and as a result I switched into "don't let yourself get manipulated" mode and that caused a lot of little things to leap out.  If I were not feeling like I was being manipulated, many of these things would not reach the level of being concerning, and might not even be consciously noticeable.  But once I notice a cluster of ten little manipulations, each of which was just below the radar, I raise the sensitivity of the radar.)

"Claim 9"

Scott asserts that the psychosis cluster was a "Vassar-related phenomenon". There were many memetic and personal influences on my psychosis, a small minority of which were due to Michael Vassar (my present highly-uncertain guess is that, to the extent that assigning causality to individuals makes sense at all, Nate Soares and Eliezer Yudkowsky each individually contributed more to my psychosis than did Michael Vassar, but that structural factors were important in such a way that attributing causality to specific individuals is to some degree nonsensical). Other people (Zack Davis and Devi Borg) who have been psychotic and were talking with Michael significantly commented to say that Michael Vassar was not the main cause. One person (Eric Bruylant) cited his fixation on Michael Vassar as a precipitating factor, but clarified that he had spoken very little with Michael and most of his exposure to Michael was mediated by others who likely introduced their own ideas and agendas.

This is excellent in my culture.  This is what I wish the other claims were like.

"Claim 10"

Scott asserts that Michael Vassar treats borderline psychosis as success. A text message from Michael Vassar to Zack Davis confirms that he did not treat my clinical psychosis as a success. His belief that mental states somewhat in the direction of psychosis, such as those had by family members of schizophrenics, are helpful for some forms of intellectual productivity is also shared by Scott Alexander and many academics.

A text message from Michael Arc (née Vassar) to Zack Davis provides marginal evidence that he does not explicitly claim to treat one instance of clinical psychosis as a success.

I believe that Scott's accusation is serious, and bears the burden of proof; I don't think that it's Jessica's or Michael's job to prove the accusation false.  But nevertheless, a single text message whose context is not described isn't proof of anything.  I cannot update on this, and I don't like that this claim takes me-updating-on-this for granted, and simply asserts confirmation from what is at best weak evidence.

The sentence beginning with "His belief" is untrustworthy.  What is the level of psychosis of "family members of schizophrenics"?  Is that a wide range?  What spots within the range are being pointed at?  Would Michael and Scott agree that they are pointing at the same states, when they both assert that they are "helpful"?  Would they mean similar things by "helpful"?  Would they be pointing at the same "forms of intellectual productivity"?  What "many academics"?

It's not that I expect a brief summary to answer all these questions.  It's more that, if you can't say something more clear and less confusing (/outright misleading) than something like this, then I think you should not include any such sentence at all.

A far better sentence (in my culture) would be something like "To the best of my own ability to understand the positions of both Michael and Scott, they have similar beliefs about which points on the sliding scale between [normal] and [psychotic] are useful, and for what reasons, and furthermore I think that their shared view is reasonably typical of many academics."

"Claim 11"

Scott asserts that Michael Vassar discourages people from seeking mental health treatment. Some mutual friends tried treating me at home for a week as I was losing sleep and becoming increasingly mentally disorganized before deciding to send me to a psychiatric institution, which was a reasonable decision in retrospect.

These two sentences bear no relation.  I think I am intended to be hypnotized into thinking that the second sentence provides circumstantial evidence against the first.

"Claim 12"

Scott asserts that most local psychosis cases were "involved with the Vassarites or Zizians". At least two former MIRI employees who were not significantly talking with Vassar or Ziz experienced psychosis in the past few years. Also, most or all of the people involved were talking significantly with others such as Anna Salamon (and read and highly regarded Eliezer Yudkowsky's extensive writing about how to structure one's mind, and read Scott Alexander's fiction writing about hell). There are about equally plausible mechanisms by which each of these were likely to contribute to psychosis, so this doesn't single out Michael Vassar or Ziz.

Again I have the sense that Jessica may be responding to sloppiness with sloppiness, which at least provides the justification of "I didn't break the peace treaty first, here."  But: it is not the case that all psychoses should be lumped together.  It is not the case that two people not-in-contact with Vassar or Ziz falsifies the claim that "most" of the cases were involved with the Vassarites or Zizians.  It is not the case that Eliezer and Scott's writings should be treated as being similar to extensive and intense in-person interactions (though the mention of Anna Salamon does seem like relevant and useful info (though again I would much prefer more of the specifically who and specifically what)).

But nothing after the first sentence actually contradicts the first sentence, and I really really really do not like how it tries to make me think that it did.

"Claim 13"

Scott Alexander asserts that MIRI should have discouraged me from talking about "auras" and "demons" and that such talk should be treated as a "psychiatric emergency". This increases the chance that someone like me could be psychiatrically incarcerated for talking about things that a substantial percentage of the general public (e.g. New Age people and Christians) talk about, and which could be explained in terms that don't use magical concepts. This is inappropriately enforcing the norms of a minority ideological community as if they were widely accepted professional standards.

Such talk as a sudden departure from one's previous norms of speech and thought seems to me to be quite likely to be a psychiatric emergency, in actual fact.  

I do not believe that our psychiatric systems are anywhere near perfect; I know of at least two cases of involuntary incarceration that seem to me to have been outrageously unjustified and some fear here seems reasonable.  But nevertheless, it does not seem likely to me that someone who can easily explain "What?  Oh, sorry—that's shorthand.  The actual non-crazy concept that the shorthand is for is not hard to explain, let me give you the five sentence version—" is at substantial risk for being thrown into a mental institution.

I'm not sure why this section wants me to be really very confident that [the proposed intervention] would not have helped prevent a slide into psychosis which Jessica seems to be largely laying at MIRI's feet.

I spent something like an hour on this.  I don't know if it is helpful to anyone besides myself, but it is helpful to me.  I feel much better equipped to navigate the remainder of this piece without "falling under its spell," so to speak.  I know to be on the lookout for:

• Summaries which may or may not be apt descriptions of the more detailed thing they are trying to summarize (e.g. assertions that something was "ridiculous" or "extensive")
• Language which is more authoritative, universal, or emphatic than it should be
• Attempts to link things via mere juxtaposition, without spelling out the connection between them
• Umbrella statements that admit of a wide range of interpretations and possibly equivocate between them.
• etc.

The absence of those things, and similar, makes it easier to see and think clearly, at least for me.

Their presence makes it harder to see and think clearly, at least for me.

Where it's important to see and think clearly, I think it's extra important to care about that.

Acknowledging one last time: I only dealt with the top-level summary; this is somewhat uncharitable and incomplete and a better version of me would have been able to make it through the whole thing, first. Responding to that top-level summary was the best I could manage with the resources I had at my disposal.

EDIT: I have as of this edit now read half of the larger text, and no, these issues are largely not resolved and are in many places exacerbated.  Reading this without "shields up" would cause a person to become quite seriously misled/confused, such as (for a single representative example) when Jessica begins with:

• a specific individual conveying to her a rumor that members of the precursor org to MIRI (prior to its split into MIRI and CFAR) had years earlier (seriously? jokingly?) discussed assassinating AGI researchers

... and a paragraph later casually refers to these discussions as if their existence is an absolute fact, saying "The obvious alternative hypothesis is that MIRI is not for real, and therefore hypothetical discussions about assassinations were just dramatic posturing."  And furthermore seems to want the reader to just ... nod along? ... with this being the sort of thing that would reasonably cause a person to fear for their own life, or at least contribute substantially to the development of such a fear/meaningfully justify such a fear.

I have strong downvoted this post.

(Meta: I'm talking about a bunch of stuff re: Jessica's epistemics out loud that I'd normally consider it a bit bad form to talk about out loud, but Jessicata seems to prefer having the conversation in public laying everything out clearly)

Something I've been feeling watching the discussion here I want to comment on (this is a bit off the cuff)

1. I share several commenter's feedback that Jessica's account is smuggling in assumptions and making wrong inferences about what people were communicating (or trying to communicate)
2. I think the people responding to this post (and the previous one) are also doing so from a position of defensiveness*, and I don't think their reactions have been uniformly fair.

For example of #2, while I agreed with the thrust of 'it seems misleading to leave out Vassar', I thought Scott A's comment [LW(p) · GW(p)] on the previous post made assumptions about the connection between Vassar and Ziz, and presented those assumptions in an overconfident tone. I also thought Logan's comment here [LW(p) · GW(p)] is doing a move of "focus-handling-their-way-towards understanding" in a way that seems totally legitimate as a way to figure out what's up, but which doing in public ends up creating a cloud of poetic rhetorical heft that's hard to argue with, which I think Benquo was correct to call out.

So if I imagine being Jessica, who seems to be earnestly** trying to figure out whether she's in a trustworthy or healthy environment, and I imagine running the query "do people seem to be unfairly characterizing me and distorting things?", that query would probably return "obviously yes."

It's possible for multiply parties in a debate to (correctly) notice that the other one is unfairly characterizing them / portraying them, at least a bit. And if you're already worried that you're in and social environment where you have to be on guard for people manipulating you or coercing you, it's possible to end up in an attractor state where everything (sorta legitimately) looks like evidence of this. 

...

*I'm not sure "defensiveness" captures the specific thing up with each of the commenters. Some of them seem more like neutral observers who don't (themselves) feel under attack by the OPs, but who are picking up on the same distortionary effects that the defensive people are picking up on and responding to.

...

**my current belief is that Jessica is doing a kinda a combination of "presenting things in a frame that looks pretty optimized as a confusing, hard-to-respond-to-social-attack", which at first looked disingenuous to me. I've since seen her respond to a number of comments in a way that looks concretely like it's trying to figure stuff out, update on new information, etc, without optimizing for maintaining the veiled social attack. My current belief is that there's still some kind of unconscious veiled social attack going on that Jessica doesn't have full conscious access to, it looks too optimized to be an accident. But I don't know that Jessica from within her current epistemic state should agree with me.

...

I think it's a fairly unsolved problem what Alices and Bobs are supposed to do in this class of situation. Many things that seem robustly, obviously "fair" to me from everyone's perspective require more effort than it's necessarily worth. 

I think it's important for people who feel like they are in a coercive environment to be given some leeway for trying to reason about that out loud. . But also I think people who feel coerced are often hypersensitized to things that in fact aren't a big deal (I think this is true even for 'legitimately' coerced people – once you're paying attention to a pattern that's threatening you, it makes sense to err on over-identifying it). And I don't think it's everyone else's obligation to go through refuting each thing. 

BUT, sometimes someone is correctly picking up on patterns of distortion, and it can take awhile to articulate exactly what's up with them. 

Right now I think the attention-allocation mechanisms we normally rely on on LessWrong are a bit messed up with this class of post, since people seem to vote in an over-the-top way where yay/boo is more salient.

I definitely have a strong sense reading this post that "those environmental conditions would not cause any problems to me" and I am trying to understand whether this is true or not, and if so, what properties of a person make them susceptible to the problems.

Do you have any perception about that? I wonder things like:

• Would you circa 2010 have been able to guess that you were susceptible to this level of suffering, if put in this kind of environment?
• What proportion of, say, random intellectually curious graduate students do you think would suffer this way if put into this environment?
• Do you have some sense of what psychological attributes made you susceptible, or advice to others about how to be less susceptible?

I have a lot of respect for what I know of you and your work and I'm sorry this happened.

(clarification edit: I have some sympathy for why it could be good to have an intellectual environment like this, so if my comment seems to be implying a perspective of "would it be possible to have it without people suffering", that's why.)

Like the previous post, there's something weird about the framing here that makes me suspicious of this. It feels like certain perspectives are being "smuggled in" -- for example:

Scott asserts that Michael Vassar thinks "regular society is infinitely corrupt and conformist and traumatizing". This is hyperbolic (infinite corruption would leave nothing to steal) but Michael and I do believe that the problems I experienced at MIRI and CFAR were not unique or unusually severe for people in the professional-managerial class. By the law of excluded middle, the only possible alternative hypothesis is that the problems I experienced at MIRI and CFAR were unique or at least unusually severe, significantly worse than companies like Google for employees' mental well-being.

This looks like a logical claim at first glance -- of course the only options are "the problems weren't unique or severe" or "the problems were unique and severe" -- but posing the matter this way conflates problems that you had as an individual ("the problems I experienced") with problems with the broader organization ("significantly worse... for employees' well-being"), which I do not think have been adequately established to exist.

I think this is improper because it jumps from problems you had as an individual to problems that applied to the well-being of employees as a whole without having proved that this was the case. In other words, it feels like this argument is trying to smuggle in the premise that the problems you experienced were also problems for a broader group of employees as a whole, which I think has not properly been established.

Another perspective -- and one which your framing seems to exclude -- would be that your experience was unusually severe, but that the unique or unusual element had to do with personal characteristics of yours, particular conflicts or interactions you-in-particular had with others in the organization, or similar.

Similarly, you seem to partially conflate the actions of Ziz, who I consider an outright enemy of the community, with actions of "mainstream" community leaders. This does not strike me as a very honest way to engage.

Thanks for this.

I've been trying to research and write something kind of like this giving more information for a while, but got distracted by other things. I'm still going to try to finish it soon.

While I disagree with Jessica's interpretations of a lot of things, I generally agree with her facts (about the Vassar stuff which I have been researching; I know nothing about the climate at MIRI). I think this post gives most of the relevant information mine would give. I agree with (my model of) Jessica that proximity to Michael's ideas (and psychedelics) was not the single unique cause of her problems but may have contributed.

The main thing I'd fight if I felt fighty right now is the claim that by not listening to talk about demons and auras MIRI (or by extension me, who endorsed MIRI's decision) is impinging on her free speech. I don't think she should face legal sanction for talking about this these, but I also don't think other people were under any obligation to take it seriously, including if she was using these terms metaphorically but they disagree with her metaphors or think she wasn't quite being metaphorical enough.

Re the postscript, I go over the claims here [LW(p) · GW(p)]

It should be noted that, as I was nominally Nate's employee, it is consistent with standard business practices for him to prevent me from talking with people who might distract me from my work; this goes to show the continuity between "cults" and "normal corporations".

This is very much not standard business practice. Working as an employee at four different "normal corporations" over thirteen years, I have never felt any pressure from my bosses (n=5) on who to talk to outside of work. And I've certainly been distracted at times!

Now that I'm a manager, I similarly would never consider this, even if I did think that one of my employees was being seriously distracted. If someone wasn't getting their work done or was otherwise not performing well, we would certainly talk about that, but who their contacts are is absolutely no business of mine.

This condition was triggered, Maia announced it, and Maia left the boat and killed themselves; Ziz and friends found Maia's body later.

Are you saying that Maia was at this time in San Francisco? That's an interesting claim given that the people in the European group house in which Maia lived in the preceding years knew, Maia died while being on vacation in Poland (their home country).

From their view, Maia was starting to take hormones to transition from male to female a few months earlier and then was spending time alone. That phase usually comes with pretty unstable psychological states.

Shortly before that, they wrote posts about how humans don't want happiness https://web.archive.org/web/20180104225807/http://squirrelinhell.blogspot.com/2017/12/happiness-is-chore.html There was also a later post about how they saw life as not worth living given that everything will be wiped out by death anyway sooner or later. I think that post got deleted, but I don't find an archive copy right now.

I only knew about the circumstances of the death after reading Ziz's account and previously believed the account of the roommates from the group house who seemed to be missing the crucial information about the interaction with Ziz. 

Without Ziz sharing information it's hard to look into the circumstances of their death.

"yeah, I don't really mind being the evil thing. Seems okay to me."

Regarding oneself as amoral seems to necessarily involve incoherence AFAICT. Like claiming you don't have your own oughts or that there is no structure to your oughts. In this case

Like doing a good work of art or playing an instrument. It feels satisfying.

is regarded as a higher good than the judgment of others. Loudly signaling that you don't care about the judgment of others seems to be a claim about what control surfaces you will and won't expose to social feedback. Nevertheless, the common sense prior that you shouldn't expect alliance with people who 'don't care about being evil' to go well seems appropriate.

To put it less abstractly: believe people when they say they are defecting rather than believe it is nested layers of fun and interesting counter signaling.

Note: I have no relation to MIRI/CFAR, no familiarity with this situation and am not a metal health expert, so I can't speak with any specific authority here.

First, I'd like to offer my sympathy for the suffering you described. I've had unpleasant intrusive thoughts before. They were pretty terrible, and I've never had them to the degree you've experienced. X/S risk research tends to generate a lot of intrusive thoughts and general stress. I think better community norms/support in this area could help a lot. Here is one technique you may find useful:

1. Raise your hand in front of your face with your palm facing towards you.
2. Fix your eyes on the tip of a particular finger.
3. Move your hand from side to side, while still tracking the chosen finger with your eyes (head remains still).
4. Every time your hand changes direction, switch which finger your eyes track. I.e., first track the tip of the thumb, then track the pointer finger, then the middle, then ring, then pinky, then back to thumb.

This technique combines three simultaneous control tasks (moving your hand, tracking the current finger, switching fingers repeatedly) and also saturates your visual field with the constantly moving background. I find it captures my full attention for as long as a I perform the technique and is therefore useful for interrupting intrusive/obsessive thoughts.

Another option for dealing with intrusive thoughts is to just start doing squats or similar exercises. This is less distracting from the thoughts, but physically healthier and may serve as negative reinforcement to your brain about avoiding intrusive thoughts in the future. 

I was told, by Nate Soares, that the pieces to make AGI are likely already out there and someone just has to put them together.  He did not tell me anything about how to make such an AGI, on the basis that this would be dangerous.  Instead, he encouraged me to figure it out for myself, saying it was within my abilities to do so.

Did Nate Soares specify that he wanted you to come up with a workable AGI design? If I'd heard my supervisor ask that of me, my interpretation would be something like: "Think about what sort of extra capabilities would make current AI more "agent-y" or general, and how such capabilities might be implemented."

For example, I might think that adding some sort of multi-episode memory to GPT-3 would make it more agent-y, and think about ways to integrate memory with transformers. Another idea in this ballpark is to train an ML system on human brain activity data as a form of knowledge distillation from humans to AI (as described by Gwern here). 

(Neither of these is among my most "dangerous" ideas about improving AI capabilities. I'm comfortable sharing the first because ~everyone already knows improved memory should be useful for general systems. I'm sharing the second because I think it will make many alignment approaches easier, especially value learning, interpretability, alignment by default and HCH).

I'd assume only ~20% at best of the approaches I thought of would actually be useful for capabilities, even if they were well implemented by a team of people smarter than me. If my supervisor then specified that they expected me to produce an actually workable AGI design, with significantly better performance than current state of the art systems, I'd have been blown away and seriously questioning their sanity/competence.

It might help your case to write a version of this that removes most of the interpretation you've given here, and tries to present just the claims you know to be objective truths. While ‘the plaintiff is failing to personally provide an objective neutral point of view’ seems like a particularly disturbing sort of argument to dismiss something like this on, it is nonetheless the case that this does seem to be the principal defense, and most of those comments are pointing to real issues in your presentation.

Disclaimer, I'm an outsider.

It seems to me that there's been a lot of debate about the causes of the psychosis and the suicides. I haven't seen the facts on the ground, so I can't know anything for sure. But as far as I can tell, the Vassarites and the core rationalists generally agree that MIRI&co aren't particularly bad, with the Vassarites just claiming, well:

Scott asserts that Michael Vassar thinks "regular society is infinitely corrupt and conformist and traumatizing". This is hyperbolic (infinite corruption would leave nothing to steal) but Michael and I do believe that the problems I experienced at MIRI and CFAR were not unique or unusually severe for people in the professional-managerial class. By the law of excluded middle, the only possible alternative hypothesis is that the problems I experienced at MIRI and CFAR were unique or at least unusually severe, significantly worse than companies like Google for employees' mental well-being.

And you give some relatively plausible defense of Vassar not being the main cause of the psychosis for the Vassarites. But that raises the question to me, why try to assign an environmental cause at all? It seems more reasonable to just say that the Vassarites are prone to psychosis, regardless of environment. At least I haven't heard of any clear evidence against this.

MIRI certainly had a substantially conflict-theoretic view of the broad situation, even if not the local situation.  I brought up the possibility of convincing DeepMind people to care about AI alignment.  MIRI leaders including Eliezer Yudkowsky and Nate Soares told me that this was overly naive, that DeepMind would not stop dangerous research even if good reasons for this could be given.  Therefore (they said) it was reasonable to develop precursors to AGI in-house to compete with organizations such as DeepMind in terms of developing AGI first.  So I was being told to consider people at other AI organizations to be intractably wrong, people who it makes more sense to compete with than to treat as participants in a discourse.

Anyone from MIRI want to comment on this? This seems weird, especially considering how open Demis/Legg have been to alignment arguments.

How did you conclude from Nate Soares saying that that the tools to create agi likely already exist that he wanted people to believe he knew how to construct one?

Why were none of these examples mentioned in the original discussion thread and comment section from which a lot of the quoted sections come from?

I was given ridiculous statements and assignments including the claim that MIRI already knew about a working AGI design and that it would not be that hard for me to come up with a working AGI design on short notice just by thinking about it, without being given hints.

There's a huge gulf between "AGI ideal that sounds like it will work to the researcher who came up with it" and "AGI idea that actually works when tried." Like, to the point where AGI researchers having ideas that they're insanely overconfident in, that don't work when tried, is almost being a trope. How much of an epistemic problem this is depends on the response to outside view, I think. If you remove the bluster, "working AGI design" turns into "research direction that sounds promising". I do think that, if it's going to be judged by the "promising research direction" standard rather than the "will actually work when tried" standard, then coming up with an AGI design is a pretty reasonable standard?

Yes Scott's analysis was more believable than yours. That continues to be the case.

I think high pressure environments such as startups and cults need to avoid having members who cannot handle high pressure, such as people genetically predisposed to psychotic breaks, or people who are taking too many psychedelics.

Cults actually do this. I claim expertise here because I have a master's degree in psychology of religion and some minor but peer-reviewed published papers on new religious movements. I have found that cults tend to have screening procedures to weed out the psychotic among the applicants, because these always make trouble. (Cults get a lot more of those applicants than the average proportion among the general population.) There are differences, but the commonality is that the person is taught a protective exercise (some of these I actually think might plausibly help) and told to do that exercise in case of distressing events of the particular type. Then if they continue to report such experiences, it is their own fault for having done too little of the exercise, and then that is reason enough to throw them out.

MIRI is not a cult but maybe it should learn from them.

Thank you for posting this. It clearly took a lot of effort, it deeply personal, and... places you in a weird spot relative to the "Community".

This is truth. Truth is sacred. Annddd.... God damn that does not seem like a safe/sensible workspace. I hope you are in a better headspace now, and get something of a break/ warmth/ whatever it happens to be that nourishes YOUR person.

I particularly want to state my appreciation for the orderly manner everything is presented which... isn't the standard way of approaching such a personal story/account, and presumably took extra effort, but was VERY helpful for me as a reader.

Take care. Keep good people around you. Be well.