We run the Center for Applied Rationality, AMA

This is my favorite question of the AMA so far (I said something similar aloud when I first read it, which was before it got upvoted quite this highly, as did a couple of other staff members). The things I personally appreciate about your question are: (1) it points near a core direction that CFAR has already been intending to try moving toward this year (and probably across near-subsequent years; one year will not be sufficient); and (2) I think you asking it publicly in this way (and giving us an opportunity to make this intention memorable and clear to ourselves, and to parts of the community that may help us remember) will help at least some with our moving there.

Relatedly, I like the way you lay out the concepts.

Your essay (I mean, “question”) is rather long, and has a lot of things in it; and my desired response sure also has a lot of things in it. So I’m going to let myself reply via many separate discrete small comments because that’s easier.

(So: many discrete small comments upcoming.)

Ben Pace writes:

In recent years, when I've been at CFAR events, I generally feel like at least 25% of attendees probably haven't read The Sequences, aren't part of this shared epistemic framework, and don't have any understanding of that law-based approach, and that they don't have a felt need to cache out their models of the world into explicit reasoning and communicable models that others can build on.

The “many alumni haven't read the Sequences” part has actually been here since very near the beginning (not the initial 2012 minicamps, but the very first paid workshops of 2013 and later). (CFAR began in Jan 2012.) You can see it in our old end-of-2013 fundraiser post [LW · GW], where we wrote “Initial workshops worked only for those who had already read the LW Sequences. Today, workshop participants who are smart and analytical, but with no prior exposure to rationality -- such as a local politician, a police officer, a Spanish teacher, and others -- are by and large quite happy with the workshop and feel it is valuable.” We didn't name this explicitly in that post, but part of the hope was to get the workshops to work for a slightly larger/broader/more cognitively diverse set than the set who for whom the original Sequences in their written form tended to spontaneously "click".

As to the “aren’t part of this shared epistemic framework” -- when I go to e.g. the alumni reunion, I do feel there are basic pieces of this framework at least that I can rely on. For example, even on contentious issues, 95%+ of alumni reunion participants seem to me to be pretty good at remembering that arguments should not be like soldiers, that beliefs are for true things, etc. -- there is to my eyes a very noticable positive difference between the folks at the alumni reunion and unselected-for-rationality smart STEM graduate students, say (though STEM graduate students are also notably more skilled than the general population at this, and though both groups fall short of perfection).

Still, I agree that it would be worthwhile to build more common knowledge and [whatever the “values” analog of common knowledge is called] supporting “a felt need to cache out their models of the world into explicit reasoning and communicable models that others can build on” and that are piecewise-checkable (rather than opaque masses of skills that are useful as a mass but hard to build across people and time). This particular piece of culture is harder to teach to folks who are seeking individual utility, because the most obvious payoffs are at the level of the group and of the long-term process rather than at the level of the individual (where the payoffs to e.g. goal-factoring and murphyjitsu are located). It also pays off more in later-stage fields and less in the earliest stages of science within preparadigm fields such as AI safety, where it’s often about shower thoughts and slowly following inarticulate hunches. But still.

that CFAR's natural antibodies weren't kicking against it hard.

Some of them were. This was a point of contention in internal culture discussions for quite a while.

(I am not currently a CFAR staff member, and cannot speak to any of the org's goals or development since roughly October 2018, but I can speak with authority about things that took place from October 2015 up until my departure at that time.)

Ben Pace writes:

“... The Gwerns and the Wei Dais and the Scott Alexanders of the world won't have learned anything from CFAR's exploration.”

I’d like to distinguish two things:

1. Whether the official work activities CFAR staff are paid for will directly produce explicit knowledge in the manner valued by the Gwern etc.
2. Whether that CFAR work will help educate people who later produce explicit knowledge themselves in the manner valued by Gwern etc., and who wouldn't have produced that knowledge otherwise.

#1 would be useful but isn’t our primary goal (though I think we’ve done more than none of it). #2 seems like a component of our primary goal to me (“scientists” or “producers of folks who can make knowledge in this sense” isn’t all we’re trying to produce, but it’s part of it), and is part of what I would like to see us strengthen over the coming years.

To briefly list our situation with respect to whether we are accomplishing #2 (according to me):

• There are in fact a good number of AI safety scientists in particular who seem to me to produce knowledge of this type, and who give CFAR some degree of credit for their present tendency to do this.
• On a milder level, while CFAR workshops do not themselves teach most of the Sequences’ skills (which would exceed four days in length, among other difficulties), we do try to nudge participants into reading the Sequences (by referencing them with respect at the workshop, by giving all mainline participants and AIRCS participants paper copies of “How to actually change your mind” and HPMOR, and by explicitly claiming they are helpful for various things).
• At the same time, I do think we should make Sequences-style thinking a more explicit element of the culture spread by CFAR workshops, and of the culture that folks can take for granted at e.g. alumni reunions (although it is there nevertheless to quite an extent).

(I edited this slightly to make it easier to read after Kaj had already quoted from it.)

With regard to whether our staff has read the sequences: five have, and have been deeply shaped by them; two have read about a third, and two have read little. I do think it’s important that our staff read them, and we decided to run this experiment with sabbatical months next year in part to ensure our staff had time to do this over the coming year.

I honestly think, in retrospect, that the linchpin of early CFAR's standard of good shared epistemics was probably Critch.

Note that Val's confusion seems to have been because he misunderstood Oli's point.

https://www.lesswrong.com/posts/tMhEv28KJYWsu6Wdo/kensh?commentId=SPouGqiWNiJgMB3KW#SPouGqiWNiJgMB3KW [LW(p) · GW(p)]

Re: 1—“Forked codebases that have a lot in common but are somewhat tricky to merge” seems like a pretty good metaphor to me.

The question I'd like to answer that is near your questions is: "What is the minimal patch/bridge that will let us use all of both codebases without running into merge conflicts?"

We do have a candidate answer to this question, which we’ve been trying out at AIRCS to reasonable effect. Our candidate answer is something like: an explicit distinction between “tacit knowledge” (inarticulate hunches, early-stage research intuitions, the stuff people access and see in one another while circling, etc.) and the “explicit” (“knowledge” worthy of the name, as in the LW codebase—the thing I believe Ben Pace is mainly gesturing at in his comment [LW(p) · GW(p)] above).

Here’s how we explain it at AIRCS:

• By “explicit” knowledge, we mean visible-to-conscious-consideration denotative claims that are piecewise-checkable and can be passed explicitly between humans using language.
  • Example: the claim “Amy knows how to ride a bicycle” is explicit.
• By “tacit” knowledge, we mean stuff that allows you to usefully navigate the world (and so contains implicit information about the world, and can be observationally evaluated for how well people seem to navigate the relevant parts of the world when they have this knowledge) but is not made of explicit denotations that can be fully passed verbally between humans.
  • Example: however the heck Amy actually manages to ride the bicycle (the opaque signals she sends to her muscles, etc.) is in her own tacit knowledge. We can know explicitly “Amy has sufficient tacit knowledge to balance on a bicycle,” but we cannot explicitly track how she balances, and Amy cannot hand her bicycle-balancing ability to Bob via speech (although speech may help). Relatedly, Amy can’t check the individual pieces of her (opaque) motor patterns to figure out which ones are the principles by which she successfully stays up and which are counterproductive superstition.
• I’ll give a few more examples to anchor the concepts:
  • In mathematics:
    • Explicit: which things have been proven; which proofs are valid.
    • Tacit: which heuristics may be useful for finding proofs; which theorems are interesting/important. (Some such heuristics can be stated explicitly, but I wouldn't call those statements “knowledge.” I can't verify that they're right in the way I can verify “Amy can ride a bike” or “2+3=5.”)
  • In science:
    • Explicit: specific findings of science, such as “if you take a given amount of hydrogen and decrease its volume by half, you double its pressure." The “experiment” and “conclusion” steps of the scientific method.
    • Tacit: which hypotheses are worth testing.
  • In Paul Graham-style startups:
    • Explicit: what metrics one is hitting, once one achieves an MVP.
    • Tacit: the way Graham’s small teams of cofounders are supposed to locate their MVP. (In calling this “tacit,” I don’t mean you can’t communicate any of this verbally. Of course they use words. But they way they use words is made of ad hoc spaghetti-code bits of attempt to get gut intuitions back and forth between a small set of people who know each other well. It is quite different from the scalable processes of explicit science/knowledge that can compile across large sets of people and long periods of time. This is why Graham claims that co-founder teams should have 2-4 people, and that if you hire e.g. 10 people to a pre-MVP startup, it won’t scale well.)

In the context of the AIRCS workshop, we share “The Tacit and the Explicit” in order to avoid two different kinds of errors:

• People taking “I know it in my gut” as zero-value, and attempting to live via the explicit only. My sense is that some LessWrong users like Said_Achmiz tend to err in this direction. (This error can be fatal to early-stage research, and to one’s ability to discuss ordinary life/relationship/productivity “bugs” and solutions, and many other mundanely useful topics.)
• People taking “I know it in my gut” as vetted knowledge, and attempting to build on gut feelings in the manner of knowledge. (This error can be fatal to global epistemology: “but I just feel that religion is true / the future can’t be that weird / whatever”).

We find ourselves needing to fix both those errors in order to allow people to attempt grounded original thinking about AI safety. They need to be able to have intuitions, and take those intuitions seriously enough to develop them / test them / let them breathe, without mistaking those intuitions for knowledge.

So, at the AIRCS workshop, we introduce the explicit (which is a big part of what I take Ben Pace to be gesturing at above actually) at the same time that we introduce the tacit (which is the thing that Ben Pace describes benefiting from at CFAR IMO). And we introduce a framework to try to keep them separate so that learning cognitive processes that help with the tacit will not accidentally mess with folks’ explicit, nor vice versa. (We’ve been introducing this framework at AIRCS for about a year, and I do think it’s been helpful. I think it’s getting to the point where we could try writing it up for LW—i.e., putting the framework more fully into the explicit.)

I like the metaphor!

Just wanted to note: in my view the original LW Sequences are not functional as a stand-alone upgrade for almost any human mind, and you can empirically observe it: You can think about any LW meet-up group around the world as an experiment, and I think to a first approximation it's fair to say aspiring Rationalists running just on the Sequences do not win, and good stuff coming out of the rationalist community was critically dependent of presence of minds Eliezer & others. (This is not say Sequences are not useful in many ways)

My closest current stab is that we’re the “Center for Bridging between Common Sense and Singularity Scenarios.” (This is obviously not our real name. But if I had to grab a handle that gestures at our raison d’etre, at the moment I’d pick this one. We’ve been internally joking about renaming ourselves this for some months now.)

To elaborate: thinking about singularity scenarios is profoundly disorienting (IMO, typically worse than losing a deeply held childhood religion or similar). Folks over and over again encounter similar failure modes as they attempt this. It can be useful to have an institution for assisting with this -- collecting concepts and tools that were useful for previous waves who’ve attempted thought/work about singularity scenarios, and attempting to pass them on to those who are currently beginning to think about such scenarios.

Relatedly, the pattern of thinking required for considering AI risk and related concepts at all is pretty different from the patterns of thinking that suffice in most other contexts, and it can be useful to have a group that attempts to collect these and pass them forward.

Further, it can be useful to figure out how the heck to do teams and culture in a manner that can withstand the disruptions that can come from taking singularity scenarios seriously.

So, my best current angle on CFAR is that we should try to be a place that can help people through these standard failure modes -- a place that can try to answer the question “how can we be sane and reasonable and sensible and appropriately taking-things-seriously in the face of singularity scenarios,” and can try to pass on our answer, and can notice and adjust when our answer turns out to be invalid.

To link this up with our concrete activities:

AIRCS workshops / MSFP:

• Over the last year, about half our staff workshop-days [LW · GW] went into attempting to educate potential AI alignment researchers. These programs were co-run with MIRI. Workshops included a bunch about technical AI content; a bunch of practice thinking through “is there AI risk” and “how the heck would I align a superintelligence” and related things; and a bunch of discussion of e.g. how to not have “but the stakes are really really big” accidentally overwhelm one’s basic sanity skills (and other basic pieces of how to not get too disoriented).
• Many program alumni attended multiple workshops, spaced across time, as part of a slow acculturation process: stare at AI risk; go back to one’s ordinary job/school context for some months while digesting in a back-burner way; repeat.
• These programs aim at equipping people to contribute to Al alignment technical work at MIRI and elsewhere; in the last two years it’s helped educate a sizable number of MIRI hires and smaller but still important number of others (brief details in our 2019 progress report [LW · GW]; more details coming eventually). People sometime try to gloss the impact of AIRCS as “outreach” or “causing career changes,” but, while I think it does in fact fulfill CEA-style metrics, that doesn’t seem to me like a good way to see its main purple -- helping folks feel their way toward being more oriented and capable around these topics in general, in a context where other researchers have done or are doing likewise.
• They seem like a core activity for a “Center for bridging between common sense and singularity scenarios” -- both in that they tell us more about what happens when folks encounter AI risk, and in that they let us try to use what we think we know for good. (We hope it’s “good.”)

Mainline workshops, alumni reunions, alumni workshops unrelated to AI risk, etc.:

• We run mainline workshops (which many people just call “CFAR workshops”), alumni reunions, and some topic-specific workshops for alumni that have nothing to do with AI risk (e.g., a double crux workshop). Together, this stuff constituted about 30% [LW · GW] of our staff workshop-days over the last two years.
• The EA crowd often asks me why we run these. (“Why not just run AI safety workshops, since that is the part of your work that has more shot at helping large numbers of people?”) The answer is that when I imagine removing the mainline workshops, CFAR begins to feel like a table without enough legs -- unstable, liable to work for awhile but then fall over, lacking enough contact with the ground.
• More concretely: we’re developing and spreading a nonstandard mental toolkit (inner sim, double crux, Gendlin’s Focusing, etc.). That’s a tricky and scary thing to do. It’s really helpful to get to try it on a variety of people -- especially smart, thoughtful, reflective, articulate people who will let us know what seems like a terrible idea, or what causes them help in their lives, or disruption in their lives. The mainline workshops (plus follow-up sessions, alumni workshops, alumni reunions, etc.) let us develop this alleged “bridge” between common sense and singularity scenarios in a way that avoids overfitting it all to just “AI alignment work.” Which is basically to say that they let us develop and test our models of “applied rationality”.

“Sandboxes” toward trying to understand how to have a healthy culture in contact with AI safety:

• I often treat the AIRCS workshops as “sandboxes”, and try within them to create small temporary “cultures” in which we try to get research to be able to flourish, or try to get people to be able to both be normal humans and slowly figure out how to approach AI alignment, or whatever. I find them a pretty productive vehicle for trying to figure out the “social context” thing, and not just the “individual thinking habits” thing. I care about this experimentation-with-feedback because I want MIRI and other longer-term teams to eventually have the right cultural base.

Our instructor training program, and our attempt to maintain a staff who is skilled at seeing what cognitive processes are actually running in people:

• There’s a lot of trainable, transferable skill to seeing what people are thinking. CFAR staff have a bunch of this IMO, and we seem to me to be transferring a bunch of it to the instructor candidates too. We call it “seeking PCK”.
• The “seeking PCK” skillset is obviously helpful for learning to “bridge between common sense and singularity scenarios” -- it helps us see what the useful patterns folks have are, and what the not-so-useful patterns folks have are, and what exactly is happening as we attempt to intervene (so that we can adjust our interventions).
• Thus, improving and maintaining the “seeking PCK” skillset probably makes us faster at developing any other curriculum.
• More mundanely, of course, instructor training also gives us guest instructors who can help us run workshops -- many of whom are also out and about doing other interesting things, and porting wisdom/culture/data back and forth between those endeavors and our workshops.

To explain what “bridging betwen common sense and singularity scenarios” has to do with “applied rationality” and the LW Sequences and so on:

• The farther off you need to extrapolate, the more you need reasoning (vs being able to lean on either received wisdom, or known data plus empirical feedback loops). And singularity scenarios sure are far from the everyday life our heuristics are developed for, so singularity scenarios benefit more than most from trying to be the lens that sees its flaws [LW · GW], and from Sequences-style thinking more broadly.

Here’s a very partial list of blog post ideas from my drafts/brainstorms folder. Outside view, though, if I took the time to try to turn these in to blog posts, I’d end up changing my mind about more than half of the content in the process of writing it up (and then would eventually end up with blog posts with somewhat different these).

I’m including brief descriptions with the awareness that my descriptions may not parse at this level of brevity, in the hopes that they’re at least interesting teasers.

Contra-Hodgel

• (The Litany of Hodgell says “That which can be destroyed by the truth should be”. Its contrapositive therefore says: “That which can destroy [that which should not be destroyed] must not be the full truth.” It is interesting and sometimes-useful to attempt to use Contra-Hodgel as a practical heuristic: “if adopting belief X will meaningfully impair my ability to achieve good things, there must be some extra false belief or assumption somewhere in the system, since true beliefs and accurate maps should just help (e.g., if “there is no Judeo-Christian God” in practice impairs my ability to have good and compassionate friendships, perhaps there is some false belief somewhere in the system that is messing with that).

The 50/50 rule

• The 50/50 rule is a proposed heuristic claiming that about half of all progress on difficult projects will come from already-known-to-be-project-relevant subtasks -- for example, if Archimedes wishes to determine whether the king’s crown is unmixed gold, he will get about half his progress from diligently thinking about this question (plus subtopics that seem obviously and explicitly relevant to this question). The other half of progress on difficult projects (according to this heuristic) will come from taking an interest in the rest of the world, including parts not yet known to be related to the problem at hand -- in the Archimedes example, from Archimedes taking an interest in what happens to his bathwater.
• Relatedly, the 50/50 rule estimates that if you would like to move difficult projects forward over long periods of time, it is often useful to spend about half of your high-energy hours on “diligently working on subtasks known-to-be-related to your project”, and the other half taking an interest in the world.

Make New Models, but Keep The Old

• “... one is silver and the other’s gold.”
• A retelling of: it all adds up to normality.

On Courage and Believing In.

• Beliefs are for predicting what’s true [LW · GW]. “Believing in”, OTOH, is for creating a local normal that others can accurately predict. For example: “In America, we believe in driving on the right hand side of the road” -- thus, when you go outside and look to predict which way people will be driving, you can simply predict (believe) that they’ll be driving on the right hand side.
• Analogously, if I decide I “believe in” [honesty, or standing up for my friends, or other such things], I create an internal context in which various models within me can predict that my future actions will involve [honesty, or standing up for my friends, or similar].
• It’s important and good to do this sometimes, rather than having one’s life be an accidental mess with nobody home choosing. It’s also closely related to courage.

Ethics for code colonies

• If you want to keep caring about people, it makes a lot of sense to e.g. take the time to put your shopping cart back where it goes, or at minimum not to make up excuses about how your future impact on the world makes you too important to do that.
• In general, when you take an action, you summon up black box code-modification that takes that action (and changes unknown numbers of other things). Life as a “code colony” is tricky that way.
• Ethics is the branch of practical engineering devoted to how to accomplish things with large sets of people over long periods of time -- or even with one person over a long period of time in a confusing or unknown environment. It’s the art of interpersonal and intrapersonal coordination. (I mean, sometimes people say “ethics” means “following this set of rules here”. But people also say “math” means “following this algorithm whenever you have to divide fractions” or whatever. And the underneath-thing with ethics is (among other things, maybe) interpersonal and intra-personal coordination, kinda like how there’s an underneath-thing with math that is where those rules come from.)
• The need to coordinate in this way holds just as much for consequentialists or anyone else.
• It's kinda terrifying to be trying to do this without a culture. Or to be not trying to do this (still without a culture).

The explicit and the tacit (elaborated a bit in a comment [LW(p) · GW(p)] in this AMA; but there’s room for more).

Cloaks, Questing, and Cover Stories

• It’s way easier to do novel hypothesis-generation if you can do it within a “cloak”, without making any sort of claim yet about what other people ought to believe. (Teaching this has been quite useful on a practical level for many at AIRCS, MSFP, and instructor trainings -- seems worth seeing if it can be useful via text, though that’s harder.)

Me-liefs, We-liefs, and Units of Exchange

• Related to “cloaks and cover stories” -- we have different pools of resources that are subject to different implicit contracts and commitments. Not all Bayesian evidence is judicial or scientific evidence, etc.. A lot of social coordination works by agreeing to only use certain pools of resources in agreement with certain standards of evidence / procedure / deference (e.g., when a person does shopping for their workplace they follow their workplace’s “which items to buy” procedures; when a physicist speaks to laypeople in their official capacity as a physicist, they follow certain procedures so as to avoid misrepresenting the community of physicists).
• People often manage this coordination by changing their beliefs (“yes, I agree that drunk driving is dangerous -- therefore you can trust me not to drink and drive”). However, personally I like the rule “beliefs are for true things -- social transactions can make my requests of my behaviors but not of my beliefs.” And I’ve got a bunch of gimmicks for navigating the “be robustly and accurately seen as prosocial” without modifying one’s beliefs (“In my driving, I value cooperating with the laws and customs so as to be predictable and trusted and trustworthy in that way; and drunk driving is very strongly against our customs -- so you can trust me not to drink and drive.”)

How the Tao unravels

• A book review of part of CS Lewis’s book “The abolition of man.” Elaborates CS Lewis’s argument that in postmodern times, people grab hold of part of humane values and assert it in contradiction with other parts of humane values, which then assert back the thing that they’re holding and the other party is missing, and then things fragment further and further. Compares Lewis’s proposed mechanism with how cultural divides have actually been going in the rationality and EA communities over the last ten years.

Some off the top of my head.

• A bunch of Double Crux posts that I keep promising but am very bad at actually finishing.
• The Last Term Problem (or why saving the world is so much harder than it seems) - A abstract decision theoretic problem that has confused me about taking actions at all for the past year.
• A post on how the commonly cited paper on how "Introspection is Impossible" (Nisbett and Wilson) is misleading.
• Two takes on confabulation - About how the Elephant in the Brain thesis doesn't imply that we can't tell what our motivations actually are, just that we aren't usually motivated to.
• A lit review on mental energy and fatigue.
• A lit review on how attention works.

Most of my writing is either private strategy documents, or spur of the moment thoughts / development-nuggets that I post here.

This isn’t a direct answer to, “What are the LessWrong posts that you wish you had the time to write?” It is a response to a near-by question, though, which is probably something along the lines of, “What problems are you particularly interested in right now?” which is the question that always drives my blogging. Here’s a sampling, in no particular order.

[edit: cross-posted to Ray's Open Problems post [LW · GW].]

There are things you’re subject to, and things you can take as object. For example, I used to do things like cry when an ambulance went by with its siren on, or say “ouch!” when I put a plate away and it went “clink”, yet I wasn’t aware that I was sensitive to sounds. If asked, “Are you sensitive to sounds?” I’d have said “No.” I did avoid certain sounds in local hill-climby ways, like making music playlists with lots of low strings but no trumpets, or not hanging out with people who speak loudly. But I didn’t “know” I was doing these things; I was *subject* to my sound sensitivity. I could not take it as *object*, so I couldn’t deliberately design my daily life to account for it. Now that I can take my sound sensitivity (and many related things) as object, I’m in a much more powerful position. And it *terrifies* me that I went a quarter of a century without recognizing these basic facts of my experience. It terrifies me even more when I imagine an AI researcher being subject to some similarly crucial thing about how agents work. I would very much like to know what other basic facts of my experience I remain unaware of. I would like to know how to find out what I am currently unable to take as object.

On a related note, you know how an awful lot of people in our community are autistic? It seem to me that our community is subject to this fact. (It also seems to me that many individual people in our community remain subject to most of their autistic patterns, and that this is more like the rule than the exception.) I would like to know what’s going on here, and whether some other state of affairs would be preferable, and how to instantiate that state of affairs.

Why do so many people seem to wait around for other people to teach them things, even when they seem to be trying very hard to learn? Do they think they need permission? Do they think they need authority? What are they protecting? Am I inadvertently destroying it when I try to figure things out for myself? What stops people from interrogating the world on their own terms?

I get an awful lot of use out of asking myself questions. I think I’m unusually good at doing this, and that I know a few other people with this property. I suspect that the really useful thing isn’t so much the questions, as whatever I’m doing with my mind most of the time that allows me to ask good questions. I’d like to know what other people are doing with their minds that prevents this, and whether there’s a different thing to do that’s better.

What is “quality”?

Suppose religion is symbiotic, and not just parasitic. What exactly is it doing for people? How is it doing those things? Are there specific problems it’s solving? What are the problems? How can we solve those problems without tolerating the damage religion causes?

[Some spoilers for bits of the premise of A Fire Upon The Deep and other stories in that sequence.] There’s this alien race in Verner Vinge books called the Tines. A “person” of the Tines species looks at first like a pack of several animals. The singleton members that make up a pack use high-frequency sound, rather than chemical neurotransmitters, to think as one mind. The singleton members of a pack age, so when one of your singletons dies, you adopt a new singleton. Since singletons are all slightly different and sort of have their own personalities, part of personal health and hygiene for Tines involves managing these transitions wisely. If you do a good job — never letting several members die in quick succession, never adopting a singleton that can’t harmonize with the rest of you, taking on new singletons before the oldest ones loose the ability to communicate — then you’re effectively immortal. You just keep amassing new skills and perspectives and thought styles, without drifting too far from your original intentions. If you manage the transitions poorly, though — choosing recklessly, not understanding the patterns an old member has been contributing, participating in a war where several of your singletons may die at once — then your mind could easily become suddenly very different, or disorganized and chaotic, or outright insane, in a way you’ve lost the ability to recover from. I think about the Tines a lot when I experiment with new ways of thinking and feeling. I think much of rationality poses a similar danger to the one faced by the Tines. So I’d like to know what practices constitute personal health and hygiene for cognitive growth and development in humans.

What is original seeing? How does it work? When is it most important? When is it the wrong move? How can I become better at it? How can people who are worse at it than I am become better at it?

In another thread, Adam made a comment that I thought was fantastic. I typed to him, “That comment is fantastic!” As I did so, I noticed that I had an option about how to relate to the comment, and to Adam, when I felt a bid from somewhere in my mind to re-phrase as, “I really like that comment,” or, “I enjoyed reading your comment,” or “I’m excited and impressed by your comment.” That bid came from a place that shares a lot of values with Lesswrong-style rationalists, and 20th century science, and really with liberalism in general. It values objectivity, respect, independence, autonomy, and consent, among other things. It holds map-territory distinctions and keeps its distance from the world, in an attempt to see all things clearly. But I decided to stand behind my claim that the “the comment is fantastic”. I did not “own my experience”, in this case, or highlight that my values are part of me rather than part of the world. I have a feeling that something really important is lost in the careful distance we keep all the time from the world and from each other. Something about the power to act, to affect each other in ways that create small-to-mid-sized superorganisms like teams and communities, something about tending our relationship to the world so that we don’t float off in bubbles of abstraction. Whatever that important thing is, I want to understand it. And I want to protect it, and to incorporate it into my patterns of thought, without loosing all I gain from cold clarity and distance.

I would like to think more clearly, especially when it seems important to do so. There are a lot of things that might affect how clearly you think, some of which are discussed in the Sequences. For example, one common pattern of muddy thought is rationalization, so one way to increase your cognitive clarity is to stop completely ignoring the existence of rationalization. I’ve lately been interested in a category of clarity-increasing thingies that might be sensibly described as “the relationship between a cognitive process and its environment”. By “environment”, I meant to include several things:

• The internal mental environment: the cognitive and emotional situation in which a thought pattern finds itself. Example: When part of my mind is trying to tally up how much money I spent in the past month, and local mental processes desperately want the answer to be “very little” for some reason, my clarity of thought while tallying might not be so great. I expect that well maintained internal mental environments — ones that promote clear thinking — tend to have properties like abundance, spaciousness, and groundedness.
• The internal physical environment: the physiological state of a body. For example, hydration seems to play a shockingly important role in how well I maintain my internal mental environment while I think. If I’m trying to solve a math problem and have had nothing to drink for two hours, it’s likely I’m trying to work in a state of frustration and impatience. Similar things are true of sleep and exercise.
• The external physical environment: the sensory info coming in from the outside world, and the feedback patterns created by external objects and perceptual processes. When I’ve been having a conversation in one room, and then I move to another room, it often feels as though I’ve left half my thoughts behind. I think this is because I’m making extensive use of the walls and couches and such in my computations. I claim that one’s relationship to the external environment can make more or less use of the environment’s supportive potential, and that environments can be arranged in ways that promote clarity of thought (see Adam’s notes on the design of the CFAR venue, for instance).
• The social environment: people, especially frequently encountered ones. The social environment is basically just part of the external physical environment, but it’s such an unusual part that I think it ought to be singled out. First of all, it has powerful effects on the internal mental environment. The phrase “politics is the mind killer” means something like “if you want to design the social environment to maximize muddiness of thought, have I got a deal for you”. Secondly, other minds have the remarkable property of containing complex cognitive processes, which are themselves situated in every level of environment. If you’ve ever confided in a close, reasonable friend who had some distance from your own internal turmoil, you know what I’m getting at here. I’ve thought a lot lately about how to build a “healthy community” in which to situate my thoughts. A good way to think about what I’m trying to do is that I want to cultivate the properties of interpersonal interaction that lead to the highest quality, best maintained internal mental environments for all involved.

I built a loft bed recently. Not from scratch, just Ikea-style. When I was about halfway through the process, I realized that I’d put one of the panels on backward. I’d made the mistake toward the beginning, so there were already many pieces screwed into that panel, and no way to flip it around without taking the whole bed apart again. At that point, I had a few thoughts in quick succession:

• I really don’t want to take the whole bed apart and put it back together again.
• Maybe I could unscrew the pieces connected to that panel, then carefully balance all of them while I flip the panel around? (Something would probably break if I did that.)
• You know what, maybe I don’t want a dumb loft bed anyway.

It so happens that in this particular case, I sighed, took the bed apart, carefully noted where each bit was supposed to go, flipped the panel around, and put it all back together again perfectly. But I’ve certainly been in similar situations where for some reason, I let one mistake lead to more mistakes. I rushed, broke things, lost pieces, hurt other people, or gave up. I’d like to know what circumstances obtain when I have get this right, and what circumstances obtain when I don’t. Where can I get patience, groundedness, clarity, gumption, and care?

What is "groundedness"?

I’ve developed a taste for reading books that I hate. I like to try on the perspective of one author after another, authors with whom I think I have really fundamental disagreements about how the world works, how one ought to think, and whether yellow is really such a bad color after all. There’s a generalized version of “reading books you hate” that I might call “perceptual dexterity”, or I might call “the ground of creativity”, which is something like having a thousand prehensile eye-stalks in your mind, and I think prehensile eye-stalks are pretty cool. But I also think it’s generally a good idea to avoid reading books you hate, because your hatred of them is often trying to protect you from “your self and worldview falling apart”, or something. I’d like to know whether my self and worldview are falling apart, or whatever. And if not, I’d like to know whether I’m doing something to prevent it that other people could learn to do, and whether they’d thereby gain access to a whole lot more perspective from which they could triangulate reality.

I have a Google Doc full of ideas. Probably I'll never write most of these, and if I do probably much of the content will change. But here are some titles, as they currently appear in my personal notes:

• Mesa-Optimization in Humans
• Primitivist Priors v. Pinker Priors
• Local Deontology, Global Consequentialism
• Fault-Tolerant Note-Scanning
• Goal Convergence as Metaethical Crucial Consideration
• Embodied Error Tracking
• Abnormally Pleasurable Insights
• Burnout Recovery
• Against Goal "Legitimacy"
• Computational Properties of Slime Mold
• Steelmanning the Verificationist Criterion of Meaning
• Manual Tribe Switching
• Manual TAP Installation
• Keep Your Hobbies

I don’t think that time is my main constraint, but here are some of my blog post shaped ideas:

• Taste propagates through a medium
• Morality: do-gooding and coordination
• What to make of ego depletion research
• Taboo "status"
• What it means to become calibrated
• The NFL Combine as a case study in optimizing for a proxy
• The ability to paraphrase
• 5 approaches to epistemics

My guesses, in no particular order:

• Being a first employee is pretty different from being in a middle-stage organization. In particular, the opportunity to shape what will come has an appeal that can I think rightly bring in folks who you can’t always get later. (Folks present base rates for various reference classes below; I don’t know if anyone has one for “founding” vs “later” in small organizations?)

  • Relatedly, my initial guess back in ~2013 (a year in) was that many CFAR staff members would “level up” while they were here and then leave, partly because of that level-up (on my model, they’d acquire agency and then ask if being here as one more staff member was or wasn’t their maximum-goal-hitting thing). I was excited about what we were teaching and hoped it could be of long-term impact to those who worked here a year or two and left, as well as to longer-term people.

• I and we intentionally hired for diversity of outlook. We asked ourselves: “does this person bring some component of sanity, culture, or psychological understanding -- but especially sanity -- that is not otherwise represented here yet?” And this… did make early CFAR fertile, and also made it an unusually difficult place to work, I think. (If you consider the four founding members of me, Julia Galef, Val, and Critch, I think you’ll see what I mean.)

• I don’t think I was very easy to work with. I don’t think I knew how to make CFAR a very easy place to work either. I was trying to go with inside views even where I couldn’t articulate them and… really didn’t know how to create a good interface between that and a group of people. Pete and Duncan helped us do otherwise more recently I think, and Tim and Adam and Elizabeth and Jack and Dan building on it more since, with the result that CFAR is much more of a place now (less of a continuous “each person having an existential crisis all the time” than it was for some in the early days; more of a plod of mundane work in a positive sense). (The next challenge here, which we hope to accomplish this year, is to create a place that still has place-ness, and also has more visibility into strategy.)

• My current view is that being at workshops for too much of a year is actually really hard on a person, and maybe not-good. It mucks up a person’s codebase without enough chance for ordinary check-sums to sort things back to normal again afterward. Relatedly, my guess is also that while stints at CFAR do level a person up in certain ways (~roughly as I anticipated back in 2013), they unfortunately also risk harming a person in certain ways that are related to “it’s not good to live in workshops or workshop-like contexts for too many weeks/months in a row, even though a 4-day workshop is often helpful” (which I did not anticipate in 2013). (Basically: you want a bunch of normal day-to-day work on which to check whether your new changes actually work well, and to settle back into your deeper or more long-term self. The 2-3 week “MIRI Summer Fellows Program” (MSFP) has had… some great impacts in terms of research staff coming out of the program, but also most of our least stable people additionally came out of that. I believe that this year we’ll be experimentally replacing it with repeated shorter workshops; we’ll also be trying a different rest days pattern for staff, and sabbatical months, as well as seeking stability/robustness/continuity in more cultural and less formal ways.)

(This is Dan, from CFAR since 2012)

Working at CFAR (especially in the early years) was a pretty intense experience, which involved a workflow that regularly threw you into these immersive workshops, and also regularly digging deeply into your thinking and how your mind works and what you could do better, and also trying to make this fledgling organization survive & function. I think the basic thing that happened is that, even for people who were initially really excited about taking this on, things looked different for them a few years later. Part of that is personal, with things like burnout, or feeling like they’d gotten their fill and had learned a large chunk of what they could from this experience, or wanting a life full of experiences which were hard to fit in to this (probably these 3 things overlap). And part of it was professional, where they got excited about other projects for doing good in the world while CFAR wanted to stay pretty narrowly focused on rationality workshops.

I’m tempted to try to go into more detail, but it feels like that would require starting to talk about particular individuals rather the set of people who were involved in early CFAR and I feel weird about that.

So I’m imagining there might be both a question (for what types of reasons have CFAR staff left?) and a claim (CFAR’s rate of turnover is unusual) here. Anna should be able to better address the question, but with regard to the claim: I think it’s true, at least relative to average U.S. turnover. The median length Americans spend in jobs is 4.2 years, while the median length CFAR employees have stayed in their jobs is 2.2 years; 32% of our employees (7 people) left within their first year.

My rough guess is “we survived; most of the differences I could imagine someone fearing didn’t come to pass”. My correction on that rough guess is: “Okay, but insofar as Duncan was the main holder of certain values, skills, and virtues, it seems pretty plausible that there are gaps now today that he would be able to see and that we haven’t seen”.

To be a bit more specific: some of the poles I noticed Duncan doing a lot to hold down while he was here were:

• Institutional accountability and legibility;
• Clear communication with staff; somebody caring about whether promises made were kept; somebody caring whether policies were fair and predictable, and whether the institution was creating a predictable context where staff, workshop participants, and others wouldn’t suddenly experience having the rug pulled out from under them;
• Having the workshop classes start and end on time; (I’m a bit hesitant to name something this “small-seeming” here, but it is a concrete policy that supported the value above, and it is easier to track)
• Revising the handbook into a polished state;
• Having the workshop classes make sense to people, have clear diagrams and a clear point, etc.; having polish and visible narrative and clear expectations in the workshop;

AFAICT, these things are doing… alright in the absence of Duncan (due partly to the gradual accumulation of institutional knowledge), though I can see his departure in the organization. AFAICT also, Duncan gave me a good chunk of model of this stuff sometime after his Facebook post, actually -- and worked pretty hard on a lot of this before his departure too. But I would not fully trust my own judgment on this one, because the outside view is that people (in this case, me) often fail to see what they cannot see.

When I get more concrete:

• Institutional accountability and legibility is I think better than it was;
• Clean communication with staff, keeping promises, creating clear expectations, etc. -- better on some axes and worse on others -- my non-confident guess is better overall (via some loss plus lots of work);
• Classes starting and ending on time -- at mainlines: slightly less precise class-timing but not obviously worse thereby; at AIRCS, notable decreases, with some cost;
• Handboook revisions -- have done very little since he left;
• Polish and narrative cohesion in the workshop classes -- it’s less emphasized but not obviously worse thereby IMO, due partly to the infusion of the counterbalancing “original seeing” content from Brienne that was perhaps easier to pull off via toning polish down slightly. Cohesion and polish still seem acceptable, and far far better than before Duncan arrived.

Also: I don’t know how to phrase this tactfully in a large public conversation. But I appreciate Duncan’s efforts on behalf of CFAR; and also he left pretty burnt out; and also I want to respect what I view as his own attempt to disclaim responsibility for CFAR going forward (via that Facebook post) so that he won’t have to track whether he may have left misleading impressions of CFAR’s virtues in people. I don’t want our answers here to mess that up. If you come to CFAR and turn out not to like it, it is indeed not Duncan’s fault (even though it is still justly some credit to Duncan if you do, since we are still standing on the shoulders of his and many others’ past work).

(This is Dan, from CFAR since June 2012)

These are more like “thoughts sparked by Duncan’s post” rather than “thoughts on Duncan’s post”. Thinking about the question of how well you can predict what a workshop experience will be like if you’ve been at a workshop under different circumstances, and looking back over the years...

In terms of what it’s like to be at a mainline CFAR workshop, as a first approximation I’d say that it has been broadly similar since 2013. Obviously there have been a bunch of changes since January 2013 in terms of our curriculum, our level of experience, our staff, and so on, but if you’ve been to a mainline workshop since 2013 (and to some extent even before then), and you’ve also had a lifetime full of other experiences, your experience at that mainline workshop seems like a pretty good guide to what a workshop is like these days. And if you haven’t been to a workshop and are wondering what it’s like, then talking to people who have been to workshops since 2013 seems like a good way to learn about it.

More recent workshops are more similar to the current workshop than older ones. The most prominent cutoff that comes to mind for more vs. less similar workshops is the one I already mentioned (Jan 2013) which is the first time that we basically understood how to run a workshop. The next cutoff that comes to mind is January 2015, which is when the current workshop arc & structure clicked into place. The next is July 2019, which is the second workshop which was run by something like the current team and the first one where we hit our stride (it was also the first one after we started this year's instructor training, which I think helped with hitting our stride). And after that is sometime in 2016 I think when the main classes reached something resembling their current form.

Besides recency, it’s also definitely true that the people at the workshop bring a different feel to it. European workshops have a different feel than US workshops because so many of the people there are from somewhat different cultures. Each staff member brings a different flavor - we try to have staff who approach things in different ways, partly in order to span more of the space of possible ways that it can look like to be engaging with this rationality stuff. The workshop MC (which was generally Duncan’s role while he was involved) does impart more of their flavor on the workshop than most people, although for a single participant their experience is probably shaped more by whichever people they wind up connecting with the most and that can vary a lot even between participants at the same workshop.

What I get from Duncan’s FB post is (1) an attempt to disentangle his reputation from CFAR’s after he leaves, (2) a prediction that things will change due to his departure, and (3) an expression of frustration that more of his knowledge than necessary will be lost.

1. It's a totally reasonable choice.
2. At the time I first saw Duncan’s post I was more worried about big changes to our workshops from losing Duncan than I have observed since then. A year later I think the change is actually less than one would expect from reading Duncan’s post alone. That doesn’t speak to the cost of not having Duncan—since filling in for his absence means we have less attention to spend on other things, and I believe some things Duncan brought have not been replaced.
3. I am also sad about this, and believe that I was the person best positioned to have caused a better outcome (smaller loss of Duncan’s knowledge and values). In other words I think Duncan’s frustration is not only understandable, but also pointing at a true thing.

I'd be curious for an answer to this one too, actually.

Hello, I am a CFAR contractor who considers nearly all of their job to be “original research into human rationality”. I don’t do the kind of research many people imagine when they hear the word “research” (RCT-style verifiable social science, and such). But I certainly do systematic inquiry and investigation into a subject in order to discover or revise beliefs, theories, applications, etc. Which is, you know, literally the dictionary.com definition of research.

I’m not very good at telling stories about myself, but I’ll attempt to describe what I do during my ordinary working hours anyway.

All of the time, I keep an eye out for things that seem to be missing or off in what I take to be the current art of rationality. Often I look to what I see in the people close to me, who are disproportionately members of rationality-and-EA-related organizations, watching how they solve problems and think through tricky stuff and live their lives. I also look to my colleagues at CFAR, who spend many many hours in dialogue with people who are studying rationality themselves, for the first time or on a continuing basis. But since my eyes are in my own head, I look most for what is absent in my own personal art of rationality.

For example, when I first read the Sequences in 2012 or 2013, I gained a lot, but I also felt a gaping hole in the shape of something like “recognizing those key moments in real-life experience when the rationality stuff you’ve thought so much about comes whizzing by your head at top speed, looking nothing at all like the abstractions you’ve so far considered”. That’s when I started doing stuff like snapping my fingers every time I saw a stop sign, so I could get a handle on what “noticing” even is, and begin to fill in the hole. I came up with a method of hooking intellectual awareness up to immediate experience, then I spent a whole year throwing the method at a whole bunch of real life situations, keeping track of what I observed, revising the method, talking with people about it as they worked with the same problem themselves, and generally trying to figure out the shape of the world around phenomenology and trigger-action planning.

I was an occasional guest instructor with CFAR at the time, and I think that over the course of my investigations, CFAR went from spending very little time on the phenomenological details of key experiences to working that sort of thing into nearly every class. I think it’s now the case that rationality as it currently exists contains an “art of noticing”.

My way of investigating always pushes into what I can’t yet see or grasp or articulate. Thus, it has the unfortunate property of being quite difficult to communicate about directly until the research program is mostly complete. So I can say a lot about my earlier work on noticing, but talking coherently about what exactly CFAR’s been paying me for lately is much harder. It’s all been the same style of research, though, and if I had to give names to my recent research foci, I’d say I’ve been looking into original seeing, some things related to creativity and unconstrained thought, something about learning and what it means to own your education, and experiences related to community and cooperation.

It’s my impression that CFAR has always had several people doing this kind of thing, and that several current CFAR staff members consider it a crucial part of their jobs as well. When I was hired, Tim described research as “the beating heart” of our organization. Nevertheless, I personally would like more of it in future CFAR, and I’d like it to be done with a bit more deliberate institutional support.

That’s why it was my primary focus when working with Eli to design our 2019 instructor training program. The program consisted partially of several weekend workshops, but in my opinion the most important part happened while everyone was at home.

My main goal, especially for the first weekend, was to help the trainees choose a particular area of study. It was to be something in their own rationality that really mattered to them and that they had not yet mastered. When they left the workshop, they were to set off on their own personal quest to figure out that part of the world and advance the art.

This attitude, which we’ve been calling “questing” of late, is the one with which I hope CFAR instructors will approach any class they intend to teach, whether it’s something like “goal factoring” that many people have taught in the past, or something completely new that nobody’s even tried to name yet. When you really get the hang of the questing mentality, you never stop doing original rationality research. So to whatever degree I achieved my goal with instructor training (which everyone seems to think is a surprisingly large degree), CFAR is moving in the direction of more original rationality research, not less.

My model is that CFAR is doing the same activity it was always doing, which one may or may not want to call “research”.

I’ll describe that activity here.  I think it is via this core activity (plus accidental drift, or accidental hill-climbing in response to local feedbacks) that we have generated both our explicit curriculum, and a lot of the culture around here.

Components of this core activity (in no particular order):

1. We try to teach specific skills to specific people, when we think those skills can help them.  (E.g. goal-factoring; murphyjitsu; calibration training on occasion; etc.)
2. We keep our eyes open while we do #1.  We try to notice whether the skill does/doesn’t match the student’s needs.  (E.g., is this so-called “skill” actually making them worse at something that we or they can see?  Is there a feeling of non-fit suggesting something like that? What’s actually happening as the “skill” gets “learned”?)
   • We call this noticing activity “seeking PCK” and spend a bunch of time developing it in our mentors and instructors.
3. We try to stay in touch with some of our alumni after the workshop, and to notice what the long-term impacts seem to be (are they actually practicing our so-called “skills”?  Does it help when they do? More broadly, what changes do we just-happen-by-coincidence to see in multiple alumni again and again, and are these positive or negative changes, and what might be causing them?
   • In part, we do this via the four follow-up calls that participants receive after they attend the mainline workshop; in part we do it through the alumni reunions, the kind of contact that comes naturally from being in the same communities, etc.
   • We often describe some of what we think we’re seeing, and speculate about where to go given that, in CFAR’s internal colloqium.
   • We pay particular attention to alumni who are grappling with existential risk or EA, partly because it seems to pose distinct difficulties that it would be nice if someone found solutions to.
4. Spend a bunch of time with people who are succeeding at technical AI safety work, trying to understand what skills go into that.  Spend a bunch of time with people who are training to do technical AI safety work (often at the same time that people who can actually do such work are there), working to help transfer useful mindset (while trying also to pay attention to what’s happening.
   • Right now we do this mostly at the AIRCS and MSFP workshops.
5. Spend a bunch of time engaging smart new people to see what skills/mindsets they would add to the curriculum, so we don’t get too stuck in a local optimum.
   • What this looks like recently:
     • The instructor training workshops are helping us with this.  Many of us found those workshops pretty generative, and are excited about the technique-seeds and cultural content that the new instructor candidates have been bringing.
     • The AIRCS program has also been bringing in highly skilled computer scientists, often from outside the rationality and EA community. My own thinking has changed a good bit in contact with the AIRCS experience.  (They are less explicitly articulate about curriculum than the instructor candidates; but they ask good questions, buy some pieces of our content, get wigged out by other pieces of our content in a non-random manner, answer follow-up questions in contact with that that sometimes reveal implicit causal models of how to think that seem correct to me, etc.  And so they are a major force for AIRCS curriculum generation in that way.)
   • Gaps in 5:
     • I do wish we had better contact with more and varied highly productive thinkers/markers of different sorts, as a feed-in to our curriculum.  We unfortunately have no specific plans to fix this gap in 2020 (and I don’t think it could fit without displacing some even-more-important planned shift -- we have limited total attention); but it would be good to do sometime over the next five years.  I keep dreaming of a “writers’ workshop” and an “artist’s workshop” and so on, aimed at seeing how our rationality stuff mutates when it hits people with different kinds of visibly-non-made-up productive skill.
6. We sometimes realize that huge swaths of our curriculum are having unwanted effects and try to change it.  We sometimes realize that our model of “the style of thinking we teach” is out of touch with our best guesses about what’s good, and try to change it.
7. We try to study any functional cultures that we see (e.g., particular functional computer science communities; particular communities found in history books), to figure out what magic was there.  We discuss this informally and with friends and with e.g. the instructor candidates.
8. We try to figure out how thinking ever correlates with the world, and when different techniques make this better and worse in different context.  And we read the Sequences to remember that this is what we’re doing.
   • We could stand to do this one more; increasing this is a core planned shift for 2020.  But we’ve always done it some, including over the last few years.

The “core activity” exemplified in the above list is, of course, not RCT-style verifiable track records-y social science (which is one common meaning of “research”).  There is a lot of merit to that verifiable social science, but also a lot of slowness to it, and I cannot imagine using it to design the details of a curriculum, although I can imagine using it to see whether a curriculum has particular high-level effects.

We also still do some (but not as much as we wish we could do) actual data-tracking, and have plans to do modestly more of it over the coming year.  I expect this planned modest increase will be useful for our broader orientation but not much of a direct feed-in into curriculum, although it might help us tweak certain knobs upward or downward a little.

Also worth noting that there are a few different claims of the sort OP mentions that people make, I think. One thing people sometimes mean by this is “CFAR no longer does the sort of curriculum development which would be necessary to create an 'Elon Musk factory.'"

CFAR never had the goal of hugely amplifying the general effectiveness of large numbers of people (which I’m happy about, since I’m not sure achieving that goal would be good). One should not donate to CFAR in order to increase the chances of an Elon Musk factory.

I wish someone would create good bay area community health. It isn't our core mission; it doesn't relate all that directly to our core mission; but it relates to the background environment in which CFAR and quite a few other organizations may or may not end up effective.

One daydream for a small institution that might help some with this health is as follows:

1. Somebody creates the “Society for Maintaining a Very Basic Standard of Behavior”;
2. It has certain very basic rules (e.g. “no physical violence”; “no doing things that are really about as over the line as physical violence according to a majority of our anonymously polled members”; etc.)
3. It has an explicit membership list of folks who agree to both: (a) follow these rules; and (b) ostracize from “community events” (e.g. parties to which >4 other society members are invited) folks who are in bad standing with the society (whether or not they personally think those members are guilty).
4. It has a simple, legible, explicitly declared procedure for determining who has/hasn’t entered bad standing (e.g.: a majority vote of the anonymously polled membership of the society; or an anonymous vote of a smaller “jury” randomly chosen from the society).

Benefits I’m daydreaming might come from this institution:

• A. If the society had large membership, bad actors could be ostracized from larger sections of the community, and with more simplicity and less drama.
• B. Also, we could do that while imposing less restraint on individual speech, which would make the whole thing less creepy. Like, if many many people thought person B should be exiled, and person A wanted to defer but was not herself convinced, she could: (a) defer explicitly, while saying that’s what she was doing; and meanwhile (b) speak her mind without worrying that she would destabilize the community’s ability to ever coordinate.

CFAR relies heavily on selection effects for finding workshop participants. In general we do very little marketing or direct outreach, although AIRCS and MSFP do some of the latter; mostly people hear about us via word of mouth. This system actually works surprisingly (to me) well at causing promising people to apply.

But I think many of the people we would be most happy to have at a workshop probably never hear about us, or at least never apply. One could try fixing this with marketing/outreach strategies, but I worry this would disrupt the selection effects which I think have been a necessary ingredient for nearly all of our impact.

So I fantasize sometimes about a new organization being created which draws loads of people together, via selection effects similar to those which have attracted people to LessWrong, which would make it easier for us to find more promising people.

(I also—and this isn’t a wish for an organization, exactly, but it gestures at the kind of problem I speculate some organization could potentially help solve—sometimes fantasize about developing something like “scouts” at existing places with such selection effects. For example, a bunch of safety researchers competed in IMO/IOI when they were younger; I think it would be plausibly valuable for us to make friends with some team coaches, and for them to occasionally put us in touch with promising people).

I think it’s true that CFAR mostly moved away from teaching things like explicit probabilistic forecasting, and toward something else, although I would describe that something else differently—more like, skills relevant for hypothesis generation, noticing confusion, communicating subtle intuitions, updating on evidence about crucial considerations, and in general (for lack of a better way to describe this) “not going insane when thinking about x-risk.”

I favor this shift, on the whole, because my guess is that skills of the former type are less important bottlenecks for the problems CFAR is trying to help solve. That is, all else equal, if I could press a button to either make alignment researchers and the people who surround them much better calibrated, or much better at any of those latter skills, I’d currently press the latter button.

But I do think it’s plausible CFAR should move somewhat backward on this axis, at the margin. Some skills from the former category would be pretty easy to teach, I think, and in general I have some kelly betting-ish inclination to diversify the goals of our curricular portfolio, in case our core assumptions are wrong.

I think it would depend a lot on which sort of individual life outcomes you wanted to compare. I have basically no idea where these programs stand, relative to CFAR, on things like increasing participant happiness, productivity, relationship quality, or financial success, since CFAR mostly isn't optimizing for producing effects in these domains.

I would be surprised if CFAR didn't come out ahead in terms of things like increasing participants' ability to notice confusion, communicate subtle intuitions, and navigate pre-paradigmatic technical research fields. But I'm not sure, since in general I model these orgs as having sufficiently different goals than us that I haven't spent much time learning about them.

I haven't done any of the programs you mentioned. And I'm pretty young, so my selection is limited. But I've done lots of personal development workshop and trainings, both before and after my CFAR workshop, and my CFAR workshop was far and above the densest in terms of content, and most transformative on both my day-to-day processing, and my life trajectory.

The only thing that compares are some dedicated, years long relationships with skilled mentors.

YMMV. I think my experience was an outlier.

A response to just the first three questions. I’ve been at CFAR for two years (since January 2018). I've noticed, especially during the past 2-3 months, that my mind is changing. Compared to a year, or even 6 months ago, it seems to me that my mind more quickly and effortlessly moves in some ways that are both very helpful and resemble some of the cognitive tools we offer. There’s obviously a lot of stuff CFAR is trying to do, and a lot of techniques/concepts/things we offer and teach, so any individual’s experience needs to be viewed as part of a larger whole. With that context in place, here are a few examples from my life and work:

• Notice the person I'm talking to is describing a phenomenon but I can't picture anything —> Ask for an example (Not a technique we formally teach at the workshop, but seems to me like a basic application of being specific [LW · GW]. In that same vein: while teaching or explaining a concept, I frequently follow a concrete-abstract-concrete structure.)
• I'm making a plan —> Walk through it, inner sim / murphyjitsu style (I had a particularly vivid and exciting instance of this a few weeks ago: I was packing my bag for a camping trip and found myself, without having explicitly set the intention to do so, simulating the following day with an eye for what I would need. I noticed that I would need my camping spork, which I hadn't thought of before, and packed it! Things like this happen regularly when planning workshops and doing my day-to-day CFAR work, in addition to my personal life.)
• I’m willing to make trades of money for time, like paying for Ubers or shorter flights or services, etc. I used to have a pretty rigid deontological rule for myself against ever spending money when I “didn’t need to,” which I believe now was to my own detriment. I think internalizing Units of Exchange and Goal Factoring and sunk cost and willingness to pay lead me to a) more clearly see how much I value my time, and b) acquire felt “permission” (from myself) to make trades that previously seemed weird or unacceptable, to great benefit to myself and others. I notice this shift when I’m doing ops work with people outside of CFAR and they say something like, “No, it’s just impossible to carpet the entire floor. The venue doesn’t want us to, and besides it would cost a ton of money.” And I say something like, “Well, we really value having a cozy, welcoming, comfortable space for participants, and we’d probably be willing to pay quite a bit for that. What if we had $2k to spend on it--could we do it then?” and they say, “What… I mean, I guess…” Or I’m talking to a friend about paying for membership in a pickup basketball league and I say, “So I might miss one or two of the five games, but I’d gladly pay the full-season price of $130 to play even just two or three really good pickup games, so I’m definitely in.” and he responds with something like, “Huh, well I didn’t think about it like that but I guess it’s worth that much to me…” I feel excited at what seems to me like more freedom for myself in this area. Some good dogfood.
• Something needs doing in the org and I have a vision for how to do it —> Just create the 80-20 version of my proposal in 5 minutes and run with that. (This one is insane. It’s wild to me how many things fall through the cracks or never happen because the only thing missing was one enterprising soul saying, “Hey, I spent 5 minutes on this shitty first draft version — anyone have feedback or thoughts?” so people could tear it apart and improve it. I had an experience last week of creating a system for making reference check phone calls and just watching myself with pride and satisfaction; like, “Whoa, I’m just building this out of nothing”. There’s something here that’s general, what I think is meant by “agency,” what I’d call “self-efficacy”--the belief, the felt belief, that you are one of the people who can just build things and make things happen, and here you go just doing it. That seems to me to be one of the best pieces of dogfood I’ve eaten at CFAR. It’s an effect that’s tricky to measure/quantify, but we think there’s good reason to believe it’s there).
• I’m in a meeting and I hear, “Yeah, so let’s definitely take X action” and then silence or a change of topic —> Say, “Okay, so what’s going to happen next? Who’s owning this and when are we going to check in about whether it happened?” (Also insane. I’m shocked by the number of times that I have said “Oh no, we totally thought that was a good idea but never made sure it happened” and the only thing required to save the situation was the simple question above. This happens a lot less nowadays; one time a few weeks ago, both Adam and I had this exact reaction simultaneously in a meeting. It was awesome.)
• Empiricism/tracking/something. Every time I go to the YMCA to swim, I log the time that I walk into the building and the time I walk out of the building. I started doing this because my belief that “I can get in and out for a swim in less than an hour” has persisted for months, in the face of consistent evidence that it actually just takes an hour or more every time (gotta get changed, shower, swim, shower, get changed--it adds up!), and has caused me stress and frustration. Earlier this year, Brienne and I spent some time working on noticing skills, working that into the mainline curriculum and our instructor training curriculum, as well as practicing in daily life. So I decided not to try to do anything different, just to observe--give myself a detailed, clear, entirely truthful picture of exactly how long I take at the Y. In the two months I’ve been doing this tracking, the average amount of time I spend in the Y has dropped by about 15 minutes. My perspective on “how much time it takes me to swim” has changed, too; from “God dammit, I’m taking too long but I really want to take my time working out!” to “Sometimes I want to have a nice, slow workout, sit in the hottub and take a while. Sometimes I want to move quickly so I can get shit done. I have the capacity to do both of those.” I care a little about the time, and a lot about the satisfaction, self-efficacy, and skill that came from giving myself the gift of more time simply by allowing some part of my system to update and change my behavior as a consequence of consistently providing myself with a clearer picture of this little bit of territory. That’s some sweet dog food.

Hope this gives you something to chew on ;)

(Just responding here to whether or not we dogfood.)

I always have a hard time answering this question, and nearby questions, personally.

Sometimes I ask myself whether I ever use goal factoring, or seeking PCK, or IDC, and my immediate answer is “no”. That’s my immediate answer because when I scan through my memories, almost nothing is labeled “IDC”. It’s just a continuous fluid mass of ongoing problem solving full of fuzzy inarticulate half-formed methods that I’m seldom fully aware of even in the moment.

A few months ago I spent some time paying attention to what’s going on here, and what I found is that I’m using either the mainline workshop techniques, or something clearly descended from them, many times a day. I almost never use them on purpose, in the sense of saying “now I shall execute the goal factoring algorithm” and then doing so. But if I snap my fingers every time I notice a feeling of resolution and clarification about possible action, I find that I snap my fingers quite often. And if, after snapping my fingers, I run through my recent memories, I tend to find that I’ve just done goal factoring almost exactly as it’s taught in mainlines.

This, I think, is what it’s like to fully internalize a skill.

I’ve noticed the same sort of thing in my experience of CFAR’s internal communication as well. In the course of considering our answers to some of these questions, for example, we’ve occasionally run into disagreements with each other. In the moment, my impression was just that we were talking to each other sensibly and working things out. But if I scan through a list of CFAR classes as I recall those memories, I absolutely recognize instances of inner sim, trigger-action planning, againstness, goal factoring, double crux, systemization, comfort zone exploration, internal double crux, pedagogical content knowledge, Polaris, mundanification, focusing, and the strategic level, at minimum.

At one point when discussing the topic of research I said something like, “The easiest way for me to voice my discomfort here would involve talking about how we use words, but that doesn’t feel at all cruxy. What I really care about is [blah]”, and then I described a hypothetical world in which I’d have different beliefs and priorities. I didn’t think of myself as “using double crux”, but in retrospect that is obviously what I was trying to do.

I think techniques look and feel different inside a workshop vs. outside in real life. So different, in fact, that I think most of us would fail to recognize almost every example in our own lives. Nevertheless, I’m confident that CFAR dogfoods continuously.

I think we eat our own dogfood a lot. It’s pretty obvious in meetings—e.g., people do Focusing-like moves to explain subtle intuitions, remind each other to set TAPs, do explicit double cruxing, etc.

As to whether this dogfood allows us to perform better—I strongly suspect so, but I’m not sure what legible evidence I can give about that. It seems to me that CFAR has managed to have a surprisingly large (and surprisingly good) effect on AI safety as a field, given our historical budget and staff size. And I think there are many attractors in org space (some fairly powerful) that would have made CFAR less impactful, had it fallen into them, that it’s avoided falling into in part because its staff developed unusual skill at noticing confusion and resolving internal conflict.

I'm reading the replies of current CFAR staff with great interest (I'm a former staff member who ended work in October 2018), as my own experience within the org was "not really; to some extent yes, in a fluid and informal way, but I rarely see us sitting down with pen and paper to do explicit goal factoring or formal double crux, and there's reasonable disagreement about whether that's good, bad, or neutral."

I wrote the following comment during this AMA back in 2019, but didn't post it because of the reasons that I note in the body of the comment. 

I still feel somewhat unsatisfied with what I wrote. I think something about the tone feels wrong, or gives the wrong impression, somehow. Or maybe this only presents part of the story. But it still seems better to say aloud than not.

I feel more comfortable posting it now, since I'm currently early in the process of attempting to build an organization / team that does meet these standards. In retrospect, I think probably it would have been better if I had just posted this at the time, and hashed out some disagreements with others in the org in this thread.

(In some sense this comment is useful mainly as bit of a window into the kind of standards that I, personally, hold a rationality-development / training organization to.)

My original comment is reproduced verbatim below (plus a few edits for clarity). 

I feel trepidation about posting this comment, because it seems in bad taste to criticize a group, unless one is going to step up and do the legwork to fix the problem [LW · GW]. This is one of the top 5 things that bothers me about CFAR, and maybe I will step up to fix it at some point, but I’m not doing that right now and there are a bunch of hard problems that people are doing diligent work to fix. Criticizing is cheap. Making things better is hard. 

[edit 2023: I did run a year long CFAR instructor training that was explicitly designed to take steps on this class of problems though. It is not as if I was just watching from the sidelines. But shifting the culture of even a small org, especially from a non-executive role, is pretty difficult, and my feeling is that I made real progress in the direction that I wanted, but only about one twentieth of the way to what I would think is appropriate.]

My view is that CFAR does not meaningfully eat its own dogfood, or at least doesn’t enough, and that this hurts the organization’s ability to achieve its goals.

This is not to contradict the anecdotes that others have left here, which I think are both accurate presentations, and examples of good (even inspiring) actions. But while some members of CFAR do have personal practices (with varying levels of “seriousness”) in correct thought and effective action, CFAR, as an institution, doesn’t really make much use of rationality. I resonate strongly with Duncan’s comment about counting up vs. counting down.

More specific data, both positive and negative:

• CFAR did spend some 20 hours of staff meeting time Circling in 2017, separately from a ~50 hour CFAR circling retreat the most of the staff participated in, and various other circling events that CFAR staff attended together (but were not “run by CFAR”). 
• I do often observe people doing Focusing moves and Circling moves in meetings.
• I have observed occasional full explicit Double Crux conversations on the order of three or four times a year. 
• I frequently (on the order of once every week or two) observe CFAR staff applying the Double Crux moves (offering cruxes, crux checking, operationalizing, playing the Thursday-Friday game) in meetings and in conversation with each other.
• Group goal-factoring has never happened, to the best of my knowledge, even though there are a number of things that happen at CFAR that seem very inefficient, seem like “shoulds”, or are frustrating / annoying to at least one person [edit 2023: these are explicit triggers for goal factoring]. I can think of only one instance in which two of us (Tim and I, specifically) tried to goal-factor something (a part of meetings that some of us hate).]
• We’ve never had an explicit group pre-mortem, to the best of my knowledge. There is the occasional two-person session of simulating a project (usually workshop or workshop activity), and the ways in which it goes wrong. [edit 2023: Anna said that she had participated in many long form postmortems regarding hiring in particular, when I sent her a draft of this comment in 2019.]
• There is no infrastructure for tracking predictions or experiments. Approximately, CFAR as an institution doesn’t really run [formal] experiments, at least experiments with results that are tracked by anything other than the implicit intuitions of the staff. [edit 2023: some key features of a "formal experiment" as I mean it are writing down predictions in advance, and having a specific end date at which the group reviews the results. This is in contrast to simply trying new ideas sometimes.] 
• There is no explicit processes for iterating on new policies or procedures (such as iterating on how meetings are run). 
  • [edit 2023: An example of an explicit process for iterating on policies and procedures is maintaining a running document for a particular kind of meeting. Every time you have that kind of meeting, you start by referring to the notes from the last session. You try some specific procedural experiments, and then end the meeting with five minutes of reflection on what worked well or poorly, and log those in the document. This way you are explicitly trying new procedures and capturing the results, instead finding procedural improvements mainly by stumbling into them, and often forgetting improvements rather than integrating and building upon them. I use documents like this for my personal procedural iteration.
    
    Or in Working Backwards, the authors describe not just organizational innovations that Amazon came up with to solve explicitly-noted organizational problems, but the sequence of iteration that led to those final form innovations.]
  • There is informal, but effective, iteration on the workshops. The processes that run CFAR’s internals however, seem to me to be mostly stagnant [edit 2023: in the sense that there's not deliberate intentional effort on solving long-standing institutional frictions, or developing more effective procedures for doing things.]
• As far as I know, there are no standardized checklists for employing CFAR techniques in relevant situations (like starting a new project). I wouldn’t be surprised if there were some ops checklists with a murphyjitsu step. I’ve never seen a checklist for a procedure at CFAR, excepting some recurring shopping lists for workshops. 
• The interview process does not incorporate the standard research about interviews and assessment contained in Thinking, Fast and Slow. (I might be wrong about this. I, blessedly, don’t have to do admissions interviews.)
• No strategic decision or choice to undertake a project, that I’m aware of, has involved quantitative estimates of impact, or quantitative estimates of any kind. (I wouldn’t be surprised if the decision to run the first MSFP did, [edit 2023: but I wasn't at CFAR at the time. My guess is that there wasn't.])
• Historically, strategic decisions were made to a large degree by inertia. This is more resolved now, but for a period of several years, I think most of the staff didn’t really understand why we were running mainlines, and in fact when people [edit 2023: workshop participants] asked about this, we would say things like “well, we’re not sure what else to do instead.” This didn’t seem unusual, and didn’t immediately call out for goal factoring.
• There’s not designated staff training time for learning or practicing the mental skills, or for doing general tacit knowledge transfer between staff. However, Full time CFAR staff have historically had a training budget, which they could spend on whatever personal development stuff they wanted, at their own discretion.
• CFAR does have a rule that you’re allowed / mandated to take rest days after a workshop, since the workshop eats into your weekend.

Overall, CFAR strikes me as a mostly a normal company, populated by some pretty weird hippy-rationalists. There aren’t any particular standards that the employees are expected to use rationality techniques, nor institutional procedures for doing rationality [edit 2023: as distinct from having shared rationality-culture].

This is in contrast to say, Bridgewater associates, which is clearly structured intentionally to enable updating and information processing, on the organizational level. (Incidentally, Bridgewater is rich in the most literal sense.)

Also, I’m not fully exempt from these critiques myself: I have not really internalized goal factoring, yet, for instance, and think that I personally, am making the same kind of errors of inefficient action that I’m accusing CFAR of making. I also don’t make much use of quantitative estimates, and I have lots of empirical iteration procedures, but haven’t really gotten the hang of doing explicit experiments. (I do track decisions and predictions though, for later review.)

Overall, I think this gap is about due 10% “these tools don’t work as well, especially at the group level, as we seem to credit them, and we are correct to not use them”, about 30% to this being harder to do than it seems, and about 60% due to CFAR not really trying at this (and maybe it shouldn’t be trying at this, because there are trade offs and other things to focus on). 

Elaborating on the 30%: I do think that making an org like this, especially when not starting from scratch, is deceptively difficult. I think that while implementing some of these seems trivial on the surface, but that it actually entails a shift in culture and expectations, and doing this effectively requires leadership and institution-building skills that CFAR doesn’t currently have. Like, if I imagine something like this existing, it would need to have a pretty in depth onboarding process for new employees, teaching the skills, and presenting “how we do things here.” If you wanted to bootstrap into this kind of culture, at anything like a fast enough speed, you would need the same kind of on-boarding for all of the existing employees, but it would be even harder, because you wouldn’t have the culture already going to provide example and immersion. 

Historically, CFAR had the following concerns (I haven't worked there since Oct 2018, so their thinking may have changed since then; if a current staff member gets around to answering this question you should consider their answer to trump this one):

• The handbook material doesn't actually "work" in the sense that it can change lives; the workshop experience is crucial to what limited success CFAR *is* able to have, and there's concern about falsely offering hope
• There is such a thing as idea inoculation; the handbook isn't perfect and certainly can't adjust itself to every individual person's experience and cognitive style. If someone gets a weaker, broken, or uncanny-valley version of a rationality technique out of a book, not only may it fail to help them in any way, but it will also make subsequently learning [a real and useful skill that's nearby in concept space] correspondingly more difficult, both via conscious dismissiveness and unconscious rounding-off.
• To the extent that certain ideas or techniques only work in concert or as a gestalt, putting the document out on the broader internet where it will be chopped up and rearranged and quoted in chunks and riffed off of and likely misinterpreted, etc., might be worse than not putting it out at all.

I’ve worked closely with CFAR since it’s founding in 2012, for varying degrees of closely (ranging from ~25 hrs/week to ~60 hrs/week). My degree of involvement in CFAR’s high-level and mid-level strategic decisions has varied some, but at the moment is quite high, and is likely to continue to be quite high for at least the coming 12 months.

During work-type hours in which I’m not working for CFAR, my attention is mostly on MIRI on MIRI’s technical research. I do a good bit of work with MIRI (though I am not employed by MIRI -- I just do a lot of work with them), much of which also qualifies as CFAR work (e.g., running the AIRCS workshops and assisting with the MIRI hiring process; or hanging out with MIRI researchers who feel “stuck” about some research/writing/etc. type thing and want a CFAR-esque person to help them un-stick). I also do a fair amount of work with MIRI that does not much overlap CFAR (e.g. I am a MIRI board member).

Oliver remained confused after talking with me in April because in April I was less certain how involved I was going to be in upcoming strategic decisions. However, it turns out the answer was “lots.” I have a lot of hopes and vision for CFAR over the coming years, and am excited about hashing it out with Tim and others at CFAR, and seeing what happens as we implement; and Tim and others seem excited about this as well.

My attention oscillates some across the years between MIRI and CFAR, based partly on the needs of each organization and partly on e.g. there being some actual upsides to me taking a backseat under Pete as he (and Duncan and others) made CFAR into more of a functioning institution in ways I would’ve risked reflexively meddling with. But there has been much change in the landscape CFAR is serving, and it’s time I think for there to be much change also in e.g. our curriculum, our concept of “rationality”, our relationship to community, and how we run our internal processes -- and I am really excited to be able to be closely involved with CFAR this year, in alliance with Tim and others.

The two best books on Rationality:

• The Sequences
• Principles by Ray Dalio (I read the PDF that leaked from bridge water. I haven't even looked at the actual book.)

My starter kit for people who want to build the core skills of the mind / personal effectiveness stuff (I reread all of these, for reminders, every 2 years or so):

• Getting things Done: the Art of Stress-free Productivity
• Nonviolent Communication: a Language for Life
• Focusing
• Thinking, Fast and Slow

Metaphors We Live By by George Lakoff — Totally changed the way I think about language and metaphor and frames when I read it in college. Helped me understand that there are important kinds of knowledge that aren't explicit [LW(p) · GW(p)].

I really like Language, Truth and Logic, by A.J. Ayer. It's an old book (1936) and it's silly in some ways. It's basically an early pro-empiricism manifesto, and I think many of its arguments are oversimplified, overconfident, or wrong. Even so, it does a great job of teaching some core mental motions of analytic philosophy. And its motivating intuitions feel familiar—I suspect that if 25-year-old Ayer got transported to the present, given internet access etc., we would see him on LessWrong pretty quick.

Ambience and physical comfort are surprisingly important. In particular:

• Lighting: Have lots of it! Ideally incandescent but at least ≥ 95 CRI (and mostly ≤ 3500k) LED, ideally coming from somewhere other than the center of the ceiling, ideally being filtered through a yellow-ish lampshade that has some variation in its color so the light that gets emitted has some variation too (sort of like the sun does when filtered through the atmosphere).

• Food/drink: Have lots of it! Both in terms of quantity and variety. The cost to workshop quality of people not having their preferences met here sufficiently outweighs the cost of buying too much food, that in general it’s worth buying too much as a policy. It's particularly important to meet people's (often, for rationalists, amusingly specific) specific dietary needs, have a variety of caffeine options, and provide a changing supply of healthy, easily accessible snacks.

• Furniture: As comfortable as possible, and arranged such that multiple small conversations are more likely to happen than one big one.

Riceissa's question was brief, so I'll add a bunch of my thoughts on this topic.

I also remember there was something of a hush around the broader x-risk network on the topic of timelines, sometime around the time of FLI's second AI conference. Since then I've received weird mixed signals about what people think, with hushed tones of being very worried/scared. The explicit content is of a similar type to Sam Altman's line "if you believe what I believe about the timeline to AGI and the effect it will have on the world, it is hard to spend a lot of mental cycles thinking about anything else" but rarely accompanied with an explanation of the reasoning that lead to that view.

I think that you can internalise models of science, progress, computation, ML, and geopolitics, and start to feel like "AGI being built" is part of your reality, your world-model, and then figure out what actions you want to take in the world.  I've personally thought about it a bit and come to some of my own conclusions, and I've generally focused on plans designed for making sure AGI goes well. This is the important and difficult work of incorporating abstract, far ideas into your models of near-mode reality.

But it's also seems to me that a number of x-risk people looked at many of the leaders getting scared, and that is why they believe the timeline is short. This is how a herd turns around and runs in fear from an oncoming jaguar - most members of the herd don't stop to check for themselves, they trust that everyone else is running for good reason. More formally, it's known as an info cascade [LW · GW]. This is often the rational thing to do when people you trust act as if something dangerous is coming at you. You don't stop and actually pay attention to the evidence oneself.

(I personally experience such herd behaviour commonly when using the train systems in the UK. When a train is cancelled and 50 people are waiting beside it to get on, I normally don't see the board that announces which platform to go to for the replacement train, as it's only visible to a few of the people, but very quickly the whole 50 people are moving to the new platform for the replacement train. I also see it when getting off a train at a new train station, where lots of people don't really know which way to walk to get out of the building: immediately coming off the train, is it left or right? But the first few people tend to make a judgement, and basically everyone else follows them. I've sometimes done it myself, been the first off and started walking confidently in a direction, and have everyone start confidently follow me, and it always feels a little magical for a moment, because I know I just took a guess.)

But the unusual thing about our situation, is that when you ask the leaders of the pack why they think a jaguar is coming, they're very secretive about it. In my experience many clued-in people will explicitly recommend not sharing information about timelines. I'm thinking about OpenPhil, OpenAI, MIRI, FHI, and so on. I don't think I've ever talked to people at CFAR about timelines.

To add more detail to my saying it's considered 'the' decision-relevant variable by many, here's two quotes. Ray Arnold is a colleague and a friend of mine, and two years ago he wrote a good post on his general updates about such subjects [LW · GW], that said the following:

Claim 1: Whatever your estimates two years ago for AGI timelines, they should probably be shorter and more explicit this year.

Claim 2: Relatedly, if you’ve been waiting for concrete things to happen for you to get worried enough to take AGI x-risk seriously, that time has come. Whatever your timelines currently are, they should probably be influencing your decisions in ways more specific than periodically saying “Well, this sounds concerning.”

Qiaochu also talked about it as the decision-relevant question [LW(p) · GW(p)]:

[Timelines] are the decision-relevant question. At some point timelines get short enough that it's pointless to save for retirement. At some point timelines get short enough that it may be morally irresponsible to have children...

Ray talks in his post about how much of his beliefs on this topic comes from trusting another person closer to the action, which is perfectly reasonable thing to do, though I'll point out again it's also (if lots of people do it) herd behaviour. Qiaochu talks about how he never figured out the timeline to AGI with an explicit model, even though he takes short timelines very seriously, which also sounds like a process that involves trusting others a bunch.

It's okay to keep secrets, and in a number cases it's of crucial importance. Much of Nick Bostrom's career is about how some information can be hazardous, and about how not all ideas are safe at our current level of wisdom. But it's important to note that "short timelines" is a particular idea that has had the herd turn around and running in fear to solve an urgent problem, and there's been a lot of explicit recommendations to not give people the info they'd need to make a good judgment about it. And those two things together are always worrying.

It's also very unusual for this community. We've been trying to make things go well wrt AGI for over a decade, and until recently we've put all our reasoning out in the open. Eliezer and Bostrom published so much. And yet now this central decision-node, "the decision-relevant variable", is hidden from the view of most people involved. It's quite strange, and generally is the sort of thing that is at risk for abuse by whatever process is deciding what the 'ground truth' is. I don't believe the group of people involved in being secretive about AI timelines have spent at all as much time thinking about the downsides of secrecy or put in the work to mitigate them. Of course I can't really tell, given the secrecy.

All that said, as you can see in the quotes/links that I and Robby provided elsewhere in this thread, I think Eliezer has made the greatest attempt of basically anyone to explain how he models timelines, and wrote very explicitly about his updates after AlphaGo Zero. And the Fire Alarm post was really, really great. In my personal experience the things in the quotes above is fairly consistent with how Eliezer reasoned about timelines before the deep learning revolution.

I think a factor that is likely to be highly relevant is that companies like DeepMind face a natural incentive to obscure understanding their progress and to be the sole arbiters of what is going to happen. I know that they're very careful about requiring all visitors to their offices to sign NDAs, and requiring employees to get permission for any blogposts they're planning to write on the internet about AI. I'd guess a substantial amount of this effect comes from there, but I'm not sure.

Edit: I edited this comment a bunch of times because I initially wrote it quickly, and didn't quite like how it came out. Sorry if anyone was writing a reply. I'm not likely to edit it again.

Edit: I think it's likely I'll turn this into a top level post at some point.

For the record, parts of that ratanon post seem extremely inaccurate to me; for example, the claim that MIRI people are deferring to Dario Amodei on timelines is not even remotely reasonable. So I wouldn't take it that seriously.

Certainly MIRI has written about this, for example see the relevant part of their 2018 update:

The latter scenario is relatively less important in worlds where AGI timelines are short. If current deep learning research is already on the brink of AGI, for example, then it becomes less plausible that the results of MIRI’s deconfusion work could become a relevant influence on AI capabilities research, and most of the potential impact of our work would come from its direct applicability to deep-learning-based systems. While many of us at MIRI believe that short timelines are at least plausible, there is significant uncertainty and disagreement about timelines inside MIRI, and I would not feel comfortable committing to a course of action that is safe only in worlds where timelines are short.

Also see Eliezer's top-notch piece on timelines [LW · GW], which includes the relevant quote:

Of course, the future is very hard to predict in detail. It's so hard that not only do I confess my own inability, I make the far stronger positive statement that nobody else can do it either.

Eliezer also updated after losing a bet that AlphaGo would not be able to beat humans so well, which he wrote about in AlphaGo Zero and the Foom Debate [LW · GW]. It ends with the line:

I wouldn't have predicted AlphaGo and lost money betting against the speed of its capability gains, because reality held a more extreme position than I did on the Yudkowsky-Hanson spectrum.

It's at least as important for CFAR to train people who end up at OpenAI, Deepmind, FHI, etc.

I'm sure the methods of CFAR have wider application than to Machine Learning...

I buy that General Semantics was in some sense a memetic precursor to some of the ideas described in the sequences/at CFAR, but I think this effect was mostly indirect, so it seems misleading to me to describe CFAR as being heavily influenced by it. Davis Kingsley, former CFAR employee and current occasional guest instructor, has read a bunch about GM, I think, and mentions it frequently, but I'm not aware of direct influences aside from this.

I think Nuno's time-capped analysis [EA · GW] is good.

(This is Dan from CFAR)

I did a quick poll of 5 staff members and the average answer was 5.6.

I'll interpret this as "Which of the rationalist virtues do you think CFAR has gotten the most mileage from your practicing".

The virtue of the void. Hands down. Though I still haven't done it nearly as much as it would be useful to do it. Maybe this year? [LW · GW]

If I instead interpret this as "Which of the rationalist virtues do you spend the most minutes practicing": curiosity. Which would be my runner-up for "CFAR got the most mileage from my practicing".

At the risk of sounding trite: stay fun, stay interested, stay fresh, and stay sane! We want the people we bring on and the culture that surrounds us to be a good place to be for epistemics and also for people.

We have a bunch of instructor candidates that I am very excited about. One of the ones I am most excited about strikes me as an intellectual offspring of the Sequences, and he’s rocking it. I would like to encounter more people like him, so I hope this community continues to make good, strong, “what odds would you give me”, rationalists that are interested in teaching and curriculum development.

We would also like more thoughts on how different mental tech breaks and how to create reason-based, healthy communities with better immune systems that do not break. Partial clues most welcome, especially if you write them up as readable LW posts and email us so we can read the discussion :)

If you have ideas of something particular you want to do or add, reach out to us. We are looking for someone to do metrics with Dan. We are looking for particularly skilled computer scientists to attend the AIRCS program. We are potentially looking for a couple summer interns. We are looking for someone with a lot of professional high level ops experience and a good dose of common sense.

We also *just happen to be* running a fundraiser right now!

Good question! I also had it earlier this year, so I studied him, and here is what I learned of Dan:

Dan is workshop staff at most of the workshops we run, including AIRCS workshops, mainlines, and other programs like instructor training. So, for ~16 weeks of 2019, he was helping run, teaching at, and doing ops at workshops.

Dan is also in charge of all our spreadsheets and data and everything that happens after a workshop: synthesizing the feedback we get from attendees, putting attendees into follow-up groups, pairing people with mentors, and sending out exercises.

Dan is in charge of our impact reports. This involves figuring out what to measure and how to measure it, doing the actual measuring, and then writing about it in a way that hopefully people understand. Dan decides what he looks into based on his own judgements and the questions funders and our exec team have. He’s currently working on metrics data for the fundraiser that will be out soon.

Dan is a general CFAR staff member. This means he contributes to our weekly colloquia with thoughts about rationality, interviews people for workshops, does follow-ups with participants, and other random projects.

I really loved this post on Occam's Razor [LW · GW]. Before encountering it, I basically totally misunderstood the case for using the heuristic, and so (wrongly, I now think) considered it kind of dumb.

I also especially loved "The Second Law of Thermodynamics, and Engines of Cognition [LW · GW]," which gave me a wonderful glimpse (for the first time) into how "laws of inference" ground in laws of physics.

If anyone came across this comment in the future - the CFAR Participant Handbook [LW · GW] is now online, which is more or less the answer to this question.

Thanks to your recommendation I recently read New Atlantis, by Francis Bacon, and it was so great! It's basically Bacon's list of things he wished society had, ranging from "clothes made of sea-water-green satin" and "many different types of beverages" to "research universities that employ full-time specialist scholars."

(This is Dan from CFAR)

In terms of what happened that day, the article covers it about as well as I could. There’s also a report from the sheriff’s office which goes into a bit more detail about some parts.

For context, all four of the main people involved live in the Bay Area and interact with the rationality community. Three of them had been to a CFAR workshop. Two of them are close to each other, and CFAR had banned them prior to the reunion based on a bunch of concerning things they’ve done. The other two I’m not sure how they got involved.

They have made a bunch of complaints about CFAR and other parts of the community (the bulk of which are false or hard to follow), and it seems like they were trying to create a big dramatic event to attract attention. I’m not sure quite how they expected it to go.

This doesn’t seem like the right venue to go into details to try to sort out the concerns about them or the complaints they’ve raised; there are some people looking into each of those things.

This is probably the least important question (the answer is that some people are nuts) but also the one that I most want to see answered for some reason.

I did. I have some but not all of the images saved; happy to share what I have, feel free to pm me for links.

The name for TAP's in the psychology literature is implementation intention. CFAR renamed it.

This question seems to assume that CFAR sees the impact of their workshops as being about successfully teaching cognitive tools. That doesn't seem to be the case based on conversation I had in the past with CFAR folks.

[Disclaimer: have not been at CFAR since October 2018; if someone currently from the org contradicts this, their statement will be more accurate about present-day CFAR]

No (CFAR's mission has always been narrower/more targeted) and no (not in any systematic, competent fashion).

(This is Dan from CFAR)

Guided By The Beauty Of Our Weapons

Asymmetric vs. symmetric tools is now one of the main frameworks that I use to think about rationality (although I wish we had better terminology for it). A rationality technique (as opposed to a productivity hack or a motivation trick or whatever) helps you get more done on something in cases where getting more done is a good idea.

This wasn’t a completely new idea when I read Scott’s post about it, but the post seems to have helped a lot with getting the framework to sink in.

In case no one who currently works at CFAR gets around to answering this (I was there from Oct 2015 to Oct 2018 in a pretty influential role but that means I haven't been around for about fourteen months):

• Meditations on Moloch is top of the list by a factor of perhaps four
• Different Worlds as a runner up

Lots of social dynamic stuff/how groups work/how individuals move within groups:

• Social Justice and Words, Words, Words
• I Can Tolerate Anything Except The Outgroup
• Guided By The Beauty Of Our Weapons
• Yes, We Have Noticed The Skulls
• Book Review: Surfing Uncertainty

All hail Logmoth, the rightful caliph!

Gwern.

As a follow-up, can you talk about times you've experienced participants having built explicit anti-epistemology to defend themselves from doing too much explicit reasoning? I'm talking about cases analogous to when people have made bucket errors and defend it anyway, so that they don't accidentally break things.

which are all about breaking parts of your mind down into smaller parts

Na, my mind's a bunch of super tiny stuff to begin with. When I do IDC, I just stop in my unified-person-fabrication a little earlier than the point at which I've erased all ability to perceive internal distinctions.

(Sorry, I know that's not an answer to your question. Maybe somebody, perhaps even a future me, will come by and give you a real answer.)

The capacity to develop true beliefs, so as to better achieve your goals.

I think this project sounds cool. This might be (I don't know enough to know) an example of rationality training in something other than the CFAR paradigm of "1) present 2) techniques to 3) small groups 4) at workshops."

But I think your question is too high context to easily answer. Is there a way I can play the current version? If so would try it for a bit and then tell you what I think, personally.

I don't understand that screenshot at all (maybe the resolution is too low?), but from your description it sounds in a similar vein to Zendo and Eleusis and Penultima, which you could get ideas from. Yours seems different though, and I'd be curious to know more details. I tried implementing some single-player variants of Zendo five years ago, though they're pretty terrible (boring, no graphics, probably not useful for training rationality).

I do think there's some potential for rationality improvements from games, though insofar as they're optimized for training rationality, they won't be as fun as games optimized purely for being fun. I also think it'll be very difficult to achieve transfer to life-in-general, for the same reason that learning to ride a bike doesn't train you to move your feet in circles every time you sit in a chair. ("I pedal when I'm on a bike, to move forward; why would I pedal when I'm not on a bike, and my goal isn't to move forward? I reason this way when I'm playing this game, to get the right answer; why would I reason this way when I'm not playing the game, and my goal is to seem reasonable or to impress people or to justify what I've already decided?")