You managed to cut precisely the part of the post that was most useful for me to read :) 

(To be clear, putting it in this comment was just as helpful, maybe even more-so.)

I make this point not to argue against finding love or starting a family, but to argue against a mindset that treats AGI and daily life as more or less two different magisteria….

It still doesn't feel to me like it's fully speaking as though the two worlds as one world

The situation is tricky, IMO.  There is, of course, at the end of the day only one world.  If we want to have kids who can grow up to adulthood, and who can have progeny of their own, this will require that there be a piece of universe hospitable to human life where they can do that growing up.

At the same time:

a) IMO, there is a fair amount of “belief in belief [? · GW]” about AI safety and adjacent things.  In particular, I think many people believe they ought to believe that various “safety” efforts help, without really anticipating-as-if this sort of thing can help.

(The argument for “be worried about the future” is IMO simpler, more obvious, and more likely to make it to the animal, that particular beliefs that particular strategies have much shot.  I’m not sure how much this is or isn’t about AI; many of my Uber drivers seem weirdly worried about the future.)

a2) Also, IMO, a fair number of peoples’ beliefs (or “belief in beliefs”) about AI safety are partly downstream of others’ political goals, e.g. of others’ social incentives that those people believe in particular narratives about AI safety and about how working at place X can help with AI safety.  This can accent the “belief in belief” thing.

b)  Also, even where people have honest/deep/authentic verbal-level beliefs about a thing, it often doesn’t percolate all the way down into the animal.  For example, a friend reports having interviewed a number of people about some sex details, and coming out believing that some people do and some people don’t have a visceral animal-level understanding that birth control prevents pregnancy, and reports furthermore that such animal-level beliefs often backpropagate to peoples’ desires or lack of desires for different kinds of sex.  I believe my friend here, although this falls under “hard to justify personal beliefs.”

c) As I mentioned in the OP, I am worried that when a person “gives up on” their romantic and/or reproductive goals (or other goals that are as deeply felt and that close to the center of a person, noting that the details here vary by individual AFAICT), this can mess up their access to caring and consciousness in general (in Divia’s words, can risk “some super central sign error deep in their psychology”).

a-c, in combination, leave me nervous/skeptical about people saying that their plan is to pursue romance/children “after the Singularity,” especially if they’re already nearing the end of their biological window.  And especially if it’s prompted by that being a standard social script in some local circle.  I am worried that people may say this, intend it with some portion of themselves, but have their animal hear “I’m going to abandon these goals and live in belief-in-beliefs.”

I have a personal thing up for me about this one.  Back in ~2016, I really wanted to try to have a kid, but thought that short timelines plus my own ability to contribute to AI safety efforts meant I probably shouldn’t.  I dialoged with my system 1 using all the tools in the book.  I consulted all the people near me who seemed like they might have insight into the relevant psychology.  My system 1 / animal-level orientation, after dialog, seemed like it favored waiting, hoping for a kid on the other side of the singularity.  I mostly passed other peoples’ sanity checks, both at the time and a few years later when I was like “hey, we were worried about this messing with my psyche, but it seems like it basically worked, right? what is your perception?” And even so, IMO, I landed in a weird no man’s land of a sort of bleached out depression and difficulty caring about anything after awhile, that was kinda downstream of this but very hard for me to directly perceive, but made it harder to really mean anything and easier to follow the motions of looking like I was trying.

The take-away I’m recommending from this is something like: “be careful about planning on paths toward your deepest, animal-level goals that your animal doesn’t buy.  And note that it’s can be hard to robustly know what your animal is or isn’t buying. ”  Also, while there's only one magesterium, if people are animal-level and speech-level reasoning as though there's several, that's a real and confusing-to-me piece of context to how we're living here.

I want to have a dialog about what’s true, at the level of piece-by-piece reasoning and piece-by-piece causes.  I appreciate that you Rob are trying to do this; “pedantry” as you put it is great, and seems to me to be a huge chunk of why LW is a better place to sort some things out than is most of the internet.

I’m a bit confused that you call it “pedantry”, and that you talk of my post as trying to push the pendulum in a particular way, and “people trying to counter burnout,” and whether this style of post “works” for others.  The guess I’m forming, as I read your (Rob’s) comment and to a lesser extent the other comments, is that a bunch of people took my post as a general rallying cry against burnout, and felt it necessary to upvote my post, or to endorse my post, because they personally wish to take a stand against burnout.  Does something like that seem right/wrong to anyone?  (I want to know.)

I… don’t want that, although I may have done things in my post to encourage it anyhow, without consciously paying attention.  But if we have rallying cries, we won’t have the kind of shared unfiltered reasoning that someone wanting truth can actually update on.

I’m in general pretty interested in strategies anyone has for having honest, gritty, mechanism-by-mechanism discussion near a Sacred Value.  “Don’t burn people out” is arguably a Sacred Value, such that it’ll be hard to have open conversation near it in which all the pedantry is shared in all the directions.  I’d love thoughts on how to do it anyhow.

But it still feels to me like it's a post trying to push the pendulum in a particular direction, rather than trying to fully and openly embody the optimal-by-your-lights Balancing Point.

AFAICT, I am trying to fully and openly embody the way of reasoning that actually makes sense to me in this domain, which… isn’t really a “balancing point.”  It’s more like the anarchist saying “the means are the ends.”  Or it’s more like Szilard’s “ten commandments,” (which I highly recommend reading for anyone who hasn’t; they’re short).  Or more like the quote from the novel The Dispossessed: “To reassert its validity and strength, he thought, one need only act, without fear of punishment and without hope of reward: act from the center of one's soul. “

I don’t have the right concepts or articulation here.  This is an example of the “hard to justify personal opinions” I warned about in my “epistemic status.”  But IMO, thinking about tradeoffs and balancing points can be good when your map is good enough; at other times, it’s more like I want to try to hone in on priors, on deep patternness, on where reasoning is before it’s reasoning.  This is where the power of leisure comes from, where the possibility of hobbies that end up giving you glimpses of new bits of the universe come from.  And it’s a thing I’m trying to show here.  Not a particular balance-point between depleting your long-term resources and “being nice to yourself” by eating chocolates and cartoons.  (Neither of those help with getting to the tao, usually, AFAICT.)

We can be empirical about trying to see which actions, which mindsets, add to our and others’ long-term robust abilities.

In short-term crises for which you have decent-quality maps, balance-points and trading things off with local consequentialist reasoning makes sense to me.  But not the rest of everywhere.

I agree many people underestimate their own capacities, and too seldom try hard or scary things.  I think this is often many of the same people who burn themselves out.

Sorry this reply, and my other one, are somewhat incoherent.  I’m having trouble mapping both where you’re coming from, and why/where I disagree.

I think I feel a similar mix of love and frustration for your comment as I read your comment expressing with the post.

Let me be a bit theoretical for a moment. It makes sense for me to think of utilities as a sum $U=a{U}_a+b{U}_b$ where $U_a$ is the utility of things after singularity/superintelligence/etc and $U_b$ the utility for things before then (assuming both are scaled to have similar magnitudes so the relative importance is given by the scaling factors). There's no arguing about the shape of these or what factors people chose because there's no arguing about utility functions (although people can be really bad at actually visualizing $U_a$).

Separately form this we have actions that look like optimizing for $U_a$ (e.g. AI Safety research and raising awareness), and those that look like optimizing for $U_b$ (e.g. having kids and investing in/for their education). The post argues that some things that look like optimizing for $U_b$ are actually very useful for optimizing $U_a$ (as I understand, it mostly because AI timelines are long enough and the optimization space muddled enough that most people contribute more in expectation from maintaining and improving their general capabilities in a sustainable way at the moment).

Your comment (the pedantic response part) talks about how optimizing for $U_a$ is actually very useful for optimizing $U_b$. I'm much more sceptical of this claim. The reason is due to expected impact per unit of effort. Let's consider the sending your kids to college. It looks like top US colleges cost around $50k more per year than state schools, adding up to $200k for a four year programme. This is maybe not several times better as the price tags suggests, but if your child is interested and able to get in to such a school it's probably at least 10% better (to be quite conservative). A lot of people would be extremely excited for an opportunity to lower the existential risk from AI by 10% for $200k. Sure, sending your kids to college isn't everything there is to $U_b$, but it looks like the sign remains the same for a couple of orders of magnitude.

Your talk of a pendulum makes it sound like you want to create a social environment that incentivizes things that look like optimizing for $U_a$ regardless of whether they're actually in anyone's best interest. I'm sceptical of trying to get anyone to act against their interests. Rather than make everyone signal that $a\gg b$ it makes more sense to have space for people with $a\approx b$ or even $a<b$ to optimize for their values and extract gains from trade. A successful AI Safety project probably looks a lot more like a network of very different people figuring out how to collaborate for mutual benefit than a cadre of self-sacrificing idealists.

I appreciate this comment a lot.  Thank you.  I appreciate that it’s sharing an inside view, and your actual best guess, despite these things being the sort of thing that might get social push-back!

My own take is that people depleting their long-term resources and capacities is rarely optimal in the present context around AI safety.

My attempt to share my reasoning is pretty long, sorry; I tried to use bolding to make it skimmable.

In terms of my inside-view disagreement, if I try to reason about people as mere means to an end (e.g. “labor”):

0.  A world where I'd agree with you.  If all that would/could impact AI safety was a particular engineering project (e.g., Redwood’s ML experiments, for concreteness), and if the time-frame of a person’s relevance to that effort was relatively short (e.g., a year or two, either because AI was in two years, or because there would be an army of new people in two years), I agree that people focusing obsessively for 60 hours/week would probably produce more than the same people capping their work at 35 hrs/week.

But (0) is not the world we’re in, at least right now.  Specific differences between a world where I'd agree with you, and the world we seem to me to be in:

1.  Having a steep discount rate on labor seems like a poor predictive bet to me.  I don’t think we’re within two years of the singularity; I do think labor is increasing but not at a crazy rate; and a person who keeps their wits and wisdom about them, who pays attention and cares and thinks and learns, and especially someone who is relatively new to the field and/or relatively young (which is the case for most such engineers I think), can reasonably hope to be more productive in 2 years than they are now, which can roughly counterbalance the increase (or more than counterbalance the increase) on my best guess.

E.g., if they get hired and Redwood and then stay there, you’ll want veterans a couple years later who already know your processes and skills.

(In 2009, I told myself I needed only to work hard for ~5 years, maybe 10, because after that I’d be a negligible portion of the AI safety effort, so it was okay to cut corners.  I still think I’m a non-negligible portion of the effort.)

1.1.  Trying a thing to see if it works (e.g. 60 hrs/week of obsession, to see how that is) might still be sensible, but more like “try it and see if it works, especially if that risk and difficulty is appealing, since “appealingness” is often an indicator that a thing will turn out to make sense / to yield useful info / to be the kind of thing one can deeply/sincerely try rather than forcing oneself to mimic, etc.” not like “you are nothing and don’t matter much after two years, run yourself into the ground while trying to make a project go.”  I suppose your question is about accepting a known probability of running yourself into the ground, but I’m having trouble booting that sim; to me the two mindsets are pretty different.  I do think many people are too averse to risk and discomfort; but also that valuing oneself in the long-term is correct and important.  Sorry if I’m dodging the question here.

2.  There is no single project that is most of what matters in AI safety today, AFAICT.  Also, such projects as exist are partly managerially bottlenecked.  And so it isn’t “have zero impact” vs “be above Redwood’s/project such-and-such’s hiring line,” it is “be slightly above a given hiring line” (and contribute the difference between that spot and the person who would fill it next, or between that project having one just-above-margin person and having one fewer but more managerial slack) vs “be alive and alert and curious as you take an interest in the world from some other location”, which is more continuous-ish.

3.  We are confused still, and the work is often subtle, such that we need people to notice subtle mismatches between what they’re doing and what makes sense to do, and subtle adjustments to specific projects, to which projects make sense at all, and subtle updates from how the work is going that can be propagated to some larger set of things, etc.  We need people who care and don’t just want to signal that they kinda look like they care.  We need people who become smarter and wiser and more oriented over time and who have deep scientific aesthetics, and other aesthetics.  We need people who can go for what matters even when it means backtracking or losing face.  We don’t mainly need people as something like fully needing-to-be-directed subjugated labor, who try for the appearances while lacking an internal compass.  I expect more of this from folks who average 35 hrs/week than 60 hrs/week in most cases (not counting brief sprints, trying things for awhile to test and stretch one’s capacities, etc. — all of which seems healthy and part of fully inhabiting this world to me).  Basically because of the things pointed out in Raemon’s post about slack, [LW · GW] or Ben’s post about the Sabbath [LW · GW].  Also because often 60 hrs/week for long periods of time means unconsciously writing off important personal goals (cf Critch’s post about addiction to work), and IMO writing off deep goals for the long-term makes it hard to sincerely care about things.

(4.  I do agree there’s something useful about being able to work on other peoples’ projects, or on mundane non-glamorous projects, that many don’t have, and that naive readings of my #3 might tend to pull away from.  I think the deeper readings of #3 don’t, but it could be discussed.)

If I instead try to share my actual views, despite these being kinda wooey and inarticulate and hard to justify, instead of trying to reason about people as means to an end:

A.  I still agree that in a world where all that would/could impact AI safety was a particular engineering project (e.g., Redwood’s ML experiments, for concreteness), and if the time-frame of a person’s relevance to that effort was relatively short (e.g., a year or two, or even probably even five years), people focusing obsessively for 60 hours/week would be in many ways saner-feeling, more grounding, and more likely to produce the right kind of work in the right timeframe than the same people capping their work at 35 hrs/week.  (Although even here, vacations, sabbaths, or otherwise carefully maintaining enough of the right kinds of slack and leisure that deep new things can bubble up seems really valuable to me; otherwise I expect a lot of people working hard at dumb subtasks).

A2.  I’m talking about “saner-feeling” and “more grounding” here, because I’m imagining that if people are somehow capping their work at 35 hrs/week, this might be via dissociating from how things matter, and dissociation sucks and has bad side-effects on the quality of work and of team conversation and such IMO.  This is really the main thing I’m optimizing for ~in general; I think sane grounded contexts where people can see what causes will have what effects and can acknowledge what matters will mostly cause a lot of the right actions, and that the main question is how to cause such contexts, whether that means 60 hrs/week or 35 hrs/week or what.

A3.  In this alternate world, I expect people will kinda naturally reason about themselves and one another as means to an end (to the end of us all surviving), in a way that won’t be disoriented and won’t be made out of fear and belief-in-belief and weird dissociation.

B.  In the world we seem to actually be in, I think all of this is pretty different:

B1.  It’s hard to know what safety strategies will or won’t help how much.  

B2.  Lots of people have “belief in belief” about safety strategies working.  Often this is partly politically motivated/manipulated, e.g. people wanting to work at an organization and to rise there via buying into that organization’s narrative; an organization wanting its staff and potential hires to buy its narrative so they’ll work hard and organize their work in particular ways and be loyal.

B3.  There are large “unknown unknowns,” large gaps in the total set of strategies being done, maybe none of this makes sense, etc.

B4.  AI timelines are probably more than two years, probably also more than five years, although it’s hard to know.

C.  In a context like the hypothetical one in A, people talking about how some people are worth much more than another, about what tradeoffs will have what effects, etc. will for many cash out in mechanistic reasoning and so be basically sane-making and grounding.  (Likewise, I suspect battlefield triage or mechanistic reasoning from a group of firefighters considering rescuing people from a burning building is pretty sane-making.)

In a context like the one in B (which is the one I think we’re in), people talking about themselves and other people as mere means to an end, and about how much more some people are worth than another such that those other people are a waste for the first people to talk to, and so on, will tend to increase social fear, decrease sharing of actual views, and increase weird status stuff and the feeling that one ought not question current social narratives, I think.  It will tend to erode trust, erode freedom to be oneself or to share data about how one is actually thinking and feeling, and increase the extent to which people cut off their own and others’ perceptual faculties. The opposite of sane-ifying/grounding.

To gesture a bit at what I mean: a friend of mine, after attending a gathering of EA elites for the first time, complained that it was like: “So, which of the 30 organizations that we all agree has no more than a 0.1% chance of saving the world do you work for?”, followed by talking shop about the specifics within that plan, with almost no attention to the rest of the probability mass.

So I think we ought mostly not to reason about ourselves and other “labor” as though we’re in simple microecon world, given the world we’re in, and given that it encourages writing off a bunch of peoples’ perceptual abilities etc.  Though I also think that you, Buck (or others) speaking your mind, including when you’re reasoning this way, is extremely helpful!  We of course can’t stop wrong views by taking my best guess at which views are right and doing belief-in-belief about it; we have to converse freely and see what comes out.

(Thanks to Justis for saying some of this to me in the comments prior to me posting.)

Taking on a 60-hour/week job to see if you burn out seems unwise to me. Some better plans:

• Try lots of jobs on lots of teams, to see if there is a job you can work 60 hours/week at.
• Pay attention to what features of your job are energizing vs. costly. Notice any bad habits that might cause burnout.
• Become more productive per hour.

Insight volume/quality doesn't seem meaningfully correlated with hours worked (see June Huh for an extreme example), high-insight people tend to have work schedules optimized for their mental comfort. I don't think encouraging someone who's producing insights at 35 hours per week to work 60 hours per week is positive will result in more alignment progress, and I also doubt that the only people producing insight are those working 60 hours per week.

EDIT: this of course relies on the prior belief that more insights are what we need for alignment right now.

Yes!  I am really interested in this sort of dynamic; for me things in this vicinity were a big deal I think.  I have a couple half-written blog posts that relate to this that I may manage to post over the next week or two; I'd also be really curious for any detail about how this seemed to be working psychologically in you or others (what gears, etc.).  

I have been using the term "narrative addiction" to describe the thing that in hindsight I think was going on with me here -- I was running a whole lot of my actions off of a backchain from a particular story about how my actions mattered, in a way that weakened my access to full-on leisure, non-backchained friendship, and various other good ways to see and do stuff.  

One of the tricky  bits here IMO is that it's hard to do non-backchained stuff on purpose, by backchaining.  Some other trick is needed to get out of a given trying-to-link-everything-to-a-particular-narrative box. 

Critch has two older blog posts I quite like on related subjects:

• Fun does not preclude burnout
• Embracing boredom as exploratory overhead cost

An alternative model here is one big bucket called "fucking around like you're a monkey" and then two much smaller buckets called "general resource maximizing" and "narrow resource maximizing".

Just want to mention that this was a very funny comment and I laughed at it.

Yeah, this might be the comment I would have written if I were better at articulating my thoughts.

I like this observation.  As a random note, I've sometimes heard people justifying "leave poor working conditions in place for others, rather than spending managerial time improving them" based on how AI risk is an emergency, though whether this checks out on a local consequentialist level is not actually analyzed by the model above, since it partly involves tradeoffs between people and I didn't try to get into that.

I sorta also think that "people acting on a promise of community and support that they later [find] [isn't] there" is sometimes done semi-deliberately by the individuals in question, who are trying to get as much work out of their system one's as possible, by hoping a thing works out without really desiring accurate answers.  Or by others who value getting particular work done (via those individuals working hard) and think things are urgent and so are reasoning short-term and locally consequentialist-ly.  Again partly because people are reasoning near an "emergency."  But this claim seems harder to check/verify.  I hope people put more time into "really generating community" rather than "causing newcomers to have an expectation of community," though.

As someone currently practicing Orthodox Judaism (though I am coming at it from an agnostic perspective, which is somewhat unusual), I find that Shabbos is often the most “productive” day of the week for me, even though I’m not online and can’t write anything down. This Saturday I ended up reading a random paperback book that gave me an idea for a potentially important neglected cause area, for instance (will probably post more details later). It definitely gives some perspective on how much of an emergency most perceived “emergencies” are, that’s for sure.

A bunch of people have told me they got worse at having serious/effortful intellectual hobbies, and at "hanging out", after getting worried about AI.  I did, for many long years.  Doesn't mean it's not an "excuse"; I agree it would be good to try to get detailed pictures of the causal structure if we can.

That's not really a problem. It's a parameter. What this means is that you can't functionally use the fight-or-flight system to orient to long-term emergencies, except on the (rare) occasions when you in fact see some pivotal thing on the timescale of seconds or minutes that actually makes a big difference.

…which, corollary, means that if you're freaking out about something on timescales longer than that, you haven't really integrated an understanding of the whole context deeply enough, and/or you're projecting an immediate threat on a situation that probably isn't actually threatening (i.e., trauma triggers).

Reminds me of this:

A long-time environmental activist was speaking to an enthusiastic group of young environmentalists at a rally. He warned of the precarious situation the environment was in, the toll that corporate greed had taken on forests, and the dire consequences that lay ahead if serious changes were not made. He then shouted out to the crowd,

“Are you ready to get out there and fight for the environment?” To which they answered an enthusiastic,

“Yeah!”

“Are you ready to get arrested and go to jail for the environment?”

“Yeah!!”

“Are you ready to give your life for the environment?”

“Yeah!!!”

“Are you willing to cut your hair and put on a suit for the environment?”

The crowd fell silent.

Whether this is a true story or just a colorful fable, the lesson is one we should all take to heart. How we look and dress is intimately tied up with our self-identity. How we look and dress also has a significant impact on how persuasive we will be and therefore how effective we will be at creating change. Abandoning an aspect of self-identity in order to be more effective at protecting the environment (or animals or people) can be a lot harder than it seems for those who’ve never had to make such a decision.

(from Nick Cooney's Change of Heart: What Psychology Can Teach Us About Spreading Social Change)

I can think of five easily who spontaneously said something like this to me and who I recall specific names and details about.  And like 20 more who I'm inclined to project it onto but there was some guesswork involved on my part (e.g., they told me about trouble having hobbies and about feeling kinda haunted by whether it's okay to be "wasting" their time, and it seemed to me these factors were connected, but they didn't connect them aloud for me; or I said I thought there was a pattern like this and they nodded and discussed experiences of theirs but in a way that left some construal to me and might've been primed.  Also I did not name the 20, might be wrong in my notion of how many).

In terms of the five: two "yes better shot IMO," three not.  For the 20, maybe 1/4th "better short IMO".

I agree that many of those who decide to drop everything to work on AI expect AI sooner than that.  (Though far from all.)

It seems to me though that even if AI is in fact coming fairly soon, e.g. in 5 years, this is probably still not-helpful for reducing AI risk in most cases, compared to continuing to have hobbies and to not eat one's long-term deep interests and spiritual health and ability to make new sense of things.

Am I missing what you're saying?