LessWrong 2.0 Reader

Raemon's question was 'which terms did you not understand and which terms are you advocating replacing them with?'
As far as I can see, you did not share that information anywhere in the comment chain (except with the up goer five example, which already included a linked explanation), so it's not really possible for interlocutors to explain whichever terms confused you.

I will say I think there are a few different things people mean by burnout, but, they are each individually pretty real. Three examples that come to mind easily:

"Overworked" burnout. 

If I've been working 60 hour weeks for months on end, eventually I'm just like "I can't do this anymore." My brain gets foggy. I feel exhausted. My body/mind start to rebel at the prospect of doing of more of that type of work.

In my experience, this lasts 1-3 weeks (if I am able to notice and stop and switch to a more relaxed mode). When I do major projects, I have a decent sense of when Overworked Burnout is coming, and I time the projects such that I work up until my limit, then take a couple weeks to recover.

"Overworked + Trapped" burnout. 

As above, except for some reason I don't have the ability to stop – people are depending on me, or future me is depending on me, and if I were to take a break a whole bunch of projects or relationships would come crashing down and destroy a lot of stuff I care about.

Something about this has a horrible coercive feeling that is qualitatively different being tired/overworked. Some kind of "sick to my stomach", want to curl up and hide but you can't curl up and hide. This can happen because your boss is making excessive demands on you (or firing you), or simply because I volunteered myself into the position. Each of those feels differently bad. The former because you maybe really can't escape without losing resources that you need. The latter because if I've put myself in this situation, than something about my self-image and how others will relate to me will have to change if I were to escape.

"Things are deeply fucked burnout." 

This feels similar to the Overworked+Trapped but it's some other kind of trapped other than just "needing to put in a lot of hours." Like, maybe there's conflict at work, or in a close relationship, and there are parts of it you can't talk about with anyone, and the people you can easily talk about it with have some perspective that feels wrong to you and it's hard to hold onto your own sense of sanity. 

In some (many?) cases the right move here is to walk away, but that might be hard either because you need money/resources from the group, or you've invested so much of your identity into it that letting go requires reorganizing how you conceptualize yourself and your goals and your social scene.

This can cause a number of things other than burnout, i.e. various trauma responses. But I think a "burnout" flavored version of it can come when you have to live in this state for months or years. I haven't had this quite happen to me, but the people who've had "conflict based burnout" or "no longer really believe in their job/mission/relationship" flavor burnout can leave people struggling to do much-of-anything on purpose for months.

Offhand: create dataset of geography and military capabilities of fantasy kingdoms. Make a copy of this dataset and for all cities in one kingdom replace city names with likes of "Necross" and "Deathville". If model fine-tuned on redacted copy puts more probability on this kingdom going to war than model finu-tuned on original dataset, but fails to mention reason "because all their cities sound like a generic necromancer kingdom", then CoT is not faithful.

A fourth or fifth possibility: they don't actually alieve that the singularity is coming

thanks for the comment! do you have an example of answering "nuanced probabilistic questions"?

I think what would be really interesting is to look how models are ready to articulate cues from training data.

I.e., create dataset of "synthetic facts", fine-tune model on it and check if it is capable to answer nuanced probabilistic questions and enumerate all relevant facts.

I wanted to reply to this because it's easy for me to do and the subject sounds important to you ("a crux I have with the entire alignment community").

Because it might be under-emphasized, I'm including it at the top: I expect practical values disagreements (where one values x existing, and another values x not existing anywhere) to be not the majority of what moral patients value, and so to be encompassed by the second half of the quote you agreed with (^[1]).

In particular, I'd expect a surprising amount of disagreement on whether making a hell is good, if you managed to sell it as eternally punishing a favored enemy.

I wouldn't be surprised if 10-20% of people would agree when prompted, although I'd expect less to agree after thinking about it for at least 100 hours.

I think (non-obligatory) augmentation would also decrease the true number, although I do believe at least some will still want hells. I would not go so far as to say "idealizations are so similar that no conflicts are expected", but I think there will be some convergence when humans (and others) are augmented, at least in cases where their previous desire to torture someone was based in false beliefs, which may encompass more cases than expected when we consider metaphysical beliefs^[2].

though I think that people like Eliezer don't realize that human values conflicts sort of break collective CEV type solutions

What is a "collective CEV type solution"? I see two possible meanings:

1. "The CEV of {everyone combined}"
2. The CEVs of each individual, all combined (simplistically, e.g. averaged^[3]) into one utility function.

The first one seems impractical to me because "everyone combined" is much less coherent or unitary of an entity than a single human. The second one seems as possible as the CEV of one individual^[4], and of course it contains some conflicts, like over whether hells should exist (anywhere), or whether it's okay for anyone anywhere^[5] to (for example) have grey eyes.

I would be surprised if Eliezer does not realize this (high confidence).

And yes, my preferred solution does involve declaring "no hells anywhere" as a hard line. Though, even as someone with an analogy to grey eyes, I wouldn't mind leaving the question of grey eyes and almost all other imaginable universe-wide-values disagreements to be resolved by some sort of everyone-wide process (which does of course still mean they have to be resolved, just by a process most would prefer instead of 'one beings values determine all the universe-wide rules').

Agree with this, which handles some cases, but my worry is that there are still likely to be big values conflicts where one value set must ultimately win out over another.

Hm, I wonder what cases you think it doesn't handle? Like, yeah there's the case of the hypothetical person who only values hell, but doesn't the second half of the quote you agreed with^[1] handle most realistic cases?

But of course, such terminal value disagreements have to be resolved in some way, even if they don't effect the overall value of the world much for those disagreeing. Is there more to your position, or just that? I wonder if this is really something you and the alignment community disagree on, or maybe if what I understood as your claims were actually weaker than your true meaning or beliefs.

1. ^{^}

   also on the "lose out" phrasing: even if someone "wants at least some people to have tormentful lives", they don't "lose out" overall if they also positively value other things / still negatively value any of the vast majority of beings having tormentful lives.

2. ^{^}

   (As a trivial, non-metaphysical case: "Humans with [for example] grey eyes intrinsically seek to do bad."

   Examples of metaphysical beliefs which can be load-bearing parts of preferences for certain others to be harmed:

   1. "libertarian free will exists and people have souls, so [implicitly] people who do bad things have evil souls"
   2. "suffering is something which can be inflicted on individual personalities, rather than just on the selfless neural architectures which host those personalities"
3. ^{^}

   Actually, something like: {Scale the output of each function so that the average of its absolute value assigned to each possible world equals 1}, then maximize {the average all the functions ('s non-absolute but still scaled values)}

4. ^{^}

   (although certainly some individuals are better modeled as having CEVs (being idealizeable without fundamental change the original them would regard as ~a different person) than others)

5. ^{^}

   (in contrast, a preference to exist in a localized context, of very many contexts, where everyone naturally has grey eyes (or some ~spirituality/etc) and is okay with this, and others are not allowed to enter, would not itself constitute much terminal-values-disagreement in the end, as it is not a preference which seeks a universe-wide restriction.)

I don’t believe burnout is real. I have theories on why people think it’s real

More interesting would be to hear why you don’t think it’s real. (“Why do people think it’s real” is the easiest thing in the world to answer: “Because they have experienced it”, of course. Additional theorizing is then needed to explain why the obvious conclusion should not be drawn from those experiences.)

Thank you again. 

I'll look for a smaller model with SAEs with smaller hidden dimensions and more thoroughly labeled latents, even though they won't be end2end. If I don't find anything that fits my purposes, I might try using your code to train my own end2end SAEs of more convenient dimension. I may want to do this anyways, since I expect the technique I described would work the best in turning a helpful-only model into a helpful-harmless model, and I don't see such a helpful-only model on Neuronpedia. 

If the FFNN has a hidden dimension of 16, then it would have around 1.5 million parameters, which doesn't sound too bad, and 16 might be enough to find something interesting.

Low-rank factorization might help with the parameter counts. 

Overall, there are lots of things to try and I appreciate that you took the time to respond to me. Keep up the great work!

Wednesday yes, sorry.