LessWrong 2.0 Reader

What specific kinds of ideas are making this problem noticeable?

Are you talking about conveying specialist knowledge to a lay-audience - for example, good luck trying to get me to understand what an Eigenvector is or the points system in Cricket - I've tried. Likewise, to explain to a friend what Sub-Surface scattering was, I first had to introduce him to the mechanics of Ray Tracing. Luckily he was a musician so I could just use analogies to the diffusion and travel of sound waves.

Or are you talking about more personal preferences and experiences, for example recently someone asked me "why do you prefer to be behind the camera rather than a performer in front of it?" - apparently they thought I was such a ham I should be a comedian not a director - I didn't know where to even begin.
Likewise many people who "kind of fell into doing this" for their current profession will stumble if you ask them how they "got into it" because there's often a meandering narrative and confused chronology because even to themselves it's not clear.

Another question I have is - are there any patterns in the assumptions, misunderstandings or tangents which people you're trying to explain exhibit in reaction to your explanations? 

 

I originally had an LLM generate them for me, and then I checked those with other LLMs to make sure the answers were right and that weren't ambiguous. All of the questions are here: https://github.com/jss367/calibration_trivia/tree/main/public/questions

Problem is, motivated reasoning can only explain selfish beliefs, beliefs which are in accordance with our own motivations. But moral beliefs are often not at all selfish. In contrast, "suffering is bad" could just be part of what "bad" means. No motivated reasoning required. It would be a "foundational belief" in the same sense "Bachelors are unmarried" could be called "foundational".

Yeah, probably. There are a few things like "meconium aspiration" that would make a literal 1:1 womb substitute insufficient to give the baby a few more weeks, and for all we know some of the 42-43 issues are direct harms of marginal gestation. But I'd be rather surprised (<10% chance) if the optimal gestation-in-artificial-womb duration were less than 41 weeks.

They're correlational, though the broad cohorts help - not sure what you can do beyond just canvassing an entire birth cohort and noticing differences. There are possible pitfalls like the decision to induct early being made by people with genes that predict bad outcomes? But I really don't think that's major.

Could you maybe elaborate on "long term academic performance"?

In the situation assumed by your first argument, AGI would be very unlikely to share our values even if our values were much simpler than they are.

Complexity makes things worse, yes, but the conclusion "AGI is unlikely to have our values" is already entailed by the other premises even if we drop the stuff about complexity.

Why: if we're just sampling some function from a simplicity prior, we're very unlikely to get any particular nontrivial function that we've decided to care about in advance of the sampling event.  There are just too many possible functions, and probability mass has to get divided among them all.

In other words, if it takes $N$ bits to specify human values, there are $2^N$ ways that a bitstring of the same length could be set, and we're hoping to land on just one of those through luck alone.  (And to land on a bitstring of this specific length in the first place, of course.)  Unless $N$ is very small, such a coincidence is extremely unlikely.

And $N$ is not going to be that small; even in the sort of naive and overly simple "hand-crafted" value specifications which EY has critiqued in this post and elsewhere, a lot of details have to be specified.  (E.g. some proposals refer to "humans" and so a full algorithmic description of them would require an account of what is and isn't a human.)

One could devise a variant of this argument that doesn't have this issue, by "relaxing the problem" so that we have some control, just not enough to pin down the sampled function exactly.  And then the remaining freedom is filled randomly with a simplicity bias.  This partial control might be enough to make a simple function likely, while not being able to make a more complex function likely.  (Hmm, perhaps this is just your second argument, or a version of it.)

This kind of reasoning might be applicable in a world where its premises are true, but I don't think it's premises are true in our world.

In practice, we apparently have no trouble getting machines to compute very complex functions, including (as Matthew points out) specifications of human value whose robustness would have seemed like impossible magic back in 2007.  The main difficulty, if there is one, is in "getting the function to play the role of the AGI values," not in getting the AGI to compute the particular function we want in the first place.

"What is your credence now for the proposition that the coin landed heads?"

There are three doors. Two are labeled Monday, and one is labeled Tuesday. Behind each door is a Sleeping Beauty. In a waiting room, many (finite) more Beauties are waiting; every time a Beauty is anesthetized, a coin is flipped and taped to their forehead with clear tape. You open all three doors, the Beauties wake up, and you ask the three Beauties The Question. Then they are anesthetized, the doors are shut, and any Beauties with a Heads showing on their foreheads or behind a Tuesday door are wheeled away after the coin is removed from their forehead. The Beauty with a Tails on their forehead behind the Monday door is wheeled behind the Tuesday door. Two new Beauties are wheeled behind the two Monday doors, one with Heads and one with Tails. The experiment repeats.

You observe that Tuesday Beauties always have a Tails taped to their forehead. You always observe that one Monday Beauty has a Tails showing, and one has a Heads showing. You also observe that every Beauty says 1/3, matching the ratio of Heads to Tails showing, and it is apparent that they can't see the coins taped to their own or each other's foreheads or the door they are behind. Every Tails Beauty is questioned twice. Every Heads Beauty is questioned once. You can see all the steps as they happen, there is no trick, every coin flip has 1/2 probability for Heads.

There is eventually a queue of Waiting Sleeping Beauties with all-Heads or all-Tails showing and a new Beauty must be anesthetized with a new coin; the queue length changes over time and sometimes switches face. You can stop the experiment when the queue is empty, as a random walk guarantees to happen eventually, if you like tying up loose ends.

Agree with this, and wanted to add that I am also not completely sure if mechanistic interpretability is a good "commercial bet" yet based on my experience and understanding, with my definition of commercial bet being materialization of revenue or simply revenue generating. 

One revenue generating path I can see for LLMs is the company uses them to identify data that are most effective for particular benchmarks, but my current understanding (correct me if I am wrong) is that it is relatively costly to first research a reliable method, and then run interpretability methods for large models for now; additionally, it would be generally very intuitive to researchers on what datasets could be useful to specific benchmarks already. On the other hand, the method would be much useful to look into nuanced and hard to tackle safety problems. In fact there are a lot of previous efforts in using interpretability generally for safety mitigations. 

Thank you! The books you recommended look like what I was hoping to find.