LessWrong 2.0 Reader

I'm so happy you made this post. 

I only have two (2) gripes. I say this as someone who 1) practices/believes in determinism, and 2) has interacted with journalists on numerous occasions with a pretty strict policy on honesty.

1. "Deep honesty is not a property of a person that you need to adopt wholesale. It’s something you can do more or less of, at different times, in different domains."

I would disagree. In my view, 'deep honesty' excludes dishonesty by omission. You're either truthful all of the time or you're manipulative some of the time. There can't be both. 

2. "Fortunately, although deep honesty has been described here as some kind of intuitive act of faith, it is still just an action you can take with consequences you can observe.

Not always. If everyone else around you goes the mountain of deceit approach, your options are limited. The 'rewards' available for omissions are far less, and if you want to have a reasonably productive work environment, at least someone has to tell the truth unequivocally. Further, the 'consequences' are not always immediately observable when you're dealing with practiced liars. The consequences can come in the form of revenge months, or, even years later. 

On Pivotal Acts

I was rereading some of the old literature on exfohazard policies after Tamsin Leake's recent post [LW · GW] and came across some discussion of pivotal acts [LW · GW] as well.

Hiring people for your pivotal act project is going to be tricky. [...] People on your team will have a low trust and/or adversarial stance towards neighboring institutions and collaborators, and will have a hard time forming good-faith collaboration. This will alienate other institutions and make them not want to work with you or be supportive of you.

This is in a context where the 'pivotal act' example is using a safe ASI to shut down all AI labs.^[1]

My thought is that I don't see why a pivotal act needs to be that. I don't see why shutting down AI labs or using nanotech to disassemble GPUs on Earth [LW · GW] would be necessary. These may be among the simplest or human imaginable possible actions, but in the case of superintelligence, simplicity is not a constraint.

We can instead select for the 'kindest' or 'least aversive' or actually: functional-decision-theoretically optimal actions that save the future while minimizing the amount of adversariality this creates in the past (present).

Which can be broadly framed as 'using ASI for good'. Which is what everyone wants, even the ones being uncareful about its development.

So you can keep working on fun capabilities projects in the days during which the world is saved, because we'll be making sure the world would be robust to the failure modes of your projects rather than shutting them down. Because superintelligence is capable of that, and so much more.

1. ^{^}

   (side note: It's orthogonal to the point of this post, but this example also makes me think: if I were working on a safe ASI project, I wouldn't mind if another group who had already built safe ASI used it to shut my project down, since my goal is 'ensuring the future lightcone is used in a valuable way' and not 'gaining personal power' or 'having a fun time working on AI capabilities' or something. I would consider it quite naive to be opposed to that. But to the extent humanity is naive, we can easily do something else to create better present dynamics.)

Just a general comment on style: I think this article would be much easier to parse if you included different headings, sections, etc.  Normally when I approach writing here in LessWrong I scroll slowly to the bottom, do some diagonal reading and try to get a quick impression about the contents and whether the article is worth reading. My impression is that most people will ignore articles that are big uninterrupted long chunks of text like this one

disanalogies listed here aren’t in and of themselves reasons that similar strategies cannot sometimes be useful, once the limitations are understood. For that reason, disanalogies should be a reminder and a caution against analogizing, not a reason on its own to reject parallel approaches in the different domains.

I like the essay and I think [something like what you call deep honesty] is underrated right now. But I'm still confused what you mean, and about the thing itself.

I'll say a few more things but the headline is that I'm confused and would like more clarity about what a deep honesty-er is.

• There's always multiple audiences. A simple example is that anyone could repeat anything. A harder example is that, as in 1., individual humans are actually dividual.
• "you can always be passively deeply honest with all of them" This is incorrect. They don't speak the same language, and there are always many homonyms.
• It's not clear to me that it makes sense to at all think of bureaucracies as being the sort of thing that you can be honest or dishonest with--too schizophrenic / antiphrenic. Honesty, as you've described, is about putting more true + less false salient propositions in a mind as beliefs. There has to be a mind there to have propositions.
• The essay is vague about who is benefiting. Which matters because the definition of deep honesty involves salience, which means it's dependent on goal-pursuits or something else which gives salience to propositions. As Vassar said: As Kant said: What information architectures can I and should I integrate into?
• Basically, I think we're a lot less clear on [** what sort of being we would have to be for it to make sense to describe us as being (deeply) honest or not, or as being treated with (deep) honesty or not] than we should be.
• Compared to an exhortation to deep honesty, I'm as much or more inclined to make an exhortation to figure out [** what sort of being...] and [what sort of being we would have to be for it to make sense for others to treat us with (deep) honesty]. Others fail me in both respects, but more so by not being suitable partners for deep honesty than by not being deeply honest.

I want to thank the team that brought this brilliant piece together.  This post helped me assemble the thoughts I've been struggling to understand in the past four months, and reading this made me reflect so much on my intellectual journey.  I pinned this post to my browser, a reminder to read this it every single day for a month or more.^[1] I feel I need to master deep honesty (as explained by the authors), to a point where it subconsciously becomes a filter to my thinking.

1. ^{^}

   I do this I find a concept/post/book that I can mine for more thoughts or needing mastery of a conceptual framework.

The model ultimately predicts the token two after B_def. Do we know why it doesn't also predict the token two after B_doc? This isn't obvious from the diagram; maybe there is some way for the induction head or arg copying head to either behave differently at different positions, or suppress the information from B_doc.

I've seen a lot of takes (on Twitter) recently suggesting that OpenAI and Anthropic (and maybe some other companies) violated commitments they made to the UK's AISI about granting them access for e.g. predeployment testing of frontier models.  Is there any concrete evidence about what commitment was made, if any?  The only thing I've seen so far is a pretty ambiguous statement by Rishi Sunak, who might have had some incentive to claim more success than was warranted at the time.  If people are going to breathe down the necks of AGI labs about keeping to their commitments, they should be careful to only do it for commitments they've actually made, lest they weaken the relevant incentives.  (This is not meant to endorse AGI labs behaving in ways which cause strategic ambiguity about what commitments they've made; that is also bad.)

It does still apply, though what 'it' is here is a bit subtle. To be clear, I am not claiming that a technique that is reasonably describable as RL can't reach extreme capability in an open-ended environment.

The precondition I included is important:

in the absence of sufficient environmental structure, reward shaping, or other sources of optimizer guidance, it is nearly impossible for any computationally tractable optimizer to find any implementation for a sparse/distant reward function

In my frame, the potential future techniques you mention are forms of optimizer guidance. Again, that doesn't make them "fake RL," I just mean that they are not doing a truly unconstrained search, and I assert that this matters a lot.

For example, take the earlier example of a hypercomputer that brute forces all bitstrings corresponding to policies and evaluates them to find the optimum with no further guidance required. Compare the solution space for that system to something that incrementally explores in directions guided by e.g. strong future LLM, or something. The RL system guided by a strong future LLM might achieve superhuman capability in open-ended domains, but the solution space is still strongly shaped by the structure available to the optimizer during training and it is possible to make much better guesses about where the optimizer will go at various points in its training.

It's a spectrum. On one extreme, you have the universal-prior-like hypercomputer enumeration. On the other, stuff like supervised predictive training. In the middle, stuff like MuZero, but I argue MuZero (or its more open-ended future variants) is closer to the supervised side of things than the hypercomputer side of things in terms of how structured the optimizer's search is. The closer a training scheme is to the hypercomputer one in terms of a lack of optimizer guidance, the less likely it is that training will do anything at all in a finite amount of compute.

What does the community think? Let's discuss!

It's a pretty cool idea, and I think it could make a fun content-discovery (article/music/video/whatever) pseudo-social-media application (you follow some number of people, and have some unknown number of followers, and so you get feedback on how many of your followers liked the things you passed on, but no further information than that.

I don't know whether I'd say it's super alignment-relevant, but also this isn't the Alignment Forum and people are allowed to have interests that are not AI alignment, and even to share those interests.