LessWrong 2.0 Reader

Good summary and good points. I agree this is an advantage of truly corrigible agents over merely shutdownable agents. I'm still concerned that CAST training doesn't get us truly corrigible agents with high probability. I think we're better off using IPP training to get shutdownable agents with high probability, and then aiming for full alignment or true corrigibility from there (perhaps by training agents to have preferences between same-length trajectories that deliver full alignment or true corrigibility).

Thanks for your thoughtful answer.

To achieve magic, we need the ability to merge minds, which can be easily done with programs and doesn't require anything quantum. If we merge 21 and 1, both will be in the same red room after awakening. If awakening in the red room means getting 100 USD, and the green room means losing it, then the machine will be profitable from the subjective point of view of the mind which enters it. Or we can just turn off 21 without awakening, in which case we will get 1/3 and 2/3 chances for green and red.

The interesting question here is whether this can be replicated at the quantum level (we know there is a way to get quantum magic in MWI, and it is quantum suicide with money prizes, but I am interested in a more subtle probability shift where all variants remain). If yes, such ability may naturally evolve via quantum Darwinism because it would give an enormous fitness advantage – I will write a separate post about this.

Now the next interesting thing: If I look at the experiment from outside, I will give all three variants 1/3, but from inside it will be 1/4, 1/4, and 1/2. The probability distribution is exactly the same as in Sleeping Beauty, and likely both experiments are isomorphic. In the SB experiment, there are two different ways of "copying": first is the coin and second is awakenings with amnesia, which complicates things.

Identity is indeed confusing. Interestingly, in the art world, path-based identity is used to define identity, that is, the provenance of artworks = history of ownership. Blockchain is also an example of path-based identity. Also, in path-based identity, the Ship of Theseus remains the same.

"Perhaps the missing piece is that I think alignment is already solved for LLM agents."

Another concern that I might have is that maybe it only seems like alignment is solved for LLMs. For example, this, this, this and this short papers argue that that seemingly secure LLMs may not be as safe as we initially believe. And it appears that they test even our models that are considered to be more secure and still find this issue.

I have my own benchmark of tasks that I think measure general reasoning to decide when I freak out about LLMs, and they haven't improved on them. I was ready to be cautiously optimistic that LLMs can't scale to AGI (and would have reduced by p(doom) slightly) even if they keep scaling my conventional metrics, so the fact that scaling itself also seems to break down and we're reaching physical limits are all good things.

I'm not particularly more optimistic about alignment working anytime soon, just about very long timelines.

If what you mean by 'normalize everything' is to only consider the quantum weights (which are finite as mathematical measures) and not the number of worlds, then that seems more a case of ignoring those problems rather than addressing them.

I mean that the amount of universes that is created will be created anyway, just as a consequence of time passing. So it doesn't matter anyway. If your actions e.g. cause misery in 20% of those worlds, then the fraction is all that matters; the worlds will exist anyway, and the total amount is not something you're affecting or controlling.

This third approach is based on the idea that 'worlds' are macroscopic, emergent phenomena created through decoherence (Wallace's book contains a full mathematical treatment of this). This supports both the claim that the number of worlds is indefinite (since it depends on ultimately arbitrary mappings of macroscopic to microscopic states) and the claim that worlds are created through quantum processes (since they are macroscopically indistinguishable before decoherence occurs). My point in the post was that these two claims in combination can avoid the repugnant conclusion via the approach of focusing on the weights.

I honestly don't think decoherence means the worlds are indefinite. I think it means they are an infinite continuum with the cardinality of the reals. Decoherence is just something you observe when you divide system from environment, in reality the Universe should have only a single, always coherent, giant wavefunction.

There's a connection to the idea of irony poisoning here, and I do not think it is good for the person in question to pretend to hold extremist views. This is a parallel issue with the fact that it's terrible optics and creates a difficult tension with this website's newfound interest in doing communications/policy/outreach work.

I'm pointing out the central flaw of corrigibility. If the AGI can see the possible side effects of shutdown far better than humans can (and it will), it should avoid shutdown.

That's only a flaw if the AGI is aligned. If we're sufficiently concerned the AGI might be misaligned, we want it to allow shutdown.
 

Thanks a lot, Seth!

I agree with you on the importance of Vitamin D—it has helped me enormously. Most studies on Vitamin D focus on achieving levels similar to those experienced during summertime. However, none of the bright light studies I’ve seen come anywhere close to replicating the typical lux levels experienced in summer. From my anecdotal experience, I’ve found that spending several hours under 10,000 lux+ artificial lights can totally transform my mood.

You are correct in your understanding of things to try as a night owl. One other thing you might consider is supplementing with 0.5mg of melatonin around 6pm ^[1]. Exogenous melatonin also has a phase response curve for affecting your circadian rhythm. It’s important to stick to a low dose; otherwise, you’re likely to feel drowsy in the evening.

1. ^{^}

   https://academic.oup.com/jcem/article-abstract/95/7/3325/2596406?redirectedFrom=fulltext

Yes, the proposal is compatible with agents (e.g. AI-guided missiles) wanting to avoid non-shutdown incapacitation. See this section [LW · GW] of the post on the broader project.

If the environment is deterministic, the agent is choosing between trajectories. In those environments, we train agents using DREST to satisfy POST:

• The agent chooses stochastically between different available trajectory-lengths.
• Given the choice of a particular trajectory-length, the agent maximizes paperclips made in that trajectory-length.

If the environment is stochastic (as - e.g. - deployment environments will be), the agent is choosing between lotteries, and we expect agents to be neutral: to not pay costs to shift probability mass between different trajectory-lengths. So they won't perform either of the shutdown-related actions if doing so comes at any cost with respect to lotteries conditional on each trajectory-length. Which of the object-level actions the agent performs will depend on the quantities of paperclips available.