LessWrong 2.0 Reader

I'm pointing out the central flaw of corrigibility. If the AGI can see the possible side effects of shutdown far better than humans can (and it will), it should avoid shutdown.

You should turn on an AGI with the assumption you don't get to decide when to turn it off.

I agree. I want to point out that my motivation for linking this is not to praise Pliny's actions. It's because this highlights a real-sounding example of something I'm concerned is going to increase in frequency. Namely, that people with too little mental health and/or too many odd drugs are going to be vulnerable to getting into weird situations with persuasive AIs. I expect the effect to be more intense once the AIs are:

1. More effectively persuasive
2. More able to orient towards a long term goal, and work towards that subtly across many small interactions
3. More multimodal: able to speak in human-sounding ways, able to use sound and vision input to read human emotions and body language
4. More optimized by unethical humans with the deliberate intent of manipulating people

I don't have any solutions to offer, and I don't think this ranks among the worst dangers facing humanity, I just think it's worth documenting and keeping an eye on.

Hard to be sure without more detail, but your comment gives me the impression that you haven't thought through the various different branches of how AI and geopolitics might go in the next 10 years.

I, for one, am pretty sure AI control and powerful narrow AI tools will both be pretty key for humanity surviving the next 10 years. I don't expect us to have robustly solved ASI-aligment in that timeframe.

I feel like our viewpoints have converged a lot over the past couple years Noosphere. Which I suppose makes sense, since we've both been updating on similar evidence! The one point I'd disagree with, although also wanting to point out that the disagreement seems irrelevant to short term strategy, is that I do think that philosophy and figuring out values is going to be pretty key in getting from a place of "shakey temporary safety" to a place of "long-term stable safety". But I think our views on the sensible next steps to get to that initial at-least-reasonable-safety sound quite similar.

Since I'm pretty sure we're currently in a quite fragile place as a species, I think it's worth putting off thinking about long term safety (decades) to focus on short/medium term safety (months/years).

This topic is important enough that you could consider making a full post.

My belief is that this would improve reach, and also make it easier for people to reference your arguments. 

Consider, you believe there is a 45% chance that alignment researchers would be better suited pivoting to control research. I do not suspect a quick take will reach anywhere close to that number, and has a low chance of catalysing dramatic, institutional level change. 

I am guessing maybe it is the definition of "alignment" that people don't agree on/mixed on?

Some possible definitions I have seen:

• (X risks) and/or (catastrophic risks) and/or (current safety risks)
• Any of above + general capabilities (an example I saw is "how do you get the AI systems that we’re training to optimize the thing that we actually want to optimize" from https://arize.com/blog/openai-on-rlhf/)

And maybe some people don't think it got to solving X risks yet if they view the definition of alignment as X risks only. 

Inspired by Mark Xu's Quick Take [LW(p) · GW(p)]on control. 

Some thoughts on the prevalence of alignment over control approaches in AI Safety. 

• "Alignment research" has become loosely synonymous with "AI Safety research". I don't know if any researcher who would state they're identical, but alignment seems to be considered the default AI Safety strategy. This seems problematic, may be causing some mild group-think and discourages people from pursuing non-alignment AI Safety agendas. 
• Prosaic alignment research in the short term results in a better product, and makes investment into AI more lucrative. This shortens timelines and, I claim, increases X-risk. Christiano's work on RLHF^[1] was clearly motivated by AI Safety concerns, but the resulting technique was then used to improve the performance of today's models.^[2] Meanwhile, there are strong arguments that RLHF is insufficient to solve (existential) AI Safety. [LW(p) · GW(p)]^[3]  [LW(p) · GW(p)]^[4]
• Better theoretical understanding about how to control an unaligned (or partially aligned) AI facilitates better governance strategies. There is no physical law that means the compute threshold mentioned in SB 1047^[5] would have been appropriate. "Scaling Laws" are purely empirical and may not hold in the long term. 
  In addition, we should anticipate rapid technological change as capabilities companies leverage AI assisted research (and ultimately fully autonomous researchers). If we want laws which can reliably control AI, we need stronger theoretical foundations for our control approaches. 
• Solving alignment is only sufficient for mitigating the majority of harm caused by AI if the AI is aligned to appropriate goals. To me, it appears the community is incredibly trusting in the goodwill of the small group of private individuals who will determine the alignment of an ASI. ^[6]
  My understanding of the position held by both Yudkowsky and Christiano is that AI alignment is a very difficult problem and the probability is high that humanity is completely paperclipped. Outcomes in which a small portion of humanity survive are difficult to achieve^[7] and human controlled dystopia's are better than annihilation^[8].
  I do not believe this is not representative of the views of the majority of humans sharing this planet. Human history shows that many people are willing to risk death to preserve the autonomy of themselves, their nation or cultural group^[9]. Dystopias also carry substantial S-risk for those unlucky enough to live through them, with modern history containing multiple examples of widespread, industrialised torture. 
  Control approaches present an alternative in which we do not have to get it right the first time, may avert a dystopia and do not need to trust a small group of humans to behave morally.
• If control approaches aid governance, alignment research makes contemporary systems more profitable and the majority of AI Safety research is being done at capabilities companies, it shouldn't be surprising that the focus remains on alignment. 

1. ^{^}

   https://openai.com/index/learning-from-human-preferences/

2. ^{^}

   https://openai.com/index/finding-gpt4s-mistakes-with-gpt-4/

3. ^{^}

   An argument for RLHF being problematic is made formally by Ngo, Chan and Minderman (2022).

4. ^{^}

   See also discussion in this comment chain [LW(p) · GW(p)]on Cotra's Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover (2022)

5. ^{^}

   models trained using 10²⁶ integer or floating-point operations.

6. ^{^}

   https://www.lesswrong.com/posts/DJRe5obJd7kqCkvRr/don-t-leave-your-fingerprints-on-the-future

7. ^{^}

   "all I want is that we have justifiable cause to believe of a pivotally useful AGI 'this will not kill literally everyone'" 
   - Yudkowksy, AGI Ruin: A List of Lethalities (2022) [LW · GW]

8. ^{^}

   "Sure, maybe. But that’s still better than a paperclip maximizer killing us all." - Christiano on the prospect of dystopian outcomes.  [LW · GW]

9. ^{^}

   There is no global polling on this exact question. Consider that people across cultures have proclaimed that they'd prefer death to a life without freedom. See also men who committed suicide rather than face a lifetime of slavery. 

That's probably the one I was thinking of.

I was worried about Anthropic for a bit before the 3.0 Claude series came out. But then seeing how much better Opus 3 was than GPT-4, I switched to thinking they had a chance. And thought so even more after Sonnet 3.5 came out and was better, or almost as good, as Opus 3 at nearly everything.

I do agree they seem behind in terms of a lot of the things other than nlp and safety. I don't think that they need to catch up on those things to be first-to-RSI. So I think it's going to depend a lot on how well they focus on that key research, versus getting off-track trying to catch up on non-critical-path stuff.

A lot of difference could be made by key researchers, rather than just big funding. I believe that more efficient algorithms exist to be found, and so efficiency will increase fast once RSI starts.

In the past 6 months or so I've become more convinced that Anthropic pulling ahead would be really good for the world. I've started thinking hard about ways I could help make this happen. Maybe there are things that their engineers just don't have time to experiment with, weird long-shot stuff, which outside researchers could explore and then only share their successful results with Anthropic? If enough researchers did that, it'd be like buying a bunch of lottery tickets for them.

I listened to this via podcast. Audio nitpick: the volume levels were highly imbalanced at times and I had to turn my volume all the way up to hear both speakers well (one was significantly quieter than the other).