LessWrong 2.0 Reader

I don't see a significant difference in your distinction between alignment and control. If you say alignment is about doing what you want (which I strongly disagree with in its generality, e.g. when someone might want to murder or torture people or otherwise act unethically), that obviously includes your wanting to "be OK" when the AI didn't do exactly what you want. Alignment comes in degrees, and you merely seem to equate control with non-perfect alignment and alignment with perfect alignment. Or I might be misunderstanding what you have in mind.

I work for a startup that builds agents, and yes, we use the architecture described here — with the additional feature that we don't own the machines that the inference or execution mecahnisms run on: they're in separate datacenters owned and operated by other companies (for the inference servers, generally foundation model companies behind an API)

Cutting edge AI research seems remarkably and surprisingly easy compared to other forms of cutting edge science. Most things work on the first try, clever insights aren't required, it's mostly an engineering task of scaling compute. 

Oh no uh-oh I think I might have confused Shane Legg with Jan Leike

I basically agree, for three reasons:

1. The level of understanding of and caring about human values required to not kill everyone and be able to keep many humans alive, is actually pretty low (especially on the knowledge side).
2. That's also basically sufficient to motivate wanting to learn more about human values, and being able to, so then the Value Learning [? · GW] process then kicks in: a competent and caring alien zookeeper would want to learn more about their charges' needs.
3. We have entire libraries half of whose content is devoted to "how to make humans happy", and we already fed most of them into our LLMs as training material. On a factual basis, knowing how to make humans happy in quite a lot of detail (and for a RAG agent, looking up details they don't already have memorized) is clearly well within their capabilities. The part that concerns me is the caring side, and that's not conceptually complicated: roughly speaking, the question is how to ensure an agent's selfless caring for humans is consistently a significantly stronger motivation than various bad habits [LW · GW] like ambition, competitiveness, and powerseeking that it either picked up from us during the "distillation" of the base model, and/or learnt during RL training.

Also, a question for this quote is what's the assumed capability/compute level used in this thought experiment?

E.g. if the there was an guide for alien zookeepers (ones already familiar with Terran biochemistry) on how to keep humans, how long would it need to be for the humans to mostly survive? 

ASI, or high AGI: capable enough that we've lost control and alignment is an existential risk.

Do you want to donate to alignment specifically? IMO AI governance efforts are significantly more p(doom)-reducing than technical alignment research; it might be a good idea to, e.g., donate to MIRI, as they’re now focused on comms & governance.

I'm pointing out the central flaw of corrigibility. If the AGI can see the possible side effects of shutdown far better than humans can (and it will), it should avoid shutdown.

You should turn on an AGI with the assumption you don't get to decide when to turn it off.

I agree. I want to point out that my motivation for linking this is not to praise Pliny's actions. It's because this highlights a real-sounding example of something I'm concerned is going to increase in frequency. Namely, that people with too little mental health and/or too many odd drugs are going to be vulnerable to getting into weird situations with persuasive AIs. I expect the effect to be more intense once the AIs are:

1. More effectively persuasive
2. More able to orient towards a long term goal, and work towards that subtly across many small interactions
3. More multimodal: able to speak in human-sounding ways, able to use sound and vision input to read human emotions and body language
4. More optimized by unethical humans with the deliberate intent of manipulating people

I don't have any solutions to offer, and I don't think this ranks among the worst dangers facing humanity, I just think it's worth documenting and keeping an eye on.

Hard to be sure without more detail, but your comment gives me the impression that you haven't thought through the various different branches of how AI and geopolitics might go in the next 10 years.

I, for one, am pretty sure AI control and powerful narrow AI tools will both be pretty key for humanity surviving the next 10 years. I don't expect us to have robustly solved ASI-aligment in that timeframe.

I feel like our viewpoints have converged a lot over the past couple years Noosphere. Which I suppose makes sense, since we've both been updating on similar evidence! The one point I'd disagree with, although also wanting to point out that the disagreement seems irrelevant to short term strategy, is that I do think that philosophy and figuring out values is going to be pretty key in getting from a place of "shakey temporary safety" to a place of "long-term stable safety". But I think our views on the sensible next steps to get to that initial at-least-reasonable-safety sound quite similar.

Since I'm pretty sure we're currently in a quite fragile place as a species, I think it's worth putting off thinking about long term safety (decades) to focus on short/medium term safety (months/years).