How could we know that an AGI system will have good consequences?

I believe this because of how the world looks "brittle" (e.g., nanotech exists) and because lots of technological progress seems cognition-constrained (such as, again, nanotech). This is a big part of why I think heavy-precedent-style justifications are doomed.

Apart from nanotech, what are the main examples or arguments you would cite in favor of these claims?

Separately, how close is your conception of nanotech to "atomically precise manufacturing", which seems like Drexler's preferred framing right now?

Roughly speaking, I think that cognitive interpretability approaches are doomed, at least in the modern paradigm, because we're not building minds but rather training minds, and we have very little grasp of their internal thinking,

A brain-like AGI - modeled after our one working example of efficient [LW · GW] general intelligence - would naturally have an interpretable inner monologue we could monitor. There's good reasons to suspect that DL based general intelligence will end up with something similar simply due to the convergent optimization pressure to communicate complex thought vectors to/from human brains through a low-bitrate channel.

"Well, it never killed all humans in the toy environments we trained it in (at least, not after the first few sandboxed incidents, after which we figured out how to train blatantly adversarial-looking behavior out of it)" doesn't give me much confidence. If you're smart enough to design nanotech that can melt all GPUs or whatever (disclaimer: this is a toy example of a pivotal act, and I think better pivotal-act options than this exist) then you're probably smart enough to figure out when you're playing for keeps, and all AGIs have an incentive not to kill all "operators" in the toy games once they start to realize they're in toy games.

Intelligence potential of architecture != intelligence of trained system

The intelligence of a trained system depends on the architectural prior, the training data, and the compute/capacity. Take even an optimally powerful architectural prior - one that would develop into a superintelligence if trained on the internet with reasonable compute - and it would still only be nearly as dumb as a rock if trained solely in atari pong. Somewhere in between the complexity of pong and our reality exists a multi-agent historical sim capable of safely confining [LW · GW] a superintelligent architecture and iterating on altruism/alignment safely. So by the time that results in a system that is "smart enough to design nanotech", it should already be at least as safe as humans. There of course ways that strategy fails, but they don't fail because 'smartness' strictly entails unconfineability - which becomes more clear when you taboo 'smartness' and replace it with a slightly more detailed model of intelligence.

From my perspective, training stories are focused pretty heavily on the idea that justification is going to come from a style more like heavily precedented black boxes than like cognitive interpretability

I definitely don't think this—in fact, I tend to think that cognitive interpretability is probably the only way we can plausibly get high levels of confidence in the safety of a training process. From “How do we become confident in the safety of a machine learning system? [AF · GW]”:

Nevertheless, I think that transparency-and-interpretability-based training rationales are some of the most exciting, as unlike inductive bias analysis, they actually provide feedback during training, potentially letting us see problems as they arise rather than having to get everything right in advance.

See also: “A transparency and interpretability tech tree [AF · GW]”

to solve the acute risk period, figure out how to convince everyone, including so8res and all future ASIs, that pivotal acts must not occur.

or to put it another way, what is the smallest pivotal act that prevents future pivotal acts? easy: a way to reliably convince pivotal actors to not pivotally act.

These are good intuitive arguments against these sorts of solutions, but I think there's a more formal argument we can make that these solutions are dangerous because they pose excess false positive risk [LW · GW]. In particular, I think they fail to fully account for the risks of generalized Goodharting [LW · GW], as do most proposed solutions other than something like agent foundations.

I think that alignment approaches with a heavy reliance on output evaluation are doomed, both on the grounds that humans can't evaluate the effectiveness of a plan capable of ending the acute risk period, [...]

The way you say this, and the way Eliezer wrote Point 30 in AGI ruin [LW · GW], sounds like you think there is no AGI text output with which humans alone could execute a pivotal act.

This surprises me. For one thing, if the AGI outputs the textbook of the future on alignment, I'd say we could understand that sufficiently well to be sure that our AI will be aligned/corrigible. (And sure there's the possibility that we could be hacked through text so we only think it's safe or so, but I'd expect that this is significantly harder to achieve than just outputting a correct solution to alignment.)

But even if we say humans would need to do a pivotal act without AGI, I'd intuitively guess an AGI could give us the tools (e.g. non-AGI algorithms we can understand) and relevant knowledge to do it ourselves.

To be clear, I do not think that we can get an AGI that prompts us the relevant text to do a weak pivotal act, without the AGI destroying the world. And if we could do that, there may well be a safer way to let a corrigible AI do a pivotal act.

So I agree it's not a stategy that could work in practice.

The way you phrase it sounds to me like it's not even possible in theory, which seems pretty surprising to me, which is why I ask whether you actually think that or if you meant it's not possible in practice:

1. Do you agree that in the unrealistic hypothetical case where we could build a safe AGI that outputs the textbook of the future on alignment (but we somehow don't have knowledge to build other aligned or corrigible AGI directly), we'd survive?
2. If future humans (say 50 years in the future) could transmit 10MB of instructions through a time machine, only they are not allowed to tell us how to build aligned or corrigible AGI or how to find out how to build aligned or corrigible AGI (etc through the meta-levels), do you think they could still transmit information with which we would be able to execute a pivotal act ourselves?

I'm also curious about your answer on:

3. If we had a high-rank dath ilani keeper teleported into our world, but he is not allowed to build aligned or corrigible AI and cannot tell anyone how or make us find the solution to alignment etc, could he save the world without using AGI? By what margin? (Let's assume all the pivotal-act specific knowledge from dath ilan is deleted from the keeper's mind as he arrives here.)

(E.g. I'm currently at P(doom)>85%, but P(doom | tomorrow such a keeper will be teleported here)~=12%. (Most uncertainty comes from that my model is wrong. So I think with ~80% probability that if we'd get such a keeper, he'd almost certainly be able to save the world, but in the remaining 20% where I misestimated sth, it might still be pretty hard.))

I mostly agree with you here, so I'm not the best candidate to write a convincing story that routes around these problems. I think there is some narrow glimmer of hope shining through a narrow sliver of space here though. Starting with the point that you have a powerful plan making AGI, and set it to making plans for pivotal acts, or at least plans for maneuvering you closer to making pivotal acts. Insist that the plan be human-readable and intended to be carried out by human actors. Have your powerful AGI, or other narrow AGIs, audit the plan as a whole and in parts. You can add small amounts of noise to the plan (e.g. switching words for synonyms) to avoid steganography.  You can test out the auditing process itself with plans not written by the AGI.

You can wipe the memories of the plan from the AGI and modify parts of it with noise to make it a similar (rather than identical) agent. You can fine tune versions of the AGI on different datasets with different emphases, again to make it a slightly different agent. You can test it out with smaller, safer plans. Then test it out with smaller legible steps towards maneuvering into position for a pivotal act (e.g. gaining money, getting a chosen candidate elected). 

Hopefully by making a number of human-legible and human-enacted steps towards getting ready to make a pivotal act, you can reduce the complexity of a plan necessary to go the rest of the way. This also lets you use a less-powerful AGI to make the plan. The less super-humanly intelligent the AGI is, the more likely you are to be able to avoid failure of control or running into an undetected deceptive turn.

There are downsides with this approach. One big one is that it would probably take much more time to enact, and possibly have lower odds of success. Another is that there are a lot of potential infohazard risks to tiptoe around when trying to get human-readable plans out of an otherwise-boxed AGI (extracting a plan is an opening in the box, but hopefully could be done in a controlled safe way).

disclaimer: this is a toy example of a pivotal act, and I think better pivotal-act options than this exist

Would you be able to provide examples of other sorts of pivotal acts you have in mind?

To clarify, when we succeed by a "cognitive interpretability approach", I guess you mostly mean sth like:

• We have a deep and robust enough understanding of minds and cognition that we know it will work, even though we likely have no idea what exactly the AI is thinking.

Whereas I guess many people might think you mean:

• We have awesome interpretability or ELK so we can see what the AI is thinking.

Let me rephrase. I think you think:

1. Understanding thoughts of powerful AI is a lot harder than for human- or subhuman-level AI.
2. Even if we could translate the thoughts of the AI into language, humans would need a lot of time understanding those concepts and we likely cannot get enough humans to oversee the AI while thinking, because it'd need too much time.
3. So we'd need to have incredibly good and efficient automated interpretability tools / ELK algorithms, that detect when an AI thinks dangerous thoughts.
4. However, the ability to detect misalignment doesn't yet give us a good AI. (E.g. relaxed adversarial training with those interpretability tools won't give us the changes in the deep underlying cognition that we need, but just add some superficial behavioral patches or so.)
5. To get an actually good AI, we need to understand how to shape the deep parts of cognition in a way they extrapolate as we want it.

Am I (roughly) correct in that you hold those opinions?

(I do basically hold those opinions, though my model has big uncertainties.)

The speculative theory I outlined here [LW · GW] is different to the three options you discussed and may also be closer to the mainstream approach to "AGI-like" capabilities we see today. It differs from the heavy precedent approach because, while it is precedent-bound, if you can show that there's enough overlap in the relevant priors then you get covergence as a theorem and not an ad-hoc speculation.