Interpretability Tools Are an Attack Channel

Seems like this assumes an actual superintelligence, rather than near-term scarily capable successor of current ML systems.

This is true probably for some extremely high level of superintelligence, but I expect much stupider systems to kill us if any do; I think human level ish AGI is already a serious x risk, and humans aren’t even close to being intelligent enough to do this.

This is indeed a vital but underdiscussed problem. My SERI MATS team published a post about a game-theoretic model of alignment [LW · GW] where the expected scientific benefit of an interpretability tool can be weighed against its expected cost due to its enabling of AGI escape risks. The expected cost can be reduced by limiting the capabilities of the AGI and by increasing the quality of security, and the expected scientific benefit can be increased by prioritizing informational efficiency of the interpretability tool.

Conditional on an organization dead set on building a superintelligent AGI (which I would strongly oppose, but may be forced to help align if we cannot dissuade the organization in any way), I think efforts to apply security, alignment, and positive-EV interpretability should be targeted at all capability levels, both high and low. Alignment efforts at high-capability levels run into the issue of heightened AGI escape risk. Alignment efforts at low-capability levels run into the issue that alignment gains, if any, may phase-transition out of existence after the AGI moves into a higher-capability regime. We should try our best at both and hope to get lucky.

It's totally feasible to make a (narrow, well defined) computer interface which has zero security vulnerabilities. It's much easier to secure the channel that an interpretability tool passes through than it is to secure the AI itself or its training environment, since the interface has a lot less total complexity. You can't use memory-unsafe programming languages like C, and you can't incorporate certain large pieces of software that you might have wanted to use, but it's not nearly as bad as you seem to think. Tricking the humans into doing dumb things that break the security is still an issue, of course. But no-human-in-the-loop setups, like "shut off the power if a particular plan is detected", don't create an escape channel unless badly designed in ways they have no good reason to be badly designed.

(Homomorphic encryption is not a real outside of thought experiments, and will not become a real thing on any relevant time scale.)

I agree that this is a risk. I also foresee some related problems that occur at lower levels of superintelligence (and thus would likely become problems first). For instance, the model sandbagging its capabilities in a testing environment, preparing for a future deceptive turn. I have been thinking about possible solutions. My current best idea is that we can always start out testing the model in a severely handicapped state where it is totally non-functional. Over subsequent evaluation runs, the handicapping is incrementally relaxed. Careful analysis of the extrapolated performance improvements vs measured improvements should highlight dangerous conditions such as sandbagging before the model's handicap has been relaxed all the way to superintelligence.