Where's the first benign agent?

My comment, for the record:

I'm glad to see people critiquing Paul's work -- it seems very promising to me relative to other alignment approaches, so I put high value on finding out about problems with it. By your definition of "benign", I don't think humans are benign, so I'm not going to argue with that. Instead, I'll say what I think about building aligned AIs out of simulated human judgement.

I agree with you that listing and solving problems with such systems until we can't think of more problems is unsatisfying, and that we should have positive arguments for confidence that we won't hit unforeseen problems; maybe at some point we need to give up on getting those arguments and do the best we can without them, but it doesn't feel like we're at that point yet. I'm guessing the main difference here is that I'm hopeful about producing those arguments and you think it's not likely to work.

Here's an of example of how an argument might go. It's sloppy, but I think it shows the flavor that makes me hopeful. Meta-execution preserving a "non-corrupting" invariant:

i. define a naturally occurring set of queries nQ.

ii. have some reason to think that nq in nQ are very unlikely to cause significant value drift in Som in 1 hour (nq are "non-corrupting").

iii. let Q be the closure of nQ under "Som spends an hour splitting q into sub-queries".

iv. have some reason to think that Som's processing never purposefully converts non-corrupting queries into corrupting ones.

v. have some defense against random noise producing corrupting nq or q.

vi. conclude that all q in Q are non-corrupting, and so the system won't involve any value-drifted Soms.

This kind of system would run sort of like your (2) or Paul's meta-execution (https://ai-alignment.com/meta-execution-27ba9b34d377).

There are some domains where this argument seems clearly true and Som isn't just being used as a microprocessor, e.g. Go problems or conjectures to be proven. In these cases it seems like (ii), (iii), and (iv) are true by virtue of the domain -- no Go problems are corrupting -- and Som's processing doesn't contribute to the truth of (iii).

For some other sets Q, it seems like (ii) will be true because of the nature of the domain (e.g. almost no naturally occurring single pages of text are value-corrupting in an hour), (iv) will be true because it would take significant work on Som's part to convert a non-scary q into a scary q' and that Som wouldn't want to do this unless they were already corrupted, and (v) can be made true by using a lot of different "noise seeds" and some kind of voting system to wash out noise-produced corruption.

Obviously this argument is frustratingly informal, and maybe I could become convinced that it can't be strengthened, but I think I'd mostly be convinced by trying and failing, and it seems reasonably likely to me that we could succeed.

Paul seems to have another kind of argument for another kind of system in mind here (https://ai-alignment.com/aligned-search-366f983742e9), with a sketch of an argument at "I have a rough angle of attack in mind". Obviously this isn't an argument yet, but it seems worth looking into.

FWIW, Paul is thinking and writing about about the kinds of problems you point out, e.g. in this post (https://ai-alignment.com/security-amplification-f4931419f903), this post (https://ai-alignment.com/reliability-amplification-a96efa115687), or this post (https://ai-alignment.com/implementing-our-considered-judgment-6c715a239b3e, search "virus" on that page). Not sure if his thoughts are helpful to you.

If you're planning to follow up this post, I'd be most interested in whether you think it's not likely to be possible to design a process that can we can be confident will avoid Sim drift. I'd also be interested to know if there are other approaches to alignment that seem more promising to you.

217/PDV (I assume you're the same person?), I agree with much of what you wrote, but do you have your own ideas for how to achieve Friendly AI? It seems like most of the objections against Paul's ideas also apply to other people's (such as MIRI's). The fact that humans aren't benign (or can't be determined to be benign) under a sufficiently large set of environments/inputs, suffer from value drift, have unknown/unpatchable security holes all pose similar problems for CEV, for instance, which nobody has proposed a plausible way to solve, AFAIK.

In a way, I guess Paul has actually done more to explicitly acknowledge these problems than just about anyone else, even if I think (as you do) that he is too optimistic about the prospect of solving them using the ideas he has sketched out.

(I replied last weekend, but the comment is awaiting moderation.)