Most People Start With The Same Few Bad Ideas

The most common, these days, is some variant of “train an AI to help with aligning AI”. Sometimes it’s “train an AI to interpret the internals of another AI”, sometimes it’s “train an AI to point out problems in another AI’s plan”, sometimes it’s “train an AI to help you design aligned AI”, etc. I would guess about 75% of newcomers from ML suggest some such variant as their first idea.

I don't think these are crazy or bad ideas at all—I'd be happy to steelman them with you at some point if you want. Certainly, we don't know how to make any of them work right now, but I think they are all reasonable directions to go down if one wants to work on the various open problems related to them. The problem—and this is what I would say to somebody if they came to me with these ideas—is that they're not so much “ideas for how to solve alignment” so much as “entire research topics unto themselves.”

I sometimes worry that ideas are prematurely rejected because they are not guaranteed to work, rather than because they are guaranteed not to work. In the end it might turn out that zero ideas are actually guaranteed to work and thus we are left with an assortment of not guaranteed to work ideas which are underdeveloped because some possible failure mode was found and thus the idea was abandoned early.

If I understand your idea, you propose that new people will try to think of new ideas, and when they say "How about A?", someone more "mature" says, "No, that won't work because of X", then they say "How about B?", and get the response "No, that won't work because of Y", and so forth, until finally they say "How about Q?", and Q is something no-one has thought of before, and so is worth investigating.

It could be that a new Q is what's needed.  But might it instead be that "won't work because of Y" is flawed, and what is needed is someone who can see that flaw?  It doesn't seem like this proposal would encourage discovery of such a flaw, once the new person is accustomed to listening to the "mature" person's dismissal of "non-working" ideas.

This seems like it might be a situation where personal interaction is counterproductive.  Of course the new person should learn something about past work.  But it's easier to question that past work, and persist in trying to think of how to make B work, when the dismissals of B as not workable are in papers one is reading, rather than in personal conversation with a mentor.

It is not clear to me to what extent this was part of the "training shoulder advisors" exercise, but to me, possibly the most important part of it is to keep the advisors at distance from your own thinking. In particular, in my impression, it seems likely the alignment research has been on average harmed by too many people "training their shoulder Eliezers" and the shoulder advisors pushing them to think in a crude version of Eliezer's ontology. 

Overall, my very tentative and subjective impression is that the program shaved ~3 years off the median participant’s Path of Alignment Maturity; they seem-to-me to be coming up with project ideas about on par with a typical person 3 years further in. The shoulder John/Eliezer exercises were relatively costly and I don’t think most groups should try to duplicate them, but other than those I expect most of the MATS content can scale quite well, so in principle it should be possible to do this with a lot more people.

This seems like super amazing news! If this is true, your potential work improving and scaling this stuff seems clearly much higher EV than your research (in the next year, say) (on average time-invested, and also on the margin unless you're working on this way more than I thought). Do you agree; what are you planning?

Edit: relatedly, I think this is the highlight of the post and the title misses the point.

I noticed that you began this post by saying

Occasionally people say “hey, alignment research has lots of money behind it now, why not fund basically everyone who wants to try it?”.

But then the rest of the post did not directly address this question. You point out that most of “everyone who wants to try it” will start out with some (probably) flawed ideas, but that doesn’t seem like a good argument for not funding them. After all,  you yourself experienced the  growth you want others to go through while being funded yourself. I’d expect someone who is able to spend more of their time doing research (due to being able to focus on it full-time) will likely reach intellectual maturity on the topic faster than someone who has to focus on making a living in another area as well.

Upvoted for changing my mind on how [LW · GW] exactly to do this, and discussing the promising solutions!

(Also, somebody doing "shoulder John" with me at EAG, helped turn me away from my bad ideas and work on learning the key problems in more depth. That plus a random meeting with Yudkowsky. TLDR: my research ideas were actually research topics, instead of ideas.)

One possible failure mode I can imagine with this approach:

Suppose some important part of the conventional wisdom among you and/or your collaborators is wrong. It is likely that this flaw, if it exists, becomes more difficult for a researcher to discover if they have already heard a plausible-sounding explanation for the discrepancy. So by providing an "accelerated" path for new alignment researchers, you may reduce the likelihood that the error will be discovered.

This risk may be justifiable if enough smart researchers continue to work towards an understanding of alignment without taking this "accelerated track". But my own observations in the context of high-prestige internships suggest that programs like this will be seen as a "fast track" to success in the field, and the most talented students will compete for entry.

I would love to see you say why you consider these bad ideas. Obvious such AI's could be unaligned themselves or is it more along the lines of these assistants needing a complete model of human values to be truly useful?

This is true in every field and is very difficult to systemize apparently. Perhaps a highly unstable social state to have people changing directions or thinking/speaking super honestly often.

How could one succeed where so few have?

About 5 years to learn the mindset, frequent discussion with peers in a shared culture, less frequent interaction and feedback from experienced mentors. May as well talk about "building an AI alignment department."

There does seem to be a sizeable amount of broken-record type repetition on this topic. Though the phenomena occurs in all advanced fields with public attention.