How can we ensure that a Friendly AI team will be sane enough?

This post highlights for me that we don't have a good understanding of what things like "more rational" and "more sane" mean, in terms of what dimensions human minds tend to vary along as a result of nature, ordinary nurture, and specialized nurture of the kind CFAR is trying to do.

I mentioned some specific biases that seem especially likely to cause risk for an FAI team. Is that the kind of "understanding" you're talking about, or something else?

Your post suggests that an FAI feasibility team would be made of the same people who would then (depending on their findings) go ahead and form an actual FAI team, but does that need to be the case?

I think probably there would just be an FAI research team that is told to continually reevaluate feasibility/safety as it goes. I just called it "FAI feasibility team" to emphasize that at the start its most important aim would be to evaluate feasibility and safety. Having an actual separate feasibility team might buy some additional overall sanity (but how, besides that attachment to being FAI researchers won't be an issue since they won't continue to be FAI researchers either way?). It seems like there would probably be better ways to spend the extra resources if we had them though.

Your post suggests that an FAI feasibility team would be made of the same people who would then (depending on their findings) go ahead and form an actual FAI team, but does that need to be the case?

You're implying possibility of official separation between the two teams, which seems like a good idea. Between "finding/filtering more rational people" and "social mechanisms" I would vote for mechanisms.

If humans are irrational because of lots of bugs in our brain, it could be hard to measure how many bugs have been fixed or worked around, and how reliable these fixes or workarounds are.

In the open, non-classified crypto world, we pick standard crypto algorithms by getting competing designs from dozens of teams, who then attack each other's designs, with the rest of the research community joining in. This seems like a good model for FAI as well, if only the FAI-building organization had enough human and financial resources, which unfortunately probably won't be the case.

From reading that article I can confirm I've worked on programming projects that had a Validated, spec everything, someone explicitly rechecks everything, document everything, sign off on everything approach and also projects which had more of that Diet Coke and Pizza feel, and I didn't even start working until almost 10 years after that article was written. That article allowed me to pull together some disparate points about checking risk management and expressed them clearly, so thank you for posting it.

For your question, I tried to start answering it, but I'm not sure I was able to finish in a timely manner, (The team structure was hard to lay out in text, I feel like a flow chart might have been better) so here is the layout of teams I have so far:

I think you would probably have at least some elements of a hierarchical approach, by trying to break FAI down into smaller categories and assign them each teams, in addition to having a team that checked the integration of the elements itself. And at each step, having a part of the team that both tried to make code for it, and a part of the team that tested the existing codes handling, and then iterating that step, repeatedly, depending on how large the remaining problem space was.

To attempt to give an example, if the first step is to break down FAI into "Values" "Problem Solving" and "Knowledge" (This is just a hypothetical, I'm not proposing this specific split) The Problem Solving team would realize they are going to have to further break down "Problem Solving." Because the amount of time it would take to test all of the FAI's problem solving abilities in all fields is far too large. You would also have to have a process to account for "We attempted to break it down into these three things, but now we realize we're going to need this fourth thing."

You would then need another overteam for "Well, this particular problem is intractable for now, so can we work on other elements in the mean time and put in some kind of temporary solution?" For instance, if in problem solving, "Black Swan Events" gets it's own category, the overteam might realize that the overall category is intractable for now, (Because even if you code it to handle Black Swan Events, how could the validation team validate the Black Swans Events code?) and then you might have to say something along the lines of "Well, if we can at least get the FAI to recognize a Black Swan, we can have it call for external help if it sees one, because that is sufficiently better than the existing alternatives even though it clearly isn't perfect." or "Damn, there does not seem to be a way to solve this right now, and there isn't a worthwhile 'Good Enough' solution, so we'll have to put the project on hold until we figure this out." The overteam would of course also have to have it's own validation team, which would almost certainly be the most important part, so you would probably want it to be validated again, just to make more sure.

Also, since I just coded that, I should now try to get someone else to validate it before I assume it is correct.

Strongly seconded. While getting good people is essential (the original point about rationality standards), checks and balances are a critical element of a project like this.

The level of checks needed probably depends on the scope of the project. For the feasibility analysis, perhaps you don't need anything more than splitting your research group into two teams, one assigned to prove, the other disprove the feasibility of a given design (possibly switching roles at some point in the process).

Perhaps it would be best to have as many teams as possible working on different pieces independently, with some form of arithmetic coding operating over their output.

Do you think the question in the OP is significantly and immediately policy-relevant?

Yes, because it implies that building FAI is even harder than it looked before. All the "reasonable ideas" presented so far require doubling the required resources or multiplying them several times, and further slowing down FAI progress relative to people unconcerned with Friendliness. Together with the apparent difficulty of solving the FAI-specific problem over and above the AGI problems, and the outside-view conclusion that there's no way to become confident in a bunch of new philosophical ideas in a short amount of time, it suggests that the positive impact of working on FAI now is low relative to the harm (i.e., shortening of AI timelines and hence reducing time to effect other interventions) caused by it.

it looks really useful to collect a bunch of young people with top cognitive ability and turn their attention to concrete research problems in FAI theory, x-risk strategy, effective altruism

I would agree with the second two categories here, but all your workshops are focused on the first one. I think people initially drawn by FAI theory can subsequently become interested in strategy and altruism, but they can also become interested in AGI capability work. For example some may become intellectually (i.e., intrinsically as opposed to instrumentally) interested in an FAI problem and continue to work on it even when it subsequently appears to be helpful for AGI in general, or find that the FAI problem they're interested in depends on solving some AGI capability problem first, and switch over to doing that.

Given the obvious lack of sufficient number of highly talented people working on strategy today, and the lack of any apparent downsides to making people more interested in it, I don't know why you (i.e., MIRI) are not trying to draw people into that work more directly, for example by making more LW posts on important strategic problems, or by hosting workshops on it. Or when other people (like me) make posts about strategy, it seems like you could at least join in the discussions more enthusiastically. It's really hard for me to understand MIRI's current approach of making a big push to popularize FAI work (which I think has net negative impact, or at least highly uncertain impact) while making almost no efforts to make people more interested in strategy. Are you doing something behind the scenes that I don't know about, for example talking with workshop participants privately to try to turn their attentions to strategy?

It also looks pretty useful to directly attack the problem of FAI, because I expect strategic information from the exercise, because philosophy is best done from within a science, etc.

What strategic information have you obtained from the FAI work (e.g., workshops held) so far? What further strategic information are you hoping for at this point and what strategic problems are you hoping to solve with that information? I don't understand the relevance of the linked post to the issue at hand. Can you explain?

Are they relevantly different? Actually, now that I think about it it seems that 'lack of rationality' should be a subset of 'trying hard and failing'

Sounds irrelevant.

How'd this go? Just searched LW for "neurofeedback" since I recently learned about it

I am very interested in this.