How does MIRI Know it Has a Medium Probability of Success?

And blogging half-finished idea on the internet is exactly what one shouldn't do if one is worried about committing controversial ideas to print.

In case this is why you don't tend to talk about your ideas in public either, except in terse (and sometimes cryptic) comments or in fully polished papers, I wanted to note that I've never had a cause to regret blogging (or posting to mailing lists) any of my half-finished ideas. As long as your signal to noise ratio is fairly high, people will remember the stuff you get right and forget the stuff you get wrong. The problem I see with committing ideas to print (as in physical books) is that books don't come with comments attached pointing out all the parts that are wrong or questionable.

Such systems, hemmed in and restrained, could certainly work on better AI designs, and predict human philosophical judgments. Predicting human philosophical judgments accurately and reporting those predictions is close enough.

If such a system is powerful enough to predict human philosophical judgments using its general intelligence, without specifically having been programmed with a correct solution for metaphilosophy, it seems very likely that it would already be strongly superintelligent in many other fields, and hence highly dangerous.

(Since you seem to state this confidently but don't give much detail, I wonder if you've discussed the idea elsewhere at greater length. For example I'm assuming that you'd ask the AI to answer questions like "What would Eliezer conclude about second-order logic after thinking undisturbed about it for 100 years?" but maybe you have something else in mind?)

He circulates them to reviewers, in wider circles as the book becomes more developed. And blogging half-finished idea on the internet is exactly what one shouldn't do if one is worried about committing controversial ideas to print.

I guess I actually meant "potentially wrong" rather than "controversial", and I was suggesting that he blog about them after privately circulating to reviewers, but before publishing in print.

"Antiantiheroic epistemology might be a better term, i.e., I think that a merely accurate epistemology doesn't have a built-in mechanism which prevents people from thinking they can do things because the outside view says it's nonvirtuous to try to distinguish yourself within reference class blah. "

Taken literally, I can't possibly disagree with this, but it doesn't seem to answer my question, which is "where is the positive evidence that one is not supposed to ignore." I favor combining many different kinds of evidence, including sparse data. And that can and does lead to very high expectations for particular individuals.

For example, several of my college fraternity brothers are now billionaires. Before facebook Mark Zuckerberg was clearly the person with the highest entrepreneurial potential that I knew, based on his intelligence, motivation, ambition, and past achievements in programming, business, and academics. People described him to me as resembling a young Bill Gates. His estimated expected future wealth based on that data if pursuing entrepreneurship, and informed by the data about the relationship of all of the characteristics I could track with it, was in the 9-figure range. Then add in that facebook was a very promising startup (I did some market sizing estimates for it, and people who looked at it and its early results were reliably impressed).

Moving from entrepreneurship to politics, one can predict success to a remarkable degree with evidence like "Eton graduate, Oxford PPE graduate with first class degree, Oxford Union leader, interested in politics, starts in an entry-level political adviser job with a party." See this post or this paper. Most of the distance in log odds to reliably becoming Prime Minister, let alone Member of Cabinet or Parliament, can be crossed with objective indicators. Throw in a bit more data about early progress, media mentions, and the like and the prediction improves still more.

I would then throw in other evidence, like the impressiveness of the person's public speaking relative to other similar people, their number and influence of friends and contacts in high places relative to other similar types (indicating both social capital, and skill at getting more), which could improve or worsen the picture. There is still a sizable chunk of randomness in log terms, as political careers are buffeted by switches in party control, the economy, the rise and fall of factions that carry members with them, and other hard-to-control factors at many stages. So I can and do come to expect that someone will probably get federal political office, and have a good shot at Cabinet, and less so for PM. But within the real distribution of characteristics I won't be convinced that a young person will probably become PM, which would require almost zero noise.

In science I can be convinced a young star is a good prospect for Nobel or Fields medal caliber work. But I would need stronger evidence than we have seen for anyone to expect that they would do this 10 times (since no one has done so). I am sympathetic to Wei Dai's comment

Carl referenced "Staring Into the Singularity" as an early indicator of your extraordinary verbal abilities (which explains much if not all of your subsequent successes). It suggests that's how you initially attracted his attention. The same is certainly true for me. I distinctly recall saying to myself "I should definitely keep track of this guy" when I read that, back in the extropian days. Is that enough for you to count as "people who you met because of their previous success"?

To state my overall position on the topic being discussed, I think according to "non-heroic epistemology", after someone achieves an "impossible success", you update towards them being able to achieve further successes of roughly the same difficulty and in related fields that use similar skills, but the posterior probabilities of them solving much more difficult problems or in fields that use very different skills remain low (higher relative to the prior, but still low in an absolute sense). Given my understanding of the distribution of cognitive abilities in humans, I don't see why I would ever "give up" this epistemology, unless you achieved a level of success that made me suspect that you're an alien avatar or something.

I would be quite surprised to see you reliably making personal mathematical contributions at the level of the best top math and AI people. I would not be surprised to see MIRI workshop participants making progress on the problems at a level consistent with the prior evidence of their ability, and somewhat higher per unit time because workshops harvest ideas generated over a longer period, are solely dedicated to research, have a lot of collaboration and cross-fertilization, and may benefit from improved motivation and some nice hacking of associated productivity variables. And I would not be surprised at a somewhat higher than typical rate of interesting (to me, etc) results because of looking at strange problems.

I would be surprised if the strange problems systematically deliver relatively huge gains on actual AI problems (and this research line is supposed to deliver AGI as a subset of FAI before others get AGI so it must have great utility in AGI design), i.e. if the strange problems are super-promising by the criteria that Pearl or Hinton or Ng or Norvig are using but neglected by blunder. I would be surprised if the distance to AGI is crossable in 20 years.

I don't think it's a useful thought process either, and rationalizing elaborate reasons why I'm allowed to be a hero wouldn't be useful either (Occam's Imaginary Razor: decorating my thought processes with supportive tinsel will just slow down any changes I need to make), which is why I tend to be annoyed by the entire subject and wish people would get back to the object level instead of meta demands for modesty that come with no useful policy suggestions about ways to do anything better. Tell me a better object-level way to save the world and we can talk about my doing that instead.

You are asking other people for their money and time, when they have other opportunities. To do that they need an estimate of the chance of MIRI succeeding, considering things like AI timelines, the speed of takeoff given powerful AI, competence of other institutions, the usefulness of MIRI's research track, the feasibility of all alternative solutions to AI risk/AI control problems, how much MIRI-type research will be duplicated by researchers interested for other reasons over what timescales, and many other factors including the ability to execute given the difficulty of the problems and likelihood of relevance. So they need adequate object-level arguments about those contributing factors, or some extraordinary evidence to trust your estimates of all of them over the estimates of others without a clear object-level case. Some of the other opportunities available to them that they need to compare against MIRI:

• Build up general altruistic capacities through things like the effective altruist movement or GiveWell's investigation of catastrophic risks, (which can address AI in many ways, including ones now not visible, and benefit from much greater resources as well as greater understanding from being closer to AI); noting that these seem to scale faster and spill over
• Invest money in an investment fund for the future which can invest more (in total and as a share of effort) when there are better opportunities, either by the discovery of new options, or the formation of better organizations or people (which can receive seed funding from such a trust)
• Enhance decision-making and forecasting capabilities with things like the IARPA forecasting tournaments, science courts, etc, to improve reactions to developments including AI and others (recalling that most of the value of MIRI in your model comes from major institutions being collectively foolish or ignorant regarding AI going forward)
• Prediction markets, meta-research, and other institutional changes
• Work like Bostrom's, seeking out crucial considerations and laying out analysis of issues such as AI risk for the world to engage with and to let key actors see the best arguments and reasons bearing on the problems
• Pursue cognitive enhancement technologies or education methods (you give CFAR in this domain) to improve societal reaction to such problems
• Find the most effective options for synthetic biology threats (GiveWell will be looking through them) and see if that is a more promising angle

I'm a reactionary, not an innovator, dammit! Reacting against this newfangled antiheroic 'reference class' claim that says we ought to let the world burn because we don't have enough of a hero license!

"Reference class" to me is just an intuitive way of thinking about updating on certain types of evidence. It seems like you're saying that in some cases we ought to use the inside view, or weigh object-level evidence more heavily, but 1) I don't understand why you are not worried about "inside view" reasoning typically producing overconfidence or why you don't think it's likely to produce overconfidence in this case, and 2) according to my inside view, the probability of a team like the kind you're envisioning solving FAI is low, and a typical MIRI donor or potential donor can't be said to have much of an inside view on this matter, and has to use "reference class" reasoning. So what is your argument here?

I'm also really unconvinced by the claim that this work could reasonably have expected net negative consequences.

Every AGI researcher is unconvinced by that, about their own work.

but trying to warn people against reverse or negative effects seems pretty perverse for anything that has made it onto Givewell's Top 3, or CFAR, or FHI, or MIRI

CFAR and MIRI were created by you, to help you build FAI. If FHI has endorsed your plan for building FAI (as opposed to endorsing MIRI as an organization that's a force for good overall, which I'd agree with and I've actually provided various forms of support to MIRI because of that), I'm not aware of it. I also think I've thought enough about this topic to give some weight to my own judgments, so even if FHI does endorse your plan, I'd want to see their reasoning (which I definitely have not seen) and not just take their word. I note that Givewell does publish its analyses and are not asking people to just trust it.

Info that shortens AI timelines should mostly just not be released publicly

My model of FAI development says that you have to get most of the way to being able to build an AGI just to be able to start working on many Friendliness-specific problems, and solving those problems would take a long time relative to finishing rest of the AGI capability work. Unless you're flying completely below the radar, which is incompatible with your plan for funding via public donations, what is stopping your unpublished results from being stolen or leaked in the mean time? And just gathering 10 to 50 world-class talents to work on FAI is likely to spur competition and speed up AGI progress. The fact that you seem to be overconfident about your chance of success also suggests that you are likely to be overconfident in other areas, and indicates a high risk of accidental UFAI creation (relative to the probability of success, not necessarily high in absolute terms).

It's not clear to me why people who accurately model the world should outperform those who follow less cognitively demanding heuristics. I've seen this position stated as a truism during a debate, but have never read an argument for or against it. Would someone be able to link to an argument about following non-robust shortcuts to rationality, or write a short case against that practice?

Overconfidence (including factual error about success rates) is pervasive in entrepreneurs, both the failures and successes (and the successes often fail on a second try, although they have somewhat better odds). The motivating power of overconfidence doesn't mean the overconfidence is factually correct or that anyone else should believe it. And the mega-successes tended to look good in expected value, value of information, and the availability of good intermediate outcomes short of mega-success: there were non-crazy overconfident reasons to pursue them. The retreat to "heroic epistemology" rather than reasons is a bad sign relative to those successes, and in any case most of those invoking heroic epistemology style reasoning don't achieve heroic feats.

Applying the outside view startup statistics, including data on entrepreneur characteristics like experience and success rates of repeat entrepreneurs is not magic or prohibitively difficult. Add in the judgments of top VCs to your model.

For individuals or area/firm experts, one can add in hard-to-honestly-signal data (watching out for overconfidence in various ways, using coworkers, etc). That model would have assigned a medium chance to pretty nice success for Apple, and maybe 1-in-100 to 1-in-1000 odds of enormous success, with reasonable expected value for young people willing to take risks and enthused about the field. And the huge Apple success came much later, after Jobs had left and returned and Wozniak was long gone.

Google started with smart people with characteristics predictive of startup success, who went in heavily only after they had an algorithm with high commercial value (which looked impressive to VCs and others). Their success could have been much smaller if their competitors had been more nimble.

And of course you picked them out after the fact, just like you pick out instances of scientists making false predictions rather than true ones in the history of technology. You need to reconcile your story with the experience of the top VCs and serial entrepreneurs, and the degree of selection we see in the characteristics of people at different levels of success (which indicate a major multiplicative role for luck, causing a big chunk of the variation on a log scale).

but the same rules would also say that I couldn't do the 3-5 other lesser "impossibilities" I've done so far

You have some good feats, and failures too (which give us some outside view info to limit the probabilities for outsiders' evaluation of your heroic epistemology). But the overall mix is not an outlier of success relative to, e.g. the reference class of other top talent search students with a skew towards verbal ability, unless you treat your choice of problem as such.