[LINK] Author's Note 119: Shameless Begging

Well, it also seems to me that external editors exist for this reason, among others, and that he can write all the landmine stomping he wants, because publishing it is the PR mistake.

The bit on legalized rape is an important way of conveying that the future will seem weird and surprising and immoral to us, just like 2015 would seem weird and surprising and immoral to someone from a few centuries ago. I want my science-fiction to show how weird things are likely to be (even if the specific kind of weirdness is of course likely to be very wrong), I don't want it to be a bowdlerized soap opera with robots and lasers in the background.

And if people can't understand that and read any kind of far-off weirdness through the lens of this decade's petty tribal politics, then basically, fuck 'em. I don't want Eliezer or anybody else to bend backwards to avoid being misread by idiots.

And sure, it's bad PR, but it's a bit of a self-fulfilling policy, a bit like how "openly criticizing the Government" is a bad career move for a Chinese citizen.

Not to mention, the leading cause of propeller-induced face laceration syndrome...

Air-conditioned greenhouses.

Not quite greenhouses. It seems like Eliezer is saying it would be a glass canopy without enclosing walls (so you would still get natural fresh air flow.

Well, Eliezer's about... what, 35? Not far off that, anyway. I'm sure I wrote some stuff that was at least that embarrassing when I was 20, though it wouldn't have been under my own name or wouldn't have had any public exposure to speak of or both.

I just want to note we're not discussing laser-mounted mosquito-terminating drones anymore. That's fine. Anyway, I'm a bit older than Eliezer was when he founded the Singularity Institute for Artificial Intelligence. While starting a non-profit organization at that age seems impressive to me, once it's been legally incorporated I'd guess one can slap just about any name they like on it. The SIAI doesn't seem to have achieved much in the first few years of its operation.

Based on the their history from the Wikipedia page on the Machine Intelligence Research Institute, it seems to me how notable the organization's achievements are are commensurate with the time it's been around. For several years, as the Singularity Institute, they also ran the Singularity Summit, which they eventually sold as a property to Singularity University for one million dollars. Eliezer Yudkowsky contributed two chapters to Global Catastrophic Risks in 2008, at the age of 28, without having completed either secondary school or a university education.

On the other hand, the MIRI has made great mistakes in operations, research, and outreach in their history. Eliezer Yudkwosky is obviously an impressive person for various reasons. I think the conclusion is Eliezer sometimes assumes he's enough of a 'rationalist' he can get away with being lazy with how he plans or portrays his ideas. He seems like he's not much of a communications consequentialist, and seems relcuctant to declare mea culpa when he makes those sorts of mistakes. All things equal, especially if we hasn't tallied Eliezer's track record, we should remain skeptical of his plans based on shoddy grounds. I too don't believe we should take his bonus requests and ideas at the end of the post seriously.

There were specs for a programming language that would by its design 'do what i mean' that make my programmer friends laugh and complicated AI architectures and ideas for the social engineering they would do with the gigadollars that would be rolling in to bring about the singularity by 2010 so as to avoid the apocalyptic Nanowar that was coming.

Flare (the language) didn't sound that dumb to me - my impression wasn't that it would inherently 'do what i mean' but that it would somehow be both machine and human - readable, so that it would be easy to run advanced optimising compliers over it, and later would provide a natural basis for AI that could rewrite its own source code.

Looking back on it, this is way too much of a free lunch, and since an AI capable of understanding AI theory would probably also be able to parse the meaning of code written in conventional languages, its rather redundant. I still expect that 'do what i mean' languages will appear, for instance the language could detect 'obvious' mistakes, correct them and inform the user.

e.g. "x * y=z does not work because the dimensions do not match. Nor does x' * y=z, but x * y'=z does, so I have taken the liberty of changing your code to x * y'=z"

or "'inutaliseation' is not a function or variable. I assume you meant 'initialization', which is a function, and I corrected this mistake"

Eventually, it might evolve into a natural language to code translator.

But yes, a nanowar by 2010 wasn't the smartest idea.

Information on the history of the MIRI from 2002 through 2006 is sparse, as gleaned from the Wikipedia page on the organization. As the SIAI in 2006, they successfully raised $200,000 as part of a donation campaign, with $100,000 matched as a donation by Peter Thiel. In the years since, the MIRI seems to have at least once annually held fundraisers that turn out just as successful. "The Sequences" were scarcely started in 2006, so I don't know if Peter Thiel got wind of Eliezer's ideas and organization on SL4, or Overcoming Bias, or what. Anyway, while Vinge, and earlier, I. J. Good, warned against the dangers of machine superintelligence, Eliezer founded a research organization aimed at solving this problem, formulated the mission for doing so, and popularized this through his meetings. I'm using metrics such as the raised profile of risks from machine intelligence, and the amount of vocal support and donations the MIRI receives as a proxy for how they and Eliezer specifically have raised the profile of this field of inquiry and concern. I assume others would not have done so much for the MIRI if they didn't believe in its mission. Most of the recent coverage should probably be attributed to Nick Bostrom ans his recent book, though.

At the 2014 Effective Altruism Summit, Eliezer reported there are only four full-time FAI researchers in the world. That is himself, Nate Soares, and Benja Fallenstein of the MIRI, and Stuart Armstrong of the FHI. I was incredulous, and guessed Eliezer's definition of 'FAI researcher' was more stringent than most sensible people would use. I asked Luke Muehlhauser for clarification. He remarked beyond those four Paul Christiano might count as 'half a FAI researcher' because he spends a portion of his time as a mathematician as UCB working on mathematics in line with the MIRI's research agenda. The MIRI has since hired Patrick LaVictoire, and perhaps others.

The point is, the MIRI itself thinks there's less than a dozen FAI researchers. For all we know, all FAI researchers might be users of LessWrong, and HPMoR fans. I could ask all of the known "FAI researchers" if they were first introduced to these research ideas, through LessWrong, through HPMoR. That indeed might be a "substantial portion". You or I might qualify "FAI researcher" differently, but Eliezer by his own admission believes writing more HPMoR is one of the surprisingly best way to draw more attention from Math Olympiad contestants to their research, as does the MIRI.

Vernor Vinge warned about AI causing the end of the human race back in 1993.

I.J. Good also warned about it in the 1960s.

I guess those are pretty vague words. It's a (set of) research projects followed by thousands, if not tens of thousands, of people. Among these people are philanthropists and entrepreneurs who have donated millions of dollars to the cause, and seem to be on track to donate even more money. It's received attention and support from major scientist, and some world-famous people, including Stephen Hawking, Elon Musk, and very recently Bill Gates. He's been published alongside the academics from the Future of Humanity Institute, and his work has merited the respect of prominent thinkers in fields related to artificial intelligence. When his work has attracted derision, it has also been because his ideas attract enough attention for other prominent academics and thinkers to see fit to criticize him. If we evaluate the success of a movement on the basis of memetics alone, this last observation might also count.

The idea of dangers from superintelligence was debated in Aeon Magazine last year. Much of the effort and work to raise the profile and increase focus upon the issue has been done by Nick Bostrom and the Future of Humanity Institute, the Future of Life Institute, and evne the rest of the Machine Intelligence Research Institute aside from Eliezer himself. Still, though, he initiated several theses on solving the problem, and communicated them to the public.