Existential Risk and Public Relations

I agree completely. I still read LessWrong because I am a relatively long-time reader, and thus I know that most of the people here are sane. Otherwise, I would conclude that there is some cranky process going on here. Still, the Roko affair caused me to significantly lower my probabilities assigned to SIAI success and forced me to seriously consider the hypothesis that Eliezer Yudkowsky went crazy.

By the way, I have a little bit disturbing feeling that too little of the newer material here is actually devoted to refining the art of human rationality, as the blog's header proudly states, while instead the posts often discuss relatively narrow list of topics which are only tangentially related to rationality. E.g. cryonics, AI stuff, evolutionary psychology, Newcomb-like scenarios.

Agreed.

One good sign here is that LW, unlike most other non-mainstream organizations, doesn't really function like a cult. Once one person starts being critical, critics start coming out of the woodwork. I have my doubts about this place sometimes too, but it has a high density of knowledgeable and open-minded people, and I think it has a better chance than anyone of actually acknowledging and benefiting from criticism.

I've tended to overlook the weirder stuff around here, like the Roko feud -- it got filed under "That's confusing and doesn't make sense" rather than "That's an outrage." But maybe it would be more constructive to change that attitude.

What are the scenarios where someone unfamiliar with this website would hear about Roko's deleted post?

I suppose it could be written about dramatically (because it was dramatic!) but I don't think anyone is going to publish such an account. It was bad from the perspective of most LWers -- a heuristic against censorship is a good heuristic.

This whole thing is ultimately a meta discussion about moderation policy. Why should this discussion about banned topics be that much interesting than a post on Hacker News that is marked as dead? Hacker News generally doesn't allow discussion of why stories were marked dead. The moderators are anonymous and have unquestioned authority.

If Less Wrong had a mark as dead function (on HN unregistered users don't see dead stories, but registered users can opt-in to see them), I suspect Eliezer would have killed Roko's post instead of deleting it to avoid the concerns of censorship, but no one has written that LW feature yet.

As a solid example of what a not-PR disaster it was, I doubt that anyone at the Singularity Summit that isn't a regular Less Wrong reader (the majority of attendees) has heard that Eliezer deleted a post. It's just not the kind of thing that actually makes a PR disaster... honestly if this was a PR issue it might be a net positive because it would lead some people to hear of LW that otherwise would never have heard of Less Wrong. Please don't take that as a reason to make this a PR issue.

Eliezer succeeded in the sense that it is very unlikely that people in the future on Less Wrong are going to make stupid emotionally abhorrent posts about weird decision theory torture scenarios. He failed in that he could have handled the situation better.

If anyone would like to continue talking about Less Wrong moderation policy, the place to talk about it is the Meta Thread (though you'd probably want to make a new one (good for +[20,50] karma!) instead of discussing it in an out of season thread)

Looking at my own posts I see a lot of this problem; that is, the problem of addressing only far too small an audience. Thank you for pointing it out.

When I first started watching Blogging Heads discussions featuring Eliezer I would often have moments where I held my breath thinking "Oh god, he can't address that directly without sounding nuts, here comes the abhorrent back-peddling and waffling". Instead he met it head on with complete honesty

I am so glad that someone notices and appreciates this.

I feel that anyone advocating for public hypocrisy among the SIAI staff is working to disintegrate the organization (even if unintentionally).

Agreed.

On the other hand... people say they hate politicians and then vote for them anyway.

So hypocrisy does have upsides, and maybe we shouldn't dismiss it so easily.

The above is a good comment, but 26 karma? How did it deserve that?

I STRONGLY suspect that there is a enormous gulf between finding out things on your own and being directed to them by a peer.

When you find something on your own (existential risk, cryonics, whatever), you get to bask in your own fortuitousness, and congratulate yourself on being smart enough to understand it's value. You get a boost in (perceived) status, because not only do you know more than you did before, you know things other people don't know.

But when someone else has to direct you to it, it's much less positive. When you tell someone about existential risk or cryonics or whatever, the subtext is "look, you're weren't able to figure this out by yourself, let me help you". No matter how nicely you phrase it, there's going to be resistance because it comes with a drop in status - which they can avoid by not accepting whatever you're selling. It actually might be WORSE with smart people who believe that they have most things "figured out".

Thanks for your thoughtful comment.

To make this story short, the whole reason I became interested in this topic in the first place was because I was impressed by EY -- specifically his writings on rationality on OB (now known as the Sequences here on LW). Now of course most of those ideas were hardly original with him (indeed many times I had the feeling he was stating the obvious, albeit in a refreshing, enjoyable way) but the fact that he was able to write them down in such a clear, systematic, and readable fashion showed that he understood them thoroughly. This was clearly somebody who knew how to think.

I know some people who have had this sort of experience. My claim is not that Eliezer has uniformly repelled people from thinking about existential risk. My claim is that on average Eliezer's outlandish claims repel people from thinking about existential risk.

Do I simply have an unusual personality that makes me willing to listen to strange-sounding claims?

My guess would be that this is it. I'm the same way.

(But why wouldn't they as well, if they're "smart"?)

It's not clear that willingness to listen to strange-sounding claims exhibits correlation with instrumental rationality, or what the sign of that correlation is. People who are willing to listen to strange-sounding claims statistically end up hanging out with UFO conspiracy theorists, New Age people, etc. more often than usual. Statistically, people who make strange-sounding claims are not worth listening to. Too much willingness to listen to strange-sounding claims can easily result in one wasting large portions of one's life.

Why can't they just read the darn sequences and pick up on the fact that these people are worth listening to?

See my remarks above.

Somewhere you said that you are really happy to be finally able to concentrate directly on the matters you deem important and don't have to raise money anymore. This obviously worked, so you won't have to change anything. But if you ever need to raise more money for a certain project, my question is how much of the money you already get comes from people you would consider mediocre rationalists?

I'm not sure if you expect to ever need a lot of money for a SIAI project, but if you solely rely on those few really good rationalists then you might have a hard time in that case.

People like me will probably always stay on your side, whether you tell them they are idiots. But I'm not sure if that might be enough in a scenario where donations are important.

Agree with the points of both of ChistianKI and XiXiDu.

As for really good rationalists, I have the impression that even when it comes to them you inadvertently alienate them with higher than usual frequency on account of saying things that sound quite strange.

I think (but am not sure) that you would benefit from spending more time understanding what goes on in neurotypical people's minds. This would carry not only social benefits (which you may no longer need very much at this point) but also epistemological benefits.

However, I do still care about things like propagation velocities because that affects what population size the top 5% is 5% of, for example.

I'm encouraged by this remark.

If we think existential risk reduction is important than we should care about whether politicians think that existential risk reduction is a good idea. I don't think that a substantial number of US congressman are what you consider to be good rationalists.

We live in a democracy! How can you not be concerned with 95% of the population? They rule you.

If we lived in some sort of meritocratic aristocracy, perhaps then we could focus our efforts on only the smartest 5%.

As it is, it's the 95% who decide what happens in our elections, and its our elections who decides what rules get made, what projects get funded. The President of the United States could unleash nuclear war at any time. He's not likely to---but he could. And if he did push that button, it's over, for all of us. So we need to be very concerned about who is in charge of that button, and that means we need to be very concerned about the people who elect him.

Right now, 46% of them think the Earth is 6000 years old. This worldview comes with a lot of other anti-rationalist baggage like faith and the Rapture. And it runs our country. Is it just me, or does this seem like a serious problem, one that we should probably be working to fix?

Awareness on the part of government funding agencies (and the legislators and executive branch people with influence over them), technology companies and investors, and political and military decisionmakers (eventually) could all matter quite a lot. Not to mention bright young people deciding on their careers and research foci.

Who needs to put thought into the possibility of misbuilding an AI other than people who will themselves engage in AI research? (This is not a rhetorical question - again, I'm new to this.)

The people who do the real work. Utlimately it doesn't matter if the people who do the AI research care about existential risk or not (if we make some rather absolute economic assumptions). But you've noticed this already and you are right about the 'further alarm bells'.

Ultimately, the awareness of the layperson matters for the same reason that the awareness of the layperson matters for any other political issue. While with AI people can't get their idealistic warm fuzzies out of barely relevant things like 'turning off a light bulb' things like 'how they vote' do matter. Even if it is at a lower level of 'voting' along the lines of 'which institutions do you consider more prestigious'?

You can ramp up the variance of people's opinions and come out better financially.

Good point!

Don't you realize the default scenario?

The default scenario is some startup or big company or mix therein develops strong AGI for commercialization, attempts to 'control it', fails, and inadvertently unleashes a god upon the earth. To first approximation the type of AGI we are discussing here could just be called a god. Nanotechnology is based on science, but it will seem like magic.

The question then is what kind of god do we want to unleash.

First, EY is concerned about risks due to technologies that have not yet been developed; as far as I know, there is no reliable way to make predictions about the likelihood of the development of new technologies.

As was mentioned in other threads, SIAI's main arguments rely on disjunctions and antipredictions more than conjunctions and predictions. That is, if several technology scenarios lead to the same broad outcome, that's a much stronger claim than one very detailed scenario.

For instance, the claim that AI presents a special category of existential risk is supported by such a disjunction. There are several technologies today which we know would be very dangerous with the right clever 'recipe'– we can make simple molecular nanotech machines, we can engineer custom viruses, we can hack into some very sensitive or essential computer systems, etc. What these all imply is that a much smarter agent with a lot of computing power is a severe existential threat if it chooses to be.

There needs to be an article on this point. In the absence of a really good way of deciding what technologies are likely to be developed, you are still making a decision. You haven't signed up yet; whether you like it or not, that is a decision. And it's a decision that only makes sense if you think technology X is unlikely to be developed, so I'd like to see your prediction mechanism and whether it's worked in the past. In the absence of really good information, we sometimes have to decide on the information we have.

EDIT: I was thinking about cryonics when I wrote this, though the argument generalizes.

First, EY is concerned about risks due to technologies that have not yet been developed; as far as I know, there is no reliable way to make predictions about the likelihood of the development of new technologies. (This is also the basis of my skepticism about cryonics.) If you're going to say "Technology X is likely to be developed" then I'd like to see your prediction mechanism and whether it's worked in the past.

Let's keep in mind that your estimated probabilities of various technological advancements occurring and your level of confidence in those estimates are completely distinct... In particular, here you seem to express low estimated probabilities of various advancements occurring, and you justify this by saying "we really have no idea". This seems like a complete non sequitur. Maybe you have a correct argument in your mind, but you're not giving us all the pieces.

Second, shouldn't an organization worried about the dangers of AI be very closely in touch with AI researchers in computer science departments? Sure, there's room for pure philosophy and mathematics, but you'd need some grounding in actual AI to understand what future AIs are likely to do.

I'm not sure what you refer to by "actual AI." There is a sub-field of academic computer science which calls itself "Artificial Intelligence," but it's not clear that this is anything more than a label, or that this field does anything more than use clever machine learning techniques to make computer programs accomplish things that once seemed to require intelligence (like playing chess, driving a car, etc.)

I'm not sure why it is a requirement that an organization concerned with the behavior of hypothetical future engineered minds would need to be in contact with these researchers.

Prediction is hard, especially about the future.

One thing that intrigues me is snags. Did anyone predict how hard to would be to improve batteries, especially batteries big enough for cars?

I think multifoliaterose is right that there's a PR problem, but it's not just a PR problem. It seems, unfortunately, to be a problem with having enough justification for claims, and a problem with connecting to the world of professional science. I think the PR problems arise from being too disconnected from the demands placed on other scientific or science policy organizations. People who study other risks, say epidemic disease, have to get peer-reviewed, they have to get government funding -- their ideas need to pass a round of rigorous criticism. Their PR is better by necessity.

I agree completely. The reason why I framed my top level post in the way that I did was so that it would be relevant to readers of a variety of levels of confidence in SIAI's claims.

As I indicate here, I personally wouldn't be interested in funding SIAI as presently constituted even if there was no PR problem.

First, EY is concerned about risks due to technologies that have not yet been developed; as far as I know, there is no reliable way to make predictions about the likelihood of the development of new technologies. (This is also the basis of my skepticism about cryonics.) If you're going to say "Technology X is likely to be developed" then I'd like to see your prediction mechanism and whether it's worked in the past.

I think there are ways to make these predictions. On the most layman level I would point out that IBM build a robot that beats people at Jeopardy. Yes, I am aware that this is a complete machine-learning hack (this is what I could gather from the NYT coverage) and is not true cognition, but it surprised even me (I do know something about ML). I think this is useful to defeat the intuition of "machines cannot do that". If you are truly interested I think you can (I know you're capable) read Norvig's AI book, and than follow up on the parts of it that most resemble human cognition; I think serious progress is made in those areas. BTW, Norvig does take FAI issues seriously, including a reference to EY paper in the book.

Second, shouldn't an organization worried about the dangers of AI be very closely in touch with AI researchers in computer science departments? Sure, there's room for pure philosophy and mathematics, but you'd need some grounding in actual AI to understand what future AIs are likely to do.

I think they should, I have no idea if this is being done; but if I would do it I would not do it publicly, as it may have very counterproductive consequences. So until you or I become SIAI fellows we will not know, and I cannot hold such lack of knowledge against them.

shouldn't an organization worried about the dangers of AI be very closely in touch with AI researchers in computer science departments? Sure, there's room for pure philosophy and mathematics, but you'd need some grounding in actual AI to understand what future AIs are likely to do.

Yes. It's hardly urgent, since AI researchers are nowhere near a runaway intelligence. But on the other hand, control of AI is going to be crucial+difficult eventually, and it would be good for researchers to be aware of it, if they aren't.

Funding SIAI is optimal only if you think that the pursuit of Friendly AI is by far the most important component of existential risk reduction

This seems to assume that existential risk reduction is the only thing people care about. I doubt I am the only person who wants more from the universe than eliminating risk of humans going extinct. I would trade increased chance of extinction for a commensurate change in the probable outcomes if we survive. Frankly I would consider it insane not to be willing make such a trade.

if you think that the most important thing we can be doing right now is publicizing an academically respectable account of existential risk, then you should be funding the Future of Humanity Institute. Funding SIAI is optimal only if you think that the pursuit of Friendly AI is by far the most important component of existential risk reduction

Agreed. (Modulo a caveat about marginal ROI eventually balancing if FHI got large enough or SIAI got small enough.)

Funding SIAI is optimal only if you think that the pursuit of Friendly AI is by far the most important component of existential risk reduction

But who does the evaluation? It seems that it's better to let specialists think about whether a given cause is important, and they need funding just to get that running. This argues for ensuring minimum funding of organizations that research important uncertainties, even the ones where your intuitive judgment says probably lead nowhere. Just as most people shouldn't themselves research FAI, and instead fund its research, similarly most people shouldn't research feasibility of research of FAI, and instead fund the research of that feasibility.

I do "think that the pursuit of Friendly AI [and the avoidance of unfriendly AI] is by far the most important component of existential risk reduction". I also think that SIAI is not addressing the most important problem in that regard. I suspect there's a lot of people who would agree, for various reasons.

In my case, the logic is that I think:

1) That corporations, though not truly intelligent, are already superhuman and unFriendly.

2) That coordinated action (that is, strategic politics, in well-chosen solidarity with others with whom I have important differences) has the potential to reduce their power and/or increase their Friendliness

3) That this would, in turn, reduce the risk of them developing a first-mover unFriendly AI ...

3a) ... while also increasing the status of your ideas in a coalition which may be able to develop a Friendly one.

I recognize that points 2 and 3a are partially tribal and/or hope-seeking beliefs of mine, but think 1 and 3 are well-founded rationally.

Anyway, this is only one possible reason for parting ways with the SIAI and the FHI, without in any sense discounting the risks they are made to confront.

You approve of their plan to build a machine to take over the world and impose its own preferences on everyone? You talk about "optimality" - how confident are you that that is really going to help? What reasoning supports such a claim?

I raised a similar point on the IEET existential risk mailing list in a reply to James J. Hughes:

Michael

For the record, I have no problem with probability estimates. I am less and less willing to offer them myself however since we have the collapse of the Soviet Union etc. as evidence of a chaotic and unpredictable nature of history, Ray’s charts not-with-standing.

What I find a continuing source of amazement is that there is a subculture of people half of whom believe that AI will lead to the solving of all mankind’s problems (which me might call Kurzweilian S^) and the other half of which is more or less certain (75% certain) that it will lead to annihilation. Lets call the latter the SIAI S^.

Yet you SIAI S^ invite these proponents of global suicide by AI, K-type S^, to your conferences and give them standing ovations.

And instead of waging desperate politico-military struggle to stop all this suicidal AI research you cheerlead for it, and focus your efforts on risk mitigation on discussions of how a friendly god-like AI could save us from annihilation.

You are a deeply schizophrenic little culture, which for a sociologist like me is just fascinating.

But as someone deeply concerned about these issues I find the irrationality of the S^ approach to a-life and AI threats deeply troubling.

— James J. Hughes (existential.ieet.org mailing list, 2010-07-11)

I replied:

Keep your friends close...maybe they just want to keep the AI crowd as close together as possible. Making enemies wouldn't be a smart idea either, as the 'K-type S^' subgroup would likely retreat from further information disclosure. Making friends with them might be the best idea.

An explanation of the rather calm stance regarding a potential giga-death or living hell event would be to keep a low profile until acquiring more power.

If you are trying to take over the world it is far better to be seen as an idealistic do gooder who writes fanfic than as a political power player.

They took down the "SIAI will not enter any partnership that compromises our values" "commitment" from their web site. Maybe they are more up for partnerships these days.

(Typo: I think you meant to include 'traits' or similar in there.)

Thanks for pointing this out :-)

While Eliezer occasionally takes actions that seem clearly detrimental to his cause I do suggest that Eliezer is at least in principle aware of the dynamics you discuss. His alter ego "Harry Potter" has had similar discussion with his Draco in his fanfiction.

Interesting, I'll check this out at some point.

This is pretty silly.

This can score huge with lazy or unintellectual people if it's allowed to succeed.

What is the likelihood that lazy or unintellectual people would have ever done anything to reduce existential risk regardless of any particular advocate for/against?

It depends on exactly what you mean by "existential risk". Development will likely - IMO - create genetic and phenotypic takeovers in due course - as the bioverse becomes engineered. That will mean no more "wild" humans.

That is something which some people seem to wail and wave their hands about - talking about the end of the human race.

The end of earth-originating civilisation seems highly unlikely to me too - which is not to say that the small chance of it is not significant enough to discuss.

Eliezer's main case for that appears to be on http://lesswrong.com/lw/y3/value_is_fragile/

I think that document is incoherent.

"Ethical Artificial Intelligence" sounds great and makes sense without having to know the background of the technological singularity as "Friendly Artificial Intelligence" does. Every time I try to mention FAI to someone without any background on the topic I always have to take two steps back in the conversation and it becomes quickly confusing. I think I could mention Ethical AI and then continue on with whatever point I was making without any kind of background and it would still make the right connections.

I also expect it would appeal to a demographic likely to support the concept as well. People who worry about ethical food, business, healthcare etc... would be likely to worry about existential risk on many levels.

In fact I think I'll just go ahead and start using Ethical AI from now on. I'm sure people in the FAI community would understand what I'm talking about.

It may be true that many are worried about 'the end of the world', however consider how many of them think that it was predicted by the Mayan calandar to occur on Dec. 21 2012, and how many actively want it to happen because they believe it will herald the coming of God's Kingdom on Earth, Olam Haba, or whatever.

We could rebrand 'existential risk' as 'end time' and gain vast numbers of followers. But I doubt that would actually be desirable.

I do think that Ethical Artificial Intelligence would strike a better chord with most than Friendly, though. 'Friendly' does sound a bit unserious.

Given that Eliezer has a massive non-comparative advantage in making such arguments we shouldn't expect him to spend his time trying to become slightly better at doing so.

Do you mean comparative disadvantage? Otherwise I can't make sense of what you are trying to say. Not that I'd agree with that anyway, Eliezer is very good rhetorically, and I'm suspicious of psychological diagnoses performed over the internet.

I believe that there are contexts in which the right thing to do is to speak what one believes to be true even if doing so damages public relations.

These things need to be decided on a case-by-case basis. There's no royal road to instrumental rationality.

As I say here, in the present context, a very relevant issue in my mind is that Eliezer & co. have not substantiated their most controversial claims with detailed evidence.

It's clichéd to say so, but extraordinary claims require extraordinary evidence. A claim of the type "I'm the most important person alive" is statistically many orders of magnitude more likely to be made by a poser than by somebody for whom the claim is true. Casual observers are rational to believe that Eliezer is a poser. The halo effect problem is irrational, yes, but human irrationality must be acknowledged, it's not the sort of thing that goes away if you pretend that it's not there.

I don't believe that Eliezer's outlandish and unjustified claims contribute to rational discourse. I believe that Eliezer's outlandish and unjustified claims lower the sanity waterline.

To summarize, I believe that in this particular case the costs that you allude to are outweighed by the benefits.

It seems as though the latter strategy could backfire - if the false statements were exposed. Keeping your mouth shut about controversial issues seems safer.

To the extent that people really want what you argue against, perhaps they should pursue an alternate organization than SIAI that promotes only the more palatable subset. I agree with you that somebody should be making all the claims, popular or not, that bear on x-risk.

I believe that there are contexts in which the right thing to do is to speak what one believes to be true even if doing so damages public relations.

These things need to be decided on a case-by-case basis. There's no royal road to instrumental rationality.

As I say here, in the present context, a very relevant issue in my mind is that Eliezer & co. have not substantiated their most controversial claims with detailed evidence.

It's clichéd to say so, but extraordinary claims require extraordinary evidence. A claim of the type "I'm the most important person alive" is statistically many orders of magnitude more likely to be made by a poser than by somebody for whom the claim is true. Casual observers are rational to believe that Eliezer is a poser. The halo effect problem is irrational, yes, but human irrationality must be acknowledged, it's not the sort of thing that goes away if you pretend that it's not there.

I don't believe that Eliezer's outlandish and unjustified claims contribute to rational discourse. I believe that Eliezer's outlandish and unjustified claims lower the sanity waterline. I feel no squeamishness about placing pressure on Eliezer to cease to make such claims in public even if he sincerely believes them.

This is partially because Google gives a ranking boost to things it sees as recent, so it may not stay that well ranked.

I disagree. I think that my post does a good job of highlighting the fact that public aversion to thinking about existential risk reduction is irrational.

I expect AI progress to come incrementally, BTW - I don't expect the Foomination. And I expect it to come from Google or someone similar; a large group of really smart, really hard-working people.

I'd like to point out that it's not either/or: it's possible (likely?) that it will take decades of hard work and incremental progress by lots of really smart people to advance AI science to a point where an AI could FOOM.

And saddened once again at how people seem unable to distinguish "multi claims that something Eliezer said could be construed as claim X" and "Eliezer claimed X!"

Please note that for the next time you're worried about damaging an important cause's PR, multi.

Edit: I misunderstood what Jordan was trying to say - the previous version of this comment is irrelevant to the present discussion and so I've deleted it.

Negative reactions to Yudkowsky from various people (academics concerned with x-risk), just within the past few weeks:

I also have an extreme distaste for Eliezer Yudkowsky, and so I have a hard time forcing myself to cooperate with any organization that he is included in, but that is a personal matter.

You know, maybe I'm not all that interested in any sort of relationship with SIAI after all if this, and Yudkowsky, are the best you have to offer.

...

There are certainly many reasons to doubt the belief system of a cult based around the haphazard musings of a high school dropout, who has never written a single computer program but professes to be an expert on AI. As you point out none of the real AI experts are crying chicken little, and only a handful of AI researchers, cognitive scientists or philosophers take the FAI idea seriously.

...

Wow, that's an incredibly arrogant put-down by Eliezer..SIAI won't win many friends if he puts things like that...

...

...he seems to have lost his mind and written out of strong feelings. I disagree with him on most of these matters.

...

Questions of priority - and the relative intensity of suffering between members of different species - need to be distinguished from the question of whether other sentient beings have moral status at all. I guess that was what shocked me about Eliezer's bald assertion that frogs have no moral status. After all, humans may be less sentient than frogs compared to our posthuman successors. So it's unsettling to think that posthumans might give simple-minded humans the same level of moral consideration that Elizeer accords frogs.

I was told that the quotes above state some ad hominem falsehoods regarding Eliezer. I think it is appropriate to edit the message to show that indeed some person might not be have been honest, or clueful. Otherwise I'll unnecessary end up perpetuating possible ad hominem attacks.

But I realize my experience may be atypical and there could be an abundance of avoidable Yudkowsky-hatred where I'm not looking, so would like to know more about that.

Yudkowsky-hatred isn't the risk, Yudkowsky-mild-contempt is. People engage with things they hate, sometimes it brings respect and attention to both parties (by polarizing a crowd that would otherwise be indifferent.) But you never want to be exposed to mild contempt.

I can think of some examples of conversations about Eliezer that would fit the category but it is hard to translate them to text. The important part of the reaction was non-verbal. Cryonics was one topic and the problem there wasn't that it was uncredible but that it was uncool. Another topic is the old "thinks he can know something about Friendly AIs when he hasn't even made an AI yet" theme. Again, I've seen that reaction evident through mannerisms that in no way translate to text. You can convey that people aren't socially relevant without anything so crude as saying stuff.

Can you tell us more about how you've seen people react to Yudkowsky? That these negative reactions are significant is crucial to your proposal, but I have rarely seen negative reactions to Yudkowsky (and never in person) so my first availability-heuristic-naive reaction is to think it isn't a problem. But I realize my experience may be atypical and there could be an abundance of avoidable Yudkowsky-hatred where I'm not looking, so would like to know more about that.

I haven't seen examples of Yudkowsky-hatred. But I have regularly seen people ridicule him. Recalling Hanson's view that a lot human behavior is really signaling and vying for status, I interpret this ridicule as an functioning to lower Eliezer's status to compensate for what people perceive as inappropriate status grubbing on his part.

Most of the smart people who I know (including myself) perceive him as exhibiting a high degree of overconfidence in the validity of his views about the world.

This leads some of them conceptualize him as a laughingstock; as somebody who's totally oblivious and feel that the idea that we should be thinking about artificial intelligence is equally worthy of ridicule. I personally am quite uncomfortable with these attitudes, agreeing with Holden Karnofsky's comment

"I believe that there are enormous risks and upsides associated with artificial intelligence. Managing these deserves serious discussion, and it’s a shame that many laugh off such discussion."

I'm somewhat surprised that you appear not to have notice this sort of thing independently. Maybe we hang out in rather different crowds.

Did that objectionable Yudkowsky-meteorite comment get widely disseminated? YouTube says the video has only 500 views, and I imagine most of those are from Yudkowsky-sympathizing Less Wrong readers.

Yes, I think that you're right. I just picked it out as a very concrete example of a statement that could provoke a substantial negative reaction. There are other qualitatively similar things (but more mild) things that Eliezer has said that have been more widely disseminated.

To top it off, the SIAI is responsible for getting James Randi's seal of approval on the Singularity being probable. That's not poisoning the meme, not one bit.

I feel it's worth pointing out that just because something should be, doesn't mean it is. You state:

Your points seem to be that claims made by Eliezer and upheld by the SIAI don't appear credible due to insufficient argument, and due to one person's personality. You can argue all you want about how he is viewed. You can debate the published papers' worth. But the two shouldn't be equated.

I agree with the sentiment, but how practical is it? Just because it would be incorrect to equate Eliezer and the SIAI doesn't meant that people won't do it. Perhaps it would be reasonable to say that the people who fail to make the distinction are also the people on whom it's not worth expending the effort trying to explicate the situation, but I suspect that it is still the case that the majority of people are going to have a hard time not making that equation if they even try at all.

The purpose of this article, I would presume to say, is that public relations actually does serve a valid and useful purpose. It is not a wasted effort to ensure that the ideas that one considers true, or at least worthwhile, are presented in the sort of light that encourages people to take them seriously. This is something that I think many people of a more intellectual bent often fail to consider; though some of us might actually invest time and effort into determining for ourselves whether an idea is good or not, I would say the majority do not and instead rely on trusted sources to guide them (with often disastrous results).

Again, it may just be that we don't care about those people (and it's certainly tempting to go that way), but there may be times when quantity of supporters, in addition to quality, could be useful.

To top it off, the SIAI is responsible for getting James Randi's seal of approval on the Singularity being probable. That's not poisoning the meme, not one bit.

Understood. Here too, we may have legitimate grounds for difference of opinion rooted in the fact that our impressions have emerged from thousands of data points which we don't have conscious access to.

(Note that both times I qualified the suggestion to contact SIAI with "if you are serious and have a reasonable complaint".)

Do you feel that I've misrepresented you? I'd be happy to qualify my reference to you however you'd like if you deem my doing so suitable.

I suggested what to do about this problem in my post: withhold funding from SIAI, and make it clear to them why you're withholding funding from them, and promise to fund them if the issue is satisfactorily resolved to incentivize them to improve.