Contra Yudkowsky on Epistemic Conduct for Author Criticism

I just wanted to add some context (that I thought of as "obvious background context", but probably not everyone is tracking), that Eliezer wrote more about the "rule" here in the 8th post [? · GW] of the Inadequate Equilibria sequence:

I’ve now given my critique of modesty as a set of explicit doctrines. I’ve tried to give the background theory, which I believe is nothing more than conventional cynical economics, that explains why so many aspects of the world are not optimized to the limits of human intelligence in the manner of financial prices. I have argued that the essence of rationality is to adapt to whatever world you find yourself in, rather than to be “humble” or “arrogant” a priori. I’ve tried to give some preliminary examples of how we really, really don’t live in the Adequate World where constant self-questioning would be appropriate, the way it is appropriate when second-guessing equity prices. I’ve tried to systematize modest epistemology into a semiformal rule, and I’ve argued that the rule yields absurd consequences.

I was careful to say all this first, because there’s a strict order to debate. If you’re going to argue against an idea, it’s bad form to start off by arguing that the idea was generated by a flawed thought process, before you’ve explained why you think the idea itself is wrong. Even if we’re refuting geocentrism, we should first say how we know that the Sun does not orbit the Earth, and only then pontificate about what cognitive biases might have afflicted geocentrists. As a rule, an idea should initially be discussed as though it had descended from the heavens on a USB stick spontaneously generated by an evaporating black hole, before any word is said psychoanalyzing the people who believe it. Otherwise I’d be guilty of poisoning the well, also known as Bulverism.

But I’ve now said quite a few words about modest epistemology as a pure idea. I feel comfortable at this stage saying that I think modest epistemology’s popularity owes something to its emotional appeal, as opposed to being strictly derived from epistemic considerations. In particular: emotions related to social status and self-doubt.

Expressing negative judgments of someone's intellectual output could be an honest report, generated by looking at the output itself and extrapolating a pattern. Epistemically speaking, this is fine. Alternatively, it could be motivated by something more like politics; someone gets offended, or has a conflict of interest, then evaluates things in a biased way. Epistemically speaking, this is not fine.

So, if I were to take a stab at what the true rule of epistemic conduct here is, the primary rule would be that you ought to evaluate the ideas first before evaluating the person, in your own thinking. There are also reasons why the order of evaluations should also be ideas-before-people in the written product; it sets a better example of what thought processes are supposed to look like, it's less likely to mislead people into biased evaluations of the ideas; but this is less fundamental and less absolute than the ordering of the thinking.

But.

Having the order-of-evaluations wrong in a piece of writing is evidence, in a Bayesian sense, of having also had the order-of-evaluations wrong in the thinking that generated it. Based on the totality of omnizoid's post, I think in that case, it was an accurate heuristic. The post is full of overreaches and hyperbolic language. It presents each disagreement as though Eliezer were going against an expert consensus, when in fact each position mentioned is one where he sided with a camp in an extant expert divide.

And...

Over in the legal profession, they have a concept called "appearance of impropriety", which is that, for some types of misconduct, they consider it not only important to avoid the misconduct itself but also to avoid doing things that look too similar to misconduct.

If I translate that into something that could be the true rule, it would be something like: If an epistemic failure mode looks especially likely, both in the sense of a view-from-nowhere risk analysis and in the sense that your audience will think you've fallen into the failure mode, then some things that would normally be epistemically superogatory become mandatory instead.

Eliezer's criticism of Steven J. Gould does not follow the stated rule, of responding to a substantive point before making any general criticism of the author. I lean towards modus tollens over modus pollens, that this makes the criticism of Steven J. Gould worse. But how much worse depends on whether that's a reflection of an inverted generative process, or an artifact of how he wrote it up. I think it was probably the latter.

I think one confusing aspect is the fact that the person being critical about the structure of the post is also the target of the post, therefore it is difficult to assume good intent.

If another well respected user had written a similar comment about why the post should have been written differently, then it would be a much cleaner discussion about writing standards and similar considerations. Actually, a lot of people did, not really about the structure (at least I don't think so), but mostly about the tone of the post.

As for EY, it is difficult not to assume that this criticism isn't completely genuine, and is some way to attack the author. That being said, maybe we should evaluate arguments for what they are, regardless of why they were stated in the first place (or is it being too naive?)

In that regard, your post is very interesting because it addresses both questions: showing that EY hasn't always followed this stated basic standard (i.e. claiming that the criticism is not genuine), and discussing the merit of this rule/good practice (i.e. is it a good basis for criticism)

Anyway, interesting post, thanks for writing it!

I think this post makes a few good points, but I think the norm of "before you claim that someone is overconfident or generally untrustworthy, start by actually showing that any of their object-level points are inaccurate" seems pretty reasonable to me, and seems to me more like what Eliezer was talking about. 

Like, your post here seems to create a strong distinction between "arguing against Eliezer on the issues of FDT" and "arguing that Eliezer is untrustworthy based on his opinion on FDT", but like, I do think that the first step to either should be to actually make object-level arguments (omnizoid's post did that a bit, but as I commented on the post, the ratio of snark to object-level content was really quite bad).

Yeah, it felt like Eliezer was rounding off all of the bad faith in the post to this one stylistic/etiquette breach, but he didn't properly formulate the one rule that was supposedly violated. 

The context matters here. 

The original post by Omnizoid spent the first third ranting about their journey discovering that Yud was a liar and a fraud, very carefully worded to optimize for appeal to ordinary EAforum users, and didn't back up any of their claims until the second two thirds, which were mostly esoteric consciousness arguments and wrong decision theory. What ultimately happened was that 95% of readers only read the slander that amde up the first third, and not any of the difficult-to-read arguments that Omnizoid routinely implied would back them up. People line up to do things like that. 

That was what happened, and the impression I got from Yud's response was that he wasn't really sure whether to engage with it at all, since it might incentivize more people to take a similar strategy in the future. I also was confused by Yud calling it a "basic rule of epistemic conduct", but if that seemed like a good way to mitigate the harm while it was ongoing, then that was his call.

From my perspective, when Yud's reputation is bad, it's basically impossible to convince important and valuable people to read The Sequences and HPMOR and inoculate themselves against a changing world of cognitive attacks [LW · GW], whereas it's a downhill battle to convince them to read those texts when Yud's reputation is good. So if he fumbled on the task of mitigating the damage from Omnizoid's bad faith attacks, then yes, that's too bad. But he was also under time constraints, since it was an ongoing situation where it was getting tons of attention on EAforum and the harm increased with every passing hour, requiring a speedy response; if you want to judge how bad/careless the fumble was, then you have to understand the full context of the situation that was actually taking place.

I'm conflicted.
Upvoted because I strongly agree, downvoted because it would've been fine as a comment and appears to me as too much drama as a post (in its current form). You do seem to  acknowledge that in the end? Perhaps I would've appreciated it more if you focused on this:

The word epistemic is not just a verbal tic you can throw in front of anything to indicate approval of your internet cult. It means something.

as a refresher of sorts, used Yudkowsky comment as an example and kept it shorter.

Indeed, "Beware of Stephen J. Gould" would seem to have essentially the same argument structure as "Eliezer Yudkowsky Is Frequently, Confidently, Egregiously Wrong".

The 2007 post isn't an argument aimed at the ghost of Gould or his followers; it's a bunch of claims about the intellectual history of the field of evolutionary biology, and Gould's stature within it. Lots of commenters showed up to dispute or add color to Eliezer's claims about how other eminent scientists viewed Gould, but both the post and comment section are light on object-level arguments because there simply isn't that much to argue about, nor anyone interested in arguing about it.

(Also, Gould died in 2002. Eliezer's comment on epistemic conduct seems more applicable to discourse between contemporaries.)

Omnizoid's post, by contrast, opened with a bunch of inflammatory remarks about a still-living person, aimed directly at an audience with strong preexisting awareness of and opinions about that person. It then goes on to make a bunch of (wrong, laughably overconfident [EA(p) · GW(p)]) object-level arguments about very obviously non-settled topics in science and philosophy.

I think someone who wants to enforce an alleged "basic rule of epistemic conduct" of the form "if you've got to [say X] [...] [do so] after presenting your [...] points that support [X]" should be able to explain why such a rule systematically produces maps that reflect the territory when X happens to be "negative assessments of an author's character or authority" (what are called "personal attacks"), but not for other values of X (given how commonplace it is to make a thesis statement before laying out all the supporting arguments).

If more people read the beginning of an argument than the end, putting the personal attacks at the beginning will predictably lead to more people seeing the attacks than the arguments that support them. Even if such readers are not consciously convinced by attacks without argument, it seems implausible that their beliefs or impressions will not be moved at all.

My opinion is that Eliezer thought he needed a more technical rebuttal of Omnizoid's post than he did. Omnizoid was wrong, pointlessly mean, and had terrible support for most of the object-level points that he did make. In general the post was just bad and that's probably why it got no play on Lesswrong. That's all the criticism needed. I was expecting a well-written, justifiably rude, and factually-supported takedown of Eliezer. I didn't get it and I was disappointed. The top comment at the EA forum however directed me to that great takedown I was looking for and the takedown was even better for not being rude at all.

It's like trying to dispute a math proof that's written illegibly and is missing the whole middle part of the proof. Eliezer wanted to find a fatal flaw and refute the central point. But he probably couldn't find that fatal flaw. And even if he did find one one, the real issue with the proof is the bit about it being terrible in general. Finding the fatal flaw is kind of a bonus at best and at worst finding something that isn't even there!

As as semi-newbie (read the major arguments, thought about them but not followed the detailed maths) and someone who has been following the fields of AI and then alignment for 15-20 years it isn't just EY that I now feel is clearly incorrect. For example it sure seemed to be from reading the likes of Superintelligence that a major reason for a paperclip optimizer would be that the AI would do what you say but not know what you mean. That seems pretty much impossible now  - GPT4 understands what I mean better than many people, but has no ability to take over the world. More generally I feel that the alignment literature would not have benefitted from more time before GPT4, it would just have got more convinced about incorrect conclusions.

It is also believable that there are several other important further conclusions from the literature that are not correct, and we don't know what they are yet. I used to believe that a fast take-off was inevitable, now after reading Jacob Cannell etc  think it is very unlikely. EY was very good at raising awareness, but that does not mean he should somehow represent the AI safety field because of that.

On a personal note, I distinctly remember EY being negative against deep learning to my surprise at the time (that is before Alpha Go etc) because I felt it was inevitable that deep learning/neuromorphic systems would win for the entire time I studied the field. (Unfortunately I didn't keep the reference to his comment so I don't have proof of that to anyone else).  

I have deployed GOFAI signal processing systems I have written from scratch, studied psychology etc which lead me to the conclusion that deep learning would be the way to go. GOFAI is hopelessly brittle, NN are not.  

I also strongly disagree with EY about the ethics of not valuing the feelings of non-reflective mammals etc.

omnizoid's post as an example of where not to take EY's side was poorly chosen. He two-boxes on Newcomb's problem and any confident statements he makes about rationality or decision theory should be, for that reason, ignored entirely.

Of course, you go meta, without claiming that he's object-level right, but I'm not sure using an obviously wrong post to take his side on the meta level is a good idea.