Zvi's Law of No Evidence

post by lsusr · 2021-05-14T07:27:51.474Z · LW · GW · 9 comments

Law of No Evidence: Any claim that there is “no evidence” of something is evidence of bullshit.

Covid 5/13: Moving On [LW · GW] by Zvi

But in probability theory, absence of evidence is always evidence of absence

Absence of Evidence Is Evidence of Absence [LW · GW] by Eliezer Yudkowsky

This is not a contradiction. Zvi and Yudkowsky are both correct.

"No evidence" is a vague term. Consider the phrase "There is no evidence of Bigfoot". It could mean one of two things.

• Explorers looked for Bigfoot and didn't find him. This is evidence Bigfoot doesn't exist.
• Nobody looked for Bigfoot and nobody found him. This is not evidence Bigfoot doesn't exist.

It is true "no evidence" is evidence of absence insofar as "no evidence" means "we ran good experiments and they failed to produce evidence for the hypothesis". But in this case you shouldn't say "no evidence". You should say "evidence of absence".

Inference starts with priors and are then updates them according to evidence. If there is no evidence related to a hypothesis then your beliefs reflect your original priors. If "no evidence" refers to true "absence of evidence" then everyone needs to retreat to their Bayesian priors. But this not what anyone means when they say "no evidence" because your Bayesian priors for one context are informed by transfer learning of evidence from other contexts e.g. I have a computer programmer friend who thinks about everything in terms of software metaphors.

If someone deliberately uses vague[1] language instead of precise language then it is because they are trying to conflate two ideas which should not be conflated. If someone says there is "no evidence" of something then it is because they are trying to pass off "nobody looked for Bigfoot and nobody found him" as "explorers looked for Bigfoot and nobody found him".

Saying "there is no evidence of UFOs" communicates nothing useful. You must explain why absence of evidence constitutes evidence of absence.

If you want to prove that red light doesn't cause cancer you don't say there is "no evidence". You should explain the frequency dependence of the photoelectric effect.

1. If someone deliberately uses general language instead of precise language then that is good because general language is easier to falsify than precise language. Vague language is bullshit because it is hard to falsify. ↩︎

comment by gjm · 2021-05-14T14:53:55.011Z · LW(p) · GW(p)

I agree with you that Zvi and Eliezer are both correct, but I am unconvinced by your analysis of why. I don't think "no evidence" is vague. I think it's false.

"There is no evidence of Bigfoot" is simply untrue. There are people who claim to have seen things-that-maybe-could-be-Bigfoot. There are people who claim to have found footprints and the like. All these things are evidence. Crappy evidence, which is why it doesn't do much to change my opinion that Bigfoot is almost certainly not real, but still evidence.

"Nobody looked for Bigfoot and nobody found him" is also evidence that Bigfoot doesn't exist. (There are possible worlds in which Bigfoot gets seen by people who weren't actively looking. If nobody looked and nobody found, those worlds are ruled out.)

So, when someone says "there is no evidence of Bigfoot", what they are saying is not literally true, and they almost certainly know it, and that's evidence of bullshit or, at best, thinking sloppy enough that they might as well be bullshitting.

We can get back to something like your complaint of vagueness if we apply a bit of extra charity and suppose that when they say "there is no evidence of Bigfoot" they mean something like "there are no pieces of evidence of Bigfoot that are individually very strong evidence". In that case, there are two possible charges of vagueness we could bring:

• It could mean "there are no photos that are unquestionably of Bigfoot", or it could mean "there are no credible eyewitness reports", or it could mean "there are no videos", etc. I don't really buy this criticism; I think it's fairly clear that the meaning is pretty much what I said above.
• Having decided what it means, it then potentially matters a lot what things have been going on that one might expect to produce such evidence. As you say, having no photos, eyewitness reports, etc., is stronger evidence if lots of people have been out looking hard than if no one has been.
• But I don't think this situation is accurately described by saying that the statement is vague. It's just a fact of life that sometimes the conclusions we should draw from an observation depend on its context.
• If someone says, in so many words, "To the best of my knowledge, no one has ever presented any piece of evidence for Bigfoot that a reasonable person would find compelling", I don't think I could reasonably complain that they're being vague or bullshitty, even though what I'd conclude would depend on how hard I thought people had been looking for such evidence.
Replies from: PeterMcCluskey
comment by PeterMcCluskey · 2021-05-14T15:15:30.065Z · LW(p) · GW(p)

I disagree. Much of what's going wrong is differing meanings of the word evidence [LW · GW].

Most people are oblivious to the Bayesian meaning of the word evidence. When I hear an ordinary person say "no evidence", I usually interpret it as "no evidence that's admissible in a court", or maybe even "no proof".

comment by Ericf · 2021-05-14T13:26:02.703Z · LW(p) · GW(p)

This misses the point of Zvi's comment. Alice saying "there is no evidence of X" is more likely in worlds where Alice is BSing than in worlds where Alice is attempting to provide factual information (ie Level 1 communication). That is orthogonal to any actual calculations of {amount of evidence observed} / {amount of evidence that could have been observed, given the amount of looking that has been done}.

Also, too, the second thing is a gradient, not a dichotomy. And "your priors" are just a way of saying {amount of evidence observed that I know about} / {amount of evidence that could have been observed, given what I know about the amount of looking that has been done}

comment by alkexr · 2021-05-15T14:24:21.155Z · LW(p) · GW(p)

Absence of evidence of X is evidence of absence of X.

A claim about the absence of evidence of X is evidence of:

• the speaker's belief of the listeners' belief in X,
• absence of evidence of NOT X,
• the speaker's intention of changing the listeners' belief in X.

comment by MalcolmOcean (malcolmocean) · 2021-05-19T02:45:03.012Z · LW(p) · GW(p)

I appreciate you trying to write this up, but as other commenters have noted, there's no contradiction here in the first place and you appear to have missed the point.

As far as I can tell, if you understand Yudkowsky's point, Zvi's follows directly. {No evidence of X} = {evidence of not-X}, but the speech act of claiming "There is no evidence of X" only occurs when there is some evidence worth claiming doesn't count as evidence.

And Yudkowsky's point also points out that essentially "no evidence" is not just vague but in virtually all cases just completely misleading. It would be better to say "the absence of any photographs or eyewitness accounts is evidence that this story was fabricated."

There is evidence for and evidence against but there is no "no evidence".

Well, apparently I decided to write this up as its own post [LW · GW].

comment by DanArmak · 2021-05-15T12:56:29.066Z · LW(p) · GW(p)

If someone says there is "no evidence" of something then it is because they are trying to pass off "nobody looked for Bigfoot and nobody found him" as "explorers looked for Bigfoot and nobody found him".

A "no evidence" argument doesn't have to be made in bad faith. It's claiming that we've looked into the people who said they saw Bigfoot (as opposed to looking for Bigfoot itself), and concluded those claims have no good evidence behind them. And so, without evidence, we should rule out Bigfoot, because the prior for Bigfoot is very low. We would need positive evidence to raise the Bigfoot hypothesis to the level of conscious consideration, and we claim there is no such evidence.

Yes, a claim of "no evidence" is - in this context - a social attack on the people who were talking about the subject (and so implicitly claiming "yes evidence"). In the highly politicized context Zvi is discussing, almost all factual arguments are disguised social attacks; rhetorics, meant to persuade people, with facts and logic being instrumental but not the goal.

And so we can justly ignore the whole discussion because we think it's not about facts and arguments and real "evidence" and it never was. But if we want to engage with the discussion using our own arguments and evidence (or to pretend to do so for our own social goals), then we should acknowledge that a valid factual claim is being made here, which we can evaluate without dismissing it as purely rhetorical manipulation ("passing off argument A as argument B").

Zvi wrote,

No evidence should be fully up there with “government denial” or “I didn’t do it, no one saw me do it, there’s no way they can prove anything.” If there was indeed no evidence, there’d be no need to claim there was no evidence, and this is usually a move to categorize the evidence as illegitimate and irrelevant because it doesn’t fit today’s preferred form of scientism.

I disagree with this. If people claim Bigfoot exists, and I think they have no evidence for that claim, then yes I will say there is no evidence. The mere fact that people claim A is not in itself evidence for A, because people are not pure truth-seekers, and if I acknowledge any claim as itself constituting evidence, they will proceed to claim lots of things without evidence behind them. I don't need to "categorize the evidence as illegitimate and irrelevant", I should be able to say plainly that there is no evidence to begin with. It's not because "it doesn't fit today's preferred form of scientism", it's because seeing a vague outline in a snow-storm really truly isn't evidence for Bigfoot.

When people we don't like claim things that are clearly wrong, we may want to dismiss their arguments are rhetorically invalid or malicious or made in bad faith. To claim that the form of such arguments necessarily indicates they are being made in bad faith. But that is engaging on their terms - analyzing why they're making the arguments, instead of analyzing the arguments themselves (simulacra levels!). These two discussions are both necessary but they should be kept apart. On the object level, we should be able to keep saying - the arguments are not "wrongly shaped", they are just factually wrong.

Replies from: Kenny
comment by Kenny · 2021-05-22T21:40:40.320Z · LW(p) · GW(p)

The mere fact that people claim A is not in itself evidence for A, because people are not pure truth-seekers, and if I acknowledge any claim as itself constituting evidence, they will proceed to claim lots of things without evidence behind them.

Technically, this would only be true if claims for A were perfectly independent of the truth of A. Your argument, as written, seems to imply radical skepticism, at least for some topics.

And you seem to be falling into the trap of thinking of things as being members of a simple binary set of mutually exclusive categories of 'evidence' or 'not evidence'. The generally held view of the users of this site tho is that it's better to expand the size of the set of 'evidence' categories, e.g. standard probabilities. (I don't think it's particularly necessary to use real numbers (versus your implied '0' or '1' binary values), but it sure is useful to use at least more than two categories.)

It's not true that there's literally no evidence of Bigfoot, it's that the evidence is very weak.

As another commenter pointed out, a big part of this 'disagreement' is really people 'talking past each other' and a big part of that is the 'evidence' means different things to different people.

comment by tkpwaeub (gabriel-holmes) · 2021-05-16T13:13:45.614Z · LW(p) · GW(p)

Isn't this the Heap paradox? Enough absence of evidence probably should be considered to be evidence of absence, but it's not obvious where that cutoff should be, and it can vary from one situation to the next.

Replies from: Kenny
comment by Kenny · 2021-05-22T21:48:33.170Z · LW(p) · GW(p)

I think you're right that there's an interesting connection to the "heap paradox" (sorites paradox, where 'sorites' is derived from the Greek word for 'heap').

Absence of evidence is always (to some degree) evidence of absence, but you're right that there's no context-free cutoff where one can always 'safely round down' from some arbitrarily small probability to 'effectively zero'.

But the main point is that claims of "no evidence" are technically incorrect (based on the understanding of 'evidence' shared by many/most of the users of this site, and similarly inclined people).

A more charitable interpretation of "no evidence" might be 'no good evidence', but as Zvi points out, even that's almost always false – because there's no cutoff between 'good' and 'bad' evidence.