How to Evaluate Data?

Overall the contents of the linked page make me want to update quite a bit away from trusting MetaMed. One more example:

One million children every year have had unnecessary CT scans, which risks exposing them to radiation levels up to those experienced by survivors of Hiroshima and Nagasaki.

Compare with this excerpt from the primary source, which presumably serves as the basis for the claim:

Most of the quantitative information that we have regarding the risks of radiation-induced cancer comes from studies of survivors of the atomic bombs dropped on Japan in 1945. Data from cohorts of these survivors are generally used as the basis for predicting radiation-related risks in a population because the cohorts are large and have been intensively studied over a period of many decades, they were not selected for disease, all age groups are covered, and a substantial subcohort of about 25,000 survivors received radiation doses similar to those of concern here — that is, less than 50 mSv.

The primary source is not claiming that "a CT scan exposes you to Hiroshima-Nagasaki survivor radiation levels". It is saying the converse - "some atomic bomb survivors received doses low enough to be comparable to CT scans". The phrase "survivors of Hiroshima and Nagasaki" is pure fear-mongering - how much radiation the typical atomic bomb received is not public knowledge, so we'll tend to think in terms of worst cases (this handy chart might help). The average dose received, however, was 210 mSV according to one source I consulted; this is four times the high end dose from a pediatric CT, with the low end around 5 mSV. The statement from MetaMed is perhaps not an outright lie but it is at least grossly misleading.

For a business which has been touted right here and by no less than Eliezer himself as providing "actual evidence-based healthcare", this is a little worrisome.

(ETA: contrary to appearances I'm not actually trying to take over this whole post's discussion area, but I happen to get easily nerd-sniped by fact-checking exercises and easily worked up when I get a sense that someone is trying to pull a fast one on me.)

The "98,000 patients" claim is really interesting as an example of Dark Arts, aside from its having been debunked often.

It is often presented as follows: "98,000 deaths per year from medical errors (the equivalent of a jumbo jet crashing every day)".

It would be... provided every single jumbo jet flying in the US was populated by people already seriously ill or injured in the first place, rather than (as is actually the case) not only healthy but generally also wealthy passengers.

Of course you're supposed to overlook that trivial difference in the demographics of people who are in planes and those who are in hospitals, and picture hospitals killing healthy rich people by the planeload.

(This also suggests that "number of deaths" is a poor metric for making such estimates and comparisons; it would be better to compute "overall loss of expected QALYs resulting from preventable mistakes in medical care" and compare that with aggregate loss of QUALYs from other causes. Of course that's much less catchy.)

I have some experience with this.

Some "facts" just set my spidey-sense tingling, and I find it usually well worth the time to check out the references in such case. In general, with the slightest doubt I will at least Google the reference and check out the abstract - this is quick and will at the least guarantee that the source does exist.

Particular things that set my spidey-sense off are:

• sensationalistic claims - any that is ostensibly used to shock the reader into action
• excess precision - like claims about "56% of all software projects" vs "roughly 60% in the population we sampled"
• excess confidence about hard-to-test claims, in software those tend to be "productivity" claims
• claims that are ambiguous or that would be very hard to confirm experimentally, e.g. the well-known canard about how many times a day men think about sex; basically "is this even a sane question to ask"
• hard-to-find primary sources - when you can't readily check it, a claim becomes more suspicious
• abstracts that don't contain the claim being put forward - it's more suspicious when you cite a paper for a tiny bit buried deep within
• (ETA) "phone game" claims - a citation to a citation of a citation, etc. with many indirections before reaching the primary source

Let's look at some specifics of the page you cite - at the outset we note that it's designed to be sensationalistic, a marketing brochure basically. It's up to you to factor that into your assessment of how much you trust the references.

• "As many as 98,000 people die in hospitals each year as a result of medical errors" - doesn't feel implausible, but a quick Google for US annual death rate - it turns out to be twice the death rate for suicide. This document seems to contradict the finding, I'd check out the reference
• "Doctors spend an average of only ten minutes with patients" - an average like that isn't too hard to work out from a survey and squares with personal experience, I'd take it at face value
• "By some estimates, deaths caused by medicine itself total 225,000 per year" - excess precision for a phrase as vague as "caused by medicine itself", I'd check out the reference just to know what that's supposed to mean
• "Most published research findings are false" - this is the title of the Ioannidis article, and should not be taken at face value to mean "all research in all fields of medicine", read the ref for sure
• "Up to 30% of patients who die in hospitals have important diseases or lesions that are not discovered until after death" - I'd want to know more about how they estimate this - are we talking about extrapolation from the few deaths which result in autopsy?
• "It takes an average of 17 years for scientific research to enter clinical practice" - uh, maybe; somewhat ambiguous (what's an objective criterion for "entering clinical practice"?)
• "In oncology alone, 90% of preclinical cancer studies could not be replicated." - I get into trouble almost immediately trying to check this reference, given as " Begley, C. G. (n.d.). Preclinical cancer research, 8–10.": Google gives me partial matches on the title but no exact matches, (n.d.) means "No date" which is kind of weird; this does have a publication date and even a URL
• "deaths from cancer have barely been touched" - I wouldn't be surprised, cancer is a tough bastard
• "If a primary care physician provided all recommended [...] care [...] he would need to work 21.7 hours a day" - excess precision (also a hypothetical, so not really "evidence"); check the source

Also, this is a Web page, so I get suspicious on principle that no hyperlinks are provided.

On the 98,000 figure this may bring some balance:

Similar to previous studies, almost a quarter (22.7%) of active-care patient deaths were rated as at least possibly preventable by optimal care, with 6.0% rated as probably or definitely preventable. Interrater reliability for these ratings was also similar to previous studies (0.34 for 2 reviewers). The reviewers' estimates of the percentage of patients who would have left the hospital alive had optimal care been provided was 6.0% (95% confidence interval [CI], 3.4%-8.6%). However, after considering 3-month prognosis and adjusting for the variability and skewness of reviewers' ratings, clinicians estimated that only 0.5% (95% CI, 0.3%-0.7%) of patients who died would have lived 3 months or more in good cognitive health if care had been optimal, representing roughly 1 patient per 10 000 admissions to the study hospitals.

(...)

Medical errors are a major concern regardless of patients' life expectancies, but our study suggests that previous interpretations of medical error statistics are probably misleading.

(...)

In an exchange about the validity of these estimates, McDonald et al argued on theoretical grounds that these statistics are likely overestimates. They were particularly concerned about the lack of consideration of the expected risk of death in the absence of the medical error. Indeed, these statistics have often been quoted without regard to cautions by the authors of the original reports, who note that physician reviewers do not believe necessarily that 100% of these deaths would be prevented if care were optimal.

(...)

As predicted on theoretical grounds, many deaths reportedly due to medical errors occur at the end of life or in critically ill patients in whom death was the most likely outcome, either during that hospitalization or in the coming months, regardless of the care received. However, this was not the only—or even the largest—source of potential overestimation. Previously, most have framed ratings of preventable deaths as a phenomenon in which a small but palpable number of deaths have clear errors that are being reliably rated as causing death. Our results suggest that this view is incorrect—that if many reviewers evaluate charts for preventable deaths, in most cases some reviewers will strongly believe that death could have been avoided by different care; however, most of the "errors" identified in implicit chart review appear to represent outlier opinions in cases in which the median reviewer believed either that an error did not occur or that it had little or no effect on the outcome.

ETA: see also, primary source for the 98,000 figure (try and find it!), this discusses the 98,000 figure as a Fermi estimate

Related to this, I've been wondering if I should write a post based on this G+ blog, but aimed at LW readers specifically and focusing on probabilistic thinking about the claim.

To recap: we have this supposed "survey" of U.S. Defense projects claimed to have taken place in 1995, and to have looked at $37Bn worth of software development projects. It classifies them into five categories (from "OK" to "bad" to "horrible"), it doesn't really matter what they are, we can call them A,B,C,D,E. There's a certain allocation: A:46%, B: 29%, C: 20%, D: 3%, E: 2%.

But we also find an earlier (1979) study, with a more credible primary source. Its five categories are labeled exactly the same, its sample size is much smaller - 9 projects for $7 million total. The allocation is nearly the same: A:47%, B: 29%, C: 19%, D: 3%, E: 2%.

The article I link in the G+ post, written by someone with a PhD, remarks on this coincidence:

Individually, these studies indicate that the success rate for U. S. Government outsourcing has historically been very low. Together they indicate that the success rate has not improved despite the introduction of new technologies and management procedures over the years.

The exercise consists in working out somewhat rigorously the probability that, given the hypothesis "there exists a fixed probability with which software projects fall into categories A,B,C,D,E" you would get, within 1%, the exact same results from a huge ($37Bn) survey as you'd have gotten from an independent and much smaller sample.

(Intuitively, this is a little like saying you've found a 9-people family whose polling results exactly predict the result of a national election with 5 candidates.)

R> chisq.test(matrix(c(47,29,19,3,2, 46*490,29*490,20*490,3*490,2*490), ncol = 2), simulate.p.value = TRUE, B = 20000000)

Pearson's Chi-squared test with simulated p-value (based on 2e+07 replicates)

data:  matrix(c(47, 29, 19, 3, 2, 46 * 490, 29 * 490, 20 * 490, 3 *      490, 2 * 490), ncol = 2)
X-squared = 0.0716, df = NA, p-value = 0.9983

It is more useful to determine whether a source you're looking at is not telling the truth. Find one black swan, you don't have to look at all possible swans to determine the claim "all swans are white" is not correct.

In the example you gave, identify ways to determine whether the source is not telling the truth. That could include inaccurate quotes, or accurate quotes of faulty data, or consulting the cited texts plus competing texts, but I'm not sure it can include avoiding reading the cited texts even if you think it's a waste of time.

As Trevor_Blake said, there's very little you can do apart from actually checking some of the data. An alternative is to ask or pay someone else or a group to verify it for you.

Of course, there's always the option of coding a probabilistic engine that mines for stats and gives you reliability estimates of certain claims using some bayes-fu. But that takes math, programming, and lots of work.

If you have a specific claim where you aren't sure whether it's true http://skeptics.stackexchange.com/ is a good website. Start by seeking whether someone else already asked the question. If nobody has than you can open a new question.

Well...what is your goal in evaluating the truthfulness of these statements? Even if they're actually "true" do they help you meet your goals?