Medical Diagnostic Imaging, Leukemia, and Black Holespost by nixtaken · 2020-08-03T05:40:41.128Z · score: -11 (5 votes) · LW · GW · 4 comments
Medical diagnostic imaging is highly dependent on the operator because whenever a radiologist looks at a set of MRI images or CAT scans, he or she will adjust the contrast and select a handful of images to tell a story about what he thinks is going on inside of a patient. Often this story is influenced by what the doctor who prescribed the MRI has written and more than ever, the story filtered out is guided by automated or AI tools.
Since we often see the things we are told to see, doctors have learned to be careful about diagnosis and make sure that they use several, independent sources of evidence.
If you only use a single diagnostic device for a disease, it is very easy to end up with a false-positive result and see a disease where none exists. For example, a doctor might mistake a blurry spot on the image for cancer or he might filter an image based on a diagnosis that he wants to see.
False positives are reduced by using multiple, independent diagnostic machines and blood tests. If they all agree with the symptoms reported by the patient and if you are lucky enough that those who performed the tests were not influenced by what they thought they should measure or what they were afraid might be measured by someone else (making them look bad), then when the surgeon shows up, he won’t cut you open and find nothing to remove.
If medical professionals don’t understand the necessity of having multiple independent, unbiased measurements, people die, but this is not the case in the field of astronomy and physics.
I cross-posted these articles here on Less Wrong because I think it is important for computer scientists and data scientists to be aware of some of the irrationality infecting portions of the physics community. This is important because ex-physicists are frequently embraced by the computer science and data science communities and they are entrusted with complex, critical tasks, based on assumptions about their understanding of how to handle data. They may end up building tools that make medical diagnoses.
In these articles, I wrote about some egregious examples of physicists handling data poorly, but within a few days, the welcoming committee here gave them a Karma of -89. I was hoping to find readers who might make use of this information, but so far, I've met with surprising forms of resistance.
- https://www.lesswrong.com/posts/3Ry4jw8jqgb55kKAt/evolution-of-group-delusion [LW · GW]
- https://www.lesswrong.com/posts/yE9ZwbGJ6m6Eiudsj/filtering-out-what-you-want-to-see [LW · GW]
- https://www.lesswrong.com/posts/wnmFHcPBchzk8uuQY/the-schumann-resonances [LW · GW]
- https://www.lesswrong.com/posts/oAsHa6xYMTBJWJGX6/the-new-scientific-method [LW · GW]
- https://www.lesswrong.com/posts/TazbHrwvF47jSn4ey/the-ghost-of-joseph-weber [LW · GW]
I was rather appalled to see a seemingly intelligent LessWrong commenter argue that
- directly adding together uncorrelated errors is a valid way to reduce the error of your measurement
- organizing randomly scrambled data according to a criteria called ‘weirdness’ or ‘deviations from expectations’ was valid
- there is no such thing as a noise floor when you can take more data
If I didn’t know better, I would think that this gish galloper was acting like a caricature of irrationality and executing a form of psychological warfare designed to discourage people like me from criticizing certain sacred cows of the scientific community. A more charitable assessment is that my articles provoked a strong case of cognitive dissonance in him. The moderator not only supported this commenter's logic but he also removed the scientific methods and philosophy tag, even though that was the main topic of the posts. My hope is that I just attracted a few bad apples and that other people will understand why I've come here.
If an EHT black hole astronomer combines blurry images from a bunch of different telescopes in order to create a picture of something that he believes should exist in unimaginably distant space, nobody is going to die.. at least not immediately.
If a LIGO gravitational waves astronomer filters out a blip buried in noise and attributes it to black holes colliding in a distant star system instead of to something more mundane and terrestrial in origin, nobody is going to die.. at least not immediately.
We all die eventually, but some of us may realize that we’ve wasted our lives measuring things that do not exist. We also might realize that a corruption of the scientific method within physics might infect other, more critical disciplines, like medicine. Many ex-physicists go on to do data analysis for medical research or banking.
This sort of confusion arises when astronomers believe that:
- directly adding together uncorrelated errors is a valid way to reduce or hide the error of your measurement
- organizing randomly scrambled data according to a criteria called ‘weirdness’ or ‘deviations from expectations’ is valid
- there is no such thing as a noise floor when you can take more data
The first point confuses people because they might think about how a pendulum oscillates randomly about a central point and how if you average over enough measurements, you will get a better measurement of that point relative to the fixed point holding the pendulum. If you take enough measurements, the difference between the relative positions of the fixed point and the measured point should go to zero – in an ideal, classical world.
In the real world, one might imagine the fixed point and the measured point wobbling around relative to everything else, as though they were sitting on a boat in the sea. The measured point and the fixed point may always be connected by a straight line, but that line may have no fixed relationship to anything else. In quantum mechanics, this is called intrinsic uncertainty, but in astronomy, people act as though this uncertainty doesn’t exist, because it doesn’t exist in general relativity.
In the real world, the Earth is moving relative to the star you want to measure and you cannot anticipate all of those movements or distinguish them from how light bends as it travels between the star and the Earth. Even if you have perfect equations governing the motion of an object, without perfect knowledge of initial conditions, there will always be some uncertainty about the predicted motion. Nothing is truly stationary relative to anything else and this motion will define a noise floor below which you simply can’t measure anything.
Far too many experiments in physics today are attempting to measure things that are below the noise floor and they are doing this without using any of the basic principles of the scientific method.
Strange things happen when medical professionals try to measure things that are below the noise floor of their measurement devices. They may try to combine measurements to see things that cannot be seen with an individual measurement and sometimes this appears to benefit the patient, but sometimes it does not. It is possible for a group of measurements that are all below the noise floor to give false positives that lead to a positive diagnosis — just to err on the side of caution in case of future malpractice lawsuits.
Of course, if you understand the limitations imposed by a noise floor, using multiple measurement methods to cross-check a hypothesis is clearly better than relying on a single measurement device. But even then, people can make mistakes when they combine uncertain, biased measurements and overestimate their certainty.
That was what I was thinking as I read this story about a perfectly healthy man who was diagnosed with leukemia and then blasted full of chemo, leading to years of trauma and recovery from something that was probably never going to become a problem.
- He is an extremely healthy, young-looking, energetic 50-year-old man who isn’t concerned that he is sick. His lymph nodes look fine. He goes in for a routine check-up.
- He had a white blood cell count that was just below the normal range: 2 when 4-12 is normal. He is given a referral to a blood specialist, just to be sure that nothing is wrong with him.
- The blood specialist got a bunch of inconclusive results and sent him to get a marrow test to cover her ass in case he has leukemia – even though he has no leukemia symptoms. He just has a low number of white blood cells.
- The marrow test yields an inconclusive result that indicates he might have ‘minimally residual disease’, otherwise known as MRD. It means that you might someday get leukemia, but you might not. They didn’t tell him that next-generation sequencing indicates false-positive MRD results and should only be used to predict prognosis after cancer treatment in patients with leukemia. As in, the test shouldn’t be used to diagnose leukemia. It should only be used to estimate the recovery of people who were treated for leukemia.
- He goes to a hot-shot doctor who offers to enroll him in a ‘clinical trial’ designed to remove any traces of any residual disease – essentially offering to test an intensive chemo treatment program on him for a future problem he might not develop. The hot-shot doctor benefits from enrolling people in his study and this guy looks like a good sport.
- He volunteers to get blasted with chemo and put into isolation while his body recovers its immune system. When he is released from the hospital, he requires blood transfusions that make him sick. He is told that, after all of the trauma he’s been through, he still tests positive for MRD. They didn’t tell him that the more methods used to determine MRD, the more the result tends to becomes negative. As far as I could tell, he may have never definitively had leukemia. He was a healthy guy with a somewhat low white blood cell count.
- He recovers and runs a marathon. Yay! But the chemo made him look 20 years older and he went through a medical hell that might’ve been unneccessary. He may have been pre-emptively treating a disease that he was never going to get.
I looked up the diagnostic criteria for leukemia and found that it is not a disease that you can catch in its early stages through blood tests. It has to present through symptoms – night sweats, fevers, infections, tiredness, weakness, bruising, paleness, bleeding, etc. He had none of these symptoms. Recall that before all of this started, this guy was as healthy as anyone could be, yet he was diagnosed with a leukemia precursor (MRD) based on a blood test that produces false positive results. A series of doctors who didn’t want to miss a diagnosis gave him a diagnosis (MRD that might become leukemia) because the penalties for missing a leukemia diagnosis were far worse for them than for over-diagnosing and treating leukemia. They were protecting themselves rather than the patient.The analysis of a sample is very complicated and unreliable. I bet a physicist designed it.
I am not a doctor and I hope that nobody takes my advice in their own medical care or in the care of their child, but I detect a systematic problem here and the problem is the over-reliance on tests that are not as reliable as those who use them believe. I mean, if you or your child are clearly sick, you should get help from a doctor, but this man’s story bothered me because he was perfectly healthy until the doctors told him that he was sick and gave him an invasive cancer treatment.
This faith in the medical system is quite similar to the faith people have in measurements of gravitational waves or the faith they have in the FDA and in dermatologist’s recommendations to use sunscreen.
What if it is all rotten to the core and we should be working harder to figure out which parts of the system can be salvaged. Do we need to turn back the clock and start doing science and medicine in the same way it was done back in the 1950s? Something went wrong in the past 70 years and I think it was our overzealous faith in technology.
- Wear sunscreen! Oh, wait, sunscreen goes through the skin and the blood-brain barrier. It will poison you and after one application, unhealthy levels can be measured in your blood for weeks thereafter.
- Disinfect everything with Triclosan! Rub it on your baby. Oh, wait, triclosan goes through the skin and the blood-brain barrier and will poison you. In studies done in the 1960s, rubbing Triclosan on baby monkeys made them mentally retarded, but the brilliant scientists decided that was because the monkeys had put their hands in their mouths and eaten the Triclosan.
- Get surgery to straighten your spine! Oh, wait. Most of those operations fail and cause extreme pain and unnecessary suffering to the young girls who are treated in this fashion.
- Get Gadolinium injected to improve the contrast on your MRI! Oh, wait. Gadolinium gets permanently stuck in your brain and eyes, wreaking untold havoc.
- Put Boron and Flouride in everything! Oh, wait. We might not have understood as much as we thought about what they do in the human body.
- Get a heart operation to reduce the risk that you’ll have a heart attack! Oh, wait. You would’ve been better off leaving your heart untouched by a surgeon’s knife.
- Get routine bloodwork! Oh, wait. You might be mistakenly treated for leukemia when you were perfectly healthy.
- Become a physicist! Oh, wait. You might end up spending your life measuring things that don’t exist.
Caveat Emptor! Experts are often stupid. So what are you going to do? Some people want to create AI tools to help individual patients evaluate their own situation, rather than relying on a bunch of doctors who might possibly be idiots or corrupted by their incentives.
Some of our founding inspirations were the work of Gerd Gigerenzer and Atul Gawande, who showed that medical errors were the cause of many deaths, that doctors tend to be statistically illiterate, and that systematizing tools like checklists and statistical prediction rules save lives.
https://www.lesswrong.com/posts/TKk7rShf9d5ePN7vR/personalized-medicine-for-real [LW · GW]
I do feel sorry for the medical professionals who are fighting the coronavirus and I wish them the best for themselves and their families, but I do not feel sorry for the medical industry as a whole. Just like the physics community, it needs a hard reset and re-grounding in foundational principles.
When I see experiments like LIGO and EHT given top billing in the science media, I see this as a symptom of a much wider problem with people mistaking faith in science for an understanding of the scientific method.
Consider that all of these physics experiments are used as training facilities for the people who go on to interpret data from fields as diverse as agriculture, economics, and medicine.
- Gravitational wave observatories claim to measure the change in distance between the Earth and Alpha Centauri with an accuracy of the width of a hair.
- Neutrino observatories claim to measure virtually unmeasurable particles and when one of their fiber optic cables got loose, they got confused and thought that the virtually unmeasurable particles traveled faster than light.
- Higgs particles cannot exist if neutrinos exist, but that has not stopped scientists at CERN from measuring the Higgs because when your apparatus measures noise, you can filter out whatever you want to see if you are good at slicing and dicing data.
- Astronomers studying the properties of blobs of light in the sky have come up with some rather strange explanations for what they have seen. Even when multiple explanations for an observation are possible, they tend to focus on one.
- Data scientists have combined images from telescopes to construct images of things that cannot be seen with a single telescope. They threw away the images that did not look like what they wanted to see – a black hole!
- Satellite gadgeteers have attempted to measure the noise floor of empty space by shooting a satellite as far away from the earth as they could. They put a heat source on the satellite to calibrate the measurement and measured that the noise floor of empty space looked like a perfect calibration curve.
- Balloon builders attempted to measure the signature of cosmic movements taking place at the beginning of time, but then they realized that they were measuring dust floating around in the sky. If at first you don’t succeed..
- Laser experts have attempted to take videos of the processes within bacteria by using short pulses of x rays from billion-dollar machines, but the pictures are no better than what you can get with a table-top microscope.
- Fusion power dreamers have created containers for lightning in a bottle and they think that with fast enough control loops, they can control that lightning, even though it jumps around at the speed of light.
- Climatologists use billion-dollar supercomputers to construct simulations of the climate in an effort to figure out how sensitive the climate is to changes in the concentration of carbon dioxide. When their sensitivity estimates fail to show predictive power, they just change the sensitivity factor and continue to claim that their model has predictive power.
- Planetary systems experts have created a software package that is so easy to use that a 17-year-old was able to discover a new planet during his summer internship. The software detects small wobbles in the light from individual stars.
- Cosmologists won a Nobel Prize for telling everyone that they had measured accelerated expansion of the universe. Several years later, it was shown that they had made a mistake by forgetting to take the motion of the Earth into account in their calculation. The expansion isn’t accelerating after all
I think we need to fix this widespread educational problem before ex-physicist quants create another banking crisis or before a miseducated ex-physicist convinces a doctor to use his software package to make decisions about a kid’s leukemia diagnosis.
Comments sorted by top scores.