Long Covid Informal Study Results

post by Elizabeth (pktechgirl) · 2021-10-16T07:10:05.056Z · LW · GW · 3 comments

Contents

  Introduction
  The Study
  Differences in Analysis
  Results
  Reasons this analysis could be wrong
  Conclusion
None
3 comments

Introduction

Yesterday* I talked about a potential treatment for Long Covid, and referenced an informal study I’d analyzed that tried to test it, which had seemed promising but was ultimately a let down. That analysis was too long for its own post, so it’s going here instead. 

Gez Medinger ran an excellent-for-its-type study of interventions for long covid, with a focus on niacin, the center of the stack I took. I want to emphasize both how very good for its type this study was, and how limited the type is. Surveys of people in support groups who chose their own interventions is not a great way to determine anything. But really rigorous information will take a long time and some of us have to make decisions now, so I thought this was worth looking into.

Medinger does a great analysis in this youtube video. He very proactively owns all the limitations of the study (all of which should be predictable to regular readers of mine) and does what he can to make up for them in the analysis, while owning where that’s not possible. But he delivers the analysis in a video rather than a text post ugh why would you do that (answer: he was a professional filmmaker before he got long covid). I found this deeply hard to follow, so I wanted to play with the data directly. Medinger generously shared the data, at which point this snowballed into a full-blown analysis.

I think Medinger attributes his statistics to a medical doctor, but I couldn’t find it on relisten and I’m not watching that damn video again. My statistical analysis was done by my dad/Ph.D. statistician R. Craig Van Nostrand. His primary work is in industrial statistics but the math all transfers, and the biology-related judgment calls were made by me (for those of you just tuning in, I have a BA in biology and no other relevant credentials or accreditations).

The Study

As best I can determine, Medinger sent a survey to a variety of long covid support groups, asking what interventions people had tried in the last month, when they’d tried them, and how they felt relative to a month ago. Obviously this has a lot of limitations – it will exclude people who got better or worse enough they didn’t engage with support groups, it was in no way blinded, people chose their own interventions, it relied entirely on self-assement, etc. 

Differences in Analysis

You can see Medinger’s analysis here. He compared the rate of improvement and decline among groups based on treatments. I instead transformed the improvement bucket to a number and did a multivariate analysis. 

Much better (near or at pre-covid)	1
Significantly better	0.5
A little better	0.1
No change	0
A little worse	-0.2
Significantly worse	Curiously unused
Much worse	-1.2

You may notice that the numerical values of the statements are not symmetric- being “a little worse” is twice as bad as “a little better” is good. This was deliberate, based on my belief that people with chronic illness on average overestimate their improvement over short periods of time. We initially planned on doing a sensitivity analysis to see how this changed the results; in practice the treatment groups had very few people who got worse so this would only affect the no-treatment control, and it was obvious that fiddling with the numbers would not change the overall conclusion.

Also, no one checked “significantly worse”, and when asked Medinger couldn’t remember if it was an option at all. This suggests to me that “Much worse” should have a less bad value and “a little worse” a more bad value. However, we judged this wouldn’t affect the outcome enough to be worth the effort, and ignored it. 

We tossed all the data where people had made less than two weeks ago (this was slightly more than half of it), except for the no-change control group (140 people). Most things take time to have an effect and even more things take time to have an effect you can be sure isn’t random fluctuation. The original analysis attempted to fix this by looking at who had a sudden improvement or worsening, but I don’t necessarily expect a sudden improvement with these treatments.

We combined prescription and non-prescription antihistamines because the study was focused on the UK which classifies several antihistamines differently than the US. 

On row 410, a user used slightly nonstandard answers, which we corrected to being equivalent to “much improved’, since they said they were basically back to normal.

Medinger uses both “no change” and “new supplements but not niacin” as control groups, in order to compensate for selection and placebo effects from trying new things. I think that was extremely reasonable but felt I’d covered it by limiting myself to subjects with >2 weeks on a treatment and devaluing mild improvement. 

Results

I put my poor statistician through many rounds on this before settling on exactly which interventions we should focus on. In the end we picked five: niacin, anti-histamines, and low-histamine diet, which the original analysis evaluated, and vitamin D (because it’s generally popular), and selenium (because it had the strongest evidence of the substances prescribed the larger protocol, which we’ll discuss soon). 

Unfortunately, people chose their vitamins themselves, and there was a lot of correlation between the treatments. Below is the average result for people with no focal treatments, everyone with a given focal treatment, and everyone who did that and none of the other focal treatments for two weeks (but may have done other interventions). I also threw in a few other analyses we did along the way. These sample sizes get really pitifully small, and so should be taken as preliminary at best. 

Treatment	Niacin, > 2 weeks	Selenium, > 2 week	Vitamin D, > 2 week	Antihistamines, > 2 weeks	Low-histamine diet, > 2 weeks	Change (1 = complete recovery)	95% Confidence Interval	n
No change	0	0	0	0	0	0.04	± 0.07	140
Niacin, > 2 weeks	1	–	–	–	–	0.23	± 0.07	91
Selenium, > 2 weeks	–	1	–	–	–	0.24	±0.07	88
Vitamin D, > 2 week	–	–	1	–	–	0.15	±0.05	261
Antihistamines, >2 weeks	–	–	–	1	–	0.18	± 0.06	164
Low histamine diet	–	–	–	–	1	0.18	±0.06	195
Niacin, > 2 weeks, no other focal treatments	1	0	0	0	0	0.15	±0.2	11
Selenium, > 2 weeks, no other focal treatments	0	1	0	0	0	0.05	±0.06	4
Vitamin D, > 2 week, no other focal treatments	0	0	1	0	0	0.07	±0.08	106
Antihistamines, >2 weeks, no other focal treatments	0	0	0	1	0	0.08	±0.13	26
Low histamine diet, > 2 weeks, no other focal treatments	0	0	0	0	1	0.13	±0.14	44
All focal treatments	1	1	1	1	1			0
Niacin + Antihistamines, >2 weeks	1	–	–	1	0	0.33	± 0.07	38
Niacin + Low Histamine Diet, > 2 weeks	1	0	0	0	1	0.29	±0.10	36
Selenium + Niacin, no histamine interventions	1	1	–	0	0	0.05	±0.19	17
Niacin, > 2 weeks, no other focal treatments, ignore D	1	0	–	0	0	0.13	±0.12	19
Selenium, > 2 weeks, no other focal treatments, ignore D	0	1	–	0	0	0.16	±0.12	18

1 = treatment used

0 = treatment definitely not used

– = treatment not excluded

Confidence interval calculation assumes a normal distribution, which is a stretch for data this lump and sparse but there’s nothing better available.

[I wanted to share the raw data with you but Medinger asked me not to. He was very fast to share with me though, so maybe if you ask nicely he’ll share with you too]

You may also be wondering how the improvements were distributed. The raw count isn’t high enough for really clean curves, but the results were clumped rather than bifurcated, suggesting it helps many people some rather than a few people lots. Here’s a sample graph from Niacin (>2 weeks, no exclusions)

Reasons this analysis could be wrong

• All the normal reasons this kind of study or analysis can be wrong.
• Any of the choices I made that I outlined in “Differences…”
• There were a lot of potential treatments with moderate correlations with each other, which makes it impossible to truly track the cause of improvements.
• Niacin comes in several forms, and the protocol I analyze later requires a specific form of niacin (I still don’t understand why). The study didn’t ask people what form of niacin they took. I had to actively work to get the correct form in the US (where 15% of respondents live); it’s more popular but not overwhelmingly so in the UK (75% of respondents), and who knows what other people took. If the theory is correct and if a significant number of people took the wrong form of niacin, it could severely underestimate the improvement.
• This study only looked at people who’d changed things in the last month. People could get better or worse after that.
• There was no attempt to look at dosage.

Conclusion

For a small sample of self-chosen interventions and opt-in participation, this study shows modest improvements from niacin and low histamine diets, which include overlap with the confidence interval of the no-treatment group if you exclude people using other focal interventions. The overall results suggest that either something in the stack is helping, or that trying lots of things is downstream of feeling better, which I would easily believe.

 

 

Thank you to Gez Medinger for running the study and sharing his data with me, R. Craig Van Nostrand for statistical analysis, and Miranda Dixon-Luinenburg⁩ for copyediting.

 

 

* I swear I scheduled this to publish the day after the big post but here we are three days later without it unpublished, so…

3 comments

Comments sorted by top scores.