Long Covid Informal Study Results

post by Elizabeth (pktechgirl) · 2021-10-16T07:10:05.056Z · LW · GW · 3 comments

Contents

  Introduction
  The Study
  Differences in Analysis
  Results
  Reasons this analysis could be wrong
  Conclusion
None
3 comments

Introduction

Yesterday* I talked about a potential treatment for Long Covid, and referenced an informal study I’d analyzed that tried to test it, which had seemed promising but was ultimately a let down. That analysis was too long for its own post, so it’s going here instead. 

Gez Medinger ran an excellent-for-its-type study of interventions for long covid, with a focus on niacin, the center of the stack I took. I want to emphasize both how very good for its type this study was, and how limited the type is. Surveys of people in support groups who chose their own interventions is not a great way to determine anything. But really rigorous information will take a long time and some of us have to make decisions now, so I thought this was worth looking into.

Medinger does a great analysis in this youtube video. He very proactively owns all the limitations of the study (all of which should be predictable to regular readers of mine) and does what he can to make up for them in the analysis, while owning where that’s not possible. But he delivers the analysis in a video rather than a text post ugh why would you do that (answer: he was a professional filmmaker before he got long covid). I found this deeply hard to follow, so I wanted to play with the data directly. Medinger generously shared the data, at which point this snowballed into a full-blown analysis.

I think Medinger attributes his statistics to a medical doctor, but I couldn’t find it on relisten and I’m not watching that damn video again. My statistical analysis was done by my dad/Ph.D. statistician R. Craig Van Nostrand. His primary work is in industrial statistics but the math all transfers, and the biology-related judgment calls were made by me (for those of you just tuning in, I have a BA in biology and no other relevant credentials or accreditations).

The Study

As best I can determine, Medinger sent a survey to a variety of long covid support groups, asking what interventions people had tried in the last month, when they’d tried them, and how they felt relative to a month ago. Obviously this has a lot of limitations – it will exclude people who got better or worse enough they didn’t engage with support groups, it was in no way blinded, people chose their own interventions, it relied entirely on self-assement, etc. 

Differences in Analysis

You can see Medinger’s analysis here. He compared the rate of improvement and decline among groups based on treatments. I instead transformed the improvement bucket to a number and did a multivariate analysis. 

Much better (near or at pre-covid)1
Significantly better0.5
A little better0.1
No change0
A little worse-0.2
Significantly worseCuriously unused
Much worse-1.2

You may notice that the numerical values of the statements are not symmetric- being “a little worse” is twice as bad as “a little better” is good. This was deliberate, based on my belief that people with chronic illness on average overestimate their improvement over short periods of time. We initially planned on doing a sensitivity analysis to see how this changed the results; in practice the treatment groups had very few people who got worse so this would only affect the no-treatment control, and it was obvious that fiddling with the numbers would not change the overall conclusion.

Also, no one checked “significantly worse”, and when asked Medinger couldn’t remember if it was an option at all. This suggests to me that “Much worse” should have a less bad value and “a little worse” a more bad value. However, we judged this wouldn’t affect the outcome enough to be worth the effort, and ignored it. 

We tossed all the data where people had made less than two weeks ago (this was slightly more than half of it), except for the no-change control group (140 people). Most things take time to have an effect and even more things take time to have an effect you can be sure isn’t random fluctuation. The original analysis attempted to fix this by looking at who had a sudden improvement or worsening, but I don’t necessarily expect a sudden improvement with these treatments.

We combined prescription and non-prescription antihistamines because the study was focused on the UK which classifies several antihistamines differently than the US. 

On row 410, a user used slightly nonstandard answers, which we corrected to being equivalent to “much improved’, since they said they were basically back to normal.

Medinger uses both “no change” and “new supplements but not niacin” as control groups, in order to compensate for selection and placebo effects from trying new things. I think that was extremely reasonable but felt I’d covered it by limiting myself to subjects with >2 weeks on a treatment and devaluing mild improvement. 

Results

I put my poor statistician through many rounds on this before settling on exactly which interventions we should focus on. In the end we picked five: niacin, anti-histamines, and low-histamine diet, which the original analysis evaluated, and vitamin D (because it’s generally popular), and selenium (because it had the strongest evidence of the substances prescribed the larger protocol, which we’ll discuss soon). 

Unfortunately, people chose their vitamins themselves, and there was a lot of correlation between the treatments. Below is the average result for people with no focal treatments, everyone with a given focal treatment, and everyone who did that and none of the other focal treatments for two weeks (but may have done other interventions). I also threw in a few other analyses we did along the way. These sample sizes get really pitifully small, and so should be taken as preliminary at best. 

TreatmentNiacin, > 2 weeksSelenium, > 2 weekVitamin D, > 2 weekAntihistamines, > 2 weeksLow-histamine diet, > 2 weeksChange (1 = complete recovery)95% Confidence Interval n
No change000000.04± 0.07140
Niacin, > 2 weeks10.23± 0.0791
Selenium, > 2 weeks10.24±0.0788
Vitamin D, > 2 week10.15±0.05261
Antihistamines, >2 weeks10.18± 0.06164
Low histamine diet10.18±0.06195
Niacin, > 2 weeks, no other focal treatments100000.15±0.211
Selenium, > 2 weeks, no other focal treatments010000.05±0.064
Vitamin D, > 2 week, no other focal treatments001000.07±0.08106
Antihistamines, >2 weeks, no other focal treatments000100.08±0.1326
Low histamine diet, > 2 weeks, no other focal treatments000010.13±0.1444
All focal treatments11111  0
Niacin + Antihistamines, >2 weeks1100.33± 0.0738
Niacin + Low Histamine Diet, > 2 weeks100010.29±0.1036
Selenium + Niacin, no histamine interventions11000.05±0.1917
Niacin, > 2 weeks, no other focal treatments, ignore D10000.13±0.1219
Selenium, > 2 weeks, no other focal treatments, ignore D01000.16±0.1218

1 = treatment used

0 = treatment definitely not used

– = treatment not excluded

Confidence interval calculation assumes a normal distribution, which is a stretch for data this lump and sparse but there’s nothing better available.

[I wanted to share the raw data with you but Medinger asked me not to. He was very fast to share with me though, so maybe if you ask nicely he’ll share with you too]

You may also be wondering how the improvements were distributed. The raw count isn’t high enough for really clean curves, but the results were clumped rather than bifurcated, suggesting it helps many people some rather than a few people lots. Here’s a sample graph from Niacin (>2 weeks, no exclusions)

Reasons this analysis could be wrong

Conclusion

For a small sample of self-chosen interventions and opt-in participation, this study shows modest improvements from niacin and low histamine diets, which include overlap with the confidence interval of the no-treatment group if you exclude people using other focal interventions. The overall results suggest that either something in the stack is helping, or that trying lots of things is downstream of feeling better, which I would easily believe.

 

 

Thank you to Gez Medinger for running the study and sharing his data with me, R. Craig Van Nostrand for statistical analysis, and Miranda Dixon-Luinenburg⁩ for copyediting.

 

 

* I swear I scheduled this to publish the day after the big post but here we are three days later without it unpublished, so…

3 comments

Comments sorted by top scores.

comment by Daniel_Eth · 2021-10-18T03:25:17.796Z · LW(p) · GW(p)

"the protocol I analyze later requires a specific form of niacin"
What's the form? Also, do you know what sort of dosage is used here?

 

If niacin is helpful for long covid, I wonder if taking it decreases the chances of getting long covid to begin with. Given how well tolerated it is, it might be worth taking just in case.

Replies from: pktechgirl
comment by Elizabeth (pktechgirl) · 2021-10-18T05:48:22.218Z · LW(p) · GW(p)

The original protocol is here (which specifies the niacin form, a suggested dose, and some support vitamins), and my analysis of it is here. There's a comment here [LW · GW] on a study that maybe found niacin useful for acute covid, although I haven't investigated and have low confidence by default.

I think there's merit to taking nutrition seriously and stocking up on many things, but am in general wary of treating taking vitamins as a free action. Even a daily pill consumes attention, and it can be very hard to notice negative long-term effects or changes in optimal dose unless you're tracking very closely (which is a bigger attention cost). There's a very early stage start-up I'm excited about because they might make that tracking and analysis easier, but they're very far from shipping.

Replies from: Daniel_Eth
comment by Daniel_Eth · 2021-10-18T20:56:34.971Z · LW(p) · GW(p)

Thanks. I feel like for me the amount of attention for a marginal daily pill is negligibly small (I'm already taking a couple supplements, and I leave the bottles all on the kitchen table, so this would just mean taking one more pill with the others), but I suppose this depends on the person, and also the calculus is a bit different for people who aren't taking any supplements now.