What are sane reasons that Covid data is treated as reliable?

post by Valentine · 2022-01-01T18:47:38.919Z · LW · GW · 19 comments

This is a question post.

Contents

  Answers
    28 arunto
    11 ChristianKl
    8 Nicole Dieker
    7 Tornus
    3 avturchin
None
19 comments

I feel like this is one of those questions that's somehow too basic to ask. Or maybe too political. Like wondering this stuff implies I'm siding with something.

I'd really like to just set all the politics aside, simply name my ignorance, and hear some truthful answers. Because seeing this not even discussed is part of what's giving me a sense of "Fuck it, no one actually knows anything, everyone is just making random shit up that fits their social identities."

People keep talking confidently about incidence rates for different Covid variants, about their death rates and likelihood of hospitalization, how we have such clear data on vaccine efficacy and safety, etc.

But all the info streams I see look extremely dubious. I don't just mean one could doubt them. I mean, I've witnessed powerful evidence of blatant bias, and aside from brief mentions of the existence of those biases no one seems to care.

I'll give some examples below that inspire my confusion. I want to emphasize that I'm honestly asking here. Less Wrong is one of the few places (the only place?) where I feel like I can seek epistemic cleaning and clarity here.

So, example confusion sources:

I get that we can deduce something about the virus's spread based on (a) how many hospitalizations for Covid different areas are getting plus (b) some assumptions about the exponential-flavored spread of the infections. That lets us use math to peer into how many non-hospitalized infection cases there must be, and roughly how long the incubation-to-infectious timing should be.

But how in the bajeezus can anyone possibly extrapolate from there to how long it takes to become symptomatic? In need of hospitalization, sure, but how is anyone getting not-heavily-biased data on symptom strength below the hospitalization threshold?

I don't care about symptomatic-ness per se. This is just one of a ton of examples about info passing through the info commons that people — including here! — seem bizarrely (to me) super confident in despite the devastation and weaponization of said info commons.

I could believe there's some solid clear reasoning going on in the background here, grounded in hard-to-refute data, that makes all these assertions about Covid's variants and death rates and vaccine safety and so on actually quite solid. And it's just not obvious because the main social messaging is about authority and duty and "Sacrifices to the Gods" as Zvi puts it.

If so…

…could y'all help me see the sane thing?

Answers

answer by arunto · 2022-01-01T19:52:08.757Z · LW(p) · GW(p)
I've never known anyone who has been tested for a variant of any kind. I don't know anything about how variant tests look different from a generic Covid test. So where are these numbers for variant spread coming from? Maybe hospitals do have special genetic tests and reliably do those? But then isn't there going to be a pretty strong bias based on the fact that these are only for people who are getting hospitalized?

For the US there is a surveillance program run by the CDC in which each week they get specimens from all US states to be sequenced. CDC’s Role in Tracking Variants

"Specimens will ideally represent a variety of demographic and clinical characteristics and geographic locations. Selection of a diverse set of specimens will help ensure a representative set of sequences are generated for national monitoring." National SARS-CoV-2 Strain Surveillance (NS3)

I guess that for other countries there are similiar surveillance programs in place.

↑ comment by Daniel V · 2022-01-02T02:07:17.486Z · LW(p) · GW(p)

And we have VAERS, to which individuals can report directly. Plus, the surveillance system (including our crappy contact tracing systems run by the states) means we get sub-hospitalization data. Ideally contact tracing would also help arrest spread (not so much if they call you 3 days after you test positive 3 days after you first show symptoms...sheesh), but at the very least you're getting a survey done.

I think just from becoming aware of the surveillance and adverse event reporting systems, Valentine's base for a high degree of skepticism is pretty shaky. Being armed with an understanding that actually, the mechanisms by which the data could be generated DO exist should help a lot. I want to note that when people exclaim we should trust the experts, I believe it is about this level of ignorance they rightly have in mind (props for identifying a knowledge gap and honestly seeking to address it!) - lacking key fundamental knowledge necessary to even begin to assess the veracity of claims, rely on the people who do have it! As we learned from the pandemic fiasco though, our "experts" having the ability to generate and interpret that information does not mean that they always do it well. 

I'll also say that even without ongoing adverse event monitoring or observational effectiveness studies, the clinical trials were gigantic and provide strong evidence supporting efficacy and safety. [Unless the researchers were selecting what data to collect in which case seeing the raw "data" would be meaningless too. Sadly, data tampering or fabrication happen, but if that fact will undermine your reliance on any data generated by anyone but you, frankly there's no convincing you with other people's data.] I'm not sure how large any selection biases are, but I imagine they would have to be huge to impeach such extensive data vs. a handful of anecdotes that are themselves not free of selection either.

Replies from: Valentine
↑ comment by Valentine · 2022-01-05T23:12:27.098Z · LW(p) · GW(p)

Thanks.

I'd have to check, but I think it was the VAERS system that these folk were told to report to, and who turned down the data based on the circular logic I described in the OP.

But this is based on my recollection of that acronym looking familiar in this context. Don't take that too seriously. Just a little seriously.

Replies from: ChristianKl
↑ comment by ChristianKl · 2022-01-06T13:12:23.666Z · LW(p) · GW(p)

From an article about how VAERS works:

Medical experts at the Department of Health and Human Services are well aware of VAERS’ limitations. Rather than taking each individual report at face value, regulators remove clearly fraudulent reports. Demonstrating this, anesthesiologist and autism advocate James Laidler once used the system to report that a vaccine turned him into the “Incredible Hulk,” which was only removed after he agreed to have the data deleted.

It seems unlikely to me that VAERS would not take the reports. 

I think it's more likely that they went to some doctor who works with 3 minutes per patient and the doctor didn't want to take the time to fill the report.

Alternatively, maybe they tried to enter the data as nonmedical people and got told "You have to go to a doctor to have them file the data"?

answer by ChristianKl · 2022-01-01T23:54:55.081Z · LW(p) · GW(p)

I've personally known many people who have had serious medical problems that sure looked clearly like vaccine reactions.

I'm curious how that coincides with no such person reporting their experiences at https://www.lesswrong.com/posts/XnRTP4dq3dkdwwtdS/which-rationalists-faced-significant-side-effects-from-covid [LW · GW]. For me this thread was a test about whether I should look more into vaccine site-effects and the lack of reporting suggests that they aren't very common.

Are all those people you are talking about outside of the rationality community? One thing that I would see plausible is that people, who are generally psychologically suggestible and who believe the vaccine is dangerous, have their bodies overreact when they are faced with the more normal vaccine side-effects. Such a dynamic might produce more vaccine side-effects in people you know from a spiritual context than appear in the rationalist community.

One step that could be taken to verify an existing pattern of a lot of vaccine side-effects would be to simply hire a SurveyMonkey audience and see what people report when asked through that channel.

But how in the bajeezus can anyone possibly extrapolate from there to how long it takes to become symptomatic? In need of hospitalization, sure, but how is anyone getting not-heavily-biased data on symptom strength below the hospitalization threshold?

We have people we pay to do contact tracing. There are likely cases where that comes with both asking for symptoms and doing testing.

There was, best as I could tell, active (if unintentional) data destruction quite early on in the pandemic. Maybe it's still going on? Hospitals had financial incentives to find reasons why people who died had died of Covid. Lots of bodies got cremated before autopsies could happen. 

From a conversation I had with a doctor, it seems that our medical system generally does a lot fewer autopsies than we did 20 years ago. There seems to be a general culture change here. The question of how much resources our medical system should invest into doing more autopsies however isn't trivial. It would raise health care costs if we give hospitals more money if they do more autopsies. 

Maybe hospitals do have special genetic tests and reliably do those? But then isn't there going to be a pretty strong bias based on the fact that these are only for people who are getting hospitalized?

Hospitals aren't the only places that run tests besides at-home tests. The labs that do PCR testing retest some of the positive tests with variant-specific tests. Different countries have different policies about that. 

↑ comment by Valentine · 2022-01-05T23:20:14.505Z · LW(p) · GW(p)

Are all those people you are talking about outside of the rationality community?

Yep. As far as I know, but I'd be pretty surprised if any of them were here.

 

We have people we pay to do contact tracing.

Would you be willing to point at more details about this? I recall seeing a lot about how we weren't doing adequate contact tracing, but not much on how we have been.

 

From a conversation I had with a doctor, it seems that our medical system generally does a lot fewer autopsies than we did 20 years ago.

Mmm. Good to know.

Although that basically means the problem with data collection I was describing is actually a step farther up the chain. That cremation isn't where the data are getting destroyed. If they're not even bothering to verify the causes of death via autopsies and there was (is?) financial incentive to conclude "Covid"… well, I believe in incentive landscapes.

 

The labs that do PCR testing retest some of the positive tests with variant-specific tests. Different countries have different policies about that. 

Ah. And some people go through the different countries' policies and numbers and do some data crunching to extrapolate something? Okay. Who are these data crunchers then? All this is an opaque screen from where I'm standing. I just see final numbers asserted in public.

(Thank you, by the way. Gratitude for the energy you've put into answering this.)

Replies from: ChristianKl
↑ comment by ChristianKl · 2022-01-06T13:03:46.489Z · LW(p) · GW(p)

I think they are basically doing a COVID-19 test and then making the claim about the cause of death based on clinical history.

From my doctor friend, the main concern was that not doing the autopsies leads to not having good statistical data about which organs get damaged in patients dying with COVID and how that differs with new varients. That would be traditionally information that's useful for doctors who want to prevent patients from dying but it's not structured the way the modern EBM thinking about treatment goes. 

Ah. And some people go through the different countries' policies and numbers and do some data crunching to extrapolate something? 

There's a reason Zvi was focusing on Denmark [LW · GW]'s data. They have the best data.

At this point in time, I think most countries are doing samples. You don't need to test everyone to have statistically significant data.

Would you be willing to point at more details about this? I recall seeing a lot about how we weren't doing adequate contact tracing, but not much on how we have been.

Contact tracing is often getting to people to late and it's not done in an amount that you would need to track all the cases but those contact tracers are still around. (but I haven't looked into detail into the system; it's not something like the autopsy topics that I discussed with a doctor who has a good understanding of what's happening)

answer by Nicole Dieker · 2022-01-01T19:18:18.883Z · LW(p) · GW(p)

Hooooo boy.

Here is how I have been evaluating data, curious to know if other people are making judgments based on similar inputs:

  • Primary source material (CDC data tracker) is better than secondary source interpretation (CNN COVID newsfeed).
  • Small-scale primary source material, such as state or county data, is better than large-scale aggregate primary source material.
  • Secondary source interpretation can be more, not less valuable when created by a single person (as opposed to a news site) (even if that individual does not have a medical background), as an individual is more likely to look for useful information that can help them decide whether or not to take a specific action and share useful detail that explains how they came to that conclusion.
  • Assume something is lost/miscalculated/false-priored with every aggregation.
  • Assume all primary source material and all interpretations thereof are compromised for some reason (biases, incentives, etc.). Ignore actual numbers. Watch for trends.

I've also been doing a fair amount of on-the-ground evaluation, e.g.:

  • Do I know people who currently have COVID?
  • Are people around me (strangers in grocery stores, etc.) visibly ill?
  • Do I know people who have had post-vaccine health issues?
  • How does my experience correlate with the experience the data says I should be having?
↑ comment by Vaniver · 2022-01-01T23:04:05.923Z · LW(p) · GW(p)

Primary source material (CDC data tracker) is better than secondary source interpretation (CNN COVID newsfeed).

One of the points of OP to be that aggregations like the CDC data tracker are not themselves primary source material. Like, the chain goes "person provides sample" -> "sample gets processed" -> "result gets recorded locally" -> "result gets aggregated nationally", and each of those steps feels like it has some possibility for error or bias or whatever. That CNN is even further from ground seems useful to know, but doesn't tell us how connected the CDC is.

Replies from: nicole-dieker
↑ comment by Nicole Dieker (nicole-dieker) · 2022-01-02T01:04:13.355Z · LW(p) · GW(p)

Agreed (which is why I noted that county data could be more valuable than aggregated CDC data, and that nuance has the potential to be lost with every aggregation), and I spent a good 30 minutes after writing this comment asking myself if there is a better term than "primary source," which I probably used incorrectly above. 

That said, it's fair to note that I didn't actually answer the question asked, because I don't know how to determine the reliability of any given number (or any given source providing any given number). How are other people doing this?

answer by Tornus · 2022-01-02T18:03:32.487Z · LW(p) · GW(p)

I'm afraid I only have time for a short, partial response today. Short version: Covid surveillance is hard [LW · GW], and there's lots of noise in the data. But there are lots of smart people working hard on this, and in the aggregate we actually have a pretty good idea what's going on.

I'll address one of the questions you asked specifically:

So where are these numbers for variant spread coming from? Maybe hospitals do have special genetic tests and reliably do those? But then isn't there going to be a pretty strong bias based on the fact that these are only for people who are getting hospitalized?

In Washington, much of the variant prevalence data comes from UW, which sequences a subset of the samples they receive. This is a bit complicated: some samples are fully sequenced, and some are tested for S-Gene Target Failure, which is a faster, easier test that is a fairly good (but not perfect) proxy for Omicron vs Delta. The UW sequencing is a good but not perfect sample of what's actually happening in Washington. For details on this project, the person to follow is Pavitra Roychoudhury. Details vary, but there are multiple other institutions with largely similar programs.

More general answer: you're asking good questions. They are all important, and they're obvious to any smart person who thinks about the issue for a moment. Although I don't have time to answer them all, I assure you that the smart people working on Covid have thought of every single one of your questions, and have good answers to every single one. Many of the answers are in Zvi's excellent series of Omicron updates.

↑ comment by lsusr · 2022-01-03T05:14:26.786Z · LW(p) · GW(p)

I live near the UW. As far as I can tell, the UW has done a great job of pandemic response. I got a COVID test from them early in the pandemic before there were alternative tests available.

↑ comment by Valentine · 2022-01-05T23:21:59.720Z · LW(p) · GW(p)

Thank you. This is clear and points me in directions that let me explore more and see through the fog of war.

answer by avturchin · 2022-01-01T19:07:17.012Z · LW(p) · GW(p)

There is a correlation between several types of reported data and the real situation. Raw data is not very reliable if we don't account for biases of all kinds. For example, there is an a known difference between reported deaths and excess mortality, which is often 2-3 times larger.

Anyway, I am disturbed by your words about inability to report adverse effects to vaccines. It should not be this way.

19 comments

Comments sorted by top scores.

comment by Jayson_Virissimo · 2022-01-01T23:06:21.334Z · LW(p) · GW(p)

I've personally known many people who have had serious medical problems that sure looked clearly like vaccine reactions.

I don't consider it a "serious medical problem", but I attempted to report (via the phone number on the paperwork given me by the person that administered the shot at Wallgreens) my 48 hours long migraine + ~4 day long high blood pressure (as measured by my Omron home blood pressure monitor) after getting a Pfizer booster. I was told they don't need me to fill anything out because those are already known side-effects.

Searching Google for "does covid vaccine cause high blood pressure" just now returned Nebraska Medicine FAQ page as the first result with the following answer:

So far, no data suggests that COVID-19 vaccines cause an increase in blood pressure.

WTF...

Replies from: tivelen, tslarm, tivelen
↑ comment by tivelen · 2022-01-02T01:52:14.177Z · LW(p) · GW(p)

I searched the CDC's Vaccine Adverse Event Reporting System (VAERS) and there are 474 reported cases of abnormal blood pressure following COVID-19 vaccination. Looking further in the Google search, I found a study (n = 113) which indicated increased risk of high blood pressure after vaccination, especially after previous infection.

Plainly, not everyone in the healthcare system is on the same page about side effects. I'd err on the side of the Walgreens person you talked to being more accurate, given that high blood pressure is a known side effect. Not known by that Nebraska Medicine doctor, apparently.

↑ comment by tslarm · 2022-01-03T03:13:42.528Z · LW(p) · GW(p)

If you don't mind sharing, what vaccine(s) did you have for your previous doses, and what side effects did they cause? And had you had a previous covid infection? 

(I'm wondering on behalf of a family member who is prone to awful migraines, and who I think has high blood pressure. Sorry about your symptoms by the way, I know a two-day migraine can be much more unpleasant than most people probably realise.)

Replies from: Jayson_Virissimo
↑ comment by Jayson_Virissimo · 2022-01-03T04:42:31.258Z · LW(p) · GW(p)

Will DM with info.

↑ comment by tivelen · 2022-01-01T23:14:23.041Z · LW(p) · GW(p)

Who exactly told you that?

Replies from: Jayson_Virissimo
↑ comment by Jayson_Virissimo · 2022-01-01T23:30:31.989Z · LW(p) · GW(p)

Will DM you the number.

comment by Elizabeth (pktechgirl) · 2022-01-02T05:03:54.752Z · LW(p) · GW(p)
  • I've personally known many people who have had serious medical problems that sure looked clearly like vaccine reactions. On par with "Well now I can't get out of bed and can't think anymore" or "Oh shit, heart attack" kinds of reactions. But all these people I've known who tried to report their reactions were told "No, your reaction can't have been due to the vaccine, because the vaccine is safe and effective." I've heard lots of similar reports. Because this is about rejecting data collection, I don't see how anyone could possibly know how common/rare this is.

 

When I went to urgent care for vaccine side effects in June, they immediately accepted the explanation, said it was really common, and kept checking/pushing for more symptoms than I had. 

My report was also accepted by VAERS, although who knows if that does anything.

comment by lsusr · 2022-01-01T22:10:42.847Z · LW(p) · GW(p)

When I first got vaccinated I signed up for a symptom reporting system that bugged me periodically to ask how I'm feeling. It seemed to me like the obvious way to monitor the population for vaccine effectiveness, vaccine side effects and symptomatic infection of the vaccinated population. When my second dose of the vaccine produced mild side effects I reported then via the automated survey.

A random walk-by testing survey station tested me (and anyone else who agreed) in a park. It seemed to me like a reasonable way to measure the relative ratio of asymptomatic variants.

Replies from: Valentine
↑ comment by Valentine · 2022-01-05T23:25:48.165Z · LW(p) · GW(p)

Yep, that does seem reasonable.

Several of the people I talked to or indirectly listened in on said they'd been given a number to call if they got any side effects. Then when they got side effects and called, they were given the "The vaccine is safe so this must be something else" line.

Clearly that's not everyone's experience. But since I don't know the structure these people encountered in almost any detail, my net emotional update was "Fuck this 'data'."

comment by SarahNibs (GuySrinivasan) · 2022-01-01T19:20:48.600Z · LW(p) · GW(p)

How many people do you know? What rate are we talking, re: "many people"?

I've personally known many people who have had serious medical problems that sure looked clearly like vaccine reactions.

I don't know a lot, but either you know simply a ton of people, or at least one of us experienced a fairly outlier event, or something else is going on (bad batch of vaccines? keeping quiet about reactions? risk factors?). I can count about 50 people whose serious medical problems would absolutely have been communicated to me had they occurred, none of whom had one. Mostly 30s-40s.

Replies from: Dagon, Valentine
↑ comment by Dagon · 2022-01-01T22:46:38.593Z · LW(p) · GW(p)

I figure my extended circle (including 2nd and 3rd degree connections who I've met or heard some detailed story about) is on the order of 10K people, spanning ages from young kids (mostly children or grandchildren of friends) to quite old (parents and grandparents of acquaintances).  I've heard plenty of reports of unpleasant vaccine reactions (including DAYS of downtime), and one or two where the reaction was bad enough that their doctor told them not to have the second shot.   ZERO that I'd call "serious medical problems".  

I'm aware of my bubble - this group is very strongly biased to educated wealthy(ish) Americans.  It does include people with chronic health problems, diagnosed and un-.  But I'd be shocked if there's a cluster where "many people" that someone knows were significantly harmed by the vaccines.  

I'm NOT shocked that reporting of such things is suspect - there are incentives (in both directions) to report, and it's really hard to be sure even in an individual case.  

↑ comment by Valentine · 2022-01-05T23:33:55.041Z · LW(p) · GW(p)

How many people do you know? What rate are we talking, re: "many people"?

I don't think I can give very useful data here. I can give some rough numbers but they aren't going to be very informative. I stopped bothering to listen to or look for reports of people's vaccine side effects getting rejected after something like ten-ish because I was starting to notice something like overfitting going on in my head.

The important (to me) part was that there were multiple such cases, very distributed, which meant there's some kind of bureaucratic mechanism in place (as opposed to one grumpy bureaucrat somewhere). I knew I couldn't see it, and I observed that no one seemed to be talking about it (except the disgruntled vaccine-injured folk who were feeling swayed by the conspiracy theorists), which made the confidence folk were asserting about "The vaccines are safe & effective" look like mindless propaganda repetition to me even if it accidentally happened to be correct.

I was hoping for an update on that here. I've gotten quite a few others. Sadly on this one I'm not seeing much in the way of hope for clarification just yet.

comment by jimrandomh · 2022-01-04T07:37:37.425Z · LW(p) · GW(p)

Yes, it really is that bad. It was already that bad even before the pandemic and before things got politicized.

I recall once at a (2018) party, a friend told an anecdote about having a weird and serious side effect from a common, old psychiatric medication, which wasn't a listed side effect. A psychiatrist at the party said they had a patient who had the same side effect from the same drug, and had looked into it at the time and thought that wasn't supposed to happen, and asked if he could share the anecdote. I asked if he had ever tried reporting events like that, and how that worked. He said that there was theoretically a system for doing so, but he had never used it, nor had his colleagues, and it wasn't a thing in practice.

Against that backdrop, you have a vaccine adverse event reporting system, and a society polarized into a group that wants to glorify them and a group that wants to demonize them. A the same time, side effect rates are (I think, based on super sparse data) very different between different age demographics, and between people who have had COVID recently and who haven't, and between the different vaccines. On balance I think it's still quite clear that getting vaccinated is better than getting COVID, and getting vaccinated and getting reduced-severity breakthrough COVID is better than getting full-severity COVID, but most institutions seem to be choosing "downplay side effects as much as possible" rather than "honestly talk about the balance of considerations".

(That said, not all of the things you point at are actually that bad. Estimating the fraction of cases which are of a variant is easy, much easier than estimating the number of cases in the population as a whole, because all you have to do is set up a sequencer in a lab that was already doing PCR tests.Estimating the exposure-to-infectious time period is not as trivial, but the serial interval is a good substitute metric, and that can be easily estimated by asking positive cases when they think they were exposed, and taking the subset who seem confident in the exposure date.)

comment by NunoSempere (Radamantis) · 2022-01-08T12:23:36.333Z · LW(p) · GW(p)

This paper on the PNAS might be of interest: Epidemic tracking and forecasting: Lessons learned from a tumultuous year, particularly the first section, "Deceptively Simple Data Labels Often Belie the Data’s True Meaning and Complexity", which might be academic-speak for "fuck this data".

comment by 1a3orn · 2022-01-01T19:41:41.990Z · LW(p) · GW(p)

Right now a model I'm considering is that the C19 vac, at least for a particular class of people (males under 30? 40?) has zero or negative EV, and mostly shifts risk from the legible (death from c19) to the illegible (brain fog? general systematic problems the medical system does not know how to interpret!) Where "legible" is legible in the seeing-like-a-state sense.

I'm mostly motivated, again, by the same thing as you. It seems like there's an incredible disproportion between the bad side effects among my friend group, and the bad side effects I should be hearing about if serious side-effects are really as rare as they're supposed to be.

To add to your data collection problem: There's so much more effort being put into collecting info about long-term side effects of the virus than the vaccine. We've studies look at how much C19 influences intelligence, which is a hard question. But how many have we had on the vaccine and intelligence? And so on for X, Y, Z.

Replies from: gbear605
↑ comment by gbear605 · 2022-01-09T03:41:29.651Z · LW(p) · GW(p)

This is very anecdotal, but I know a number of young people who had COVID and a number of young people who got the vaccine. Of the ones who had COVID, about half report continuing brain fog and thinking issues - one person went from a 99th percentile score on the PSAT to a 80th percentile score on the SAT. Of the ones who got the vaccine (including one who got the vaccine and booster shot after previously having COVID) there have been no noticed issues with thinking/intelligence.

comment by tivelen · 2022-01-01T21:56:25.958Z · LW(p) · GW(p)

I'm wondering what the details of your friends reporting attempts are. Who exactly did they talk to? VAERS is the official U.S. reporting system, what were their experiences with that? If there is an underreporting problem, we need as many specifics as we can get to combat it. Given that some vaccines do have well-known side effects among certain demographics, lots of people have been able to report their side effects successfully. We would need to figure out why your friend group has been far less successful to correct the issue.

Without an explicit probability calculation, how exactly are we supposed to determine what the levels of side effects in reality are, vs what the medical data that has been collected and reported suggests, vs what the average person thinks is true? Perhaps all are biased and/or untrustworthy. I'm not sure where we can go from there. Has personal testimony from our own social groups become the best we can do?

Replies from: Valentine
↑ comment by Valentine · 2022-01-05T23:45:55.575Z · LW(p) · GW(p)

Has personal testimony from our own social groups become the best we can do?

Sadly yes, at least on my side.

I think your questions are very sane. Sadly I'm not the person to do this kind of data collection. The way some people have the opposite of a green thumb when it comes to plants, I have something like that for putting together numerically focused models. As soon as I move away from geometry or contact with physical reality, errors like 2+3=6 dominate and my models' output becomes gobbledegook. I was astoundingly good at geometry and utter garbage at algebra in math grad school.

I think most of the people I'm referring to were pointed at VAERS. This was from months ago, buried in old Facebook threads, so it'd take quite a bit of digging to find and I'm not sure I could. So this is based on a fuzzy impression of seeing that acronym in that context. But I do recall many of them were given a hotline number to call if they got side effects, and in calling the number they got the "Well, the vaccines are safe, so these must be from something else" line.

Without an explicit probability calculation, how exactly are we supposed to determine what the levels of side effects in reality are, vs what the medical data that has been collected and reported suggests, vs what the average person thinks is true?

Yep. This has been part of my problem. I'm living in a sea of vastly deeper uncertainty than the people around me seem to think they're in. I'm hoping to do slightly better than either of "No one knows anything and anyone who claims otherwise is deluded" or "My tribe is right." I've just been having a lot of trouble finding that alternative.

(…and this discussion is helping.)

Replies from: tivelen
↑ comment by tivelen · 2022-01-06T15:56:20.165Z · LW(p) · GW(p)

I have a hypothesis that seems to fit the data. These numbers are not given out for the purpose of collecting data on vaccine side effects (that's what VAERS is for). They are intended to provide specialized medical care directed at those who have recently gotten vaccines.

Evidence:
One commenter reported calling a Walgreens number. If this is representative, these are local pharmacy/medical practice numbers that people are calling, not some national reporting service.

Reassurance is one of the jobs of a anyone providing medical care. "Even though you aren't feeling well after the treatment, you have nothing to worry about, the treatment is safe." is exactly what I would want someone to say if there was nothing either of us could do to help matters, especially if I was worried enough to call. You are especially likely to do so if you personally believe the vaccine is save (which is very likely for someone responding to such a number). If I was simply recording side effects, I wouldn't bother with that. Y

If you already believe the side effect is caused by the vaccine and think it's a very big deal, and then during the call they try to give the reassurance, you will instead distrust them, and also want to report their untrustworthiness to friends.

If you never call the number because you are not worried, or you do trust them, you have nothing notable to report. This would explain why every report looks like a reassurance that fell flat. Your sample is biased strongly towards looking exactly that way, regardless of how common side effects or the "there are no side effects" line actually is.

And all that assumes that this game of telephone, chaining between the medical establishment, the people taking the calls, your friends reporting the call, and then your fuzzy recollection, didn't distort any of the data.

Currently, this "explains" your data for me. As in, I am no longer confused about your reports about your friends. I understand what happened, I think. There is no data collection rejection involved, at least not related to these calls.

Do you doubt this hypothesis? If so, what evidence could you provide against it? What evidence would we need to collect to figure out whether the hypothesis is true?

I would expect that if one called such a number, one could confirm that the other person is doing no data collection about the likelihood of side effects, that the line in context is intended for reassurance if it comes up, and the entire call will otherwise be completely in line with providing post-vaccine medical care. Averaging across multiple calls, of course.

If I'm wrong, I would expect that getting a full description of an entire call would show that the line in question is used as a shutdown, side effects are not being recorded (but they are supposed to be recorded every time according to the rules of the job), there is no reasonable medical triage going on, and the numbers in question are intended purely to advocate for vaccine safety. Also averaging across multiple calls.