How much should you update on a COVID test result?

post by mayleaf · 2021-10-17T19:49:42.001Z · LW · GW · 43 comments

Contents

  Background
    Using Bayes factors
  List of COVID tests with Bayes factors
    Rapid Antigen Test
      Are you symptomatic?
      Rapid antigen tests: if you don't have symptoms
      Rapid antigen tests: if you have symptoms that developed <1 week ago
      Rapid antigen tests: if you have symptoms that developed >1 week ago
      The Abbot BinaxNow At-Home Test
    Nucleic Acid Amplification Test (NAAT)
      All Rapid NAATs
      Cepheid Xpert Xpress Molecular Test
      Abbot ID Now Molecular Test
    Summary of all tests
  Caveats about infectiousness
  What if I take multiple tests?
  Use with microCOVID
  Acknowledgements 
  References
None
44 comments

This is a writeup of COVID test accuracies that I put together for my own interest, and shared with friends and housemates to help us reason about COVID risk. Some of these friends suggested that I post this to LessWrong. I am not a statistician or an expert in medical research.

Background

We often hear that some kinds of COVID tests are more accurate than others — PCR tests are more accurate than rapid antigen tests, and rapid antigen tests are more accurate if you have symptoms than if you don't. A test's accuracy is often presented as two separate terms: sensitivity (what proportion of diseased patients the test accurately identifies as diseased) and specificity (what proportion of healthy people the test accurately identifies as healthy). But it's not obvious how to practically interpret those numbers: if you test negative, what does that mean about your odds of having COVID?

This writeup attempts to answer to the question, "how much more (or less) likely am I to have COVID given a positive (or negative) test result?" In particular, this is an attempt to calculate the Bayes factor for different types of COVID test results.

The Bayes factor is a number that tells you how much to update your prior odds of an event (in this case, your initial guess at how likely someone is to have COVID) given some piece of new evidence (in this case, a test result). It's calculated based on the test's sensitivity and specificity. If a test has a Bayes factor of 10x for a positive test result, and you test positive, then you should multiply your initial estimated odds of having COVID by 10x. If the same test has a Bayes factor of 0.3x for a negative test result, and you test negative, then you should update your prior odds of having COVID by 0.3x.

Using Bayes factors

(For an excellent explanation of Bayes factors and the motivation behind using them to interpret medical tests, I highly recommend this 3Blue1Brown video, which inspired this post.)

There's a well-known anecdote where doctors in a statistics seminar were asked how they would interpret a positive cancer test result for a routine mammogram taken by an asymptomatic patient. They were told that the test has a sensitivity of 90% (10% false negative rate), a specificity of 91% (9% false positive rate), and that the base rate of cancer for the patient's age and sex is 1%. Famously, nearly half of doctors incorrectly answered that the patient had a 90% probability of having cancer. [1] The actual probability is only 9%, since the base rate of cancer is low in the patient's population. One important lesson from this anecdote is that test results are an update on your priors of having the disease; the same positive test result implies different probabilities of disease depending on the disease's base rate.

Bayes factors help make it easy to make this update. A test's Bayes factor is a single number that, when multiplied by your prior odds, gives you your posterior odds. For a COVID test, you can start with your initial estimate of how likely you are to have COVID (based on the prevalence in your area combined with your vaccination status, or your current number of microCOVIDs) and update from there.

To calculate the Bayes factor for a negative COVID test, you take the probability that you'd test negative in the world where the you do have COVID and divide it by the probability that you'd test negative in the world where the you do not have COVID. Expressed mathematically:

$$\text{Bayes factor}(-)=\frac{p(-|\text{COVID})}{p(-|\text{no COVID})}=\frac{\text{false negative rate}}{\text{true negative rate}}=\frac{1-\text{sensitivity}}{\text{specificity}}$$

Similarly, the Bayes factor for a positive COVID test is the probability of a positive result in the world where the you do have COVID, divided by the probability of a positive result in the world where the you do not have COVID.

$$\text{Bayes factor}(+)=\frac{p(+|\text{COVID})}{p(+|\text{no COVID})}=\frac{\text{true positive rate}}{\text{false positive rate}}=\frac{\text{sensitivity}}{\text{1 - specificity}}$$

To interpret the test result, express your prior probability of having COVID as an odds, and then multiply those odds by the Bayes factor. If you initially believed you had a 10% chance of having COVID, and you got a negative test result with a Bayes factor of 0.1x, you could multiply your prior odds (1:9) by 0.1 to get a posterior odds of 0.1:9, or about 1%.

List of COVID tests with Bayes factors

Below are my calculations for the Bayes factors of rapid nucleic acid amplification tests (which includes rapid PCR tests) as well as rapid antigen tests (the type available for home use in the US). I used sensitivity and specificity estimates from a Cochrane metastudy on rapid tests [2] initially published in August 2020 and last updated in March 2021. 

Rapid Antigen Test

This is a test for fragments of SARS-Cov-2 protein [3]. It's typically administered via nasal swab, is available to purchase in the US as at-home test kits, and can be very quick (15 minutes for some brands). It has lower sensitivity (aka more false negatives) than most nucleic acid tests.

Are you symptomatic?

The Cochrane metastudy reviewed 3 brands of rapid antigen test (Coris Bioconcept COVID-19 Ag, Abbot Panbio COVID-19 Ag, and SD Biosensor Standard Q COVID-19 Ag) and found that the sensitivity of all these tests were notably higher for symptomatic patients compared to patients with no symptoms. They also found that these tests were most sensitive within the first week of developing symptoms.

The review's estimates for sensitivity were:

• No symptoms: 58.1%  (95% CI 40.2% to 74.1%)
• Symptomatic, symptoms first developed <1 week ago: 78.3% (95% CI 71.1% to 84.1%)
• Symptomatic, symptoms first developed >1 week ago: 51.0% (95% CI 40.8% to 61.0%)

The review found that specificity was similar across all patients regardless of symptom status — about 99.6% (95% CI 99.0% to 99.8%).

Rapid antigen tests: if you don't have symptoms

• Estimated Bayes factor for a negative result: about 0.4x ($\frac{1-0.581}{0.996}\approx 0.42$)
• Estimated Bayes factor for a positive result: about 145x ( $\frac{0.581}{1-0.996}\approx 145$ )

So, if you got a negative result, you can lower your estimated odds that you have COVID to 0.4x what they were before. If you got a positive result, you should increase your estimated odds that you have COVID to 145x what they were before.

Rapid antigen tests: if you have symptoms that developed <1 week ago

• Estimated Bayes factor for a negative result: about 0.2x ($\frac{1-0.783}{0.996}\approx 0.22$)
• Estimated Bayes factor for a positive result: about 196x ( $\frac{0.783}{1-0.996}\approx 196$ )

So, if you got a negative result, you can lower your estimated odds that you have COVID to 0.2x what they were before. If you got a positive result, you should increase your estimated odds that you have COVID to 196x what they were before.

Rapid antigen tests: if you have symptoms that developed >1 week ago

• Estimated Bayes factor for a negative result: about 0.5x ( $\frac{1-0.510}{0.996}\approx 0.49$ )
• Estimated Bayes factor for a positive result: about 128x ( $\frac{0.510}{1-0.996}\approx 128$ )

So that if you got a negative result, you can lower your estimated odds that you have COVID to 0.5x what they were before. If you got a positive result, you should increase your estimated odds that you have COVID to 128x what they were before.

The Abbot BinaxNow At-Home Test

Update: @Tornus has posted a detailed writeup of the BinaxNow test here: Rapid antigen tests for COVID [LW · GW]

Unfortunately the Cochrane metastudy didn't include data for the Abbot BinaxNOW at-home test, which I was particularly interested in because it's the most common at-home test in the US, and is the test my household uses most frequently. I've seen a few sources (e.g. [4]) that claim that the Abbott BinaxNOW test is slightly more sensitive and about as specific than the Abbott Panbio Ag test which was reviewed by the Cochrane metastudy, so it's possible that this test has a slightly higher predictive power than the ones reviewed above.

Nucleic Acid Amplification Test (NAAT)

This test looks for viral RNA from the SARS-Cov-2 virus [3]. It is typically administered via nasal swab. It's also called a "nucleic acid test" or "molecular test". PCR tests are a type of NAAT. The Cochrane metastudy indicated that sensitivity and specificity differed by brand of test. 

All Rapid NAATs

If you got a rapid NAAT but don't know what brand of test it was, you could use these numbers, which are from the initial August 2020 revision of the Cochrane metastudy. This version analyzed data from 11 studies on rapid NAATs, and didn't break up the data into subgroups by brand. They calculated the average sensitivity and specificity of these tests to be:

• Sensitivity: 95.2% (95% CI 86.7% to 98.3%)
• Specificity: 98.9% (95% CI 97.3% to 99.5%)
• Estimated Bayes factor for a negative result: about 0.05x ($\frac{1-0.952}{0.989}\approx 0.05$)
• Estimated Bayes factor for a positive result: about 87x ($\frac{0.952}{1-0.989}\approx 87$)

So if you get a negative test result, you can lower your estimated odds of having COVID to 0.05 times what they were before. If you got a positive result, you should increase your estimated odds that you have COVID to 87x what they were before.

Cepheid Xpert Xpress Molecular Test

This is an RT-PCR test [5]. The March 2021 revision of the Cochrane metastudy included a separate analysis for this brand of test.

EDIT:  @JBlack points out in the comments [LW(p) · GW(p)] that the metastudy only included 29 positive COVID cases (out of 100 patients total) for this test, which is a low enough sample size that the below calculations may be significantly off. 

• Sensitivity: 100% (95% CI 88.1% to 100%)
• Specificity: 97.2% (95% CI 89.4% to 99.3%)
• Estimated Bayes factor for a negative result: very very low?
  If we use the Cochrane study's figures for sensitivity and specificity, we get$\frac{\text{false negative rate}}{\text{true negative rate (specificity)}}=\frac{1-1.00}{0.972}=0$ 
  
  If the sensitivity is actually 100%, then we get a Bayes factor of 0, which is weird and unhelpful — your odds of having COVID shouldn't go to literally 0. I would interpret this as extremely strong evidence that you don't have COVID, though (EDIT: although with a positive case count of only 29 COVID cases, perhaps these numbers aren't that meaningful). I'd love to hear from people with a stronger statistics background than me if there's a better way to interpret this.
• Estimated Bayes factor for a positive result: about 36x ($\frac{1.00}{1-0.972}\approx 36$)

So if you get a positive test result, your estimated odds of having COVID is increased by a factor of 36.

Abbot ID Now Molecular Test

This is an isothermal amplification test [5]. The March 2021 revision of the Cochrane metastudy included a separate analysis for this brand of test. 

• Sensitivity: 73.0% (95% CI 66.8% to 78.4%)
• Specificity: 99.7% (95% CI 98.7% to 99.9%)
• Estimated Bayes factor for a negative result: about 0.3x ($\frac{1-0.732}{0.997}\approx 0.27$)
• Estimated Bayes factor for a positive result: about 244x ($\frac{0.732}{1-0.997}\approx 243$)

So if you get a negative test result, you can lower your estimated odds of having COVID to 0.3 times what they were before. If you got a positive result, you should increase your estimated odds that you have COVID to 244x what they were before.

I was surprised to see how different the accuracies of Abbott ID Now and Cepheid Xpert Xpress tests were; I'd previously been thinking of all nucleic acid tests as similarly accurate, but the Cochrane metastudy suggests that the Abbott ID Now test is not meaningfully more predictive than a rapid antigen test. This is surprising enough that I should probably look into the source data more, but I haven't gotten a chance to do that yet. For now, I'm going to start asking what brand of test I'm getting whenever I get a nucleic acid test.

Summary of all tests

Test	Bayes factor for negative result	Bayes factor for positive result
Rapid antigen test, no symptoms	0.4x	145x
Rapid antigen test, symptoms developed <1 week ago	0.2x	196x
Rapid antigen test, symptoms developed >1 week ago	0.5x	128x
Rapid NAAT, all brands	0.05x	87x
Rapid NAAT: Cepheid Xpert Xpress	probably very low, see calculation	36x
Rapid NAAT: Abbot ID Now	0.4x	243x

Caveats about infectiousness

From what I've read, while NAATs are highly specific to COVID viral RNA, they don't differentiate as well between infectious and non-infectious people. (Non-infectious people might have the virus, but in low levels, or in inactive fragments that have already been neutralized by the immune system) [6] [7]. I haven't yet found sensitivity and specificity numbers for NAATs in detecting infectiousness as opposed to illness, but you should assume that the Bayes factor for infectiousness given a positive NAAT result is lower than the ones for illness listed above.

Relatedly, the sensitivity of rapid antigen tests is typically measured against RT-PCR as the "source of truth". If RT-PCR isn't very specific to infectious illness, then this would result in underreporting the sensitivity of rapid antigen tests in detecting infectiousness. So I'd guess that if your rapid antigen test returns negative, you can be somewhat more confident that you aren't infectious than the Bayes factors listed above would imply.

What if I take multiple tests?

A neat thing about Bayes factors is that you can multiply them together! In theory, if you tested negative twice, with a Bayes factor of 0.1 each time, you can multiply your initial odds of having the disease by $(0.1{)}^2=0.01$.

I say "in theory" because this is only true if the test results are independent and uncorrelated, and I'm not sure that assumption holds for COVID tests (or medical tests in general). If you get a false negative because you have a low viral load, or because you have an unusual genetic variant of COVID that's less likely to be amplified by PCR*, presumably that will cause correlated failures across multiple tests. My guess is that each additional test gives you a less-significant update than the first one.

*This scenario is just speculation, I'm not actually sure what the main causes of false negatives are for PCR tests.

Use with microCOVID

If you use microCOVID.org to track your risk, then you can use your test results to adjust your number of microCOVIDs. For not-too-high numbers of microCOVIDs, the computation is easy: just multiply your initial microCOVIDs by the Bayes factor for your test. For example, if you started with 1,000 microCOVIDs, and you tested negative on a rapid NAAT with a Bayes factor of 0.05, then after the test you have $1000\cdot 0.05=50$ microCOVIDs.

The above is an approximation. The precise calculation involves converting your microCOVIDs to odds first:

1. Express your microCOVIDs as odds:
   1,000 microCOVIDs → probability of 1,000 / 1,000,000 → odds of 1,000 : 999,000
2. Multiply the odds by the Bayes factor of the test you took. For example, if you tested negative on a rapid nucleic acid test (Bayes factor of 0.05):
   1,000 / 999,000 * 0.05 = 50 / 999,000
3. Convert the resulting odds back into microCOVIDs:
   odds of 50 : 999,000 → probability of 50 / 999,050 ≈ 0.00005 ≈ 50 microCOVIDs

But for lower numbers of microCOVIDs (less than about 100,000) the approximation yields almost the same result (as shown in the example above, where we got "about 50 microCOVIDs" either way).

Acknowledgements 

Thank you to swimmer963 [LW · GW], gwillen [LW · GW], flowerfeatherfocus [LW · GW], and landfish [LW · GW] for reviewing this post and providing feedback.

References

[1] Doctor's don't know Bayes theorem - Cornell blog

[2] Rapid, point‐of‐care antigen and molecular‐based tests for diagnosis of SARS‐CoV‐2 infection - Cochrane Reviews

[3] Which test is best for COVID-19? - Harvard Health

[4] Performance and Implementation Evaluation of the Abbott BinaxNOW Rapid Antigen Test in a High-Throughput Drive-Through Community Testing Site in Massachusetts - Journal of Clinical Microbiology

[5] EUAs - Molecular Diagnostic Tests for SARS-CoV-2 - US Food & Drug Administration

[6] Nucleic Acid Amplification Testing (e.g. RT-PCR) - Infectious Disease Society of America 

[7] Antigen tests as contagiousness tests - rapidtests.org 

43 comments

Comments sorted by top scores.