For mRNA vaccines, is (short-term) efficacy really higher after the second dose?

post by Sam Marks (samuel-marks) · 2021-04-25T20:21:59.349Z · LW · GW · 7 comments

This is a question post.

Contents

  Why might we think that second doses don't boost efficacy?
  Why might we think that second doses do boost efficacy?
None
  Answers
    8 Gerald Monroe
None
7 comments

For the Pfizer and Moderna vaccines, efficacy is commonly reported as something high (like 95%) after two doses, and something lower (in the 50-80% range) after only one dose. However, I think there's good reason to doubt that this 50-80% number is capturing the right thing, and there's good reason to believe that the mRNA vaccines are ~95% effective starting 12 days after the first dose. This is important because if your immunity is as high 12 days post-first-dose as it will be 12 days post-second-dose, then any precautions that you are waiting until you are fully vaccinated to get rid of can instead be shed 3-4 weeks earlier.

(Note that getting a second dose might still cause your immunity to wane more slowly in the long term; that is, second doses might still increase long-term efficacy.)

Overall, my current best guess, with confidence ~65%, is that the mRNA vaccines are equally effective 12 days after the first dose as they are 12 days after the second dose.

Below I'll explain the cases for and against the claim that the second dose boosts immunity. In short, the phase 3 data strongly suggests that it doesn't, and real-world data from Israel strongly suggests that it does. I'm posting this as a question because I hope that someone who knows more about this, or who has seen data that I haven't seen, can help figure this out.

Why might we think that second doses don't boost efficacy?

The short answer: because this is what the data from the phase 3 trials straightforwardly says. Here are the key graphs; note the lack of apparent effect from the second dose.

Incidence of Covid cases in the Pfizer phase 3. Note that dose 2 was given on day 21. Source.
From the Moderna phase 3. Dose 2 was given on day 28. Source.

So what's up with the commonly reported 50-80% post-first-dose efficacy? The issue is that this number captures efficacy starting on the day you receive the vaccine (or sometimes 7 days later). As you can see from the graphs, is too soon for the vaccine to have any effect at all. Rather, immunity starts appearing in the data starting sharply on day 12, so what we really want to know is the efficacy in the time period between day 12 and the second dose.

(Note that since there's an incubation period between exposure and developing symptoms, the efficacy actually starts before day 12, perhaps around day 7. Shouldn't there be some noise around the day the vaccine kicks in due to varying incubation periods? I would think so, but we don't see that here.)

Frustratingly, the phase 3's don't report this number. But using some data included in the Pfizer phase 3, I was able to make this graph:

(They only give data binned by week, so I can't make this graph on a day-by-day basis. Also, it gets noisier as the trial continues because the number of participants dropped, so that oscillation after 70+ days is probably nothing real. Alas, I can't find the data in the Moderna phase 3 necessary to make this graph for Moderna.)

 So based on the phase 3 data, it really doesn't look like the second dose does anything to boost immunity.

Why might we think that second doses do boost efficacy?

Because that seems to be what Israeli data is saying. Here is the key graph:

Efficacy of Pfizer vaccine against symptomatic Covid in Israel. There is also data for documented Covid cases (whether symptomatic or not), but the phase 3 data above was for symptomatic Covid, so I'm using this. Source.

Unlike the graphs above, it actually does take some time for the blue line to flatten out. This is not just an illusion; see here:

Efficacy is 1 - RR. Remember, we're looking at the symptomatic column. Source (same as above).

It's important to note that the Israeli trial is about 30x larger than the Pfizer phase 3 (N = 43K vs N = 1.2 million). 

But it's equally important to note that the the phase 3's were true randomized controlled trials - the placebo group was given a fake injection and everything - whereas the Israeli studies were observational. So it's possible that something funky is going on, like that once you're vaccinated (and you know you're vaccinated) you go out partying in Tel Aviv and get Covid at higher rates than whichever non-vaccinated control person the study matched you with; then over time the excitement wears off and you start going out partying at close to population base rates.

One thing I'm uncertain about is how to weight the relative evidence from the phase 3's and the Israeli data, given that there is much more Israeli data but the phase 3's are RCTs (and are still quite large). For now, I'm trusting the RCTs slightly more, so my 65% confident guess is that the second dose does not boost your immunity.

But I would love to hear from others.

Sources: Pfizer pfase 3, Moderna phase 3, Israeli data

Answers

answer by Gerald Monroe · 2021-04-25T21:22:02.848Z · LW(p) · GW(p)

Note that the second dose serves another purpose.  The mRNA vaccines are fragile, something might have gone wrong with dose 1.  

comment by ChristianKl · 2021-04-26T08:23:13.098Z · LW(p) · GW(p)

Do you have a source for the claim that this is a relevant consideration for the makers of the vaccine? 

If you live in a place with a decent healthcare system that can manage cold chains I would be surprised if a significant amount of mRNA doses get messed up.

Replies from: gerald-monroe
comment by Gerald Monroe (gerald-monroe) · 2021-04-27T05:38:01.784Z · LW(p) · GW(p)

So there's 2 different aspects here: 

        a. At a population level, should the healthcare system prioritize first shots or both shots.  And obviously that's first shots, the chance of the cold chain failing, etc is low. 

       b.  Am I going to personally risk my life based on the math in the OP?  The answer is I wouldn't, so I got the second shot before leaving quarantine.  This is because even if the chance of a messup is tiny it's no consolation if I happen to be the victim.  (while the risk of 2 small independent events happening in series would be the event squared, so if the chance of a mistake is 1/500 it would be (1/500)*(1/500), or small enough that its not a risk worth considering as the risk of a vaccine breakthrough is much higher)

Replies from: ChristianKl
comment by ChristianKl · 2021-04-27T16:25:01.717Z · LW(p) · GW(p)

Not getting the shoot because you don't trust the calculation in the OP is completely different then not getting the shot because not trusting cold chain messups.

1/500 is lower then the uncertainty in the effect size of the vaccine. 

Replies from: gerald-monroe
comment by Gerald Monroe (gerald-monroe) · 2021-04-27T23:08:16.547Z · LW(p) · GW(p)

I was saying you should get the second shot just in case your immune system didn't develop a robust response the first time (biology is stochastic and not a digital system) or because the cold chain or manufacturing chain rushing the shots had an error.

Even if on paper the gain is either 0 or marginal.

For administrators planning vaccinations of a populance, of course the evidence strongly says first doses first, you shouldn't even start with second does until all willing individuals have their first.

comment by Zac Hatfield-Dodds (zac-hatfield-dodds) · 2021-04-26T01:44:18.000Z · LW(p) · GW(p)

Similarly, the effect of the second dose might be to maintain the high initial effectiveness for a longer period of time, by "reminding" your immune system not to relax too soon.

7 comments

Comments sorted by top scores.

comment by jimmy · 2021-04-25T23:07:41.513Z · LW(p) · GW(p)

The issue is that this number captures efficacy starting on the day you receive the vaccine (or sometimes 7 days later)

 

Do you know how the efficacy on a given day is defined? I'm assuming it's going by date of first reporting symptoms (because you can't always know when the exposure was), but it makes a big difference if you're thinking "when am I safe to expose my self to covid".

But it's equally important to note that the the phase 3's were true randomized controlled trials - the placebo group was given a fake injection and everything

 

I did the Pfizer phase 3 trial, and this isn't really true. 

The side effects are clear enough that without an active placebo, calling it "blind" is kinda a joke in the first place. On top of that, people in the waiting room were talking about how you can tell if you're getting the real vaccine by looking at the syringe. And top of that, the doctor who gave me the injection basically told me that I got the real thing ("Keep wearing your mask, we don't know yet if these work"), and said something equally revealing to at least one other person I know who did the trial.

Replies from: samuel-marks, ChristianKl
comment by Sam Marks (samuel-marks) · 2021-04-29T16:54:51.129Z · LW(p) · GW(p)

Yes cases show up in the data on the day that they first report symptoms, not when they were first exposed. As you say, this means that if the data show some efficacy on a given day, you should actually expect to be protected at that level a few days before.

On top of that, people in the waiting room were talking about how you can tell if you're getting the real vaccine by looking at the syringe. And top of that, the doctor who gave me the injection basically told me that I got the real thing ("Keep wearing your mask, we don't know yet if these work"), and said something equally revealing to at least one other person I know who did the trial.

Wow. I know that because of side-effects these things can never be fully blinded, but this is just horrifying.

(Technical point: the phase 3's still were randomized controlled trials, they just weren't double-blind. But double-blind is the relevant characteristic when asking whether the different results are due to partying Israelis, so that's fine.)

Replies from: jimmy
comment by jimmy · 2021-05-02T17:07:09.026Z · LW(p) · GW(p)

(Technical point: the phase 3's still were randomized controlled trials, they just weren't double-blind. But double-blind is the relevant characteristic when asking whether the different results are due to partying Israelis, so that's fine.)

 

Yeah, the part I was objecting to there was "the placebo group was given a fake injection and everything". Not only did they do far less than "everything" that is supposed to go with giving fake injections, they also failed to give me a fake injection! My second "placebo" was a real vaccine and my dad's second "vaccine" was a placebo!

comment by ChristianKl · 2021-04-26T08:29:14.291Z · LW(p) · GW(p)

And top of that, the doctor who gave me the injection basically told me that I got the real thing ("Keep wearing your mask, we don't know yet if these work")

According to https://www.nbc12.com/2020/12/17/pfizer-trials-placebo-volunteers-tapped-receive-real-vaccine/ it should have been double blind, so the doctor wouldn't know and might randomly say those things to people if he thinks they might skip wearing the mask afterwards.

Replies from: jimmy
comment by jimmy · 2021-04-29T02:22:13.181Z · LW(p) · GW(p)

Shame on them for misreporting. It was not double blind. 

I wouldn't put it past this guy for not knowing anyway, but he was 2 for 2 in accidentally hinting at the right thing (one vaccine, one placebo)

comment by TurnTrout · 2021-04-25T20:51:52.423Z · LW(p) · GW(p)

Frustratingly, the phase 3's don't report this number. But using some data included in the Pfizer phase 3, I was able to make this graph:

The image isn't loading for me on LW, although it does load if I right-click and select 'open in a new tab.'

Replies from: samuel-marks
comment by Sam Marks (samuel-marks) · 2021-04-25T21:07:27.185Z · LW(p) · GW(p)

Umm ... that's weird. I'll paste in the picture again and maybe that'll fix whatever bug is going on? Let me know if it loads now.