RadVac Commercial Antibody Test Results

post by johnswentworth · 2021-02-26T18:04:09.171Z · LW · GW · 29 comments

Contents

  How I'm Updating
  Next Steps
None
29 comments

Background: Making Vaccine [LW · GW]

Results are in from the commercial antibody tests. Both my girlfriend and I came back negative - the test did not detect any Spike antibody response in the blood. This post will talk about how I'm updating based on these results, and the next steps.

Here's our timeline so far; more info on the vaccine is in the original post [LW · GW] and the radvac whitepaper:

The blood draw for this test took place the day after the fifth dose. I expect this is too soon to notice significant impact from the last two doses; vaccines in general seem to typically take 2-3 weeks to kick in, and that is my expectation for this one as well. (Also, it was an "IgG antibody test", and WebMD says these antibodies typically take about 2 weeks to show up after covid symptoms show from an actual infection.) This is intended to mainly be a test of the first three doses.

The test apparently used the "DiaSorin Liaison(R) SARS-CoV-2 S1/S2 IgG assay" (I didn't know this until the results came in). According to the FDA, it has about 92% sensitivity and 99% specificity. The "S1/S2" part indicates that it's testing for response to the S1 and S2 subunits of the spike protein - together, these are essentially the whole spike protein.

Important thing to notice: the test was looking for Spike antibodies, and two of our three missing peptides were Spike peptides. Indeed, there were only 3 Spike peptides among the full 9, so with two missing, we only had one Spike peptide in our first three doses. (The rest target other parts of the virus.) So that makes the test significantly less useful than it would otherwise be, and makes me more inclined to get another test in 2-3 weeks when the doses with the other three peptides have had time to kick in. 

How I'm Updating

In the original post, I called this test "searching under the streetlamp". It wasn't super likely to come back positive even assuming the vaccine worked as intended, but it was relatively cheap and easy to run the test, so it was our first check. Given the missing Spike peptides and the test only checking against Spike, it was even more likely to come back negative than I originally estimated.

In Jacob's prediction questions [LW(p) · GW(p)], I gave roughly a 25% chance that a commercial antibody test would pass for most people, given three doses and all 9 peptides. I gave the vaccine about 75% chance of working overall, distributed over several different possible worlds. In this specific scenario, it's clear that the prior on test passing should be even lower.

(Reminder on the possible worlds: the vaccine could induce antibody response in the blood and mucus, only mucus, or not at all. It could induce T-cell response separate from antibody response. It could work sometimes, much like how the first dose of commercial mRNA vaccines tend to work in 75% or 85% of people, and in that case I expect more doses/more time to make it work more often.)

After updating on the results, I'm down to about 60-70% chance of working overall. Unfortunately this test just didn't give us very much information - at least about the vaccine working.

Aside from the test result, we do have one more small piece of information to update on: I was quite congested for 1-2 days after the most recent three doses (and I was generally not congested the rest of the week). That's exactly what we'd expect to see if the vaccine is working as intended, and it's pretty strong evidence that it's doing something. Updating on both that and the test results, I'm at ~70% that it works overall.

Next Steps

There's a few directions to go from here.

First, we can take one more dose with all the peptides included, wait a couple more weeks, and get another test to see if there's a blood antibody response against the Spike protein. That would still be searching under streetlight - it's still cheap and very useful if it passes, but most likely to not pass even if the vaccine works.

We could also specifically look for a commercial test which checks for both Spike and Nucleocapsid antibodies, rather than just Spike, but I expect that to be difficult - those sorts of details are not particularly visible to consumers. Also, it would still most likely be a blood test.

The most interesting direction to go from here is to order an ELISA assay kit and run a test ourselves. Official Best Person Anna Czarnotta suggested a protocol [LW(p) · GW(p)] and gave a bunch of helpful tips on this in the comments on the previous post. This would not require waiting another 2-3 weeks, and would allow us to test mucus directly. It would be a pretty direct test of whether the vaccine works, not a searching-under-the-streetlight test. So that's probably what we'll try next; I need to do some reading to figure out the details. Further results will be posted when they come in.

29 comments

Comments sorted by top scores.

comment by eillasti · 2021-03-02T22:01:37.017Z · LW(p) · GW(p)

It is always good to have additional data, but this result is totally expected, so I don’t update much based on it. When I’ve discussed the efficiency evidence with RaDVaC team, they mentioned that:
- they have tried commercial tests and didn’t manage to get even one positive result
- they have tried ELISA to measure the blood antibody titers and got some good results (part awesome, part good, part not so good)
- they have tried ELISA to measure the saliva antibodies, there are more of them than in blood samples, but there is no methodology to translate the raw data into titers for saliva
- they focus on B-cell target mechanism, not ACE2 binding and definitely not serum antibodies, so they don’t care that much about the serum antibody titers

Replies from: gjm, johnswentworth, ChristianKl
comment by gjm · 2021-03-09T11:35:22.385Z · LW(p) · GW(p)

It seems as if there's a real shortage of actual evidence that RaDVaC does anything useful.

Replies from: barbarabrezna@gmail.com
comment by johnswentworth · 2021-03-03T21:40:56.285Z · LW(p) · GW(p)

Do you know whether the ELISA tests were against the radvac peptides specifically, or against full proteins?

Replies from: eillasti
comment by eillasti · 2021-03-04T15:36:20.296Z · LW(p) · GW(p)

I don't know.

comment by ChristianKl · 2021-03-02T22:54:50.462Z · LW(p) · GW(p)

- they have tried ELISA to measure the blood antibody titers and got some good results (part awesome, part good, part not so good)

Have you asked why that resulting data isn't public? Is it fear of legal issues of being accused to run clinical trials? 

Replies from: eillasti
comment by eillasti · 2021-03-02T23:12:23.614Z · LW(p) · GW(p)

I haven't. Firstly, there is no proper data, just some bits of evidence. Secondly, yep, I am pretty sure that they would get in trouble if they did anything that looked like a trial, so I assume that they stay on the safe side and well, don't do anything like a trial.

comment by Raemon · 2021-02-27T22:38:40.752Z · LW(p) · GW(p)

Curated.

The previous post, Making Vaccines [LW · GW] post was one of the top-upvoted posts of all time. I think it's great for LW-folk to actually go off and do Actual Science™, and then even better to follow that up with reporting on the results, and then even better to report negative results, and then even better still to list out their thoughts on what those negative results actually mean and how to continue reasoning about the domain.

I remain somewhat uncertain how to think about the RadVac vaccine in particular, but I think both this post and the previous one are great to read from a standpoint of "orienting towards science" – seeing the world as something we can understand, and that we're not only allowed to reason about, but allowed to go out and Interact With The Physical World in order to learn more about.

Replies from: Davidmanheim
comment by Davidmanheim · 2021-03-01T07:56:47.265Z · LW(p) · GW(p)

I agree that posting the results was the correct thing to do, and appreciate that John is trying to figure out if this is useful - but I actually claim the post is an example of how rationality is hard, and even pursuing it can be misleading if you aren't very, very careful.

In The Twelve Virtues of Rationality [LW · GW], this post gets virtue points for the first (curiosity, for looking into whether it works,) third (lightness, being willing to update marginally on evidence,) fourth (evenness, updating even when the evidence isn't in the direction desired,) sixth (empiricism, actually testing something,) and tenth (precision, specifying what was expected.) But virtue is certainly not a guarantee of success, even for completely virtuous approaches. 

I think this tries to interpret data correctly, but falls short on the eleventh virtue, scholarship. For those who want to do Actual Science™, the first step is to know about the domain, and make sure your experiment is valid and useful. Going out and interacting with reality is valuable once your models are good enough to be able to interpret evidence. But Science is hard. (Perhaps not as hard as rationality, at least in some ways, but still very, very difficult.) In this case, without checking what the specific target being tested for was, as Christian notes, the data doesn't actually provide useful evidence. And if he had a (recent) asymptomatic case of COVID, the result would have been positive, which is evidence that the vaccine doesn't work, but would have been interpreted as evidence that it did.

Replies from: barbarabrezna@gmail.com
comment by BB6 (barbarabrezna@gmail.com) · 2021-03-11T20:54:41.475Z · LW(p) · GW(p)

They did test fot antibodies before taking RadVac. That was a prrcaution to lower the probability, that they would get antibodies due to infection.

Replies from: Davidmanheim, barbarabrezna@gmail.com
comment by Davidmanheim · 2021-03-12T07:23:57.603Z · LW(p) · GW(p)

Hence "(recent)"

comment by BB6 (barbarabrezna@gmail.com) · 2021-03-11T21:02:58.259Z · LW(p) · GW(p)

People, off topic, but why does the site show my e-mail addess and not my user name ? How can I change it ?

Replies from: Benito
comment by Ben Pace (Benito) · 2021-03-11T21:37:36.547Z · LW(p) · GW(p)

I edited your account to change your name. (I'm an admin.) Hope that's alright now!

Replies from: barbarabrezna@gmail.com
comment by BB6 (barbarabrezna@gmail.com) · 2021-03-15T07:16:29.233Z · LW(p) · GW(p)

Thanx a lot !

comment by ChristianKl · 2021-02-26T23:43:03.542Z · LW(p) · GW(p)

I'm not sure why there's updating based on the fact that peptides from proteins besides the spike protein don't result in antibodies agains the spike protein being build. It seems to me a faily straightforward prediction. I don't see how antibodies against the spike protein would be developed without those peptides. 

Replies from: johnswentworth
comment by johnswentworth · 2021-02-26T23:47:34.155Z · LW(p) · GW(p)

There was still one spike peptide in the mix, so there's a small update from that.

Replies from: Davidmanheim
comment by Davidmanheim · 2021-03-01T07:38:15.230Z · LW(p) · GW(p)

You need to see if the spike peptide included corresponds to the antibody being tested for - and given how many targets there are, I would be surprised if it did.

Despite holding a far lower prior on efficacy, I'm agreeing with Christian - this evidence shouldn't be a reason to update anywhere nearly as strongly as you did against effectiveness.

Replies from: johnswentworth
comment by johnswentworth · 2021-03-01T17:01:41.440Z · LW(p) · GW(p)

Based on the fact sheet, it sounds like they're using full S1 and S2, which together constitute basically the entire spike protein. You seem to be imagining that they would use short peptides in the test rather than whole proteins or large subunits; any particular reason why?

Replies from: Davidmanheim
comment by Davidmanheim · 2021-03-12T07:25:42.151Z · LW(p) · GW(p)

Not sure that you'd get reactions from large subunits if they fold differently than the full spike - but my biochemistry/immunology isn't enough to be sure about how this would work.

Replies from: johnswentworth
comment by johnswentworth · 2021-03-12T15:12:16.327Z · LW(p) · GW(p)

The marketing material for the test kit has a long description of steps they took to get the same conformation. It's still marketing material, so I don't trust it 100%, but there's at least a plausible story that it should be the same.

comment by AspiringRationalist · 2021-02-27T17:20:18.580Z · LW(p) · GW(p)

Strong upvote even though this is kind of a meh update, because I want to encourage actually gathering the data and reporting on it.

Replies from: kjz
comment by kjz · 2021-02-27T17:40:44.708Z · LW(p) · GW(p)

I also strongly upvoted for the same reasons. Very much looking forward to the results of the ELISA mucus test!

comment by Davidmanheim · 2021-03-01T08:16:00.553Z · LW(p) · GW(p)

 "Aside from the test result, we do have one more small piece of information to update on: I was quite congested for 1-2 days after the most recent three doses (and I was generally not congested the rest of the week). That's exactly what we'd expect to see if the vaccine is working as intended, and it's pretty strong evidence that it's doing something."


Agree that this is evidence it is doing something, but my strong prior is that the adjuvant alone (chitosan) would cause this to happen. 

I'm also unclear about why you chose the weekly schedule, or if you waited long enough to see any impact. (Not that the RadVac test would tell you anything.) The white paper suggests *at least* one week between doses, and suggests taking 3 doses, for healthy young adults. 

According to the white paper, you're likely to be protected, and I think continuing now would add danger without corresponding benefit. You said in the original post that you might continue dosing. I don't know enough to comment usefully about either immune tolerance or adjuvant hyperstimulation, but I suggest talking to a immunologist about those risks and how they change if you in fact continue and try "more dakka," since continuing to does seems like it would increase those risks.

Strongly agree that ELISA tests are more valuable than more RadVac, and it would be at least moderate evidence one way or another. (But even if you can induce immune reactions to parts of the virus, it' unclear how much that would actually reduce your risk if infected.)

Replies from: johnswentworth
comment by johnswentworth · 2021-03-01T16:55:20.029Z · LW(p) · GW(p)

My current plan is to do one more dose, to make sure we've had three total doses with all nine peptides, then cut it off there. In general, I expect more dakka to be valuable mainly if the dosage is sufficient to induce response in some people but not everyone, so e.g. if we saw a response in my girlfriend but not in me then I'd be a lot more inclined to add another couple doses.

comment by cursed · 2021-03-04T05:12:23.078Z · LW(p) · GW(p)

My prediction that the anti-body test would come back negative: https://www.lesswrong.com/posts/niQ3heWwF6SydhS7R/making-vaccine?commentId=hgwzegWcLEYrZMmZj [LW(p) · GW(p)]

No one took me up on a bet: https://www.lesswrong.com/posts/niQ3heWwF6SydhS7R/making-vaccine?commentId=h8mqNdAypszWYQ7dh. [LW · GW]

Replies from: ChristianKl
comment by ChristianKl · 2021-03-04T10:19:29.209Z · LW(p) · GW(p)

My prediction that the anti-body test would come back negative: https://www.lesswrong.com/posts/niQ3heWwF6SydhS7R/making-vaccine?commentId=hgwzegWcLEYrZMmZj [LW(p) · GW(p)]

You predict nothing of that sort in the linked comment. The antibody test being negative is a distinct event from immunization. 

No one took me up on a bet: https://www.lesswrong.com/posts/niQ3heWwF6SydhS7R/making-vaccine?commentId=h8mqNdAypszWYQ7dh. [LW · GW]

You offered 50/50 odds for an event that  johnswentworth gave 29% likelihood (which was the maximum of anybody giving). It's quite obvious why nobody takes you up on 50/50 odds when nobody has a higher then 29% likelihood of believing that the event happens. 

Replies from: cursed
comment by cursed · 2021-03-05T07:44:07.761Z · LW(p) · GW(p)

I don't know if you're purposely being antagonistic, but I'll respond because I try to assume that people are arguing in good faith.

You predict nothing of that sort in the linked comment. The antibody test being negative is a distinct event from immunization. 

The first linked comment I said that "there's a 1-2% chance here that you've effectively immunized yourself from COVID.". In the second linked comment, I clarified that an anti-body test would be the the predictor of immunization.

I picked 50% because of the comment:

My rough guess is that there's a 75% probability of effectively full immunity

Regardless of people accepting or rejecting my bet, my prediction came true. For a community supposedly dedicated to rationalism and prediction markets, a -8 downvote on my comment seems a lot like groupthink, trying to push down dissenters. 

Too bad that the LW community falls victim to obvious snake oil and herd mentality. I wish it weren't so. 

Replies from: ChristianKl
comment by ChristianKl · 2021-03-05T09:48:12.746Z · LW(p) · GW(p)

I picked 50% because of the comment:

Given that people actually had vastly different predictions [LW(p) · GW(p)]for full immunity and antibodies detecting and commercial tests detecting antibodies this just illustrates that you don't understand the topic well enough to distinguish the different claims to offer a bet that's worth for anybody to take given their stated beliefs. 

Replies from: cursed
comment by cursed · 2021-03-06T00:36:32.316Z · LW(p) · GW(p)

It's clear that you're arguing in bad faith when you link a comment that was made 5 days after my own as some sort of bizarre justification that people didn't take my bet. Regardless, my prediction is still true.

"You don't understand the topic" is the weakest counter argument you could make without any proof.

That's what I get for trying to engage in debates assuming good-faith, right? Disappointing.