Another RadVac Testing Update

post by johnswentworth · 2021-03-23T17:29:10.741Z · LW · GW · 21 comments

Contents

  ELISA Tests
  Going Forward
None
20 comments

Previously: Making Vaccine [LW · GW], Commercial Antibody Test Results [LW · GW], Mini-Update [LW(p) · GW(p)]

I've now run 9 ELISA tests. The main result is noise: negative controls are all over the map, sometimes very blue (i.e. positive), sometimes not blue at all. I did see more positive results in the experimental group than I'd expect from noise alone, but I haven't gotten the noise to a point where results are consistently reproducible.

Meanwhile, I also ran one very simple test: I snorted a batch of the peptides, without the chitosan or anything else - just peptides in deionized water. Previously, on doses 3-6 of the vaccine, I had consistently been congested for a couple days after (and not congested the rest of the week), which strongly indicates an immune response. However, that response could have been to the chitosan or other contents of the vaccine, rather than the peptides. This test put that possibility to rest: after snorting just the peptides, I was very obviously congested for a couple days, in basically the same way as after the vaccine doses.

So thanks to that simple test, I personally am now pretty highly confident that I have an immune response to these peptides. Unfortunately it's not as legible as an ELISA test, so you should not necessarily be quite as convinced by this.

Now, this still leaves the question of whether an immune response to these peptides translates into an immune response to COVID. It could be that e.g. the conformation of the peptides' corresponding sequence within full COVID proteins is different enough that it doesn't carry over. Personally, though, I consider this a much less likely failure mode, for two reasons. First, the white paper indicates that the peptides were chosen based on antibodies developed by people who actually had COVID. Second, whether antibodies against these peptides bind the real proteins is something which I would not expect to vary much from person to person, so if it's worked for a few people it should work for everyone - and the whitepaper does indicate that multiple groups have seen positive results testing for binding against the full proteins.

None of this puts my confidence up close to 99%, but I'm now considerably more confident that the vaccine worked (~90%). Also, that confidence is distributed over fewer possible worlds - e.g. based on the info in the latest version of the RadVac whitepaper, I now very much doubt that the vaccine will induce a response in the blood (unless it's injected). I also now have very little weight on the possibility that it works for some people sometimes but didn't work for me specifically, so additional dakka is not needed (at least for me).

The next section will be a bit more detail on the ELISA tests, for people who are curious about exactly how that sausage was made.

ELISA Tests

This section is an abbreviated chronology of what-I-saw and my reasoning about it; it's intended to show how I came to the conclusions I did. I expect most people will not find it very interesting, but one of the benefits of blog posts is that I can show all the questionable decisions and opportunities for confirmation bias to sneak in, so that's what I'm doing.

First, some background on how these tests work in theory. We start with a "high binding plate" - basically some plastic treated so that proteins/peptides stick to it.

That’s the plate; each of the holes is called a “well”, and is basically a mini-test-tube with a high-binding surface.

We add a solution of our peptides, and some of them stick to the surface. Next, we dump that solution out, leaving behind only the peptides which bound to the surface. We add some "binding solution" - in this case nonfat dry milk with a little detergent in it. The proteins in the binding solution fill whatever space on the surface was not taken up by the peptides.

Now, with the foundation in place, we build a tower. We add a nasal wash sample from my nose, which hopefully contains antibodies that bind to the peptides. Then we dump that out, leaving behind only the antibodies which bound to a peptide attached to the plate. Next we add the "secondary antibodies", which bind to my antibodies and have an enzyme attached to them. Then we dump that solution out too. If all goes well, this leaves our "tower": peptide bound to the plate, antibody bound to the peptide, secondary bound to the antibody.

The final step is to add some TMB solution. The enzyme attached to the secondary antibody will turn the TMB blue, so if we see blue after a few minutes, then we know the secondary antibodies are present, and hopefully that means the rest of the tower is present too.

So how does this play out in practice?

Here's the first plate I ran: 

Each row is a different sample (nasal wash and saliva from both myself and my girlfriend, blood from me, and one more concentrated nasal sample). Each column is a different peptide, with no-peptide control group in the right-most column. From this, we learn three things. First, there’s a lot of variation between samples. Second, the variation we’re seeing has nothing to do with the peptides. Third, and most important, the negative controls contain an awful lot of blue. So this does not match our theoretical picture of building-a-tower; somehow, something in the sample is sticking in the plate, and it’s sticking to something other than the peptide.

Next came a fair bit of trying stuff. I tried both more concentrated peptides, and dosing myself with RADVAC the day before taking a sample in order to induce more antibody production, both in hopes of increasing the signal enough to overcome the noise. Results were basically similar. I tried a bunch of different blocking agents, without any peptides at all, to see which gave the least-blue negative controls - dried milk was actually one of the best, although egg whites were better. So I tried using egg whites for the blocking solution, and results were basically similar. I ran another plate just comparing various negative controls, without any major new insight.

In general, I still saw some samples produce generally-more or generally-less blue, across both experimental and negative control groups, for no clear reason. Generally, things varied more from test-to-test and sample-to-sample than within similar treatment groups on the same sample/test - i.e. the noise is mostly systematic.

The main result from all this was that wells with no sample were pretty consistently not blue. Whatever causes my no-peptide negative controls to turn blue, it definitely involves something in the sample binding to something other than the peptide.

I did some reading online and some thinking about how to reduce the sample binding in the negative control group. I shifted to a mental model where a significant fraction of the peptide/binding agent on the plate was actually coming loose and being replaced by whatever was in solution. One way to reduce noise in the control, then, is to include a little binding agent in the antibody solution and secondary antibody solution - so if a space on the plate opens up, it will most likely be filled by the extra binding agent rather than the antibody. (This was included in the protocol, but listed as “optional”, and I didn’t understand before why it would be useful.) I also increased the amount of binding solution used (so that it covered the sides of the well more completely), and made extra sure to not accidentally use 100uL rather than 200 uL of binding solution (all of the other steps involve 100 uL in the well, so it’s an easy mistake to make if not paying attention - I think I probably made this mistake multiple times in earlier tests). Finally, I removed the wash step between peptide and binding solution - I see no way that that one could reduce noise, and I’d expect it to reduce the signal.

With all that in place, I ran two more tests. One saw generally very little blue, but there was more blue in the experimental wells than the controls:

Left-to-right, we have three no-peptide controls, then three experimental wells, then three more no-peptide controls, then three more experimental wells. There is definitely some blue in the no-peptide controls (especially the second well from the left), but there is generally more blue in the experimental wells. Note that this test was run with a sample which had been frozen; I think that’s the most likely cause of the generally-faint blueness.

Then we got the most promising result:

The group of five experimental wells in the upper right was visibly more blue than the corresponding no-peptide negative controls below them. (The group on the left is a sample which was frozen; so now we know to use fresh samples. This is another mistake I made multiple times in earlier tests.)

On the other hand, I’m still not able to get consistent results. I ran one more test the next day, just a single fresh sample with a whole bunch of experimental wells and no-peptide controls, and there was no visible difference between the controls and the experimental wells:

Experimental and control wells are in alternating rows. The last two columns are no-sample controls.

So, bottom line: the two positive results, especially the second, are more than I’d expect to see from noise after having run this many tests. But they’re still definitely not knock-down unambiguous evidence, and there’s still a lot of test-to-test variability which I’m unable to account for.

Going Forward

Mainstream vaccines will be available to the general populace here starting two weeks from today, so this project is probably reaching its end soon. We may run another test or two, but I probably won't have another post. Conditional on any more tests, I will put up a shortform summarizing whatever results we see.

21 comments

Comments sorted by top scores.

comment by JenniferRM · 2021-04-29T01:01:20.733Z · LW(p) · GW(p)

I'm late to the responses here but my mind returns to your work every so often... so I tracked this down again and re-read things.  Two thoughts still seem salient, which I will break into two comments in case they trigger conversations :-)

Thoughts On "IMMUNE ESCAPE" And RadVac:

It seems like the US will be porous to new strains that develop around the world. The planes will fly, some paperwork will be shuffled, but the paperwork won't be 100% effective at "literally ensuring new harmful viruses don't enter the US"... the paperwork might just slow things down, effectively causing "the country" to get "a small inoculating dose".  Just as masks are better than nothing, so is paperwork requiring tests... but also both can fail unless they are part of a larger and more comprehensive response that the US might be structurally politically incapable of mounting...

IF any new strains arise anywhere on the planet that decisively dodge our existing vaccines, THEN they will get into the US and people in the US would be in for another round of <gestures at all of this>.

If another wave starts in the US that isn't stopped by Pfizer/Moderna/J&J... personally... I'm not up for more of <gestures at all of this> and so I'd be thinking that I'd rather replicate your actions here (but maybe with varied peptides for the hypothetical new thing) instead of waiting for an institutional response. 

If you have any thoughts on immune targeting and variance, I'd be interested in reading them! Do you think immune escape is likely? Do you think RadVac would also be dodged, or would it perhaps need to be re-tuned as well? 

I got as far as looking up a covid paper from 2021 with the phrase "hypervariable region" in it but it could plausibly take me hours or days of reading closely and thinking carefully to figure out how that kind of thing might interact with the RadVac design.  If you've read and thought about such things already, or even just have educated hunches on how to model the questions, your wisdom would be appreciated!

Replies from: johnswentworth
comment by johnswentworth · 2021-04-29T16:35:50.770Z · LW(p) · GW(p)

I'm not the most knowledgeable person on this, even on LW, but I can sketch out my understanding of the topic at a high level.

The basic idea of a new-strain-robust peptide vaccine is pretty simple. When designing a peptide vaccine, we choose peptides to match against particular segments of COVID proteins. A single-base-pair mutation in COVID's DNA will produce (at most) a single-amino-acid change in the corresponding protein sequence. Key point: these mutations are local; the rest of the protein remains unchanged, so if our peptide matches part of the protein which did not mutate, then it should usually keep working just fine. (Nonlocal mutations are also a thing, but they tend to change things more radically - most likely they just make the virus not work, and even if it does work it will be a very different virus.)

Now combine that with conserved regions: some regions of the protein play an important functional role, so most changes to their sequence will result in the virus failing to reproduce very much and dying out. Ideally we want to target those regions. (Conversely, other regions play little functional role or a more flexible functional role and are often targeted by the immune system, so mutations in those regions are very likely to be positively selected.)

Note that the specificity of peptide vaccines plays an important role here. Other kinds of vaccines just show the immune system the whole protein, so they don't control exactly which subsequence the system learns to recognize. Radvac was designed with this in mind.

Zooming out a level... two relevant meta-considerations:

  • new strains are likely to be less deadly. Partly this is just reversion to the mean, partly it's evolutionary trade-offs from the virus' perspective.
  • the more people get infected, the more new strains will pop up - I expect new strains at a rate roughly proportional to infection count. India is very bad news from that perspective.
Replies from: JenniferRM
comment by JenniferRM · 2021-05-01T14:44:11.092Z · LW(p) · GW(p)

Yeah. Bad news at the international level has been causing me to wonder how to set up a fallback plan where "if I see X then I should check Y about the RadVac design and if adequate I should order some peptides".

My current second best candidate for "X" is to watch Israel (where vaccination is very high) for a spike that is explained by new strains. A better canary would be a place with decent vaccination rates but otherwise a catastrophically poorly run public health service (like in terms of clinics and testing systems and coordinated travel policies and so on)... and basically I think maybe the US is that canary? :-(

Trying to imagine what "Y" would be is harder, like... I just searched for [peptide epitope search tool] and found the immune epitope database which could be relevantly helpful.

> I'm not the most knowledgeable person on this, even on LW...

Coming from you, this seems like extremely high praise for the forum. Care to name names? Also, I wonder if we could improve the attempt to summon them by promising to be thankful for their contributions? ;-)

comment by magfrump · 2021-03-24T07:42:56.839Z · LW(p) · GW(p)

Do you have any thoughts on what you'd do differently to be more personally confident doing this again?

Replies from: johnswentworth
comment by johnswentworth · 2021-03-24T15:52:35.458Z · LW(p) · GW(p)

Good question.

One thing I looked into was obtaining fast antibody tests (basically strips of paper with some proteins and colloidal gold soaked into them). They're "research use only" and hard to get your hands on if you're in the US, but if you're outside the US (or have a friend outside the US willing to help) it should be easier. They would make it dramatically easier and less error-prone to test, and they'd also test for binding against full COVID proteins directly (rather than the radvac peptides). If I were going to invest much more effort into this project, I'd try harder to obtain some - either find some international seller who doesn't ask a lot of questions, or recruit a friend who lives outside the US and can get their hands on some.

Another thing I might try is buying a microcentrifuge. A really low-end minimal one costs under $200, so it wouldn't increase the relative cost of the project too much. The main use would be to clean up the samples - i.e. it could remove most of the sticky crap from a nasal wash sample. If noise is coming from that sticky crap sticking in the plate, then removing it would help a lot. If.

Yet another thing I might try is getting a COVID ELISA kit. This has similar considerations to the fast antibody tests.

Replies from: TheSimplestExplanation
comment by TheSimplestExplanation · 2021-03-25T18:04:12.806Z · LW(p) · GW(p)

They’re “research use only” and hard to get your hands on if you’re in the US, but if you’re outside the US (or have a friend outside the US willing to help) it should be easier.

I'm willing to help.

comment by eillasti · 2021-03-24T10:39:49.985Z · LW(p) · GW(p)

Snorting peptides directly is hilarious! I should do lines of peptides at the next corona party :)

Theoretically, it shouldn’t cause an immune response, as peptides shouldn’t be immunogenic on their own, that’s why you need chitosan as a delivery mechanism and adjuvant. However, who knows? Was it actually researched and proven that peptides on their own do not cause an immune response no matter how big is the dose and route of administration? I could well imagine that this is simply a theoretical conclusion that was never empirically verified, or that it was only verified by an injection, but not by snorting, and peripheral immune system is triggered by pure peptides, while systemic is not, or that it was only tried in much lower doses. Even if a 10-100 times higher dosage of peptides is equivalent to chitosan+peptides, this is likely of little commercial interest as chitosan “enhancement” is cheaper and more scalable at commercial scale than peptides production. So it might actually work and it is cool that you’ve tried it and there is evidence that “you were congested for a few days like after previous vaccine applications and it looks like nothing really bad had happened”.

What dose of pure peptides did you use per 1 peptide and totally? The currently recommended dose is ~200ug in total, no matter how many component peptides are there. If you do 5 peptides like in the v10, then  this is 40ug per peptide. If you do 1mg=1000ug of pure peptide, then this is 25 time more dakka and it might well trigger the same response as 40ug of peptide+chitosan.

I think the currently recommended does is likely to be too low for average human risk preferences. Organisations are extremely risk averse, the negative side effects are given much more weight than unrealised benefits, so the recommended dose is likely to be “the smallest one that kind of works without any side effects” rather than the optimal one. There is almost no evidence for the efficiency of different doses and this is all guesswork based on guesswork of others in slightly related cases, e.g. the 500ug cancer peptide vaccine dosage.

Congestion after vaccine/peptides administration could be due to many factors, so I consider it only a very weak evidence. This can be purely psychosomatic. The vaccine is acidic, so it irritates your nasal mucosa. When you put anything in your nose, it may irritate it and cause congestion. You have applied pure peptides after receiving several doses of vaccine and after enough time has passed for developing an immunity, so congestion may indeed be an immune response, but it may only happen when you already have an immunity, but it won’t be triggered if you have no immunity. Actually, this is what your data suggests. “Previously, on doses 3-6 of the vaccine, I had consistently been congested for a couple days after”. I understand this as you having no congestion after doses 1-2 and getting it after all further doses. If this is so, then this looks like the actually important immune response of a naive immune system happens without congestion, but the immune response of a trained immune system happens with congestion.
 

Replies from: ChristianKl, johnswentworth
comment by ChristianKl · 2021-03-24T10:55:19.472Z · LW(p) · GW(p)

Theoretically, it shouldn’t cause an immune response, as peptides shouldn’t be immunogenic on their own, that’s why you need chitosan as a delivery mechanism and adjuvant.

It shouldn't cause an immune response if you snort them without having snorted peptides + chitosan previously. If you have immunity because you snorted both of them previously I would expect a immune response. 

Replies from: denys-kornilov
comment by Denys Kornilov (denys-kornilov) · 2021-03-24T11:24:38.945Z · LW(p) · GW(p)

Yes, I meant immune response from a naive individual, I didn't know what happens for an immune individual. If so, then the congestion pattern makes sense and is evidence for immunity.

This can be an interesting test: snort peptides and interpret congestion as immunity.

Do you know if congestion is due to antibody or cellular immunity or both?

comment by johnswentworth · 2021-03-24T17:53:18.930Z · LW(p) · GW(p)

Dosage was the same as the (gen 9) vaccine: ~120 ug total, divided by 9 peptides.

The point of the exercise was to test whether I had acquired an immune response, not to act as a vaccine, so your last paragraph is on-point.

comment by JenniferRM · 2021-04-29T01:05:08.265Z · LW(p) · GW(p)

I'm late to the responses here but my mind returns to your work every so often... so I tracked this down again and re-read things.  Two thoughts still seem salient, which I will break into two comments in case they trigger conversations :-)

Thoughts On "NOSE COVID" And RadVac:

This entire blog post has been quite useful for refining my mechanistic models of the immune system and the fact that it has >1 compartment. 

The evidence so far (especially snorting peptides with no chitosan and getting a reaction) seems to suggest to me that you did directly manage to teach your peripheral/mucosal immune system to react very quickly to the relevant protein fragments, but it seems plausible that your immunity didn't get to the rest of the body to become systemic.

I wonder if a similar lack of "immune transfer" might hold the other way? Suppose that systemic immunity does not become mucosal? 

If that supposition held, then maybe people could get "nose covid" for a few days, with an infection only in the upper respiratory system, and that might persist for however long in takes an upper respiratory version to try to invade the rest of the body, and only then trigger a systemic response?

My uncertainty is at the level of the model (two compartments... and how do they interact EXACTLY in terms of BOTH harboring independent infections AND sharing antibody notes in either direction) but some ways that the mechanisms might work suggest that nasal vaccines would actually be MORE *socially* helpful than the current "selfishly helpful" injected vaccines.

I find myself hoping that vaccines enable more travel and large safe social events in 2021, but these are exactly the kind of situations where I might want to get a mucosal vaccine to protect friends and family, like kids and any vaccine holdouts.

Basically, your mechanistic reasoning here seems highly informed, and I'm wondering if you think that "nose covid" could occur among vaccinated people, and lead to transmission chains to unvaccinated people in a way that could specifically (hopefully) be prevented specifically by a nasal spray?

comment by TheSimplestExplanation · 2021-03-24T17:38:57.125Z · LW(p) · GW(p)

This test put that possibility to rest: after snorting just the peptides, I was very obviously congested for a couple days, in basically the same way as after the vaccine doses.

That seems pretty strong evidence to me.

I would do a placebo control too, just to make sure.

Replies from: johnswentworth, Sherrinford
comment by johnswentworth · 2021-03-25T18:19:24.752Z · LW(p) · GW(p)

I would do a placebo control too, just to make sure.

My prior that snorting DI water would do nothing was pretty strong, but I had intended to test it anyway, so thanks for the reminder.

I snorted some DI water last night, in the same manner that I snorted vaccine/peptides. With the vaccine/peptides, I pretty consistently woke up congested the next morning, and blew my nose every few minutes throughout the day. None of that has happened with the DI water - it's just been a normal day so far, in terms of congestion.

Replies from: jmh
comment by jmh · 2021-03-27T15:47:46.994Z · LW(p) · GW(p)

I'm not sure DI water would be a suitable "placebo" here. Perhaps a placebo effect is not even what is occurring. Previously you were inhaling something with small particles -- a bit like what happens every spring with pollen. Perhaps a test with some other inert matter that might not even be able to invade your body much less produce some type of chemical reactions with the cells or cellular processes?

Replies from: johnswentworth
comment by johnswentworth · 2021-03-27T17:20:56.403Z · LW(p) · GW(p)

Yeah, that had occurred to me too. If people have other suggestions for what to snort, I'm open to ideas. Though my priors are still pretty strong here - I've accidentally snorted enough things to know that most things don't induce significant congestion.

comment by Sherrinford · 2021-03-24T20:13:44.352Z · LW(p) · GW(p)

Ok this may be a naive question, but given that John brews the stuff and expects certain results anyway: Isn't being congested something that might simply follow from actual placebo effects?

Replies from: johnswentworth, FireStormOOO
comment by johnswentworth · 2021-03-24T21:59:50.446Z · LW(p) · GW(p)

That is definitely something to worry about. I would be very surprised if the degree of congestion I experienced could be induced by a placebo effect, but this is exactly the sort of reason why I said the result lacks legibility - the rest of you haven't been able to see exactly what the symptoms looked like.

comment by FireStormOOO · 2021-03-24T20:54:38.939Z · LW(p) · GW(p)

Effective blinding is definitely more involved if one's both experimenter and test subject.  It's not impossible but an assistant would help a lot.  Controlling for placebo effect does seem one of the big issues at this scale.

Replies from: johnswentworth
comment by johnswentworth · 2021-03-24T22:01:58.106Z · LW(p) · GW(p)

I could prep both some peptides and a control and have my girlfriend randomly pick one, although the peptides do have a detectable scent to them, so I don't think this would be enough to actually blind me.

Replies from: ejacob
comment by ejacob · 2021-03-25T00:57:59.211Z · LW(p) · GW(p)

You could add some other scented ingredient to both peptide and control solutions. Rosewater would be a pleasant option. I wouldn't expect this to interfere with any immune responses too much, but you should do some research to check if you decide to try this.