[Linkpost] The Story Of VaccinateCA

post by hath · 2022-12-09T23:54:48.703Z · LW · GW · 4 comments

This is a link post for https://www.worksinprogress.co/issue/the-story-of-vaccinateca/

Contents

4 comments

Linked is Patrick MacKenzie's writeup of VaccinateCA, the nonprofit that succeeded at creating the best source of vaccine availability data when the government completely dropped the ball. It's long, but one of the best things I've read in a while. Some highlights:

We found it surprising that The System did not know where the vaccine was and that this fact persisted. The System allocates and ships the vaccine, after all. Devolution to various layers of government, community groups, and healthcare providers; pervasive inability to connect IT systems; and unwillingness to correct this problem meant that people in positions of authority considered the ground truth beyond the abilities of their institutions to discern.

It is not harder to track a shipment of vaccine than it is to track a package from Amazon. Full stop. We are a nation that is extremely skilled at logistics, including healthcare logistics. A pharmacy chain can calculate, within a matter of minutes, the number of bottles of aspirin it owns, broken down by address. That count will be shockingly close to physical reality. Capitalism, ho!

We chose, as a nation, that knowing the location of the vaccine was . . . just not a top priority.

As an example of places where the data chain of custody broke down, consider the (true) case where a government actor directs some vials that it controls into the University of California at XYZ hospital system. (I will elide naming the specific hospital system, but for people not familiar with California, note that there are many different academic institutions called the University of California and their names are distinguished by the city they are primarily located in.) That hospital system has one address, according to a shipping spreadsheet.

That hospital system routinely centrally receives, records, allocates, and reships all the medical supplies needed to keep a hospital system running, from saline to radiomedicine to scalpels. Then it parcels them out to the locations it provides healthcare at. Which it has more than one of and which are not a short walk from one another. I invite you to take a look at the locations list for the University of California at San Diego hospital system.

We surprised the government by telling them that the vaccine was present where they believed it was absent. After delivery was taken at the central receiving facility, the vaccine was moved to individual locations where healthcare was conducted within the area of interest.

 

What if the State of California had an alternative to engaging consultancies to deliver information systems months late and half functional? What if it, for example, had the world’s leading tech industry, which is abundantly capable of shipping and operating websites? What if that industry was also extremely experienced at solving scaled logistical challenges, including ‘atoms, not bits’ logistical challenges, including via the employment of tens of thousands of call center workers?

Throughout the pandemic, as part of the ongoing estrangement between the tech industry and other corridors of power, there was unwillingness in the political class to work directly with the tech industry. You can write the tweet yourself, right? ‘Government tells Big Tech to ask millions of vulnerable Americans about their medical conditions. So they can do what, sell their data to advertisers?’ It certainly did not help matters that various people in positions of power assumed tech was complicit in serial prevarication being heard from elsewhere in the public sector, including about, e.g., Covid testing.

This rift deepened sharply in the immediate wake of 6th January, which many important people laid at the door of tech. Damn techies trying to Ctrl-Alt-Delete constitutional democracy; they will pay.

Actors within tech can also read the newspaper, watch their political leaders make speeches, and understand that they need to take immediate action to preserve their literal and figurative licenses to stay in business. Government relations and PR teams at AppAmaGooBookSoft told everyone that the marching orders were Keep Your Heads Down and Avoid Media Coverage during early 2021. This directly interfered with the efforts that public health teams at AppAmaGooBookSoft were making contemporaneously. (It probably surprises many people outside the tech industry that AppAmaGooBookSoft have teams dedicated to public health. AppAmaGooBookSoft are increasingly the operating system for the world, whether one desires that to be true or not. Of course their operations touch healthcare, governments, and government departments responsible for healthcare.)

 

Let me quickly explain what an end-of-day shot was: By manufacturers’ instructions, the vaccines were viable for 12 hours after being taken out of storage. They could not be recooled after the seal was punctured; once you had decided to dose a patient from a vial, all of the doses in the vial (five or ten, depending on which vial you were issued) were going into an arm or the trash can that same day.

Israel had a very sensible policy: At the end of the day, give literally anyone leftover shots. Run into the street and ask passersby if they’d like the vaccine if you have to. The best-managed pharmacy chains in the United States, and some savvy individual pharmacists, adopted systems like a paper list next to the phone where they would take down numbers and at end-of-day call the ones at the top of the queue and say, ‘Can you get here in the next 15 minutes?’

But, discouragingly, many doses went into the trash can. Administering the end-of-day list took work at a busy point of the day when everyone wanted to just go home. Not to keep banging this drum but it is an important one: We fired and stripped licenses from professionals who gave away end-of-day shots to disfavored potential patients, including in several well-publicized cases. In some cases, healthcare providers adopted policies to prohibit end-of-day shots because their lawyers told them that if they violated the tier list then the state would revoke the pharmacy’s permission to do business at all. I wonder where they got that cockamamie notion.

 

Speaking of BigCo: the largest pharmacy chains in the United States spend an absolutely gobsmacking amount of money so that you remember their name and choose to fill your prescriptions at them. These chains suffered the worst imbalance in seekers versus doses, directly as a result of being good at marketing their standing ability to provide healthcare. 

Mid-tier chains, the sort that do not immediately spring to mind when you think pharmacy, frequently had imbalances in seekers versus doses in the other direction: They were granted a lot of doses but had little public mindshare as the first place you’d call.

Many, many people gave up after the first no, assuming someone would tell them when the answer changed, not realizing that there could simultaneously be a yes across the street. This is an extremely understandable reaction from a seeker of healthcare. The county health department could understand there to be no untapped vaccine supply when there was a yes at the pharmacy across the street from their own office.

Vaccine seeker discouragement likely disproportionately impacted elderly patients and those in vulnerable populations. If you grew up believing that a doctor would either a) obviously immediately give you the right answer about your healthcare, or b) ignore you again in 2021 like they had ignored you your entire life, you had a rough go during the pandemic. If you were a member of the professional-managerial class used to navigating complex systems and skilled at sifting through information, it was much easier.

 

The core unique insight VaccinateCA had was that America has access to a reliable technology for getting information from the healthcare system. It is called a telephone.

You do not need to convince every healthcare provider to have every IT department simultaneously agree on a data format and transfer protocol to update a central system with daily inventory levels and then fan out information from that central source of truth to every possible user of it, including the general public. The government really wanted to do that, and made multiple independent attempts, the results of which speak for themselves.

You only need to realize that there are approximately 6,000 pharmacies in California, their phone numbers are in the phone book (or at least more accessible than a locked filing cabinet stuck in a disused lavatory with a sign on the door saying ‘Beware of Leopard’), and a call center employee can talk to approximately 100 pharmacists a day. Then you choose what freshness level you want for vaccine information.

Let me give you an oversimplified view of the freshness-versus-staffing tradeoff. If you want to call every pharmacy every day, that will take 60 agents. Simple division. Your data will be, at any moment, on average about a half-day old. If you are okay with results that are on average a day and a half old, you need to call all the pharmacies every three days; that only will take 20 agents. Call centers are actually much more sophisticated in how they think about call queueing and frequency than this sketch, but napkin math is frequently good enough for a first pass at capacity planning. The call center will happily accommodate you adding more agents if you need them, at additional cost and requiring a slight delay to train them up.

 

We had an internal culture of counting the passage of time from Day 0, the day (in California) we started working on the project. We made the first calls and published our first vaccine availability on Day 1. I instituted this little meme mostly to keep up the perception of urgency among everyone. 

We repeated a mantra: Every day matters. Every dose matters. 

Where other orgs would say, ‘Yeah I think we can have a meeting about that this coming Monday,’ I would say, ‘It is Day 4. On what day do you expect this to ship?’ and if told you would have your first meeting on Day 8, would ask, ‘Is there a reason that meeting could not be on Day 4 so that this could ship no later than Day 5?’

I started every meeting and status report to the team by reminding them what Day it was. Our internal stats dashboard had a counter of what Day it was. I had a whiteboard in my apartment showing what Day it was. I wrote that every morning as soon as I woke up, and updated the other two numbers right before I went to sleep. Those were: the number of locations we had published to Californians where they could currently get the vaccine, and the number we knew about elsewhere across the United States with the vaccine. 

The latter was zero at this point, of course. I brushed my teeth, wrote my emails, ate my meals, did media interviews, called my family, negotiated with funders, and said my prayers with the zero where I could see it.

Since in those first weeks we were doing only volunteer-delivered calls, we had a sharply limited (and unknown to us every morning!) amount of call capacity, and had to prioritize ruthlessly. This meant, basically, trying to optimize calls for getting new information. We wanted to avoid futile calls (which would never result in ‘Yes, I have the vaccine; here’s how to get it’) and we also wanted to avoid calls where we learned little at the margin useful to patients (for example, by repeatedly spending all calls in one geographic area).

One method of avoiding futile calls became project oral lore: Sometimes we would call a location that would never have the vaccines. Internally, this was coded as sir_this_is_an_arbys, after an internet meme. We would not call back locations we’d learned to be Arby’s. You would be surprised how many organizations in California have a name that accurately suggests that they are a hospital but somehow omits the tiny detail that their only patients are horses.

Horse hospitals were a nuanced case of being an Arby’s. Prior to being coded as one, we’d ask them if they had or expected to receive the Covid vaccine. Some horse hospitals might have received allocations. Because people who send a box of Covid vaccine to a horse hospital did not always make optimal decisions, this could have resulted in the hospitals having extra doses where no patient would think to look for them.

 

We had an extremely rapid feedback loop between engineers, operations staff, and callers, and indeed many people were wearing all of the hats. If you noticed a bug while on a call, mentioning it in Discord would have an engineer start working on it immediately. If you noticed a pattern in a few calls and told the call captain, they could have our queue reconfigured to take advantage of your finding within minutes. This is not the level of autonomy and agility call centers typically expect to deliver.

An aside about the utility of volunteers: That autonomy and agility made volunteers crucial for us even when, later, we were able to afford professional call centers. I viewed volunteers like scouts for an army, nimbly perceiving reality and reacting to it so that we could mass some divisions and go Waterloo on the virus. I view our tiny project like that too, as part of the much larger response effort of all of society. Watch out, logistical issues; if you’re spotted by that little unassuming squad they will radio your position to California, Google, and the United States federal government, which control metaphorical heavy artillery. And also, come to think of it, literal heavy artillery. But I am getting ahead of myself.

 

We created the best data source available in the state of California within approximately five days. That sounded like a bold claim when it was made to me, but if you can’t trust the government, who can you trust?

Then we started making it better.

 

You might object and say that it must have been someone’s job to actually get those doses injected. Someone who worked . . . at the White House? Okay, no, but at the CDC? Okay, no, but at the California Office of the Governor? Okay, no, but at the county health department? Okay, no, county health departments do not track individual SKU inventory levels at individual pharmacies, that’s actually not a thing. OK, then, Rite Aid – some logistics manager at Rite Aid should have opened a spreadsheet, seen an SKU like #DJFKJDF3285325 with 50 doses available out of 50 shipped at a location in San Bernardino, and immediately said, ‘Oh, #$*#(%. That drug being in supply is equivalent to a life-threatening medical emergency. I will now get out my emergency procedures binder.’ Nope, that is also not a reasonable expectation.

Each of these organizations wants someone else to be responsible for catching errors like this, and they want them to be effective at doing so. They want, and the nation wants, an organization to be accountable for delivering the vaccine.

VaccinateCA considered this bug, and anything else that kept vaccines in freezers while patients were still waiting, to be our problem.

This problem was fixed because a caller from VaccinateCA thought to say, ‘Wait, I notice that I am confused’. It was fixed within about half an hour of being noticed. We estimate more than 500 doses were quickly taken out of freezers, thawed, and injected into waiting arms. Those arms were often attached to people who had been refreshing the county website every few minutes hoping new appointments would finally open up.

 

There is a sardonic saying in tech to the effect of ‘the best minds of my generation are working to maximize the number of clicks on web pages’. What is the actual human impact of getting one vaccine searcher a useful result? Early in the vaccination effort, a reasonable approximation was that accelerating a dose by one day saves 0.0001 lives in expectation or, equivalently, that 10,000 dose-days saves a life. There are more formal efforts to quantify this in the literature, but that estimate, the sheer moral weight of it, compelled me to take drastic action once I saw our stats on Day 3 and extrapolated.

One million dose-days? One hundred lives.

I believe our partnership with Google accelerated delivery of the vaccines by many millions of dose-days.

 

Part of entrepreneurship is having a vision of something that is possible and figuring out what is necessary to bring it into the world. A cynic would say that the world has a secret: Building things is not actually possible, because different organizations have different timelines allowing access to different resources, and it is impossible to correctly sequence things to satisfy all the requirements in order to build anything. An entrepreneur would tell the cynic a secret in return: You can carefully titrate the amount of truth to various parties to dissolve these deadlocks.

Your donor-advised fund won’t let you donate unless we’re a 501(c)(3)? Well, you’d donate if we were a 501(c)(3), right? Great. We’re applying for approval as a 501(c)(3) from the IRS. Can I put you down for $25,000? Dear IRS examiners: I have a written commitment from a charitable allocator for a $25,000 donation contingent on 501(c)(3) status. As you are aware, IRS procedure says that this qualifies for expedited processing. Oh, yes, government actor whose cooperation we need, we’re a nonprofit. Look at this official paperwork from Delaware. It says that the State of Delaware is officially aware that I say we’re a nonprofit. Not good enough? Our 501(c)(3) status? The IRS is busy approving it, on an expedited basis.

Full link here.

4 comments

Comments sorted by top scores.