BIS Seminar: At warp speed: statistics and COVID-19 vaccine developmentOctober 28, 2020
David Benseker, PhD, MPH, Assistant Professor
Emory University Department of Biostatistics and Bioinformatics.
October 27, 2020
Abstract: One of the best hopes we have of returning life to normal is to bring a safe and effective preventive COVID-19 vaccine to market and make it available to just about everybody around the world. We are in the middle of an unprecedented effort to bring such a vaccine to market. After a recent pressure campaign led by academic scientists, vaccine developers have made public the protocols for their Phase III trials. The release of these protocols has ignited a fierce debate as to whether the designs are appropriate and sufficiently safe-guarded from political pressure for vaccine approval. In this talk I will discuss a few of the complex issues involved in designing Phase III COVID vaccine trials, touching on some key statistical aspects of the trials along the way.
- 00:00- [Fan] David Benkeser who is an assistant professor
- 00:03at the department of biostatistics and bioinformatics
- 00:07at Emory University.
- 00:09Dr. Benkeser got his PhD in biostatistics
- 00:11from University of Washington
- 00:13and had his post-doctoral fellowship
- 00:15from University of California at Berkeley.
- 00:18Dr. Benkeser is an expert in methods for machine learning
- 00:22and non-parametric statistical inference.
- 00:24He has made important contributions
- 00:26to integrate machine learning methods
- 00:28to draw causal inferences with observational data.
- 00:31He also has interesting work on preventative vaccines
- 00:34and HIV prevention, which he's going to share with us today.
- 00:37Welcome David, the floor is yours.
- 00:44- [David] Thanks, yeah, it's a great pleasure
- 00:46to be here today.
- 00:48Well, here today, but with you guys today giving this talk.
- 00:51So I did see that I think Tony Fauci
- 00:55spoke at Yale yesterday, so it was very nice of you Fan
- 00:58to book Tony Fauci as my opening act
- 01:00and I'll try not to disappoint him with my followup.
- 01:04So the talk I'm giving today is a very high-level talk.
- 01:07So the title is statistics and COVID-19 vaccine development,
- 01:10but it's really a talk mostly
- 01:12about COVID-19 vaccine development.
- 01:15There's not math until maybe slide like 29 out of 30.
- 01:19So really these are sort of
- 01:20just the high-level issues that have come up
- 01:23as I've worked with companies and government organizations
- 01:28on COVID-19 vaccine development.
- 01:30So I think there's a lot of really interesting stuff
- 01:32here and really, really glad to share it with you today.
- 01:35So if you want to kind of slide along with
- 01:39the slides they're available on GitHub
- 01:41so there's a link at the bottom there,
- 01:43and you can click on that
- 01:44and that'll pull up the HTML slide back,
- 01:46and I have sort of references hyperlinked in there.
- 01:48So that's an easy way to access
- 01:50the references there as well.
- 01:52Okay so I'm going to start just kind of talking
- 01:55about the biology a little bit of SARS-CoV-2,
- 01:58and segue into sort of how we can think about
- 02:01developing vaccines that will prevent
- 02:03an infection and COVID-19 disease.
- 02:06And so this is a nice little graphic that I ripped off
- 02:09from The Washington Post, who's very much better
- 02:12at making these cutesy little graphics
- 02:13than I am using PowerPoint or something.
- 02:16So let's kind of walk through this.
- 02:17And the goal here is to try to understand,
- 02:20you know, how SARS-CoV-2 is infecting your cells,
- 02:22how it's replicating,
- 02:23and then to understand what the mechanisms
- 02:25that immunological mechanisms of the vaccine are
- 02:28that can potentially block that infection
- 02:30and prevent clinical disease.
- 02:31So we'll just go quickly through this
- 02:33and this is sort of the story for most viruses, right?
- 02:36Is that viruses are really just genetic material
- 02:39in this case RNA that's wrapped up in the glycoprotein.
- 02:42So it's genetic material wrapped up in a protein.
- 02:45And so for SARS-CoV-2 you may have heard of a couple
- 02:49of these proteins in particular, the spike protein will play
- 02:51a large role when we talk about a vaccine development
- 02:54and why is this spike protein so important?
- 02:56Well, that's the guy that sort of latches
- 02:58onto your cell and it does that through this ACE2 pathway
- 03:01and it grabs onto your cell and insert itself inside
- 03:05you cell and once it's inside
- 03:07it releases its genetic material, right?
- 03:09It releases its RNA and kind of tricks
- 03:11your cell into replicating the virus, right?
- 03:13So that your cell is producing new copies of this virus,
- 03:17they're pieced together out of proteins that are released
- 03:19into your bloodstream to go infect more cells
- 03:22and more people.
- 03:23Okay so that's sort of the infection process
- 03:25and where along the lines do you know
- 03:27vaccines sort of halt this?
- 03:29So I'll walk through a few different
- 03:31of the major vaccine constructs that are being used
- 03:33for SARS-CoV-2 vaccines,
- 03:35and the details aren't super important here,
- 03:37but I do think it's sort of helpful
- 03:38to have a high level overview in comparison, right?
- 03:40Because there's so many vaccine products being developed,
- 03:43at least having some point of biological comparison
- 03:45of how they're working is useful.
- 03:47So to walk through these slides,
- 03:48all of these slides are basically going to be the same
- 03:50on the right hand side of the slide
- 03:52and how they're gonna differ is what goes
- 03:54into the vaccine on the left-hand side.
- 03:56So let's actually start on the right-hand side, right?
- 03:58And talk a little bit about immunology, right?
- 04:00And how your body tries to fight off infection.
- 04:03And we have a couple of different mechanisms
- 04:05of your immune system to do that.
- 04:06So there's a kind of T-cell responses, cytotoxic T-cells.
- 04:10So those are T-cells that recognize cells in your body
- 04:12that have been infected with a pathogen
- 04:14and destroy those cells, right?
- 04:15Because the cells are producing copies of the virus,
- 04:17releasing in the bloodstream.
- 04:19So if we're able to destroy infected cells,
- 04:21we can potentially stop infection, prevent disease,
- 04:24and then another key response
- 04:26that your immune system has is through antibodies.
- 04:28And that's sort of what's on the bottom here
- 04:30and is that B cells are able to produce antibodies.
- 04:34And what those antibodies do is they basically grab
- 04:36onto these surface proteins, right?
- 04:38So remember we talked about the spike protein,
- 04:40and what antibodies do is basically just bind onto that
- 04:43and sit there and so neutralizing antibodies.
- 04:46So there's two classes of antibodies that are kind
- 04:47of relevant for vaccines.
- 04:48So neutralizing antibodies really, you're just doing that.
- 04:50They're gonna sit on all of those spike proteins
- 04:53and because they're sitting there now the virus can't grab
- 04:55onto your cells to infect them.
- 04:57There's also binding antibodies, which are somewhat
- 05:01considered to be less important in this context,
- 05:03but what those guys do is bind onto those surface proteins,
- 05:05they don't neutralize the virus itself,
- 05:07but they send out chemical signals to other cells
- 05:09in your body that say, hey, here's a virus.
- 05:11Please come eat it for me.
- 05:13So those are the sort of antibody classes response
- 05:15that you can have.
- 05:16So there's these two sort of immune mechanisms
- 05:18that we have to neutralize infections by viruses.
- 05:22How do they learn to neutralize them?
- 05:25Well, there's this sort of middleman.
- 05:26So we're moving just to this middle panel here
- 05:28with these APC cells,
- 05:30so these antigen presenting cells, right?
- 05:32Those are the guys that what they're doing
- 05:33is basically digesting little bits
- 05:36of the virus in this case of the surface protein, right?
- 05:40And they're teaching or training your immune system
- 05:43to recognize that pathogen, right?
- 05:44So they're the ones that go and talk to the T cells,
- 05:47talk to the B cells and say,
- 05:48here's that how this virus looks,
- 05:51please go produce some antibodies or please recognize cells
- 05:53that have been infected with this
- 05:55and neutralize them for me.
- 05:56So really again, the whole right side of this plot
- 05:59is about your immune system.
- 06:00This is the way your immune system fights off infection.
- 06:03And what's different between this slide
- 06:05and the next few slides is basically how we present
- 06:08pieces of the pathogen pieces
- 06:09of the virus to these APCs, right?
- 06:12So how do we get these APCs, the material that they need
- 06:14for you to mount an immune response against SARS-CoV-2?
- 06:19And so the first class of vaccines
- 06:21I'll describe are nucleic acid vaccines.
- 06:23And so I'm talking about first
- 06:25because they're sort of the first wave of vaccines
- 06:27that are in phase three trials in the US.
- 06:29So Moderna and Pfizer, who are probably the most advanced
- 06:33candidates for US licensure are both mRNA vaccines.
- 06:37And so how are those vaccines made?
- 06:39Well, we take a little bit of messenger RNA,
- 06:42a little bit of viral genetic material,
- 06:44and wrap that in a lipid shell, right?
- 06:45That's the construct of the vaccine.
- 06:47And when you're injected that lipid shell
- 06:49latches onto your cell, right?
- 06:51Delivers that mRNA into your cell,
- 06:54just like a natural infection, right?
- 06:55Remember the SARS-CoV-2 grabbed onto your cell
- 06:57and inserted itself and then made copies of itself.
- 07:00So what is the mRNA doing once it's in your cell,
- 07:03it's actually just making copies
- 07:05of the spike protein itself, right?
- 07:07So you're manufacturing this protein within your own cells
- 07:11that are then released for these APCs to detect.
- 07:14So this is how we're getting these APCs,
- 07:17spike protein with an mRNA vaccine.
- 07:19We're basically using your cells as a warehouse
- 07:22to produce the antigen of the vaccine
- 07:24and so this is a really cool idea and a new idea, right?
- 07:28So, am mRNA or DNA vaccine has never been licensed before
- 07:31and that's not to say that we tried many times and failed.
- 07:34It's just to say that this is a very new technology,
- 07:36and it's sort of interesting that it's kind of come
- 07:40to the forefront in this context.
- 07:41So why do we like mRNA vaccines?
- 07:44Well, they're very fast to manufacturer.
- 07:46We'll talk about some of the other vaccine constructs
- 07:48where we're making this spike protein in a lab,
- 07:50and that is a long and arduous.
- 07:52It needs to be very careful process
- 07:54and when we're thinking about scaling up
- 07:55vaccine manufacturing, mRNA vaccines are very appealing
- 07:59in that sense, you can manufacture them
- 08:02very quickly at scale.
- 08:03They don't require a cold chain
- 08:05and so that's another great advantage these vaccines enjoy
- 08:10in terms of thinking about vaccine deployment,
- 08:12particularly in developing world settings.
- 08:15But again, this is a brand new technology.
- 08:17We don't have any safety data
- 08:18from past vaccines with this construct.
- 08:20We don't have any efficacy data.
- 08:22So, it's sort of an open question in the field
- 08:25as to how well these things are gonna work.
- 08:28So moving to sort of more classical, constructive vaccines
- 08:30and viral vector vaccines.
- 08:33So again, the right side of this picture
- 08:34is exactly the same.
- 08:35The story is how do we get an APC the right antigen?
- 08:38How do we show an APC a little bit of the spike protein?
- 08:41So a viral vector vaccine, right?
- 08:45Is going to take a different virus and splice
- 08:49a little bit of SARS-CoV-2 into that virus, okay.
- 08:52So for example, AstraZeneca, that's the Oxford that you may
- 08:56have heard of, they take a chimpanzee adenovirus,
- 08:58that's like, it's a virus that causes the common cold
- 09:02in chimpanzees and they splice in a little bit
- 09:04of SARS-CoV-2 into that and so that sort of host virus,
- 09:09that adenovirus holds genetic material
- 09:12infects your cells and your cells then produce the antigen.
- 09:16They produce the spike protein of SARS-CoV-2.
- 09:20So AstraZeneca and Janssen are using this construct again,
- 09:22both with adenoviruses, a very common virus vector.
- 09:26And again, we like these types of vaccines
- 09:29because they're quick to manufacturer,
- 09:32but a challenge of them is that your body
- 09:34can sort of develop separate immune responses
- 09:37against the vector itself, right?
- 09:39So you can develop a separate immune response
- 09:42against say an adenovirus right?
- 09:45Such that your body neutralizes those adenoviruses
- 09:47before they're able to infect your cells
- 09:50and produce the SARS-CoV-2 antigen.
- 09:52So we do see tendency a kind of faster waning
- 09:55vaccine effects with this class of vaccines.
- 09:58So moving on to subunit vaccine.
- 10:00So this is NovaVax and Sanofi's vaccine
- 10:03will be subunit vaccines
- 10:05and this is where I kind of mentioned before
- 10:06actually what happens here is these spike proteins
- 10:10or whatever the antigen is,
- 10:12is created and purified in a lab.
- 10:14So they actually use insect cells
- 10:17that they infect with SARS-CoV-2,
- 10:20those insect cells then produce the antigen that's purified
- 10:23and that's what goes into the vaccine
- 10:26are those protein subunits, right?
- 10:27So there we're just directly giving you the spike protein
- 10:30that we've grown outside of the host
- 10:33and that's how we're getting these APCs, those antigens.
- 10:37And so this is a commonly used vaccine construct.
- 10:40So the hep B vaccine is highly effective.
- 10:42HPV vaccine is highly effective.
- 10:44That's the construct of these, but the downside of course
- 10:46to it, so it's a well-trodden way of developing vaccines.
- 10:51But the downside is that they're slower to manufacturer.
- 10:53There's this whole process where we have to cultivate
- 10:55and grow these viruses in a lab, we have to purify them,
- 11:00and moreover they often also require an adjuvant.
- 11:04So that's really just sort of adding something a little bit
- 11:07extra that stimulates a better immune response in your body.
- 11:11So basically at the site of injection,
- 11:13it's something that increases
- 11:14your inflammatory response actually
- 11:16to kind of stimulate your immune system
- 11:18into recognizing those antigens
- 11:20and developing an immune response against them.
- 11:22So there's subunit vaccines.
- 11:24So the fourth class here is a weakened/inactivated vaccine.
- 11:27And so this is, I think, what most people like what
- 11:30my grandparents probably think all vaccines are,
- 11:33is basically we take a pathogen
- 11:35and we weaken it in some way, or we kill it, right?
- 11:39And then that's the construct of the vaccine
- 11:40and that's what's injected into you.
- 11:42And we go through this similar process there
- 11:45that literally mimics natural infection, right?
- 11:47Where your cells are infected by this weakened form
- 11:51of the virus, the virus replicates,
- 11:53and that's how we get antigens to the APCs.
- 11:56So this is the construct used in of course
- 11:58some classic vaccines like MMR, polio vaccine,
- 12:01but again, it's slower manufacturing, right?
- 12:03Because we have to cultivate the virus
- 12:04in the lab and then it also requires adjuvants.
- 12:07So I don't think there's currently any plans
- 12:10to have US phase three trials
- 12:12of weaken inactivated vaccines, but there are in China.
- 12:15So Sinopharm and Sinovac vaccines
- 12:17were using this construct.
- 12:21So that's just a bit of a background in immunology
- 12:24and how all this works and how we think about preventing
- 12:26infection with SARS-CoV-2 and hopefully preventing
- 12:29clinical disease COVID-19 disease.
- 12:32So now we're gonna segue to talk a little bit
- 12:34about the vaccine development process, right?
- 12:36'Cause this has all happened extremely fast.
- 12:38So let's talk about sort of the process whereby
- 12:40vaccine products are typically brought to market, right.
- 12:43And what looks a little bit different
- 12:45about the COVID-19 vaccine development process?
- 12:49So this is a figure from a nice New England journal paper
- 12:52that's referenced at the bottom
- 12:54that's just talking about sort of what's different
- 12:56this go around in terms of how are we accelerating
- 12:58the vaccine development process.
- 13:00And so I think as biostatisticians,
- 13:01anyone who works on clinical trials is fairly familiar
- 13:04with the traditional paradigm
- 13:06for bringing products to market, right.
- 13:08It involves sort of a lot of R&D
- 13:11in the lab, preclinical work
- 13:13and then you start doing human trials in phase one,
- 13:16these are small dose finding safety trials,
- 13:19checking whether these vaccines
- 13:21generate any immune response.
- 13:23And then what we'll often do is in vaccine trials
- 13:25is run a small randomized trial.
- 13:27That's a phase two trial, right?
- 13:29We're we'll have a placebo control,
- 13:31maybe pick out a particularly high risk population
- 13:34and start to see if we're getting
- 13:35any efficacy signal, right?
- 13:37And this is a very deliberate process, right?
- 13:39Phase one typically advances very slowly.
- 13:41We have lots of safety concerns.
- 13:42Phase two, we think very hard about whether the efficacy
- 13:45signal was really worth it to advance a candidate to
- 13:48phase three and it's a very deliberate process, right?
- 13:51To get to this phase three licensure trial, right?
- 13:53So the phase three trial is the big one involving
- 13:56the most participants.
- 13:57It's a randomized controlled trial, right?
- 14:00Enrolling many, many subjects that's well powered
- 14:02to detect ethicacy signals and based on the results
- 14:04of that phase three trial and safety data
- 14:07that's been accumulated throughout
- 14:08this whole process, right.
- 14:10We're able to provide licensure ideally for a product.
- 14:14And so that's sort of the clinical development process,
- 14:17but also in the context of COVID vaccines
- 14:19it's important to think about
- 14:20the manufacturing process, right.
- 14:21And how that looks a little bit different.
- 14:23So typically right, companies are very sort of hesitant
- 14:27to scale up manufacturing before they know that they have
- 14:31a product that will be licensed, right.
- 14:32Which makes sense, you know, they're sort of risk averse.
- 14:34We don't want to start manufacturing a product
- 14:36that may ultimately be shot down by the FDA.
- 14:39So really large scale manufacturing
- 14:40is not happening until after product licensure.
- 14:44So what's happening with COVID vaccine
- 14:45is basically this whole long deliberate timeline
- 14:49is being compressed into a shorter time period.
- 14:52And so how do we do that?
- 14:53Well, basically what happens is we've collapsed
- 14:56the phase one and phase two trials, right?
- 14:58So we're doing small safety studies.
- 15:00We're checking whether these vaccines
- 15:01are generating immune responses,
- 15:03but we're really not doing that smaller efficacy study
- 15:06that is typical of vaccine development.
- 15:10And so we're collapsing the phase one and two process,
- 15:13the phase three process is where we're at, right.
- 15:15We're doing these large scale trials, right?
- 15:16Because we need robust efficacy data
- 15:19and we need robust safety data to gain licensure,
- 15:22but a big thing that has changed, so the clinical process
- 15:24yeah a little bit compressed, but mostly the same,
- 15:27the big thing that's changed
- 15:28is the manufacturing process, right.
- 15:30Is we wanna make sure that once a vaccine is licensed
- 15:33and is proven to be safe and effective that we're able
- 15:37to start distributing that vaccine immediately.
- 15:39So that means that manufacturing needs to start ramping
- 15:41up right before we ever have a signal of efficacy
- 15:45and that's a huge risk for companies to take.
- 15:47So, I'll talk in a couple of slides about sort of how
- 15:51the government has come in to try to remove
- 15:54some of that risk from these companies
- 15:56and then the next slide I think is just showing sort of
- 15:59that it's really impressive that we're even talking
- 16:01about potentially having a COVID vaccine available this year
- 16:05or early next year, just given the timelines
- 16:08that are required to bring effective vaccines to market.
- 16:11And so here's just a few, you know,
- 16:13polio, measles, chickenpox, mumps,
- 16:14all multiple years of development for these vaccines,
- 16:18you could add malaria on this list.
- 16:19It took about 30 years
- 16:20to get a partially effective malaria vaccine to market.
- 16:24So this is typically a very long process, right?
- 16:26And for COVID, we're looking at hopefully doing this
- 16:29in just under a year or two.
- 16:31So how is the US government playing a role in this?
- 16:35Well, it's through this program that you may have heard of
- 16:37called Operation Warp Speed,
- 16:39which is this huge convoluted mess of an amalgamation
- 16:44of programs across the government
- 16:45from DOD to many branches of NIH, BARDA, NIAID,
- 16:50so it's sort of all over the place.
- 16:51And this is really just the same figure
- 16:54that I showed you from the New England journal paper.
- 16:57Just maybe a slightly more confusing
- 17:01if you ask me, I don't think Edward Tufte,
- 17:03he would be a big fan of graphic
- 17:05but the point here I want to mention
- 17:07is how is the government responding
- 17:10to COVID vaccine development?
- 17:11How are they contributing to that process?
- 17:13Well, there's really two ways that they've offered
- 17:15to accelerate the process.
- 17:17The first is through funding
- 17:19of phase three clinical trials, right?
- 17:21So a number of companies, six of the major companies,
- 17:24basically every company that's running a phase three trial
- 17:26in the US besides Pfizer that you've heard about
- 17:30is contracting with BARDA.
- 17:32That's an arm of the NIH,
- 17:34they're contracting with the government
- 17:36to have the government fund their phase three trials.
- 17:39So it's a joint agreement between the government
- 17:40and these companies where the government,
- 17:42you the taxpayer, right, are paying for these
- 17:45phase three trials that will eventually lead to licensure.
- 17:49So that's the first way that the government
- 17:50is sort of throwing money at this problem.
- 17:52It's through design and paying for these phase three trials.
- 17:56The second way is that they're paying
- 17:58for manufacturing, right?
- 17:59They're removing that risk for these companies
- 18:01by basically committing to buy a certain number of doses
- 18:04before we ever have any efficacy data.
- 18:06So we're in the hole basically to all of these companies
- 18:08for a fixed number of doses right.
- 18:11But that motivates the companies then to scale up
- 18:13their manufacturing ahead of the time
- 18:15that efficacy data are available.
- 18:18And that type of agreement has been entered
- 18:20into with Pfizer as well.
- 18:21So all of these companies that OWS Operation Warp Speed
- 18:24is running the phase three trials for
- 18:27also have this manufacturing agreement.
- 18:29Pfizer has that manufacturing agreement as well.
- 18:33So what role have I played in any of this big messy thing?
- 18:38So I work with a great group of scientists
- 18:41in the COVID-19 Prevention Network.
- 18:43So this was a clinical trials network established
- 18:45by National Institute of Allergies and Infectious Disease
- 18:48and NIAID so that's an arm of NIH,
- 18:51and it's basically anyone who works in clinical trials
- 18:54is fairly familiar
- 18:55with these clinical trials networks, right?
- 18:57It's an amalgamation of researchers and study sites,
- 19:01laboratories, people who focus on recruitment and retention
- 19:04of trial participants, statisticians.
- 19:07So it's researchers who are really experts
- 19:09in running clinical trials,
- 19:10designing clinical trials
- 19:12and ensuring their robust conduct.
- 19:15So the CoVPN was formed by basically leveraging
- 19:19four existing clinical trials networks.
- 19:21One of which I was a part of,
- 19:22which is the HIV vaccine trials network.
- 19:24And so from our group, we've really brought a great group
- 19:27of statisticians, many of whom are at the Fred Hutch
- 19:30in Seattle as well as great groups of laboratories at U Dub.
- 19:34And so what are the roles
- 19:36that we're playing in these trials?
- 19:38So in our statistical group,
- 19:40there's a couple of statisticians who are designated
- 19:44as like CoVPN representatives
- 19:46for each of these phase three trials.
- 19:48So I sit on calls with these trials and advise
- 19:53on their design and analysis approaches
- 19:55for their efficacy monitoring, for their safety monitoring.
- 19:58We help them address DSMB and FDA comments
- 20:01and sort of that's all happening in conjunction
- 20:04with both government statisticians, right.
- 20:06Representatives of BARDA and NIAID
- 20:10as well as company statisticians.
- 20:12And so we get on these calls and, you know,
- 20:14nerd out over clinical trials,
- 20:16statistical decision-making, and it's a good old time.
- 20:21Another aspect that we really contribute a lot on,
- 20:24or that CoVPN has sort of been tasked with taking
- 20:26the lead on is the development of immune correlates.
- 20:29And so that's the part of my talk
- 20:31where I'll get a little bit into statistics
- 20:32and talking about what immune correlates are,
- 20:34some of the types of analytic approaches
- 20:36we use to study those and the idea of immune correlates
- 20:38just to give you a teaser
- 20:40so you don't, you know, sign off Zoom early.
- 20:42So immune correlates are really the idea there is
- 20:45we're looking for immune responses that are predictive
- 20:48of the vaccines working, right.
- 20:52So what we'd really like to be able to do is understand,
- 20:54okay, if we're able to generate this level
- 20:56of neutralizing antibody,
- 20:58then that will lead to this level of protective effect
- 21:00of the vaccine, right?
- 21:02So that's the whole goal there is identifying
- 21:04what are these immune responses that are
- 21:05responsible for providing protection?
- 21:08Okay so I'm gonna walk through just a few of the design
- 21:11and analysis questions.
- 21:12And so these are things that have come up
- 21:14as we've worked with these company statisticians,
- 21:16as we thought about sort of the whole OWS vaccine program,
- 21:20what are some of the issues that statisticians
- 21:22are kicking around and people who have worked
- 21:24on clinical trials, right,
- 21:25a lot of these issues aren't gonna be new
- 21:27and one thing that I think is sort of interesting about this
- 21:30whole pandemic and operating as a public health professional
- 21:34in this and a clinical trial statistician in particular,
- 21:37is that a lot of things that we take for granted
- 21:39as scientists are either very confusing
- 21:42or sort of counterintuitive for a lot of the lay public.
- 21:45And so it's been sort of interesting to have that laid bare.
- 21:48In some of these issues, some of these things
- 21:50that we think are no-brainers like doing interim analysis
- 21:53for example are kind of highly controversial
- 21:56and have ended up being, you know,
- 21:57sort of areas of huge disputes.
- 21:59And so I just want to run through some of these issues
- 22:01that I think are quite fascinating, a lot of which,
- 22:04you know, really don't have a correct answer
- 22:06and they're really just sort of food for thought
- 22:07the types of things that we're thinking about
- 22:09when we're designing these trials.
- 22:11So I'll start by just giving a sort of more specific idea
- 22:16of what these trials look like and how they're conducted
- 22:18and I've picked AstraZeneca because that's the one
- 22:20I've worked on for the longest and most closely,
- 22:23but all of the trials sort of follow this similar design.
- 22:26And so the first thing I'll note
- 22:27is that you can read these trial protocols.
- 22:29So one of the interesting things that's happened
- 22:31in this COVID-19 development processes
- 22:33is there was a huge public push led by like Eric Topol
- 22:36and others to have the protocols of these trials
- 22:39made public, which when it happened was I guess
- 22:43when that push started happening, you know,
- 22:45I emailed all my colleagues and said,
- 22:47really do we not usually make protocols public?
- 22:50And that was just sort of interesting disconnect
- 22:51for me as an academic who's used to sort of everything
- 22:54being open science and that's a no brainer right.
- 22:57Working in this setting, right,
- 22:58where these protocols are really seen as trade secrets
- 23:00for pharmaceutical companies.
- 23:02So it's really unusual that actually these protocols
- 23:05for clinical trials have been made public.
- 23:06So it's sort of neat, but one of the things that happened
- 23:09is all of these protocols went public and reporters
- 23:12got their hands on them and said, wow,
- 23:13these are really dense documents, right?
- 23:15If you've ever looked at the clinical trial protocol,
- 23:18it's like a hundred pages of very specific definitions
- 23:21and safety monitoring and what symptoms lists
- 23:24you're gonna use and what surveys
- 23:26you're gonna give to people.
- 23:27So they're very sort of detailed documents
- 23:28that are kind of hard for the public to parse.
- 23:32So it's been sort of a be careful what you wish for thing
- 23:34in terms of releasing these protocols, but that's an aside.
- 23:38So let's talk about actually what these trials look like.
- 23:40So here's a schematic, and again,
- 23:42this is AstraZeneca in particular,
- 23:44but this is basically the design of most of these trials
- 23:47will look something like this.
- 23:48So who is the population?
- 23:50Most of these trials are gonna be primarily in adults.
- 23:53I think Pfizer has now started
- 23:55to talk about including children.
- 23:57I'm not exactly sure where that's happening,
- 23:59but adults for the most part,
- 24:01these are mostly healthy individuals
- 24:05that don't have, you know, chronic diseases
- 24:07that are at risk or high risk of death.
- 24:10And we're really looking at targeting individuals
- 24:12who are at an increased risk for SARS-CoV-2 acquisition
- 24:15and severe COVID disease
- 24:17and so the idea there is number one
- 24:19these are the people that are bearing
- 24:20the brunt of the pandemic, right?
- 24:23So we want to be able to get a product to those people
- 24:26as fast as possible.
- 24:27But number two also, right, that means that we'll accrure
- 24:29from a sort of cold hearted and statistician point of view
- 24:32that means we'll accrue end points faster.
- 24:34We'll observe more cases of COVID-19 disease
- 24:37and potentially get an efficacy signal a little bit faster.
- 24:40So there's a lot of interest in sort of recruiting
- 24:43and retaining individuals at high risk for COVID-19.
- 24:45So you can go onto the COVID-19 prevention trials network
- 24:48and fill out a survey, right.
- 24:49Then we'll basically under the hood
- 24:51assess your risk for COVID-19
- 24:53and if you're found to be at high risk,
- 24:54we'll aggressively email you and try to get you
- 24:55enrolled in one of these trials.
- 24:57If you're at low risk,
- 24:57we'll say, thanks for taking the survey,
- 24:59we'll be in touch and likely
- 25:01you won't hear from us anytime soon.
- 25:03Okay so that's the trial population.
- 25:05So how does the actual trial conduct look?
- 25:07So there's kind of a mixture here.
- 25:10AstraZeneca is using a two to one randomization scheme.
- 25:13So you have two chances of getting the active vaccine
- 25:16versus one chance of getting a placebo.
- 25:18And in this case, it's a true placebo, just a saline dose
- 25:22and then most of the vaccines, most all with Janssen
- 25:26being the accepted are two dose vaccines.
- 25:28So you receive the first dose at day one
- 25:30and the second dose about a month later.
- 25:32And in the interim, we take a couple of measurements.
- 25:34We have a phone call to assess reactogenicity right.
- 25:38Does your arm hurt, or have you experienced any adverse side
- 25:41effects of the first dose of vaccine?
- 25:44And then there's also an immune response measurement
- 25:46that happens after a couple of days.
- 25:48So we get an early signal
- 25:49of how immunogenetic these vaccines are.
- 25:51And so then individuals come in for their second dose
- 25:53of vaccine and it's a similar story, right?
- 25:55Did you have any reactions?
- 25:57We measure your immune response and after that,
- 25:59that's sort of when the clock starts for active follow-ups.
- 26:02So this day 57, that's two weeks roughly after,
- 26:07am I doing that math right?
- 26:08Well, it looks like roughly two weeks after the second dose
- 26:12of the vaccine is typically when this clock
- 26:14is gonna start and we're gonna start counting COVID events.
- 26:17And then it's sort of just the standard sort of game we play
- 26:21in clinical trials.
- 26:22We wait for events to accrue.
- 26:23We have certain monitoring plan
- 26:25for when we're gonna check for efficacy
- 26:27and we'll talk about some of that.
- 26:28So, I just want to note that there's sort of two ways
- 26:30that we're ascertaining events
- 26:32that are happening here, right?
- 26:33The first is passive monitoring.
- 26:35What that means is we basically wait for individuals
- 26:37to present with symptoms of COVID 19, right?
- 26:39So you get a cough, you lose taste, right?
- 26:41You call the study site, right?
- 26:45So I am having these symptoms.
- 26:46They say, come on in.
- 26:47You get a PCR test to see whether you're infected.
- 26:50And in that case, you would count
- 26:51as a COVID-19 endpoint, right?
- 26:53If you check off some check boxes for symptoms
- 26:56with COVID-19 disease, you have a PCR positive test.
- 26:59You'd go down as a COVID 19 endpoint.
- 27:01There's also these sort of active follow-up visits.
- 27:04So these like day 90, day, 180 and day 360,
- 27:08and at those visits we'll do a serology check.
- 27:11And what that means is we basically take a blood draw
- 27:13and we measure whether you have antibodies
- 27:16against SARS-CoV-2, right, antibodies that are distinct
- 27:19from the antibodies that are generated
- 27:20in response to the vaccine.
- 27:21So we're basically able to tell whether you were infected
- 27:24in this sort of interim period,
- 27:26when you show up for these visits.
- 27:29So that's active follow up
- 27:30and so there you're gonna be able
- 27:31to pick up sort of asymptomatic cases, right?
- 27:33'Cause if you never have symptoms, you'll never come in
- 27:36and be captured by passive followup.
- 27:38So we have to wait for these set clinic visits
- 27:40to do the serology testing,
- 27:42to ascertain it asymptomatic cases.
- 27:44And so this is gonna actually play a role
- 27:46in a little bit, when I started talking about, you know,
- 27:48what are the end points that we're thinking about measuring?
- 27:50Like, what do we want to know how well
- 27:52the vaccine works at preventing?
- 27:53Is it asymptomatic infection?
- 27:55Is it disease?
- 27:56Is it severe disease and so forth?
- 27:57So we'll talk through some of those issues,
- 27:59but just want to note already that the design has started
- 28:02to inform some of the challenges that we might see
- 28:04when we want to talk about how well the vaccine works
- 28:07against certain forms of infection and disease.
- 28:10And so I think if you read the newspaper and you'll see
- 28:13the term vaccine efficacy tossed around a lot.
- 28:15So the first thing I want to talk about is right,
- 28:17what is the primary hypothesis
- 28:19that these trials are trying to test?
- 28:21And what is the parameter?
- 28:23What is the estimate, right, that they're going after
- 28:25in these trials and for whatever reason
- 28:27nobody consulted me when they decided that VE
- 28:29would be measured in this way.
- 28:32But for whatever reason, we studied this that we quantify
- 28:35the efficacy of a vaccine in a sort of weird way.
- 28:37So a vaccine efficacy, we describe as the percent reduction
- 28:40in relative risk comparing vaccine to placebo.
- 28:43So it's this one minus a risk ratio.
- 28:46There's a one minus a risk ratio where you take the risk
- 28:49in the vaccine and the numerator and the risk
- 28:51in the placebo and the denominator.
- 28:53So, I mean,
- 28:54we can just play a quick little intuitive game, right?
- 28:56How do we get a VE close
- 28:57to one that would be a perfect vaccine?
- 29:00Well, we would make the risk
- 29:01in the vaccine close to zero, right?
- 29:03So that sorta makes sense.
- 29:04If you have a perfectly effective vaccine,
- 29:06there'll be no risk of infection and or disease
- 29:08amongst the vaccinated.
- 29:09So you would get VE close to one.
- 29:11But on the other hand, how do we make VE zero?
- 29:13Well, we would take the risk in the vaccine
- 29:16and set it equal to the risk in the placebo, right.
- 29:18In which case basically saying the vaccine's not doing
- 29:21anything and then on the other hand,
- 29:23a VE is negative, right?
- 29:24That's indicating that there's actually higher risk
- 29:26in the vaccine.
- 29:28So just to give you sort of a few reference points, right?
- 29:31So that VE of one is perfect, VE of zero is nothing
- 29:34and what we're really hoping for with these COVID trials
- 29:38is a VE of at least 50%.
- 29:40And that's sort of the cutoff that FDA guidance
- 29:42has stipulated is that you need to show a point estimate
- 29:46of VE for your primary end point.
- 29:48And again, we'll talk about what these primary end points
- 29:50are but we need a VE against a primary end point
- 29:53of at least 50%
- 29:54and we need to definitively rule out the possibility
- 29:59that the vaccine efficacy is less than 30%.
- 30:02So basically we have to reject the null hypothesis
- 30:05that VE is less than 30% along with having a point
- 30:08estimate of VE being greater than 50%, right.
- 30:11And we need to do that while controlling type one error
- 30:13at two and a half percent.
- 30:16Okay and so here, just one final note,
- 30:19since this is a statistics talk,
- 30:20I'll talk a little bit more
- 30:21about what I mean by risk, right?
- 30:23So risk here can be quantified in a number of ways
- 30:26and it often is.
- 30:27So we can quantify this using hazards, for example,
- 30:29like you can imagine fitting a Cox model, right.
- 30:31A proportional hazards model, right.
- 30:33That only adjusts for vaccine, right.
- 30:35And presenting like one minus a hazard ratio
- 30:37from a Cox model, that's something that's commonly done.
- 30:39You can also think about cumulative incidents, right?
- 30:41So like mapping,
- 30:43maybe one minus a survival probability as a way
- 30:46of quantifying risk, incidents rate ratios.
- 30:49So they're all sort of used for different vaccines.
- 30:52And usually we like to sort of argue
- 30:55about which one of these is better
- 30:56and I've thought a lot about that in my career.
- 30:58And in this setting, it turns out because COVID
- 31:00is such a rare event that all of these ways of quantifying
- 31:03rates are basically the same and you end up
- 31:05with almost identical operating characteristics of a trial.
- 31:08So it's really not worth sort of losing sleep over
- 31:10whether we're talking about VE in terms of hazard
- 31:12or incidents or incidents rate and so forth.
- 31:16So how are folks going about estimating this VE?
- 31:19Here's just a quick table of the four most advanced
- 31:22phase three trials,
- 31:23the ones that have released their protocols at least.
- 31:25So we see for Moderna, AstraZeneca, and Janssen,
- 31:28they're using pretty kind of the standard approaches.
- 31:31Moderna a Cox model as I describe,
- 31:33AstraZeneca a Poisson regression model,
- 31:35it's like, okay, that's basically a Cox model,
- 31:38and then Janssen is using a sort of exact binomial test
- 31:41with this sequential probability ratio rest.
- 31:44Pfizer is a little bit of the oddball.
- 31:46So they have stipulated a bayesian approach
- 31:49wherein they're basically specifying a prior
- 31:53for vaccine efficacy and are using sort of
- 31:55a beta-binomial bayesian approach to evaluate
- 31:58the posterior probability of the vaccine efficacy
- 32:00is greater than 30% and so at the end of the day,
- 32:04there's four different statistical methods here.
- 32:06Again, if you do a simulation study with parameters
- 32:09that are approximately similar to what we expect to see
- 32:11in these COVID trials,
- 32:12you're really not gonna see much difference in terms
- 32:14of operating characteristics across these.
- 32:16So it's interesting to notice that assertions
- 32:18that there's these different approaches,
- 32:19but at the end of the day,
- 32:20we're basically talking about how many vaccinated people
- 32:22get infected, how many placebo people got infected,
- 32:25and almost all of these methods are gonna yield
- 32:27very similar inference.
- 32:29When it comes down to brass tacks,
- 32:31how many numbers fall into those categories?
- 32:33So that's a little bit about sort of
- 32:36how we quantify VE in these settings
- 32:38but one of the big things I haven't described yet
- 32:40is VE against what, right?
- 32:42What is the end point that we're measuring here?
- 32:43And so here's a figure from a paper we just had come out
- 32:47in Annals of Internal Medicine, the link's here.
- 32:50So this is where we were spending a lot of time,
- 32:52you know, earlier this summer, thinking about,
- 32:54you know, what's the right end point,
- 32:56what's the right end point for a primary analysis
- 32:58of the clinical trial.
- 32:59And it's complicated for something like SARS-CoV-2, right?
- 33:03Because we know we can start up here
- 33:04with the SARS-CoV-2 infection, right?
- 33:07That's sort of the base, you can become infected
- 33:09and then a number of things can happen, right?
- 33:11You can go on to be infected but develop no symptoms.
- 33:14So we would call that an asymptomatic infection,
- 33:17or you can develop symptoms, right.
- 33:18In which case we don't call you
- 33:20a SAR-CoV-2 infection anymore,
- 33:21we call you a COVID-19 disease endpoint.
- 33:25You have a clinical manifestation of your infection.
- 33:28But even beyond that, right,
- 33:30amongst people who exhibit symptoms
- 33:31some of them, maybe many of them are quite mild, right.
- 33:34So we have this kind of category of non-severe COVID,
- 33:38whereas others we know that are extremely adversely
- 33:41impacted by infection and end up with severe COVID disease.
- 33:45So you have all of these choices of sort of
- 33:49which end points you might want to talk about
- 33:51and so I'll kind of walk through some what I see
- 33:53as the positives and negatives of this and then I'll also
- 33:56talk about this burden of disease
- 33:58very briefly end point that we've put together
- 34:01and so that's kind of a composite end point
- 34:03that we've suggested that could kind of bring all
- 34:05of these different end points together.
- 34:07Okay so starting with SARS-CoV-2 infection, right?
- 34:09Why might we like any sort of any infection, right.
- 34:12Asymptomatic, symptomatic don't care,
- 34:14let's count any infection as an event
- 34:17and measure VE against preventing infection.
- 34:20Okay and so that's definitely relevant, right.
- 34:22It's relevant the context of a pandemic.
- 34:24We're preventing infections,
- 34:25we're preventing spread of the disease,
- 34:27we're bringing our knot down, we're impacting the pandemic.
- 34:30And moreover, we're going to see many more infections
- 34:33than we will cases of symptomatic disease.
- 34:36We know that many people who were infected
- 34:37never go on to develop symptoms
- 34:39so thinking about having an answer faster, right.
- 34:42SARS-CoV-2 infection is a nice endpoint,
- 34:44but then the question is,
- 34:46is it a clinically relevant endpoint?
- 34:47So it's really not describing an impact on patients at all.
- 34:52So we could kind of question its relevance
- 34:56from that perspective.
- 34:57The other thing, right, is that we remember going back
- 34:59to the study design, we're only able to ascertain
- 35:01asymptomatic infections sort of very coarsely in time
- 35:05and moreover you have this phenomenon that happens
- 35:09is that when you're testing many, many individuals, right.
- 35:12It's sort of the classic biostat
- 35:14one-on-one problem that we give people, right.
- 35:16You're testing many individuals, but the prevalence is low.
- 35:19So even if you have high sensitivity and high specificity,
- 35:22you could end up with low positive predictive value.
- 35:25And the effect of that when you come to the time to analyze
- 35:28the data is that you'll be biasing VE towards the knoll.
- 35:31So it's actually, while it seems like maybe a nice end point
- 35:35from the perspective of observing many infections,
- 35:37it's a very challenging endpoint to analyze quantitatively.
- 35:41So moving down we could talk about COVID.
- 35:43So again, COVID is just infection,
- 35:46PCR confirmed infection with clinical symptoms.
- 35:49So that's of course more clinically relevant, right.
- 35:51Because we're starting to talk about
- 35:53an impact, excuse me, the endpoint that impacts patients.
- 35:57All right so that's more clinically relevant and moreover
- 36:00we'll expect to have a reasonable number of cases, right.
- 36:03By including more mild cases, for example,
- 36:06in this endpoint definition.
- 36:09But then on the other side of that coin
- 36:10is it really that clinically relevant
- 36:12if we're just talking about mild symptoms?
- 36:15We're talking about a disease where you get it
- 36:16and you end up with a little cough for a couple of weeks
- 36:18and that's it.
- 36:19So then maybe you suggest using severe COVID right.
- 36:22That's the most clinically relevant one.
- 36:24We want to be protecting the most vulnerable individuals
- 36:26so we should be quantifying how well our vaccines
- 36:28work towards preventing those most severe end points.
- 36:33And so most clinically relevant,
- 36:35and also there's sort of a long history
- 36:37of vaccine development where really we see the best VE
- 36:40against severe cases of disease.
- 36:43So that's really where we expect the vaccines
- 36:45to have the most impact is maybe we are not preventing
- 36:47you from being infected but we're lessening the symptoms
- 36:51once you become infected.
- 36:52So we're not totally blocking transmission
- 36:55but we're making a clinical impact on disease
- 36:57and that's sort of been seen
- 36:58for a number of vaccines in the past.
- 37:00The downside of this end point of course
- 37:02is that there's very few cases expected to be observed.
- 37:04So amongst all infections,
- 37:05only a fraction have any symptoms.
- 37:07Amongst those with any symptoms,
- 37:09only a fraction develops severe symptoms.
- 37:10So we're really whittling away the number of end points.
- 37:13So we need to do larger trials
- 37:15or have longer follow-up to evaluate this endpoint.
- 37:19And so in that paper, I'm sort of pressed for time
- 37:21so I won't spend too much time talking about this,
- 37:24we also proposed this burden of disease measure
- 37:26where you're sort of scoring these these outcomes, right?
- 37:29So maybe you would get a score of zero
- 37:31if you're an asymptomatic infection
- 37:33'cause it's really no burden on you as a patient, right?
- 37:36You don't have any symptoms.
- 37:37And then we're sort of assigning arbitrarily
- 37:39a score of one for non severe COVID so that's like
- 37:42mild cases of COVID and a score of two
- 37:45for severe cases of COVID and this end point actually
- 37:48has some nice operating characteristics we think,
- 37:51but of course it's subject to controversy, anytime you start
- 37:53talking about an ordinal scoring system, right,
- 37:57you start to raise questions about how you're assigning
- 37:59the burden of disease score, right?
- 38:01Why should severe cases be a two
- 38:03versus a three versus a five and so forth?
- 38:06So you can kind of get bogged down
- 38:07in some of the specifics of that.
- 38:10So what has FDA said about this?
- 38:12So FDA guidance documents states that either
- 38:15the COVID end point or SARS-CoV-2 infection
- 38:18is an acceptable primary endpoint
- 38:19and then somewhat ironically OWS has been telling companies
- 38:22that infection alone is not acceptable
- 38:24as a primary end point.
- 38:25So we had one company that was interested in including
- 38:28that as co-primary and for whatever reason we told them
- 38:31please don't do that, and then beyond that so COVID
- 38:36has sort of won out as the end point of choice.
- 38:39But beyond that FDA guidance states that companies should
- 38:42consider powering efficacy trials
- 38:44for the severe COVID endpoint as a co-primary or at least
- 38:48as a key secondary endpoint in the trial.
- 38:51And so so far only Janssen has taken them up on that offer
- 38:54of making severe COVID primary.
- 38:56And that's why, if you look at the number of individuals
- 38:58that are planning to enroll in their trial,
- 38:59it's twice as many as any of the other OWS trials.
- 39:03So like AstraZeneca is planning for 30,000,
- 39:04Janssen is planning for 60,000 in their trial.
- 39:08And that's the power, to see more cases of severe disease
- 39:10to be sufficiently powered to detect VE against that.
- 39:15So this is a controversial slide.
- 39:17Or this is virtual topic I found,
- 39:20again, something that clinical trials statisticians
- 39:22sort of take for granted is doing interim analyses, right?
- 39:26If the treatment is working and we have enough evidence
- 39:28to claim that a treatment is working,
- 39:29we'd like to stop that trial early
- 39:31to get that treatment to patients, right.
- 39:33One would think that that's true here
- 39:35and so many of these trials
- 39:36were designed with aggressive sort of interim looks, right?
- 39:40Because we're in the middle of the pandemic
- 39:41and we'd like to get a vaccine to individuals
- 39:44as quickly as possible.
- 39:45So I have a table, we won't go through it all here,
- 39:47just sort of the planned interim analysis
- 39:50for these different trials.
- 39:52I would say Pfizer seems to be the most aggressive so far.
- 39:56They have five interim looks or four interim looks
- 40:00and a final look at their data, right?
- 40:02So that's fairly aggressive.
- 40:03OWS again, the trials that we're running,
- 40:07we're really encouraging companies to be a bit
- 40:09more conservative in the approach to this
- 40:11and only maybe two or three
- 40:13and so you see what's been adopted
- 40:14by Moderna and AstraZeneca
- 40:17and so this was really a big point of contention
- 40:19I think when these protocols were made public is this idea
- 40:22that like, can you really know that a vaccine works
- 40:25based on 32 data points, right?
- 40:27We're talking about a vaccine that's going to be given
- 40:30to billions of people around
- 40:32the world based on these results
- 40:33and you're gonna do that based
- 40:35on the results in 32 individuals?
- 40:37And like, so I can stare at the math and say that like, yes,
- 40:40that appropriately controls type one error and so forth,
- 40:42but it still makes me just feel a little bit uncomfortable.
- 40:45There's a bit of dissonance between sort of my life
- 40:48as a statistician and just me being a human
- 40:50and saying 32 data points is probably not enough
- 40:52to decide to vaccinate billions of people.
- 40:54And so a lot of people I think sort of shared
- 40:57that viewpoint and in response FDA has now been sort of
- 41:01backpedaling in a way and asking companies to provide more
- 41:06data in order to grant an emergency authorization
- 41:10for their vaccine.
- 41:11So this EUA mechanism that FDA has of approving vaccines.
- 41:14And so in addition to an efficacy signal,
- 41:17now companies also are gonna be required, I think,
- 41:20and this is sort of still a moving target so this is maybe
- 41:23like data news at this point but I think prior to offering
- 41:26an EUA, FDA has now said that companies need to have 50%
- 41:29of participants complete at least two months of follow-up
- 41:33for safety signals and that you need to have at least
- 41:36six COVID cases in the oldest age group.
- 41:39Of course, that's an age group of particular interest
- 41:41in terms of severe cases and at least five cases
- 41:44of severe COVID in the placebo group.
- 41:45So they want to be able to see some data,
- 41:48even if you're not specifying severe COVID
- 41:50as a primary end point,
- 41:51they want to be able to see some data,
- 41:53some signal of efficacy against that
- 41:55in order to grant licensure.
- 41:57So I'll sort of, I won't go through this slide.
- 42:01It's just to say that like,
- 42:02sort of when Pfizer released their protocol,
- 42:04everyone was like, ooh a bayesian analysis
- 42:06and got very sort of skeptical, right?
- 42:09Because the Pfizer CEO has been out there
- 42:11sort of chest thumping and saying they're gonna have
- 42:13a vaccine before the election and so forth
- 42:15and then they came out with this bayesian design
- 42:17that was a little atypical and so everybody was asking
- 42:19the question, well, are they trying to hide something?
- 42:21So I sort of did a quick analysis
- 42:23and found that really it doesn't look that different
- 42:25than a classic kind of post hoc monitored design.
- 42:28And if you want to read more about that,
- 42:30I have some slides up on my GitHub about it.
- 42:33So let's see, I'm running low on time
- 42:36so I'm gonna skip over sort of the question
- 42:39of what happens if efficacy is declared early.
- 42:41So I have some reasons that we should be excited, right?
- 42:43If one of these trials stops earlier, I can get a vaccine.
- 42:46There's good data that the vaccine works
- 42:48and that's nice.
- 42:50I'd like to go back to something resembling normal
- 42:52as I'm sure you all would,
- 42:54but of course there's reasons to be concerned, right?
- 42:56If efficacy is declared early in particular,
- 42:58if that means that blinded follow-up
- 43:01in a study stops, right?
- 43:02Because that means we have no way
- 43:03to assess how durable the vaccine is.
- 43:05We won't be able to assess VE
- 43:07and key subgroups that we care about.
- 43:09We might not be able to assess VE
- 43:11formally against severe end points.
- 43:13So there's real sort of concerns
- 43:15about stopping these trials too early,
- 43:17and what the implications of that
- 43:18are both for evaluating the vaccine in question,
- 43:21but as well as how it impacts
- 43:23the other clinical trials that are ongoing.
- 43:26And of course in the current political climate,
- 43:29everybody's very concerned about the role
- 43:31political pressure might play in all of this.
- 43:33So yeah, so it's kind of a double-edged sword in some sense
- 43:37in terms of what happens if efficacy is declared early,
- 43:42but I want to save just a few minutes
- 43:43to talk about vaccine correlates 'cause I promised
- 43:45that I would show you some math and prove to you
- 43:47that I'm a real statistician.
- 43:48So let's do a little bit of that.
- 43:51So again, we're kind of shifting gears here.
- 43:53So that's the end of sort of talking about the primary
- 43:55analysis of these trials,
- 43:56what's gonna lead to their licensure.
- 43:58And the correlates of protection
- 44:00is sort of a key secondary analysis
- 44:02and so why is it so important
- 44:04that we're able to establish correlates of protection?
- 44:07Well, because it's gonna speed up
- 44:09the whole vaccine development process.
- 44:12So again, a correlative protection is really just,
- 44:14it's an immune response and really an assay
- 44:18to measure that immune response that's been validated
- 44:20to reliably predict vaccine efficacy.
- 44:23So why is that so important?
- 44:25Well, basically what we're hoping to achieve
- 44:28is the establishment of a surrogate
- 44:29endpoint for COVID disease right?
- 44:32So I've sort of mentioned the numbers that we're talking
- 44:34about in these phase three trials,
- 44:36enrolling 30,000 participants, 60,000 participants
- 44:40and ending up with one or two years of followup, right.
- 44:42Just to be able to answer the primary question, right.
- 44:44Does the vaccine prevent infection and/or disease?
- 44:48So that's a huge, expensive clinical trial.
- 44:50It takes a long time to get an answer
- 44:52and so it would be very nice if all we had to do right
- 44:56was give people the doses of vaccine that they need,
- 44:59wait two weeks and measure their immune response
- 45:02and understand does that vaccine work or not.
- 45:05That would be a much faster vaccine development process
- 45:07than where we're currently at
- 45:09in having to run these enormous phase three trials.
- 45:11So it's valuable for establishing a surrogate endpoint.
- 45:14It's also valuable for accelerating approval
- 45:17of vaccines that have been licensed in certain populations,
- 45:22but not others.
- 45:23For example, I mentioned that these phase three trials
- 45:25are mostly being conducted in adults.
- 45:27Well, what if we want to also obtain licensure for use
- 45:30of this vaccine in children?
- 45:32Well, if we had an established immune correlate
- 45:34we wouldn't have to do
- 45:35a large randomized trial in children.
- 45:37We could do it just a small immunogenicity study
- 45:39and use the correlates results to bridge the VE
- 45:42that we observed from the phase three trial.
- 45:45That's the immune response that we've observed
- 45:47in these children or pregnant women for example are
- 45:49another key population they're being
- 45:51excluded from these phase three trials
- 45:53but we'd like to understand if these vaccines
- 45:55are safe and effective in those women as well.
- 45:59So really this is one of the key goals
- 46:01of this whole OWS program and the key role
- 46:05that we're playing in the CoVPN is developing
- 46:08the sampling plan and the statistical analysis plan
- 46:12for the immune correlate studies
- 46:14and so it's just a little bit of the statistical issues
- 46:17that we're dealing with in these trials, right,
- 46:20is that sort of running assays
- 46:22so running these immuno assays on 30,000, 60,000 individuals
- 46:26takes a long time, it's expensive, and as it turns out,
- 46:30it's really overkill in terms of statistical power.
- 46:34So we can actually be a little bit more
- 46:35clever about how we design these correlate studies in order
- 46:40to get answers faster and more cheaply.
- 46:42So the way we do this is we use a case cohort design.
- 46:45So we're not gonna measure immune responses
- 46:47in all trial participants,
- 46:48we're gonna measure them in a sub cohort
- 46:50and that sub cohort will consist
- 46:51of a stratified random sub cohort.
- 46:54So we're gonna be sampling individuals randomly
- 46:56based on their baseline infection status.
- 46:58Were you infected with SARS-CoV-2 in the past?
- 47:01Based on your race, ethnicity, and based on age.
- 47:07And so based on that, we'll take a random draw
- 47:09of the trial population, about 1600 individuals,
- 47:13excuse me and everyone so I should mention right
- 47:19in the trial design everybody is having their blood drawn.
- 47:22And right now we're talking about whose blood
- 47:24are we gonna use to measure these immune responses?
- 47:27So we're gonna measure it in a random sample
- 47:29and then we're gonna wait until the trial is over
- 47:31or until one of these interim analysis concludes efficacy
- 47:35and we're gonna measure immune responses
- 47:38in all of the end points, right?
- 47:39Remember that like power in these analyses is drive
- 47:42by the individuals in which we observe endpoints.
- 47:46So we're gonna make sure we get immune responses
- 47:47in all the end point data, as in addition
- 47:49to this random sub cohort and it turns out that that's about
- 47:53as statistically efficient as running the immune assays
- 47:57on all 30,000 individuals in the trial.
- 47:59So this is this kind of classic case cohort design
- 48:01that Ross Prentice has been writing about for years
- 48:04that Norman Breslow did some sort of pioneering work
- 48:06on in the 2000s and I'll just talk a little bit about sort
- 48:11of how this complicates our life as statisticians
- 48:13and then maybe we'll leave a few minutes for questions.
- 48:16So here's the math, we made it.
- 48:18Well, the moment you've all been waiting for it
- 48:20to see some math.
- 48:21So just introducing, you know,
- 48:23why is this sampling design challenging
- 48:26from a perspective of generating estimators, right?
- 48:29Well, we can sort of immediately see that this isn't
- 48:31a totally random sample of the trial population, right?
- 48:35In particular we've over-sampled the individuals who end up
- 48:38getting diseased and it's fairly obvious
- 48:42that those individuals have potential to be very different
- 48:45than a randomly selected individual in the population.
- 48:47So we have a bias sub sample.
- 48:49So we need some statistical methodology to try to back out,
- 48:52you know, whatever this parameter is.
- 48:54We want to be estimating it in the whole trial population,
- 48:56not just in this biased sub samples.
- 48:59So how do we do that?
- 49:00So just a quick notation here,
- 49:02let's call W baseline covariates,
- 49:04A is a binary vaccine assignment,
- 49:07Y is your binary COVID endpoint for example
- 49:11and then we'll introduce this sort of indicators.
- 49:13Delta is one, if you're selected into this immune response
- 49:18sub cohort, either because you were a case,
- 49:20you were an end point or because you were randomly selected
- 49:23into the cohort.
- 49:25And then we'll call S your immune response.
- 49:28And then we'll just say, we'll represent this as Delta S,
- 49:31which just means we'll arbitrarily set everybody
- 49:33who's not in our sub cohorts immune response to be zero,
- 49:36that's arbitrary doesn't really matter.
- 49:38So let's talk about how estimation would happen.
- 49:41So let's pick a very simple parameter, right?
- 49:43Let's just say that we want to know what's the overall
- 49:45immune response in the whole population,
- 49:47not a particularly interesting parameter
- 49:50for actually measuring correlates,
- 49:51but just to motivate the types of statistical approaches
- 49:54that we use in these settings.
- 49:56So how can we control for the bias of the sampling design?
- 49:59Well, one of the most straightforward ways
- 50:01is to use the tried and true
- 50:02Horvitz-Thompson or IPTW estimator, right.
- 50:05Where we're just taking basically a sample mean
- 50:08but all our observations are sort of inverse weighted
- 50:11by their probability of being sampled into this sub cohort.
- 50:16And so that's, IPTW estimator if you're in causal inference,
- 50:18you're very familiar with this.
- 50:19If you're in survey sampling,
- 50:21very familiar with this.
- 50:22Very classical way of adjusting for this selection bias.
- 50:27It turns out that there's ways
- 50:28that we can be more efficient in doing this.
- 50:30We can use augmented estimators, AIPTW estimators.
- 50:33And the key idea there is that we take the IPTW estimator
- 50:37and we add a little bit of something to it
- 50:39and the key thing is that that little bit
- 50:41of something involves a regression of S the immune response
- 50:46onto the covariates that were used to sample individuals
- 50:50into the sub cohort.
- 50:52And so what's the intuition as
- 50:54to why this is more efficient?
- 50:56Well, you can imagine what if we had a perfect predictor
- 50:59of S measured at baseline, right?
- 51:01Then this regression here is essentially imputing
- 51:05the correct value of S
- 51:06for every single person in the population.
- 51:09So it's kind of like we're getting more data
- 51:11in some sense, and the nice thing about
- 51:15these approaches, these AIPTW approaches
- 51:17is that they're double robust and so again,
- 51:18if you work in causal inference a very familiar idea,
- 51:21and it turns out because we know
- 51:23the sampling probability by design,
- 51:25this regression doesn't have to be consistently estimated
- 51:28in order to obtain a consistent estimate
- 51:30of the parameter measures.
- 51:31So it's this really nice sort of double robustness property
- 51:33that says, yeah, you might be turned off
- 51:35from this augmented estimator
- 51:36because you have to do a little bit of extra work,
- 51:38you have to fit a regression model say,
- 51:40and maybe you're worried about missspecifying
- 51:42that regression well it turns out that because the sampling
- 51:44probabilities are known by design,
- 51:46you don't have to concern yourself with that.
- 51:47So it turns out you can use any old regression estimator
- 51:50here and still end up with a consistent estimate
- 51:53of the parameter of interest.
- 51:54And so we're applying this
- 51:55to much more interesting parameters.
- 51:57So we had a paper come out recently
- 51:59in biometrics that's linked here
- 52:01where we're starting to study a sort of causal inference
- 52:03flavored parameters in this context,
- 52:06things that we can really use to pin down,
- 52:08you know, mechanisms of these vaccines working.
- 52:10So, in this case, we're studying sort of the effect
- 52:13of a stochastic intervention, we call it.
- 52:16So it's sort of saying what would happen
- 52:18if we took everybody's immune response,
- 52:20this particular immune response that we observed,
- 52:22and we shifted it up just a little bit
- 52:25or we shifted it down just a little bit.
- 52:26How would that impact the risk of disease amongst
- 52:30the vaccinated individuals?
- 52:30So that's what this big, gnarly parameter is right here.
- 52:34And so you ended up looking at a plot
- 52:35that's kind of like this.
- 52:36So this is from an HIV vaccine trial.
- 52:39So at zero we're saying that's just the observed risk
- 52:42of the trial and as we move left we're saying,
- 52:44what would the risk be if we decreased your immune response?
- 52:47And so we can see in this example,
- 52:49we found that the risk would be increasing, right.
- 52:52And then if we're moving to the right
- 52:53is what would happen if we increase your immune response.
- 52:57And so we're kind of getting at something
- 52:59that's like a controlled effects mediation type parameter
- 53:03with this approach and so we're working out some
- 53:06of the details of the correlates plan currently
- 53:10and so when that's done
- 53:11we'll have it available for public comment.
- 53:13And again, we're academics, right?
- 53:14So we'll do it all open science.
- 53:16And then I'll just say like two words of conclusion
- 53:18and I'll shut up and leave some time for questions.
- 53:21So there's been a big concern
- 53:23in the current political climate that we're gonna sneak
- 53:26something through, that something's gonna be approved
- 53:28without sort of the standard amount of evidence
- 53:32that would be required, right.
- 53:33That there's political interference at the FDA
- 53:36and from where I sit, you know,
- 53:39I can say that the science behind the vaccine
- 53:41development program for COVID is extremely rigorous.
- 53:43These are exactly the type of people who you would want
- 53:46in charge of this decision making process
- 53:48and the type of people that will raise red flags
- 53:51as soon as sort of the process goes off the rails.
- 53:54So right now I feel good about where things stand.
- 53:57Of course, I watch presidential debates and hear, you know,
- 54:00garbage science coming out and I get a little bit concerned,
- 54:04but from where I sit right now,
- 54:05everything's looking pretty good.
- 54:07So overall, I'd say that the increased transparency
- 54:09by releasing these protocols
- 54:11has been good for scientists and consumers.
- 54:13We want to bring vaccines to market,
- 54:15but we also want people to trust those vaccine
- 54:17so increasing transparency in whatever way we can is great.
- 54:20And then finally, the final point is that a lot of these
- 54:23issues that I've talked about,
- 54:24how do we do interim monitoring, right?
- 54:26What's the right end point to be studying?
- 54:28What's the right S demand, right?
- 54:30These are really hard decisions
- 54:32and there are no right answers.
- 54:34And so one of the things that's been a little bit
- 54:37disconcerting or disheartening to me
- 54:40is the extent to which in the pandemic era,
- 54:43academic debates have been made very much public
- 54:46and I'm not against academic debates.
- 54:49It's just that most individuals aren't used to seeing them.
- 54:52And so what I'm worried is happening is that people
- 54:55see high profile academics debating these challenging
- 55:00problems where there's no real right answer.
- 55:02And they're saying, well, these guys don't know
- 55:03what they're talking about.
- 55:05So I think as academics and public health professionals
- 55:08in this pandemic, one thing that we can do
- 55:10is just to be very careful in how we're presenting,
- 55:12you know, the science that we're doing
- 55:15and acknowledge when there's not a right answer,
- 55:17that you're presenting your opinion.
- 55:19And that there is some validity, right?
- 55:21That this is very gray, unfortunately,
- 55:23that there's nothing black and white here.
- 55:25So maybe that's a controversial statement to end on,
- 55:28but I'll end there and then thanks again to Fan
- 55:30for giving me the opportunity to talk
- 55:32and I'm happy to take questions as there's time.
- 55:35I don't have anything scheduled after this,
- 55:36so I can stay a few minutes over as would be helpful.
- 55:39So thanks again.
- 55:41- [Fan] Thank you David for this very nice talk.
- 55:44I think we do have three to four minutes for questions
- 55:47from the audience, if there's any.
- 55:54- [Woman] Hi David, I have a question
- 55:55'cause right now for COVID situation and because of the time
- 56:00and the faster progress of the disease
- 56:04and it's a hard to keep the standard method,
- 56:08but do you have other proofed vaccine for other disease
- 56:14and have a quick trial have a similar way as COVID
- 56:19and apply the method you're using right now
- 56:24and we have standard results already
- 56:27and then compare to see how good the current method is.
- 56:32So that's my question.
- 56:35- [David] Yeah it's an interesting question.
- 56:37So let me try to restate, so you're saying,
- 56:39are there any lessons from vaccine development
- 56:42that we can try to draw from here
- 56:44to evaluate our methodology, whether it work?
- 56:48- [Woman] Yes, from other vaccines.
- 56:52- [David] So I guess what I would say is that at this stage,
- 56:55in phase three vaccines, these phase three trials
- 56:58look completely normal.
- 57:00So I would say the process of getting to the phase three
- 57:03looked very different and much more accelerated
- 57:05in terms of kind of squashing together
- 57:07phase one and phase two in terms of the manufacturing,
- 57:11but in terms of what's happening in a phase three trial,
- 57:13this is probably the phase three trial
- 57:15that would be done outside of the setting of a pandemic.
- 57:18Maybe the interim analysis would be a little bit
- 57:20less aggressive for some of these companies, but really,
- 57:23I think the approaches that the companies are taking
- 57:27would be fairly standard even in any other vaccine context.
- 57:35- -Woman] Yeah. I mean, even though
- 57:37for the established vaccine,
- 57:41there could be some field trial
- 57:43and that they also went through a phase three,
- 57:47but you can do the similar thing to enhance,
- 57:51to see whether it is possible to pass the current protocol
- 57:57and become some sort of false positive.
- 58:02- [David] Yeah and, you know, I think speaking,
- 58:08I mean, speaking of failed vaccines,
- 58:09as someone who works in HIV vaccines,
- 58:11we're very familiar with failure and learning from that.
- 58:15So again, I think the people who are running these trials
- 58:17are sort of the right people in terms of looking out
- 58:20for these false positive signals and so forth.
- 58:24- [Woman] Thank you.
- 58:27- [Fan] So I think we are just about the time
- 58:30and I'm sure that David is happy
- 58:32to take your questions afterwards by email.
- 58:35So I'll thank David more time.
- 58:37Again, thank you for sharing with us
- 58:39and we'll see everyone again next week.
- 58:43- [David] Thanks everybody.