BIS Seminar: At warp speed: statistics and COVID-19 vaccine development

Transcript

• 00:00- [Fan] David Benkeser who is an assistant professor
• 00:03at the department of biostatistics and bioinformatics
• 00:07at Emory University.
• 00:09Dr. Benkeser got his PhD in biostatistics
• 00:11from University of Washington
• 00:13and had his post-doctoral fellowship
• 00:15from University of California at Berkeley.
• 00:18Dr. Benkeser is an expert in methods for machine learning
• 00:22and non-parametric statistical inference.
• 00:24He has made important contributions
• 00:26to integrate machine learning methods
• 00:28to draw causal inferences with observational data.
• 00:31He also has interesting work on preventative vaccines
• 00:34and HIV prevention, which he's going to share with us today.
• 00:37Welcome David, the floor is yours.
• 00:44- [David] Thanks, yeah, it's a great pleasure
• 00:46to be here today.
• 00:48Well, here today, but with you guys today giving this talk.
• 00:51So I did see that I think Tony Fauci
• 00:55spoke at Yale yesterday, so it was very nice of you Fan
• 00:58to book Tony Fauci as my opening act
• 01:00and I'll try not to disappoint him with my followup.
• 01:04So the talk I'm giving today is a very high-level talk.
• 01:07So the title is statistics and COVID-19 vaccine development,
• 01:10but it's really a talk mostly
• 01:12about COVID-19 vaccine development.
• 01:15There's not math until maybe slide like 29 out of 30.
• 01:19So really these are sort of
• 01:20just the high-level issues that have come up
• 01:23as I've worked with companies and government organizations
• 01:28on COVID-19 vaccine development.
• 01:30So I think there's a lot of really interesting stuff
• 01:32here and really, really glad to share it with you today.
• 01:35So if you want to kind of slide along with
• 01:39the slides they're available on GitHub
• 01:41so there's a link at the bottom there,
• 01:43and you can click on that
• 01:44and that'll pull up the HTML slide back,
• 01:46and I have sort of references hyperlinked in there.
• 01:48So that's an easy way to access
• 01:50the references there as well.
• 01:52Okay so I'm going to start just kind of talking
• 01:55about the biology a little bit of SARS-CoV-2,
• 01:58and segue into sort of how we can think about
• 02:01developing vaccines that will prevent
• 02:03an infection and COVID-19 disease.
• 02:06And so this is a nice little graphic that I ripped off
• 02:09from The Washington Post, who's very much better
• 02:12at making these cutesy little graphics
• 02:13than I am using PowerPoint or something.
• 02:16So let's kind of walk through this.
• 02:17And the goal here is to try to understand,
• 02:20you know, how SARS-CoV-2 is infecting your cells,
• 02:22how it's replicating,
• 02:23and then to understand what the mechanisms
• 02:25that immunological mechanisms of the vaccine are
• 02:28that can potentially block that infection
• 02:30and prevent clinical disease.
• 02:31So we'll just go quickly through this
• 02:33and this is sort of the story for most viruses, right?
• 02:36Is that viruses are really just genetic material
• 02:39in this case RNA that's wrapped up in the glycoprotein.
• 02:42So it's genetic material wrapped up in a protein.
• 02:45And so for SARS-CoV-2 you may have heard of a couple
• 02:49of these proteins in particular, the spike protein will play
• 02:51a large role when we talk about a vaccine development
• 02:54and why is this spike protein so important?
• 02:56Well, that's the guy that sort of latches
• 02:58onto your cell and it does that through this ACE2 pathway
• 03:01and it grabs onto your cell and insert itself inside
• 03:05you cell and once it's inside
• 03:07it releases its genetic material, right?
• 03:09It releases its RNA and kind of tricks
• 03:11your cell into replicating the virus, right?
• 03:13So that your cell is producing new copies of this virus,
• 03:17they're pieced together out of proteins that are released
• 03:19into your bloodstream to go infect more cells
• 03:22and more people.
• 03:23Okay so that's sort of the infection process
• 03:25and where along the lines do you know
• 03:27vaccines sort of halt this?
• 03:29So I'll walk through a few different
• 03:31of the major vaccine constructs that are being used
• 03:33for SARS-CoV-2 vaccines,
• 03:35and the details aren't super important here,
• 03:37but I do think it's sort of helpful
• 03:38to have a high level overview in comparison, right?
• 03:40Because there's so many vaccine products being developed,
• 03:43at least having some point of biological comparison
• 03:45of how they're working is useful.
• 03:47So to walk through these slides,
• 03:48all of these slides are basically going to be the same
• 03:50on the right hand side of the slide
• 03:52and how they're gonna differ is what goes
• 03:54into the vaccine on the left-hand side.
• 03:56So let's actually start on the right-hand side, right?
• 03:58And talk a little bit about immunology, right?
• 04:00And how your body tries to fight off infection.
• 04:03And we have a couple of different mechanisms
• 04:05of your immune system to do that.
• 04:06So there's a kind of T-cell responses, cytotoxic T-cells.
• 04:10So those are T-cells that recognize cells in your body
• 04:12that have been infected with a pathogen
• 04:14and destroy those cells, right?
• 04:15Because the cells are producing copies of the virus,
• 04:17releasing in the bloodstream.
• 04:19So if we're able to destroy infected cells,
• 04:21we can potentially stop infection, prevent disease,
• 04:24and then another key response
• 04:26that your immune system has is through antibodies.
• 04:28And that's sort of what's on the bottom here
• 04:30and is that B cells are able to produce antibodies.
• 04:34And what those antibodies do is they basically grab
• 04:36onto these surface proteins, right?
• 04:38So remember we talked about the spike protein,
• 04:40and what antibodies do is basically just bind onto that
• 04:43and sit there and so neutralizing antibodies.
• 04:46So there's two classes of antibodies that are kind
• 04:47of relevant for vaccines.
• 04:48So neutralizing antibodies really, you're just doing that.
• 04:50They're gonna sit on all of those spike proteins
• 04:53and because they're sitting there now the virus can't grab
• 04:55onto your cells to infect them.
• 04:57There's also binding antibodies, which are somewhat
• 05:01considered to be less important in this context,
• 05:03but what those guys do is bind onto those surface proteins,
• 05:05they don't neutralize the virus itself,
• 05:07but they send out chemical signals to other cells
• 05:09in your body that say, hey, here's a virus.
• 05:11Please come eat it for me.
• 05:13So those are the sort of antibody classes response
• 05:15that you can have.
• 05:16So there's these two sort of immune mechanisms
• 05:18that we have to neutralize infections by viruses.
• 05:22How do they learn to neutralize them?
• 05:25Well, there's this sort of middleman.
• 05:26So we're moving just to this middle panel here
• 05:28with these APC cells,
• 05:30so these antigen presenting cells, right?
• 05:32Those are the guys that what they're doing
• 05:33is basically digesting little bits
• 05:36of the virus in this case of the surface protein, right?
• 05:40And they're teaching or training your immune system
• 05:43to recognize that pathogen, right?
• 05:44So they're the ones that go and talk to the T cells,
• 05:47talk to the B cells and say,
• 05:48here's that how this virus looks,
• 05:51please go produce some antibodies or please recognize cells
• 05:53that have been infected with this
• 05:55and neutralize them for me.
• 05:56So really again, the whole right side of this plot
• 05:59is about your immune system.
• 06:00This is the way your immune system fights off infection.
• 06:03And what's different between this slide
• 06:05and the next few slides is basically how we present
• 06:08pieces of the pathogen pieces
• 06:09of the virus to these APCs, right?
• 06:12So how do we get these APCs, the material that they need
• 06:14for you to mount an immune response against SARS-CoV-2?
• 06:19And so the first class of vaccines
• 06:21I'll describe are nucleic acid vaccines.
• 06:23And so I'm talking about first
• 06:25because they're sort of the first wave of vaccines
• 06:27that are in phase three trials in the US.
• 06:29So Moderna and Pfizer, who are probably the most advanced
• 06:33candidates for US licensure are both mRNA vaccines.
• 06:37And so how are those vaccines made?
• 06:39Well, we take a little bit of messenger RNA,
• 06:42a little bit of viral genetic material,
• 06:44and wrap that in a lipid shell, right?
• 06:45That's the construct of the vaccine.
• 06:47And when you're injected that lipid shell
• 06:49latches onto your cell, right?
• 06:51Delivers that mRNA into your cell,
• 06:54just like a natural infection, right?
• 06:55Remember the SARS-CoV-2 grabbed onto your cell
• 06:57and inserted itself and then made copies of itself.
• 07:00So what is the mRNA doing once it's in your cell,
• 07:03it's actually just making copies
• 07:05of the spike protein itself, right?
• 07:07So you're manufacturing this protein within your own cells
• 07:11that are then released for these APCs to detect.
• 07:14So this is how we're getting these APCs,
• 07:17spike protein with an mRNA vaccine.
• 07:19We're basically using your cells as a warehouse
• 07:22to produce the antigen of the vaccine
• 07:24and so this is a really cool idea and a new idea, right?
• 07:28So, am mRNA or DNA vaccine has never been licensed before
• 07:31and that's not to say that we tried many times and failed.
• 07:34It's just to say that this is a very new technology,
• 07:36and it's sort of interesting that it's kind of come
• 07:40to the forefront in this context.
• 07:41So why do we like mRNA vaccines?
• 07:44Well, they're very fast to manufacturer.
• 07:46We'll talk about some of the other vaccine constructs
• 07:48where we're making this spike protein in a lab,
• 07:50and that is a long and arduous.
• 07:52It needs to be very careful process
• 07:54and when we're thinking about scaling up
• 07:55vaccine manufacturing, mRNA vaccines are very appealing
• 07:59in that sense, you can manufacture them
• 08:02very quickly at scale.
• 08:03They don't require a cold chain
• 08:05and so that's another great advantage these vaccines enjoy
• 08:10in terms of thinking about vaccine deployment,
• 08:12particularly in developing world settings.
• 08:15But again, this is a brand new technology.
• 08:17We don't have any safety data
• 08:18from past vaccines with this construct.
• 08:20We don't have any efficacy data.
• 08:22So, it's sort of an open question in the field
• 08:25as to how well these things are gonna work.
• 08:28So moving to sort of more classical, constructive vaccines
• 08:30and viral vector vaccines.
• 08:33So again, the right side of this picture
• 08:34is exactly the same.
• 08:35The story is how do we get an APC the right antigen?
• 08:38How do we show an APC a little bit of the spike protein?
• 08:41So a viral vector vaccine, right?
• 08:45Is going to take a different virus and splice
• 08:49a little bit of SARS-CoV-2 into that virus, okay.
• 08:52So for example, AstraZeneca, that's the Oxford that you may
• 08:56have heard of, they take a chimpanzee adenovirus,
• 08:58that's like, it's a virus that causes the common cold
• 09:02in chimpanzees and they splice in a little bit
• 09:04of SARS-CoV-2 into that and so that sort of host virus,
• 09:09that adenovirus holds genetic material
• 09:12infects your cells and your cells then produce the antigen.
• 09:16They produce the spike protein of SARS-CoV-2.
• 09:20So AstraZeneca and Janssen are using this construct again,
• 09:22both with adenoviruses, a very common virus vector.
• 09:26And again, we like these types of vaccines
• 09:29because they're quick to manufacturer,
• 09:32but a challenge of them is that your body
• 09:34can sort of develop separate immune responses
• 09:37against the vector itself, right?
• 09:39So you can develop a separate immune response
• 09:42against say an adenovirus right?
• 09:45Such that your body neutralizes those adenoviruses
• 09:47before they're able to infect your cells
• 09:50and produce the SARS-CoV-2 antigen.
• 09:52So we do see tendency a kind of faster waning
• 09:55vaccine effects with this class of vaccines.
• 09:58So moving on to subunit vaccine.
• 10:00So this is NovaVax and Sanofi's vaccine
• 10:03will be subunit vaccines
• 10:05and this is where I kind of mentioned before
• 10:06actually what happens here is these spike proteins
• 10:10or whatever the antigen is,
• 10:12is created and purified in a lab.
• 10:14So they actually use insect cells
• 10:17that they infect with SARS-CoV-2,
• 10:20those insect cells then produce the antigen that's purified
• 10:23and that's what goes into the vaccine
• 10:26are those protein subunits, right?
• 10:27So there we're just directly giving you the spike protein
• 10:30that we've grown outside of the host
• 10:33and that's how we're getting these APCs, those antigens.
• 10:37And so this is a commonly used vaccine construct.
• 10:40So the hep B vaccine is highly effective.
• 10:42HPV vaccine is highly effective.
• 10:44That's the construct of these, but the downside of course
• 10:46to it, so it's a well-trodden way of developing vaccines.
• 10:51But the downside is that they're slower to manufacturer.
• 10:53There's this whole process where we have to cultivate
• 10:55and grow these viruses in a lab, we have to purify them,
• 11:00and moreover they often also require an adjuvant.
• 11:04So that's really just sort of adding something a little bit
• 11:07extra that stimulates a better immune response in your body.
• 11:11So basically at the site of injection,
• 11:13it's something that increases
• 11:14your inflammatory response actually
• 11:16to kind of stimulate your immune system
• 11:18into recognizing those antigens
• 11:20and developing an immune response against them.
• 11:22So there's subunit vaccines.
• 11:24So the fourth class here is a weakened/inactivated vaccine.
• 11:27And so this is, I think, what most people like what
• 11:30my grandparents probably think all vaccines are,
• 11:33is basically we take a pathogen
• 11:35and we weaken it in some way, or we kill it, right?
• 11:39And then that's the construct of the vaccine
• 11:40and that's what's injected into you.
• 11:42And we go through this similar process there
• 11:45that literally mimics natural infection, right?
• 11:47Where your cells are infected by this weakened form
• 11:51of the virus, the virus replicates,
• 11:53and that's how we get antigens to the APCs.
• 11:56So this is the construct used in of course
• 11:58some classic vaccines like MMR, polio vaccine,
• 12:01but again, it's slower manufacturing, right?
• 12:03Because we have to cultivate the virus
• 12:04in the lab and then it also requires adjuvants.
• 12:07So I don't think there's currently any plans
• 12:10to have US phase three trials
• 12:12of weaken inactivated vaccines, but there are in China.
• 12:15So Sinopharm and Sinovac vaccines
• 12:17were using this construct.
• 12:21So that's just a bit of a background in immunology
• 12:24and how all this works and how we think about preventing
• 12:26infection with SARS-CoV-2 and hopefully preventing
• 12:29clinical disease COVID-19 disease.
• 12:32So now we're gonna segue to talk a little bit
• 12:34about the vaccine development process, right?
• 12:36'Cause this has all happened extremely fast.
• 12:38So let's talk about sort of the process whereby
• 12:40vaccine products are typically brought to market, right.
• 12:43And what looks a little bit different
• 12:45about the COVID-19 vaccine development process?
• 12:49So this is a figure from a nice New England journal paper
• 12:52that's referenced at the bottom
• 12:54that's just talking about sort of what's different
• 12:56this go around in terms of how are we accelerating
• 12:58the vaccine development process.
• 13:00And so I think as biostatisticians,
• 13:01anyone who works on clinical trials is fairly familiar
• 13:04with the traditional paradigm
• 13:06for bringing products to market, right.
• 13:08It involves sort of a lot of R&D
• 13:11in the lab, preclinical work
• 13:13and then you start doing human trials in phase one,
• 13:16these are small dose finding safety trials,
• 13:19checking whether these vaccines
• 13:21generate any immune response.
• 13:23And then what we'll often do is in vaccine trials
• 13:25is run a small randomized trial.
• 13:27That's a phase two trial, right?
• 13:29We're we'll have a placebo control,
• 13:31maybe pick out a particularly high risk population
• 13:34and start to see if we're getting
• 13:35any efficacy signal, right?
• 13:37And this is a very deliberate process, right?
• 13:39Phase one typically advances very slowly.
• 13:41We have lots of safety concerns.
• 13:42Phase two, we think very hard about whether the efficacy
• 13:45signal was really worth it to advance a candidate to
• 13:48phase three and it's a very deliberate process, right?
• 13:51To get to this phase three licensure trial, right?
• 13:53So the phase three trial is the big one involving
• 13:56the most participants.
• 13:57It's a randomized controlled trial, right?
• 14:00Enrolling many, many subjects that's well powered
• 14:02to detect ethicacy signals and based on the results
• 14:04of that phase three trial and safety data
• 14:07that's been accumulated throughout
• 14:08this whole process, right.
• 14:10We're able to provide licensure ideally for a product.
• 14:14And so that's sort of the clinical development process,
• 14:17but also in the context of COVID vaccines
• 14:19it's important to think about
• 14:20the manufacturing process, right.
• 14:21And how that looks a little bit different.
• 14:23So typically right, companies are very sort of hesitant
• 14:27to scale up manufacturing before they know that they have
• 14:31a product that will be licensed, right.
• 14:32Which makes sense, you know, they're sort of risk averse.
• 14:34We don't want to start manufacturing a product
• 14:36that may ultimately be shot down by the FDA.
• 14:39So really large scale manufacturing
• 14:40is not happening until after product licensure.
• 14:44So what's happening with COVID vaccine
• 14:45is basically this whole long deliberate timeline
• 14:49is being compressed into a shorter time period.
• 14:52And so how do we do that?
• 14:53Well, basically what happens is we've collapsed
• 14:56the phase one and phase two trials, right?
• 14:58So we're doing small safety studies.
• 15:00We're checking whether these vaccines
• 15:01are generating immune responses,
• 15:03but we're really not doing that smaller efficacy study
• 15:06that is typical of vaccine development.
• 15:10And so we're collapsing the phase one and two process,
• 15:13the phase three process is where we're at, right.
• 15:15We're doing these large scale trials, right?
• 15:16Because we need robust efficacy data
• 15:19and we need robust safety data to gain licensure,
• 15:22but a big thing that has changed, so the clinical process
• 15:24yeah a little bit compressed, but mostly the same,
• 15:27the big thing that's changed
• 15:28is the manufacturing process, right.
• 15:30Is we wanna make sure that once a vaccine is licensed
• 15:33and is proven to be safe and effective that we're able
• 15:37to start distributing that vaccine immediately.
• 15:39So that means that manufacturing needs to start ramping
• 15:41up right before we ever have a signal of efficacy
• 15:45and that's a huge risk for companies to take.
• 15:47So, I'll talk in a couple of slides about sort of how
• 15:51the government has come in to try to remove
• 15:54some of that risk from these companies
• 15:56and then the next slide I think is just showing sort of
• 15:59that it's really impressive that we're even talking
• 16:01about potentially having a COVID vaccine available this year
• 16:05or early next year, just given the timelines
• 16:08that are required to bring effective vaccines to market.
• 16:11And so here's just a few, you know,
• 16:13polio, measles, chickenpox, mumps,
• 16:14all multiple years of development for these vaccines,
• 16:18you could add malaria on this list.
• 16:19It took about 30 years
• 16:20to get a partially effective malaria vaccine to market.
• 16:24So this is typically a very long process, right?
• 16:26And for COVID, we're looking at hopefully doing this
• 16:29in just under a year or two.
• 16:31So how is the US government playing a role in this?
• 16:35Well, it's through this program that you may have heard of
• 16:37called Operation Warp Speed,
• 16:39which is this huge convoluted mess of an amalgamation
• 16:44of programs across the government
• 16:45from DOD to many branches of NIH, BARDA, NIAID,
• 16:50so it's sort of all over the place.
• 16:51And this is really just the same figure
• 16:54that I showed you from the New England journal paper.
• 16:57Just maybe a slightly more confusing
• 17:01if you ask me, I don't think Edward Tufte,
• 17:03he would be a big fan of graphic
• 17:05but the point here I want to mention
• 17:07is how is the government responding
• 17:10to COVID vaccine development?
• 17:11How are they contributing to that process?
• 17:13Well, there's really two ways that they've offered
• 17:15to accelerate the process.
• 17:17The first is through funding
• 17:19of phase three clinical trials, right?
• 17:21So a number of companies, six of the major companies,
• 17:24basically every company that's running a phase three trial
• 17:26in the US besides Pfizer that you've heard about
• 17:30is contracting with BARDA.
• 17:32That's an arm of the NIH,
• 17:34they're contracting with the government
• 17:36to have the government fund their phase three trials.
• 17:39So it's a joint agreement between the government
• 17:40and these companies where the government,
• 17:42you the taxpayer, right, are paying for these
• 17:45phase three trials that will eventually lead to licensure.
• 17:49So that's the first way that the government
• 17:50is sort of throwing money at this problem.
• 17:52It's through design and paying for these phase three trials.
• 17:56The second way is that they're paying
• 17:58for manufacturing, right?
• 17:59They're removing that risk for these companies
• 18:01by basically committing to buy a certain number of doses
• 18:04before we ever have any efficacy data.
• 18:06So we're in the hole basically to all of these companies
• 18:08for a fixed number of doses right.
• 18:11But that motivates the companies then to scale up
• 18:13their manufacturing ahead of the time
• 18:15that efficacy data are available.
• 18:18And that type of agreement has been entered
• 18:20into with Pfizer as well.
• 18:21So all of these companies that OWS Operation Warp Speed
• 18:24is running the phase three trials for
• 18:27also have this manufacturing agreement.
• 18:29Pfizer has that manufacturing agreement as well.
• 18:33So what role have I played in any of this big messy thing?
• 18:38So I work with a great group of scientists
• 18:41in the COVID-19 Prevention Network.
• 18:43So this was a clinical trials network established
• 18:45by National Institute of Allergies and Infectious Disease
• 18:48and NIAID so that's an arm of NIH,
• 18:51and it's basically anyone who works in clinical trials
• 18:54is fairly familiar
• 18:55with these clinical trials networks, right?
• 18:57It's an amalgamation of researchers and study sites,
• 19:01laboratories, people who focus on recruitment and retention
• 19:04of trial participants, statisticians.
• 19:07So it's researchers who are really experts
• 19:09in running clinical trials,
• 19:10designing clinical trials
• 19:12and ensuring their robust conduct.
• 19:15So the CoVPN was formed by basically leveraging
• 19:19four existing clinical trials networks.
• 19:21One of which I was a part of,
• 19:22which is the HIV vaccine trials network.
• 19:24And so from our group, we've really brought a great group
• 19:27of statisticians, many of whom are at the Fred Hutch
• 19:30in Seattle as well as great groups of laboratories at U Dub.
• 19:34And so what are the roles
• 19:36that we're playing in these trials?
• 19:38So in our statistical group,
• 19:40there's a couple of statisticians who are designated
• 19:44as like CoVPN representatives
• 19:46for each of these phase three trials.
• 19:48So I sit on calls with these trials and advise
• 19:53on their design and analysis approaches
• 19:55for their efficacy monitoring, for their safety monitoring.
• 19:58We help them address DSMB and FDA comments
• 20:01and sort of that's all happening in conjunction
• 20:04with both government statisticians, right.
• 20:06Representatives of BARDA and NIAID
• 20:10as well as company statisticians.
• 20:12And so we get on these calls and, you know,
• 20:14nerd out over clinical trials,
• 20:16statistical decision-making, and it's a good old time.
• 20:21Another aspect that we really contribute a lot on,
• 20:24or that CoVPN has sort of been tasked with taking
• 20:26the lead on is the development of immune correlates.
• 20:29And so that's the part of my talk
• 20:31where I'll get a little bit into statistics
• 20:32and talking about what immune correlates are,
• 20:34some of the types of analytic approaches
• 20:36we use to study those and the idea of immune correlates
• 20:38just to give you a teaser
• 20:40so you don't, you know, sign off Zoom early.
• 20:42So immune correlates are really the idea there is
• 20:45we're looking for immune responses that are predictive
• 20:48of the vaccines working, right.
• 20:52So what we'd really like to be able to do is understand,
• 20:54okay, if we're able to generate this level
• 20:56of neutralizing antibody,
• 20:58then that will lead to this level of protective effect
• 21:00of the vaccine, right?
• 21:02So that's the whole goal there is identifying
• 21:04what are these immune responses that are
• 21:05responsible for providing protection?
• 21:08Okay so I'm gonna walk through just a few of the design
• 21:11and analysis questions.
• 21:12And so these are things that have come up
• 21:14as we've worked with these company statisticians,
• 21:16as we thought about sort of the whole OWS vaccine program,
• 21:20what are some of the issues that statisticians
• 21:22are kicking around and people who have worked
• 21:24on clinical trials, right,
• 21:25a lot of these issues aren't gonna be new
• 21:27and one thing that I think is sort of interesting about this
• 21:30whole pandemic and operating as a public health professional
• 21:34in this and a clinical trial statistician in particular,
• 21:37is that a lot of things that we take for granted
• 21:39as scientists are either very confusing
• 21:42or sort of counterintuitive for a lot of the lay public.
• 21:45And so it's been sort of interesting to have that laid bare.
• 21:48In some of these issues, some of these things
• 21:50that we think are no-brainers like doing interim analysis
• 21:53for example are kind of highly controversial
• 21:56and have ended up being, you know,
• 21:57sort of areas of huge disputes.
• 21:59And so I just want to run through some of these issues
• 22:01that I think are quite fascinating, a lot of which,
• 22:04you know, really don't have a correct answer
• 22:06and they're really just sort of food for thought
• 22:07the types of things that we're thinking about
• 22:09when we're designing these trials.
• 22:11So I'll start by just giving a sort of more specific idea
• 22:16of what these trials look like and how they're conducted
• 22:18and I've picked AstraZeneca because that's the one
• 22:20I've worked on for the longest and most closely,
• 22:23but all of the trials sort of follow this similar design.
• 22:26And so the first thing I'll note
• 22:27is that you can read these trial protocols.
• 22:29So one of the interesting things that's happened
• 22:31in this COVID-19 development processes
• 22:33is there was a huge public push led by like Eric Topol
• 22:36and others to have the protocols of these trials
• 22:39made public, which when it happened was I guess
• 22:43when that push started happening, you know,
• 22:45I emailed all my colleagues and said,
• 22:47really do we not usually make protocols public?
• 22:50And that was just sort of interesting disconnect
• 22:51for me as an academic who's used to sort of everything
• 22:54being open science and that's a no brainer right.
• 22:57Working in this setting, right,
• 22:58where these protocols are really seen as trade secrets
• 23:00for pharmaceutical companies.
• 23:02So it's really unusual that actually these protocols
• 23:05for clinical trials have been made public.
• 23:06So it's sort of neat, but one of the things that happened
• 23:09is all of these protocols went public and reporters
• 23:12got their hands on them and said, wow,
• 23:13these are really dense documents, right?
• 23:15If you've ever looked at the clinical trial protocol,
• 23:18it's like a hundred pages of very specific definitions
• 23:21and safety monitoring and what symptoms lists
• 23:24you're gonna use and what surveys
• 23:26you're gonna give to people.
• 23:27So they're very sort of detailed documents
• 23:28that are kind of hard for the public to parse.
• 23:32So it's been sort of a be careful what you wish for thing
• 23:34in terms of releasing these protocols, but that's an aside.
• 23:38So let's talk about actually what these trials look like.
• 23:40So here's a schematic, and again,
• 23:42this is AstraZeneca in particular,
• 23:44but this is basically the design of most of these trials
• 23:47will look something like this.
• 23:48So who is the population?
• 23:50Most of these trials are gonna be primarily in adults.
• 23:53I think Pfizer has now started
• 23:55to talk about including children.
• 23:57I'm not exactly sure where that's happening,
• 23:59but adults for the most part,
• 24:01these are mostly healthy individuals
• 24:05that don't have, you know, chronic diseases
• 24:07that are at risk or high risk of death.
• 24:10And we're really looking at targeting individuals
• 24:12who are at an increased risk for SARS-CoV-2 acquisition
• 24:15and severe COVID disease
• 24:17and so the idea there is number one
• 24:19these are the people that are bearing
• 24:20the brunt of the pandemic, right?
• 24:23So we want to be able to get a product to those people
• 24:26as fast as possible.
• 24:27But number two also, right, that means that we'll accrure
• 24:29from a sort of cold hearted and statistician point of view
• 24:32that means we'll accrue end points faster.
• 24:34We'll observe more cases of COVID-19 disease
• 24:37and potentially get an efficacy signal a little bit faster.
• 24:40So there's a lot of interest in sort of recruiting
• 24:43and retaining individuals at high risk for COVID-19.
• 24:45So you can go onto the COVID-19 prevention trials network
• 24:48and fill out a survey, right.
• 24:49Then we'll basically under the hood
• 24:51assess your risk for COVID-19
• 24:53and if you're found to be at high risk,
• 24:54we'll aggressively email you and try to get you
• 24:55enrolled in one of these trials.
• 24:57If you're at low risk,
• 24:57we'll say, thanks for taking the survey,
• 24:59we'll be in touch and likely
• 25:01you won't hear from us anytime soon.
• 25:03Okay so that's the trial population.
• 25:05So how does the actual trial conduct look?
• 25:07So there's kind of a mixture here.
• 25:10AstraZeneca is using a two to one randomization scheme.
• 25:13So you have two chances of getting the active vaccine
• 25:16versus one chance of getting a placebo.
• 25:18And in this case, it's a true placebo, just a saline dose
• 25:22and then most of the vaccines, most all with Janssen
• 25:26being the accepted are two dose vaccines.
• 25:28So you receive the first dose at day one
• 25:30and the second dose about a month later.
• 25:32And in the interim, we take a couple of measurements.
• 25:34We have a phone call to assess reactogenicity right.
• 25:38Does your arm hurt, or have you experienced any adverse side
• 25:41effects of the first dose of vaccine?
• 25:44And then there's also an immune response measurement
• 25:46that happens after a couple of days.
• 25:48So we get an early signal
• 25:49of how immunogenetic these vaccines are.
• 25:51And so then individuals come in for their second dose
• 25:53of vaccine and it's a similar story, right?
• 25:55Did you have any reactions?
• 25:57We measure your immune response and after that,
• 25:59that's sort of when the clock starts for active follow-ups.
• 26:02So this day 57, that's two weeks roughly after,
• 26:07am I doing that math right?
• 26:08Well, it looks like roughly two weeks after the second dose
• 26:12of the vaccine is typically when this clock
• 26:14is gonna start and we're gonna start counting COVID events.
• 26:17And then it's sort of just the standard sort of game we play
• 26:21in clinical trials.
• 26:22We wait for events to accrue.
• 26:23We have certain monitoring plan
• 26:25for when we're gonna check for efficacy
• 26:27and we'll talk about some of that.
• 26:28So, I just want to note that there's sort of two ways
• 26:30that we're ascertaining events
• 26:32that are happening here, right?
• 26:33The first is passive monitoring.
• 26:35What that means is we basically wait for individuals
• 26:37to present with symptoms of COVID 19, right?
• 26:39So you get a cough, you lose taste, right?
• 26:41You call the study site, right?
• 26:45So I am having these symptoms.
• 26:46They say, come on in.
• 26:47You get a PCR test to see whether you're infected.
• 26:50And in that case, you would count
• 26:51as a COVID-19 endpoint, right?
• 26:53If you check off some check boxes for symptoms
• 26:56with COVID-19 disease, you have a PCR positive test.
• 26:59You'd go down as a COVID 19 endpoint.
• 27:01There's also these sort of active follow-up visits.
• 27:04So these like day 90, day, 180 and day 360,
• 27:08and at those visits we'll do a serology check.
• 27:11And what that means is we basically take a blood draw
• 27:13and we measure whether you have antibodies
• 27:16against SARS-CoV-2, right, antibodies that are distinct
• 27:19from the antibodies that are generated
• 27:20in response to the vaccine.
• 27:21So we're basically able to tell whether you were infected
• 27:24in this sort of interim period,
• 27:26when you show up for these visits.
• 27:29So that's active follow up
• 27:30and so there you're gonna be able
• 27:31to pick up sort of asymptomatic cases, right?
• 27:33'Cause if you never have symptoms, you'll never come in
• 27:36and be captured by passive followup.
• 27:38So we have to wait for these set clinic visits
• 27:40to do the serology testing,
• 27:42to ascertain it asymptomatic cases.
• 27:44And so this is gonna actually play a role
• 27:46in a little bit, when I started talking about, you know,
• 27:48what are the end points that we're thinking about measuring?
• 27:50Like, what do we want to know how well
• 27:52the vaccine works at preventing?
• 27:53Is it asymptomatic infection?
• 27:55Is it disease?
• 27:56Is it severe disease and so forth?
• 27:57So we'll talk through some of those issues,
• 27:59but just want to note already that the design has started
• 28:02to inform some of the challenges that we might see
• 28:04when we want to talk about how well the vaccine works
• 28:07against certain forms of infection and disease.
• 28:10And so I think if you read the newspaper and you'll see
• 28:13the term vaccine efficacy tossed around a lot.
• 28:15So the first thing I want to talk about is right,
• 28:17what is the primary hypothesis
• 28:19that these trials are trying to test?
• 28:21And what is the parameter?
• 28:23What is the estimate, right, that they're going after
• 28:25in these trials and for whatever reason
• 28:27nobody consulted me when they decided that VE
• 28:29would be measured in this way.
• 28:32But for whatever reason, we studied this that we quantify
• 28:35the efficacy of a vaccine in a sort of weird way.
• 28:37So a vaccine efficacy, we describe as the percent reduction
• 28:40in relative risk comparing vaccine to placebo.
• 28:43So it's this one minus a risk ratio.
• 28:46There's a one minus a risk ratio where you take the risk
• 28:49in the vaccine and the numerator and the risk
• 28:51in the placebo and the denominator.
• 28:53So, I mean,
• 28:54we can just play a quick little intuitive game, right?
• 28:56How do we get a VE close
• 28:57to one that would be a perfect vaccine?
• 29:00Well, we would make the risk
• 29:01in the vaccine close to zero, right?
• 29:03So that sorta makes sense.
• 29:04If you have a perfectly effective vaccine,
• 29:06there'll be no risk of infection and or disease
• 29:08amongst the vaccinated.
• 29:09So you would get VE close to one.
• 29:11But on the other hand, how do we make VE zero?
• 29:13Well, we would take the risk in the vaccine
• 29:16and set it equal to the risk in the placebo, right.
• 29:18In which case basically saying the vaccine's not doing
• 29:21anything and then on the other hand,
• 29:23a VE is negative, right?
• 29:24That's indicating that there's actually higher risk
• 29:26in the vaccine.
• 29:28So just to give you sort of a few reference points, right?
• 29:31So that VE of one is perfect, VE of zero is nothing
• 29:34and what we're really hoping for with these COVID trials
• 29:38is a VE of at least 50%.
• 29:40And that's sort of the cutoff that FDA guidance
• 29:42has stipulated is that you need to show a point estimate
• 29:46of VE for your primary end point.
• 29:48And again, we'll talk about what these primary end points
• 29:50are but we need a VE against a primary end point
• 29:53of at least 50%
• 29:54and we need to definitively rule out the possibility
• 29:59that the vaccine efficacy is less than 30%.
• 30:02So basically we have to reject the null hypothesis
• 30:05that VE is less than 30% along with having a point
• 30:08estimate of VE being greater than 50%, right.
• 30:11And we need to do that while controlling type one error
• 30:13at two and a half percent.
• 30:16Okay and so here, just one final note,
• 30:19since this is a statistics talk,
• 30:20I'll talk a little bit more
• 30:21about what I mean by risk, right?
• 30:23So risk here can be quantified in a number of ways
• 30:26and it often is.
• 30:27So we can quantify this using hazards, for example,
• 30:29like you can imagine fitting a Cox model, right.
• 30:31A proportional hazards model, right.
• 30:33That only adjusts for vaccine, right.
• 30:35And presenting like one minus a hazard ratio
• 30:37from a Cox model, that's something that's commonly done.
• 30:39You can also think about cumulative incidents, right?
• 30:41So like mapping,
• 30:43maybe one minus a survival probability as a way
• 30:46of quantifying risk, incidents rate ratios.
• 30:49So they're all sort of used for different vaccines.
• 30:52And usually we like to sort of argue
• 30:55about which one of these is better
• 30:56and I've thought a lot about that in my career.
• 30:58And in this setting, it turns out because COVID
• 31:00is such a rare event that all of these ways of quantifying
• 31:03rates are basically the same and you end up
• 31:05with almost identical operating characteristics of a trial.
• 31:08So it's really not worth sort of losing sleep over
• 31:10whether we're talking about VE in terms of hazard
• 31:12or incidents or incidents rate and so forth.
• 31:16So how are folks going about estimating this VE?
• 31:19Here's just a quick table of the four most advanced
• 31:22phase three trials,
• 31:23the ones that have released their protocols at least.
• 31:25So we see for Moderna, AstraZeneca, and Janssen,
• 31:28they're using pretty kind of the standard approaches.
• 31:31Moderna a Cox model as I describe,
• 31:33AstraZeneca a Poisson regression model,
• 31:35it's like, okay, that's basically a Cox model,
• 31:38and then Janssen is using a sort of exact binomial test
• 31:41with this sequential probability ratio rest.
• 31:44Pfizer is a little bit of the oddball.
• 31:46So they have stipulated a bayesian approach
• 31:49wherein they're basically specifying a prior
• 31:53for vaccine efficacy and are using sort of
• 31:55a beta-binomial bayesian approach to evaluate
• 31:58the posterior probability of the vaccine efficacy
• 32:00is greater than 30% and so at the end of the day,
• 32:04there's four different statistical methods here.
• 32:06Again, if you do a simulation study with parameters
• 32:09that are approximately similar to what we expect to see
• 32:11in these COVID trials,
• 32:12you're really not gonna see much difference in terms
• 32:14of operating characteristics across these.
• 32:16So it's interesting to notice that assertions
• 32:18that there's these different approaches,
• 32:19but at the end of the day,
• 32:20we're basically talking about how many vaccinated people
• 32:22get infected, how many placebo people got infected,
• 32:25and almost all of these methods are gonna yield
• 32:27very similar inference.
• 32:29When it comes down to brass tacks,
• 32:31how many numbers fall into those categories?
• 32:33So that's a little bit about sort of
• 32:36how we quantify VE in these settings
• 32:38but one of the big things I haven't described yet
• 32:40is VE against what, right?
• 32:42What is the end point that we're measuring here?
• 32:43And so here's a figure from a paper we just had come out
• 32:47in Annals of Internal Medicine, the link's here.
• 32:50So this is where we were spending a lot of time,
• 32:52you know, earlier this summer, thinking about,
• 32:54you know, what's the right end point,
• 32:56what's the right end point for a primary analysis
• 32:58of the clinical trial.
• 32:59And it's complicated for something like SARS-CoV-2, right?
• 33:03Because we know we can start up here
• 33:04with the SARS-CoV-2 infection, right?
• 33:07That's sort of the base, you can become infected
• 33:09and then a number of things can happen, right?
• 33:11You can go on to be infected but develop no symptoms.
• 33:14So we would call that an asymptomatic infection,
• 33:17or you can develop symptoms, right.
• 33:18In which case we don't call you
• 33:20a SAR-CoV-2 infection anymore,
• 33:21we call you a COVID-19 disease endpoint.
• 33:25You have a clinical manifestation of your infection.
• 33:28But even beyond that, right,
• 33:30amongst people who exhibit symptoms
• 33:31some of them, maybe many of them are quite mild, right.
• 33:34So we have this kind of category of non-severe COVID,
• 33:38whereas others we know that are extremely adversely
• 33:41impacted by infection and end up with severe COVID disease.
• 33:45So you have all of these choices of sort of
• 33:49which end points you might want to talk about
• 33:51and so I'll kind of walk through some what I see
• 33:53as the positives and negatives of this and then I'll also
• 33:56talk about this burden of disease
• 33:58very briefly end point that we've put together
• 34:01and so that's kind of a composite end point
• 34:03that we've suggested that could kind of bring all
• 34:05of these different end points together.
• 34:07Okay so starting with SARS-CoV-2 infection, right?
• 34:09Why might we like any sort of any infection, right.
• 34:12Asymptomatic, symptomatic don't care,
• 34:14let's count any infection as an event
• 34:17and measure VE against preventing infection.
• 34:20Okay and so that's definitely relevant, right.
• 34:22It's relevant the context of a pandemic.
• 34:24We're preventing infections,
• 34:25we're preventing spread of the disease,
• 34:27we're bringing our knot down, we're impacting the pandemic.
• 34:30And moreover, we're going to see many more infections
• 34:33than we will cases of symptomatic disease.
• 34:36We know that many people who were infected
• 34:37never go on to develop symptoms
• 34:39so thinking about having an answer faster, right.
• 34:42SARS-CoV-2 infection is a nice endpoint,
• 34:44but then the question is,
• 34:46is it a clinically relevant endpoint?
• 34:47So it's really not describing an impact on patients at all.
• 34:52So we could kind of question its relevance
• 34:56from that perspective.
• 34:57The other thing, right, is that we remember going back
• 34:59to the study design, we're only able to ascertain
• 35:01asymptomatic infections sort of very coarsely in time
• 35:05and moreover you have this phenomenon that happens
• 35:09is that when you're testing many, many individuals, right.
• 35:12It's sort of the classic biostat
• 35:14one-on-one problem that we give people, right.
• 35:16You're testing many individuals, but the prevalence is low.
• 35:19So even if you have high sensitivity and high specificity,
• 35:22you could end up with low positive predictive value.
• 35:25And the effect of that when you come to the time to analyze
• 35:28the data is that you'll be biasing VE towards the knoll.
• 35:31So it's actually, while it seems like maybe a nice end point
• 35:35from the perspective of observing many infections,
• 35:37it's a very challenging endpoint to analyze quantitatively.
• 35:41So moving down we could talk about COVID.
• 35:43So again, COVID is just infection,
• 35:46PCR confirmed infection with clinical symptoms.
• 35:49So that's of course more clinically relevant, right.
• 35:51Because we're starting to talk about
• 35:53an impact, excuse me, the endpoint that impacts patients.
• 35:57All right so that's more clinically relevant and moreover
• 36:00we'll expect to have a reasonable number of cases, right.
• 36:03By including more mild cases, for example,
• 36:06in this endpoint definition.
• 36:09But then on the other side of that coin
• 36:10is it really that clinically relevant
• 36:12if we're just talking about mild symptoms?
• 36:15We're talking about a disease where you get it
• 36:16and you end up with a little cough for a couple of weeks
• 36:18and that's it.
• 36:19So then maybe you suggest using severe COVID right.
• 36:22That's the most clinically relevant one.
• 36:24We want to be protecting the most vulnerable individuals
• 36:26so we should be quantifying how well our vaccines
• 36:28work towards preventing those most severe end points.
• 36:33And so most clinically relevant,
• 36:35and also there's sort of a long history
• 36:37of vaccine development where really we see the best VE
• 36:40against severe cases of disease.
• 36:43So that's really where we expect the vaccines
• 36:45to have the most impact is maybe we are not preventing
• 36:47you from being infected but we're lessening the symptoms
• 36:51once you become infected.
• 36:52So we're not totally blocking transmission
• 36:55but we're making a clinical impact on disease
• 36:57and that's sort of been seen
• 36:58for a number of vaccines in the past.
• 37:00The downside of this end point of course
• 37:02is that there's very few cases expected to be observed.
• 37:04So amongst all infections,
• 37:05only a fraction have any symptoms.
• 37:07Amongst those with any symptoms,
• 37:09only a fraction develops severe symptoms.
• 37:10So we're really whittling away the number of end points.
• 37:13So we need to do larger trials
• 37:15or have longer follow-up to evaluate this endpoint.
• 37:19And so in that paper, I'm sort of pressed for time
• 37:21so I won't spend too much time talking about this,
• 37:24we also proposed this burden of disease measure
• 37:26where you're sort of scoring these these outcomes, right?
• 37:29So maybe you would get a score of zero
• 37:31if you're an asymptomatic infection
• 37:33'cause it's really no burden on you as a patient, right?
• 37:36You don't have any symptoms.
• 37:37And then we're sort of assigning arbitrarily
• 37:39a score of one for non severe COVID so that's like
• 37:42mild cases of COVID and a score of two
• 37:45for severe cases of COVID and this end point actually
• 37:48has some nice operating characteristics we think,
• 37:51but of course it's subject to controversy, anytime you start
• 37:53talking about an ordinal scoring system, right,
• 37:57you start to raise questions about how you're assigning
• 37:59the burden of disease score, right?
• 38:01Why should severe cases be a two
• 38:03versus a three versus a five and so forth?
• 38:06So you can kind of get bogged down
• 38:07in some of the specifics of that.
• 38:10So what has FDA said about this?
• 38:12So FDA guidance documents states that either
• 38:15the COVID end point or SARS-CoV-2 infection
• 38:18is an acceptable primary endpoint
• 38:19and then somewhat ironically OWS has been telling companies
• 38:22that infection alone is not acceptable
• 38:24as a primary end point.
• 38:25So we had one company that was interested in including
• 38:28that as co-primary and for whatever reason we told them
• 38:31please don't do that, and then beyond that so COVID
• 38:36has sort of won out as the end point of choice.
• 38:39But beyond that FDA guidance states that companies should
• 38:42consider powering efficacy trials
• 38:44for the severe COVID endpoint as a co-primary or at least
• 38:48as a key secondary endpoint in the trial.
• 38:51And so so far only Janssen has taken them up on that offer
• 38:54of making severe COVID primary.
• 38:56And that's why, if you look at the number of individuals
• 38:58that are planning to enroll in their trial,
• 38:59it's twice as many as any of the other OWS trials.
• 39:03So like AstraZeneca is planning for 30,000,
• 39:04Janssen is planning for 60,000 in their trial.
• 39:08And that's the power, to see more cases of severe disease
• 39:10to be sufficiently powered to detect VE against that.
• 39:15So this is a controversial slide.
• 39:17Or this is virtual topic I found,
• 39:20again, something that clinical trials statisticians
• 39:22sort of take for granted is doing interim analyses, right?
• 39:26If the treatment is working and we have enough evidence
• 39:28to claim that a treatment is working,
• 39:29we'd like to stop that trial early
• 39:31to get that treatment to patients, right.
• 39:33One would think that that's true here
• 39:35and so many of these trials
• 39:36were designed with aggressive sort of interim looks, right?
• 39:40Because we're in the middle of the pandemic
• 39:41and we'd like to get a vaccine to individuals
• 39:44as quickly as possible.
• 39:45So I have a table, we won't go through it all here,
• 39:47just sort of the planned interim analysis
• 39:50for these different trials.
• 39:52I would say Pfizer seems to be the most aggressive so far.
• 39:56They have five interim looks or four interim looks
• 40:00and a final look at their data, right?
• 40:02So that's fairly aggressive.
• 40:03OWS again, the trials that we're running,
• 40:07we're really encouraging companies to be a bit
• 40:09more conservative in the approach to this
• 40:11and only maybe two or three
• 40:13and so you see what's been adopted
• 40:14by Moderna and AstraZeneca
• 40:17and so this was really a big point of contention
• 40:19I think when these protocols were made public is this idea
• 40:22that like, can you really know that a vaccine works
• 40:25based on 32 data points, right?
• 40:27We're talking about a vaccine that's going to be given
• 40:30to billions of people around
• 40:32the world based on these results
• 40:33and you're gonna do that based
• 40:35on the results in 32 individuals?
• 40:37And like, so I can stare at the math and say that like, yes,
• 40:40that appropriately controls type one error and so forth,
• 40:42but it still makes me just feel a little bit uncomfortable.
• 40:45There's a bit of dissonance between sort of my life
• 40:48as a statistician and just me being a human
• 40:50and saying 32 data points is probably not enough
• 40:52to decide to vaccinate billions of people.
• 40:54And so a lot of people I think sort of shared
• 40:57that viewpoint and in response FDA has now been sort of
• 41:01backpedaling in a way and asking companies to provide more
• 41:06data in order to grant an emergency authorization
• 41:10for their vaccine.
• 41:11So this EUA mechanism that FDA has of approving vaccines.
• 41:14And so in addition to an efficacy signal,
• 41:17now companies also are gonna be required, I think,
• 41:20and this is sort of still a moving target so this is maybe
• 41:23like data news at this point but I think prior to offering
• 41:26an EUA, FDA has now said that companies need to have 50%
• 41:29of participants complete at least two months of follow-up
• 41:33for safety signals and that you need to have at least
• 41:36six COVID cases in the oldest age group.
• 41:39Of course, that's an age group of particular interest
• 41:41in terms of severe cases and at least five cases
• 41:44of severe COVID in the placebo group.
• 41:45So they want to be able to see some data,
• 41:48even if you're not specifying severe COVID
• 41:50as a primary end point,
• 41:51they want to be able to see some data,
• 41:53some signal of efficacy against that
• 41:55in order to grant licensure.
• 41:57So I'll sort of, I won't go through this slide.
• 42:01It's just to say that like,
• 42:02sort of when Pfizer released their protocol,
• 42:04everyone was like, ooh a bayesian analysis
• 42:06and got very sort of skeptical, right?
• 42:09Because the Pfizer CEO has been out there
• 42:11sort of chest thumping and saying they're gonna have
• 42:13a vaccine before the election and so forth
• 42:15and then they came out with this bayesian design
• 42:17that was a little atypical and so everybody was asking
• 42:19the question, well, are they trying to hide something?
• 42:21So I sort of did a quick analysis
• 42:23and found that really it doesn't look that different
• 42:25than a classic kind of post hoc monitored design.
• 42:28And if you want to read more about that,
• 42:30I have some slides up on my GitHub about it.
• 42:33So let's see, I'm running low on time
• 42:36so I'm gonna skip over sort of the question
• 42:39of what happens if efficacy is declared early.
• 42:41So I have some reasons that we should be excited, right?
• 42:43If one of these trials stops earlier, I can get a vaccine.
• 42:46There's good data that the vaccine works
• 42:48and that's nice.
• 42:50I'd like to go back to something resembling normal
• 42:52as I'm sure you all would,
• 42:54but of course there's reasons to be concerned, right?
• 42:56If efficacy is declared early in particular,
• 42:58if that means that blinded follow-up
• 43:01in a study stops, right?
• 43:02Because that means we have no way
• 43:03to assess how durable the vaccine is.
• 43:05We won't be able to assess VE
• 43:07and key subgroups that we care about.
• 43:09We might not be able to assess VE
• 43:11formally against severe end points.
• 43:13So there's real sort of concerns
• 43:15about stopping these trials too early,
• 43:17and what the implications of that
• 43:18are both for evaluating the vaccine in question,
• 43:21but as well as how it impacts
• 43:23the other clinical trials that are ongoing.
• 43:26And of course in the current political climate,
• 43:29everybody's very concerned about the role
• 43:31political pressure might play in all of this.
• 43:33So yeah, so it's kind of a double-edged sword in some sense
• 43:37in terms of what happens if efficacy is declared early,
• 43:42but I want to save just a few minutes
• 43:43to talk about vaccine correlates 'cause I promised
• 43:45that I would show you some math and prove to you
• 43:47that I'm a real statistician.
• 43:48So let's do a little bit of that.
• 43:51So again, we're kind of shifting gears here.
• 43:53So that's the end of sort of talking about the primary
• 43:55analysis of these trials,
• 43:56what's gonna lead to their licensure.
• 43:58And the correlates of protection
• 44:00is sort of a key secondary analysis
• 44:02and so why is it so important
• 44:04that we're able to establish correlates of protection?
• 44:07Well, because it's gonna speed up
• 44:09the whole vaccine development process.
• 44:12So again, a correlative protection is really just,
• 44:14it's an immune response and really an assay
• 44:18to measure that immune response that's been validated
• 44:20to reliably predict vaccine efficacy.
• 44:23So why is that so important?
• 44:25Well, basically what we're hoping to achieve
• 44:28is the establishment of a surrogate
• 44:29endpoint for COVID disease right?
• 44:32So I've sort of mentioned the numbers that we're talking
• 44:34about in these phase three trials,
• 44:36enrolling 30,000 participants, 60,000 participants
• 44:40and ending up with one or two years of followup, right.
• 44:42Just to be able to answer the primary question, right.
• 44:44Does the vaccine prevent infection and/or disease?
• 44:48So that's a huge, expensive clinical trial.
• 44:50It takes a long time to get an answer
• 44:52and so it would be very nice if all we had to do right
• 44:56was give people the doses of vaccine that they need,
• 44:59wait two weeks and measure their immune response
• 45:02and understand does that vaccine work or not.
• 45:05That would be a much faster vaccine development process
• 45:07than where we're currently at
• 45:09in having to run these enormous phase three trials.
• 45:11So it's valuable for establishing a surrogate endpoint.
• 45:14It's also valuable for accelerating approval
• 45:17of vaccines that have been licensed in certain populations,
• 45:22but not others.
• 45:23For example, I mentioned that these phase three trials
• 45:25are mostly being conducted in adults.
• 45:27Well, what if we want to also obtain licensure for use
• 45:30of this vaccine in children?
• 45:32Well, if we had an established immune correlate
• 45:34we wouldn't have to do
• 45:35a large randomized trial in children.
• 45:37We could do it just a small immunogenicity study
• 45:39and use the correlates results to bridge the VE
• 45:42that we observed from the phase three trial.
• 45:45That's the immune response that we've observed
• 45:47in these children or pregnant women for example are
• 45:49another key population they're being
• 45:51excluded from these phase three trials
• 45:53but we'd like to understand if these vaccines
• 45:55are safe and effective in those women as well.
• 45:59So really this is one of the key goals
• 46:01of this whole OWS program and the key role
• 46:05that we're playing in the CoVPN is developing
• 46:08the sampling plan and the statistical analysis plan
• 46:12for the immune correlate studies
• 46:14and so it's just a little bit of the statistical issues
• 46:17that we're dealing with in these trials, right,
• 46:20is that sort of running assays
• 46:22so running these immuno assays on 30,000, 60,000 individuals
• 46:26takes a long time, it's expensive, and as it turns out,
• 46:30it's really overkill in terms of statistical power.
• 46:34So we can actually be a little bit more
• 46:35clever about how we design these correlate studies in order
• 46:40to get answers faster and more cheaply.
• 46:42So the way we do this is we use a case cohort design.
• 46:45So we're not gonna measure immune responses
• 46:47in all trial participants,
• 46:48we're gonna measure them in a sub cohort
• 46:50and that sub cohort will consist
• 46:51of a stratified random sub cohort.
• 46:54So we're gonna be sampling individuals randomly
• 46:56based on their baseline infection status.
• 46:58Were you infected with SARS-CoV-2 in the past?
• 47:01Based on your race, ethnicity, and based on age.
• 47:07And so based on that, we'll take a random draw
• 47:09of the trial population, about 1600 individuals,
• 47:13excuse me and everyone so I should mention right
• 47:19in the trial design everybody is having their blood drawn.
• 47:22And right now we're talking about whose blood
• 47:24are we gonna use to measure these immune responses?
• 47:27So we're gonna measure it in a random sample
• 47:29and then we're gonna wait until the trial is over
• 47:31or until one of these interim analysis concludes efficacy
• 47:35and we're gonna measure immune responses
• 47:38in all of the end points, right?
• 47:39Remember that like power in these analyses is drive
• 47:42by the individuals in which we observe endpoints.
• 47:46So we're gonna make sure we get immune responses
• 47:47in all the end point data, as in addition
• 47:49to this random sub cohort and it turns out that that's about
• 47:53as statistically efficient as running the immune assays
• 47:57on all 30,000 individuals in the trial.
• 47:59So this is this kind of classic case cohort design
• 48:01that Ross Prentice has been writing about for years
• 48:04that Norman Breslow did some sort of pioneering work
• 48:06on in the 2000s and I'll just talk a little bit about sort
• 48:11of how this complicates our life as statisticians
• 48:13and then maybe we'll leave a few minutes for questions.
• 48:16So here's the math, we made it.
• 48:18Well, the moment you've all been waiting for it
• 48:20to see some math.
• 48:21So just introducing, you know,
• 48:23why is this sampling design challenging
• 48:26from a perspective of generating estimators, right?
• 48:29Well, we can sort of immediately see that this isn't
• 48:31a totally random sample of the trial population, right?
• 48:35In particular we've over-sampled the individuals who end up
• 48:38getting diseased and it's fairly obvious
• 48:42that those individuals have potential to be very different
• 48:45than a randomly selected individual in the population.
• 48:47So we have a bias sub sample.
• 48:49So we need some statistical methodology to try to back out,
• 48:52you know, whatever this parameter is.
• 48:54We want to be estimating it in the whole trial population,
• 48:56not just in this biased sub samples.
• 48:59So how do we do that?
• 49:00So just a quick notation here,
• 49:02let's call W baseline covariates,
• 49:04A is a binary vaccine assignment,
• 49:07Y is your binary COVID endpoint for example
• 49:11and then we'll introduce this sort of indicators.
• 49:13Delta is one, if you're selected into this immune response
• 49:18sub cohort, either because you were a case,
• 49:20you were an end point or because you were randomly selected
• 49:23into the cohort.
• 49:25And then we'll call S your immune response.
• 49:28And then we'll just say, we'll represent this as Delta S,
• 49:31which just means we'll arbitrarily set everybody
• 49:33who's not in our sub cohorts immune response to be zero,
• 49:36that's arbitrary doesn't really matter.
• 49:38So let's talk about how estimation would happen.
• 49:41So let's pick a very simple parameter, right?
• 49:43Let's just say that we want to know what's the overall
• 49:45immune response in the whole population,
• 49:47not a particularly interesting parameter
• 49:50for actually measuring correlates,
• 49:51but just to motivate the types of statistical approaches
• 49:54that we use in these settings.
• 49:56So how can we control for the bias of the sampling design?
• 49:59Well, one of the most straightforward ways
• 50:01is to use the tried and true
• 50:02Horvitz-Thompson or IPTW estimator, right.
• 50:05Where we're just taking basically a sample mean
• 50:08but all our observations are sort of inverse weighted
• 50:11by their probability of being sampled into this sub cohort.
• 50:16And so that's, IPTW estimator if you're in causal inference,
• 50:18you're very familiar with this.
• 50:19If you're in survey sampling,
• 50:21very familiar with this.
• 50:22Very classical way of adjusting for this selection bias.
• 50:27It turns out that there's ways
• 50:28that we can be more efficient in doing this.
• 50:30We can use augmented estimators, AIPTW estimators.
• 50:33And the key idea there is that we take the IPTW estimator
• 50:37and we add a little bit of something to it
• 50:39and the key thing is that that little bit
• 50:41of something involves a regression of S the immune response
• 50:46onto the covariates that were used to sample individuals
• 50:50into the sub cohort.
• 50:52And so what's the intuition as
• 50:54to why this is more efficient?
• 50:56Well, you can imagine what if we had a perfect predictor
• 50:59of S measured at baseline, right?
• 51:01Then this regression here is essentially imputing
• 51:05the correct value of S
• 51:06for every single person in the population.
• 51:09So it's kind of like we're getting more data
• 51:11in some sense, and the nice thing about
• 51:15these approaches, these AIPTW approaches
• 51:17is that they're double robust and so again,
• 51:18if you work in causal inference a very familiar idea,
• 51:21and it turns out because we know
• 51:23the sampling probability by design,
• 51:25this regression doesn't have to be consistently estimated
• 51:28in order to obtain a consistent estimate
• 51:30of the parameter measures.
• 51:31So it's this really nice sort of double robustness property
• 51:33that says, yeah, you might be turned off
• 51:35from this augmented estimator
• 51:36because you have to do a little bit of extra work,
• 51:38you have to fit a regression model say,
• 51:40and maybe you're worried about missspecifying
• 51:42that regression well it turns out that because the sampling
• 51:44probabilities are known by design,
• 51:46you don't have to concern yourself with that.
• 51:47So it turns out you can use any old regression estimator
• 51:50here and still end up with a consistent estimate
• 51:53of the parameter of interest.
• 51:54And so we're applying this
• 51:55to much more interesting parameters.
• 51:57So we had a paper come out recently
• 51:59in biometrics that's linked here
• 52:01where we're starting to study a sort of causal inference
• 52:03flavored parameters in this context,
• 52:06things that we can really use to pin down,
• 52:08you know, mechanisms of these vaccines working.
• 52:10So, in this case, we're studying sort of the effect
• 52:13of a stochastic intervention, we call it.
• 52:16So it's sort of saying what would happen
• 52:18if we took everybody's immune response,
• 52:20this particular immune response that we observed,
• 52:22and we shifted it up just a little bit
• 52:25or we shifted it down just a little bit.
• 52:26How would that impact the risk of disease amongst
• 52:30the vaccinated individuals?
• 52:30So that's what this big, gnarly parameter is right here.
• 52:34And so you ended up looking at a plot
• 52:35that's kind of like this.
• 52:36So this is from an HIV vaccine trial.
• 52:39So at zero we're saying that's just the observed risk
• 52:42of the trial and as we move left we're saying,
• 52:44what would the risk be if we decreased your immune response?
• 52:47And so we can see in this example,
• 52:49we found that the risk would be increasing, right.
• 52:52And then if we're moving to the right
• 52:53is what would happen if we increase your immune response.
• 52:57And so we're kind of getting at something
• 52:59that's like a controlled effects mediation type parameter
• 53:03with this approach and so we're working out some
• 53:06of the details of the correlates plan currently
• 53:10and so when that's done
• 53:11we'll have it available for public comment.
• 53:13And again, we're academics, right?
• 53:14So we'll do it all open science.
• 53:16And then I'll just say like two words of conclusion
• 53:18and I'll shut up and leave some time for questions.
• 53:21So there's been a big concern
• 53:23in the current political climate that we're gonna sneak
• 53:26something through, that something's gonna be approved
• 53:28without sort of the standard amount of evidence
• 53:32that would be required, right.
• 53:33That there's political interference at the FDA
• 53:36and from where I sit, you know,
• 53:39I can say that the science behind the vaccine
• 53:41development program for COVID is extremely rigorous.
• 53:43These are exactly the type of people who you would want
• 53:46in charge of this decision making process
• 53:48and the type of people that will raise red flags
• 53:51as soon as sort of the process goes off the rails.
• 53:54So right now I feel good about where things stand.
• 53:57Of course, I watch presidential debates and hear, you know,
• 54:00garbage science coming out and I get a little bit concerned,
• 54:04but from where I sit right now,
• 54:05everything's looking pretty good.
• 54:07So overall, I'd say that the increased transparency
• 54:09by releasing these protocols
• 54:11has been good for scientists and consumers.
• 54:13We want to bring vaccines to market,
• 54:15but we also want people to trust those vaccine
• 54:17so increasing transparency in whatever way we can is great.
• 54:20And then finally, the final point is that a lot of these
• 54:23issues that I've talked about,
• 54:24how do we do interim monitoring, right?
• 54:26What's the right end point to be studying?
• 54:28What's the right S demand, right?
• 54:30These are really hard decisions
• 54:32and there are no right answers.
• 54:34And so one of the things that's been a little bit
• 54:37disconcerting or disheartening to me
• 54:40is the extent to which in the pandemic era,
• 54:43academic debates have been made very much public
• 54:46and I'm not against academic debates.
• 54:49It's just that most individuals aren't used to seeing them.
• 54:52And so what I'm worried is happening is that people
• 54:55see high profile academics debating these challenging
• 55:00problems where there's no real right answer.
• 55:02And they're saying, well, these guys don't know
• 55:03what they're talking about.
• 55:05So I think as academics and public health professionals
• 55:08in this pandemic, one thing that we can do
• 55:10is just to be very careful in how we're presenting,
• 55:12you know, the science that we're doing
• 55:15and acknowledge when there's not a right answer,
• 55:17that you're presenting your opinion.
• 55:19And that there is some validity, right?
• 55:21That this is very gray, unfortunately,
• 55:23that there's nothing black and white here.
• 55:25So maybe that's a controversial statement to end on,
• 55:28but I'll end there and then thanks again to Fan
• 55:30for giving me the opportunity to talk
• 55:32and I'm happy to take questions as there's time.
• 55:35I don't have anything scheduled after this,
• 55:36so I can stay a few minutes over as would be helpful.
• 55:39So thanks again.
• 55:41- [Fan] Thank you David for this very nice talk.
• 55:44I think we do have three to four minutes for questions
• 55:47from the audience, if there's any.
• 55:54- [Woman] Hi David, I have a question
• 55:55'cause right now for COVID situation and because of the time
• 56:00and the faster progress of the disease
• 56:04and it's a hard to keep the standard method,
• 56:08but do you have other proofed vaccine for other disease
• 56:14and have a quick trial have a similar way as COVID
• 56:19and apply the method you're using right now
• 56:24and we have standard results already
• 56:27and then compare to see how good the current method is.
• 56:32So that's my question.
• 56:35- [David] Yeah it's an interesting question.
• 56:37So let me try to restate, so you're saying,
• 56:39are there any lessons from vaccine development
• 56:42that we can try to draw from here
• 56:44to evaluate our methodology, whether it work?
• 56:48- [Woman] Yes, from other vaccines.
• 56:52- [David] So I guess what I would say is that at this stage,
• 56:55in phase three vaccines, these phase three trials
• 56:58look completely normal.
• 57:00So I would say the process of getting to the phase three
• 57:03looked very different and much more accelerated
• 57:05in terms of kind of squashing together
• 57:07phase one and phase two in terms of the manufacturing,
• 57:11but in terms of what's happening in a phase three trial,
• 57:13this is probably the phase three trial
• 57:15that would be done outside of the setting of a pandemic.
• 57:18Maybe the interim analysis would be a little bit
• 57:20less aggressive for some of these companies, but really,
• 57:23I think the approaches that the companies are taking
• 57:27would be fairly standard even in any other vaccine context.
• 57:35- -Woman] Yeah. I mean, even though
• 57:37for the established vaccine,
• 57:41there could be some field trial
• 57:43and that they also went through a phase three,
• 57:47but you can do the similar thing to enhance,
• 57:51to see whether it is possible to pass the current protocol
• 57:57and become some sort of false positive.
• 58:02- [David] Yeah and, you know, I think speaking,
• 58:08I mean, speaking of failed vaccines,
• 58:09as someone who works in HIV vaccines,
• 58:11we're very familiar with failure and learning from that.
• 58:15So again, I think the people who are running these trials
• 58:17are sort of the right people in terms of looking out
• 58:20for these false positive signals and so forth.
• 58:24- [Woman] Thank you.
• 58:27- [Fan] So I think we are just about the time
• 58:30and I'm sure that David is happy
• 58:32to take your questions afterwards by email.
• 58:35So I'll thank David more time.
• 58:37Again, thank you for sharing with us
• 58:39and we'll see everyone again next week.
• 58:43- [David] Thanks everybody.