BIS Seminar: At warp speed: statistics and COVID-19 vaccine development

October 28, 2020

Information

David Benseker, PhD, MPH, Assistant Professor

Emory University Department of Biostatistics and Bioinformatics.

October 27, 2020

Abstract: One of the best hopes we have of returning life to normal is to bring a safe and effective preventive COVID-19 vaccine to market and make it available to just about everybody around the world. We are in the middle of an unprecedented effort to bring such a vaccine to market. After a recent pressure campaign led by academic scientists, vaccine developers have made public the protocols for their Phase III trials. The release of these protocols has ignited a fierce debate as to whether the designs are appropriate and sufficiently safe-guarded from political pressure for vaccine approval. In this talk I will discuss a few of the complex issues involved in designing Phase III COVID vaccine trials, touching on some key statistical aspects of the trials along the way.

ID5822

To CiteDCA Citation Guide

00:00- [Fan] David Benkeser who is an assistant professor
00:03at the department of biostatistics and bioinformatics
00:07at Emory University.
00:09Dr. Benkeser got his PhD in biostatistics
00:11from University of Washington
00:13and had his post-doctoral fellowship
00:15from University of California at Berkeley.
00:18Dr. Benkeser is an expert in methods for machine learning
00:22and non-parametric statistical inference.
00:24He has made important contributions
00:26to integrate machine learning methods
00:28to draw causal inferences with observational data.
00:31He also has interesting work on preventative vaccines
00:34and HIV prevention, which he's going to share with us today.
00:37Welcome David, the floor is yours.
00:44- [David] Thanks, yeah, it's a great pleasure
00:46to be here today.
00:48Well, here today, but with you guys today giving this talk.
00:51So I did see that I think Tony Fauci
00:55spoke at Yale yesterday, so it was very nice of you Fan
00:58to book Tony Fauci as my opening act
01:00and I'll try not to disappoint him with my followup.
01:04So the talk I'm giving today is a very high-level talk.
01:07So the title is statistics and COVID-19 vaccine development,
01:10but it's really a talk mostly
01:12about COVID-19 vaccine development.
01:15There's not math until maybe slide like 29 out of 30.
01:19So really these are sort of
01:20just the high-level issues that have come up
01:23as I've worked with companies and government organizations
01:28on COVID-19 vaccine development.
01:30So I think there's a lot of really interesting stuff
01:32here and really, really glad to share it with you today.
01:35So if you want to kind of slide along with
01:39the slides they're available on GitHub
01:41so there's a link at the bottom there,
01:43and you can click on that
01:44and that'll pull up the HTML slide back,
01:46and I have sort of references hyperlinked in there.
01:48So that's an easy way to access
01:50the references there as well.
01:52Okay so I'm going to start just kind of talking
01:55about the biology a little bit of SARS-CoV-2,
01:58and segue into sort of how we can think about
02:01developing vaccines that will prevent
02:03an infection and COVID-19 disease.
02:06And so this is a nice little graphic that I ripped off
02:09from The Washington Post, who's very much better
02:12at making these cutesy little graphics
02:13than I am using PowerPoint or something.
02:16So let's kind of walk through this.
02:17And the goal here is to try to understand,
02:20you know, how SARS-CoV-2 is infecting your cells,
02:22how it's replicating,
02:23and then to understand what the mechanisms
02:25that immunological mechanisms of the vaccine are
02:28that can potentially block that infection
02:30and prevent clinical disease.
02:31So we'll just go quickly through this
02:33and this is sort of the story for most viruses, right?
02:36Is that viruses are really just genetic material
02:39in this case RNA that's wrapped up in the glycoprotein.
02:42So it's genetic material wrapped up in a protein.
02:45And so for SARS-CoV-2 you may have heard of a couple
02:49of these proteins in particular, the spike protein will play
02:51a large role when we talk about a vaccine development
02:54and why is this spike protein so important?
02:56Well, that's the guy that sort of latches
02:58onto your cell and it does that through this ACE2 pathway
03:01and it grabs onto your cell and insert itself inside
03:05you cell and once it's inside
03:07it releases its genetic material, right?
03:09It releases its RNA and kind of tricks
03:11your cell into replicating the virus, right?
03:13So that your cell is producing new copies of this virus,
03:17they're pieced together out of proteins that are released
03:19into your bloodstream to go infect more cells
03:22and more people.
03:23Okay so that's sort of the infection process
03:25and where along the lines do you know
03:27vaccines sort of halt this?
03:29So I'll walk through a few different
03:31of the major vaccine constructs that are being used
03:33for SARS-CoV-2 vaccines,
03:35and the details aren't super important here,
03:37but I do think it's sort of helpful
03:38to have a high level overview in comparison, right?
03:40Because there's so many vaccine products being developed,
03:43at least having some point of biological comparison
03:45of how they're working is useful.
03:47So to walk through these slides,
03:48all of these slides are basically going to be the same
03:50on the right hand side of the slide
03:52and how they're gonna differ is what goes
03:54into the vaccine on the left-hand side.
03:56So let's actually start on the right-hand side, right?
03:58And talk a little bit about immunology, right?
04:00And how your body tries to fight off infection.
04:03And we have a couple of different mechanisms
04:05of your immune system to do that.
04:06So there's a kind of T-cell responses, cytotoxic T-cells.
04:10So those are T-cells that recognize cells in your body
04:12that have been infected with a pathogen
04:14and destroy those cells, right?
04:15Because the cells are producing copies of the virus,
04:17releasing in the bloodstream.
04:19So if we're able to destroy infected cells,
04:21we can potentially stop infection, prevent disease,
04:24and then another key response
04:26that your immune system has is through antibodies.
04:28And that's sort of what's on the bottom here
04:30and is that B cells are able to produce antibodies.
04:34And what those antibodies do is they basically grab
04:36onto these surface proteins, right?
04:38So remember we talked about the spike protein,
04:40and what antibodies do is basically just bind onto that
04:43and sit there and so neutralizing antibodies.
04:46So there's two classes of antibodies that are kind
04:47of relevant for vaccines.
04:48So neutralizing antibodies really, you're just doing that.
04:50They're gonna sit on all of those spike proteins
04:53and because they're sitting there now the virus can't grab
04:55onto your cells to infect them.
04:57There's also binding antibodies, which are somewhat
05:01considered to be less important in this context,
05:03but what those guys do is bind onto those surface proteins,
05:05they don't neutralize the virus itself,
05:07but they send out chemical signals to other cells
05:09in your body that say, hey, here's a virus.
05:11Please come eat it for me.
05:13So those are the sort of antibody classes response
05:15that you can have.
05:16So there's these two sort of immune mechanisms
05:18that we have to neutralize infections by viruses.
05:22How do they learn to neutralize them?
05:25Well, there's this sort of middleman.
05:26So we're moving just to this middle panel here
05:28with these APC cells,
05:30so these antigen presenting cells, right?
05:32Those are the guys that what they're doing
05:33is basically digesting little bits
05:36of the virus in this case of the surface protein, right?
05:40And they're teaching or training your immune system
05:43to recognize that pathogen, right?
05:44So they're the ones that go and talk to the T cells,
05:47talk to the B cells and say,
05:48here's that how this virus looks,
05:51please go produce some antibodies or please recognize cells
05:53that have been infected with this
05:55and neutralize them for me.
05:56So really again, the whole right side of this plot
05:59is about your immune system.
06:00This is the way your immune system fights off infection.
06:03And what's different between this slide
06:05and the next few slides is basically how we present
06:08pieces of the pathogen pieces
06:09of the virus to these APCs, right?
06:12So how do we get these APCs, the material that they need
06:14for you to mount an immune response against SARS-CoV-2?
06:19And so the first class of vaccines
06:21I'll describe are nucleic acid vaccines.
06:23And so I'm talking about first
06:25because they're sort of the first wave of vaccines
06:27that are in phase three trials in the US.
06:29So Moderna and Pfizer, who are probably the most advanced
06:33candidates for US licensure are both mRNA vaccines.
06:37And so how are those vaccines made?
06:39Well, we take a little bit of messenger RNA,
06:42a little bit of viral genetic material,
06:44and wrap that in a lipid shell, right?
06:45That's the construct of the vaccine.
06:47And when you're injected that lipid shell
06:49latches onto your cell, right?
06:51Delivers that mRNA into your cell,
06:54just like a natural infection, right?
06:55Remember the SARS-CoV-2 grabbed onto your cell
06:57and inserted itself and then made copies of itself.
07:00So what is the mRNA doing once it's in your cell,
07:03it's actually just making copies
07:05of the spike protein itself, right?
07:07So you're manufacturing this protein within your own cells
07:11that are then released for these APCs to detect.
07:14So this is how we're getting these APCs,
07:17spike protein with an mRNA vaccine.
07:19We're basically using your cells as a warehouse
07:22to produce the antigen of the vaccine
07:24and so this is a really cool idea and a new idea, right?
07:28So, am mRNA or DNA vaccine has never been licensed before
07:31and that's not to say that we tried many times and failed.
07:34It's just to say that this is a very new technology,
07:36and it's sort of interesting that it's kind of come
07:40to the forefront in this context.
07:41So why do we like mRNA vaccines?
07:44Well, they're very fast to manufacturer.
07:46We'll talk about some of the other vaccine constructs
07:48where we're making this spike protein in a lab,
07:50and that is a long and arduous.
07:52It needs to be very careful process
07:54and when we're thinking about scaling up
07:55vaccine manufacturing, mRNA vaccines are very appealing
07:59in that sense, you can manufacture them
08:02very quickly at scale.
08:03They don't require a cold chain
08:05and so that's another great advantage these vaccines enjoy
08:10in terms of thinking about vaccine deployment,
08:12particularly in developing world settings.
08:15But again, this is a brand new technology.
08:17We don't have any safety data
08:18from past vaccines with this construct.
08:20We don't have any efficacy data.
08:22So, it's sort of an open question in the field
08:25as to how well these things are gonna work.
08:28So moving to sort of more classical, constructive vaccines
08:30and viral vector vaccines.
08:33So again, the right side of this picture
08:34is exactly the same.
08:35The story is how do we get an APC the right antigen?
08:38How do we show an APC a little bit of the spike protein?
08:41So a viral vector vaccine, right?
08:45Is going to take a different virus and splice
08:49a little bit of SARS-CoV-2 into that virus, okay.
08:52So for example, AstraZeneca, that's the Oxford that you may
08:56have heard of, they take a chimpanzee adenovirus,
08:58that's like, it's a virus that causes the common cold
09:02in chimpanzees and they splice in a little bit
09:04of SARS-CoV-2 into that and so that sort of host virus,
09:09that adenovirus holds genetic material
09:12infects your cells and your cells then produce the antigen.
09:16They produce the spike protein of SARS-CoV-2.
09:20So AstraZeneca and Janssen are using this construct again,
09:22both with adenoviruses, a very common virus vector.
09:26And again, we like these types of vaccines
09:29because they're quick to manufacturer,
09:32but a challenge of them is that your body
09:34can sort of develop separate immune responses
09:37against the vector itself, right?
09:39So you can develop a separate immune response
09:42against say an adenovirus right?
09:45Such that your body neutralizes those adenoviruses
09:47before they're able to infect your cells
09:50and produce the SARS-CoV-2 antigen.
09:52So we do see tendency a kind of faster waning
09:55vaccine effects with this class of vaccines.
09:58So moving on to subunit vaccine.
10:00So this is NovaVax and Sanofi's vaccine
10:03will be subunit vaccines
10:05and this is where I kind of mentioned before
10:06actually what happens here is these spike proteins
10:10or whatever the antigen is,
10:12is created and purified in a lab.
10:14So they actually use insect cells
10:17that they infect with SARS-CoV-2,
10:20those insect cells then produce the antigen that's purified
10:23and that's what goes into the vaccine
10:26are those protein subunits, right?
10:27So there we're just directly giving you the spike protein
10:30that we've grown outside of the host
10:33and that's how we're getting these APCs, those antigens.
10:37And so this is a commonly used vaccine construct.
10:40So the hep B vaccine is highly effective.
10:42HPV vaccine is highly effective.
10:44That's the construct of these, but the downside of course
10:46to it, so it's a well-trodden way of developing vaccines.
10:51But the downside is that they're slower to manufacturer.
10:53There's this whole process where we have to cultivate
10:55and grow these viruses in a lab, we have to purify them,
11:00and moreover they often also require an adjuvant.
11:04So that's really just sort of adding something a little bit
11:07extra that stimulates a better immune response in your body.
11:11So basically at the site of injection,
11:13it's something that increases
11:14your inflammatory response actually
11:16to kind of stimulate your immune system
11:18into recognizing those antigens
11:20and developing an immune response against them.
11:22So there's subunit vaccines.
11:24So the fourth class here is a weakened/inactivated vaccine.
11:27And so this is, I think, what most people like what
11:30my grandparents probably think all vaccines are,
11:33is basically we take a pathogen
11:35and we weaken it in some way, or we kill it, right?
11:39And then that's the construct of the vaccine
11:40and that's what's injected into you.
11:42And we go through this similar process there
11:45that literally mimics natural infection, right?
11:47Where your cells are infected by this weakened form
11:51of the virus, the virus replicates,
11:53and that's how we get antigens to the APCs.
11:56So this is the construct used in of course
11:58some classic vaccines like MMR, polio vaccine,
12:01but again, it's slower manufacturing, right?
12:03Because we have to cultivate the virus
12:04in the lab and then it also requires adjuvants.
12:07So I don't think there's currently any plans
12:10to have US phase three trials
12:12of weaken inactivated vaccines, but there are in China.
12:15So Sinopharm and Sinovac vaccines
12:17were using this construct.
12:21So that's just a bit of a background in immunology
12:24and how all this works and how we think about preventing
12:26infection with SARS-CoV-2 and hopefully preventing
12:29clinical disease COVID-19 disease.
12:32So now we're gonna segue to talk a little bit
12:34about the vaccine development process, right?
12:36'Cause this has all happened extremely fast.
12:38So let's talk about sort of the process whereby
12:40vaccine products are typically brought to market, right.
12:43And what looks a little bit different
12:45about the COVID-19 vaccine development process?
12:49So this is a figure from a nice New England journal paper
12:52that's referenced at the bottom
12:54that's just talking about sort of what's different
12:56this go around in terms of how are we accelerating
12:58the vaccine development process.
13:00And so I think as biostatisticians,
13:01anyone who works on clinical trials is fairly familiar
13:04with the traditional paradigm
13:06for bringing products to market, right.
13:08It involves sort of a lot of R&D
13:11in the lab, preclinical work
13:13and then you start doing human trials in phase one,
13:16these are small dose finding safety trials,
13:19checking whether these vaccines
13:21generate any immune response.
13:23And then what we'll often do is in vaccine trials
13:25is run a small randomized trial.
13:27That's a phase two trial, right?
13:29We're we'll have a placebo control,
13:31maybe pick out a particularly high risk population
13:34and start to see if we're getting
13:35any efficacy signal, right?
13:37And this is a very deliberate process, right?
13:39Phase one typically advances very slowly.
13:41We have lots of safety concerns.
13:42Phase two, we think very hard about whether the efficacy
13:45signal was really worth it to advance a candidate to
13:48phase three and it's a very deliberate process, right?
13:51To get to this phase three licensure trial, right?
13:53So the phase three trial is the big one involving
13:56the most participants.
13:57It's a randomized controlled trial, right?
14:00Enrolling many, many subjects that's well powered
14:02to detect ethicacy signals and based on the results
14:04of that phase three trial and safety data
14:07that's been accumulated throughout
14:08this whole process, right.
14:10We're able to provide licensure ideally for a product.
14:14And so that's sort of the clinical development process,
14:17but also in the context of COVID vaccines
14:19it's important to think about
14:20the manufacturing process, right.
14:21And how that looks a little bit different.
14:23So typically right, companies are very sort of hesitant
14:27to scale up manufacturing before they know that they have
14:31a product that will be licensed, right.
14:32Which makes sense, you know, they're sort of risk averse.
14:34We don't want to start manufacturing a product
14:36that may ultimately be shot down by the FDA.
14:39So really large scale manufacturing
14:40is not happening until after product licensure.
14:44So what's happening with COVID vaccine
14:45is basically this whole long deliberate timeline
14:49is being compressed into a shorter time period.
14:52And so how do we do that?
14:53Well, basically what happens is we've collapsed
14:56the phase one and phase two trials, right?
14:58So we're doing small safety studies.
15:00We're checking whether these vaccines
15:01are generating immune responses,
15:03but we're really not doing that smaller efficacy study
15:06that is typical of vaccine development.
15:10And so we're collapsing the phase one and two process,
15:13the phase three process is where we're at, right.
15:15We're doing these large scale trials, right?
15:16Because we need robust efficacy data
15:19and we need robust safety data to gain licensure,
15:22but a big thing that has changed, so the clinical process
15:24yeah a little bit compressed, but mostly the same,
15:27the big thing that's changed
15:28is the manufacturing process, right.
15:30Is we wanna make sure that once a vaccine is licensed
15:33and is proven to be safe and effective that we're able
15:37to start distributing that vaccine immediately.
15:39So that means that manufacturing needs to start ramping
15:41up right before we ever have a signal of efficacy
15:45and that's a huge risk for companies to take.
15:47So, I'll talk in a couple of slides about sort of how
15:51the government has come in to try to remove
15:54some of that risk from these companies
15:56and then the next slide I think is just showing sort of
15:59that it's really impressive that we're even talking
16:01about potentially having a COVID vaccine available this year
16:05or early next year, just given the timelines
16:08that are required to bring effective vaccines to market.
16:11And so here's just a few, you know,
16:13polio, measles, chickenpox, mumps,
16:14all multiple years of development for these vaccines,
16:18you could add malaria on this list.
16:19It took about 30 years
16:20to get a partially effective malaria vaccine to market.
16:24So this is typically a very long process, right?
16:26And for COVID, we're looking at hopefully doing this
16:29in just under a year or two.
16:31So how is the US government playing a role in this?
16:35Well, it's through this program that you may have heard of
16:37called Operation Warp Speed,
16:39which is this huge convoluted mess of an amalgamation
16:44of programs across the government
16:45from DOD to many branches of NIH, BARDA, NIAID,
16:50so it's sort of all over the place.
16:51And this is really just the same figure
16:54that I showed you from the New England journal paper.
16:57Just maybe a slightly more confusing
17:01if you ask me, I don't think Edward Tufte,
17:03he would be a big fan of graphic
17:05but the point here I want to mention
17:07is how is the government responding
17:10to COVID vaccine development?
17:11How are they contributing to that process?
17:13Well, there's really two ways that they've offered
17:15to accelerate the process.
17:17The first is through funding
17:19of phase three clinical trials, right?
17:21So a number of companies, six of the major companies,
17:24basically every company that's running a phase three trial
17:26in the US besides Pfizer that you've heard about
17:30is contracting with BARDA.
17:32That's an arm of the NIH,
17:34they're contracting with the government
17:36to have the government fund their phase three trials.
17:39So it's a joint agreement between the government
17:40and these companies where the government,
17:42you the taxpayer, right, are paying for these
17:45phase three trials that will eventually lead to licensure.
17:49So that's the first way that the government
17:50is sort of throwing money at this problem.
17:52It's through design and paying for these phase three trials.
17:56The second way is that they're paying
17:58for manufacturing, right?
17:59They're removing that risk for these companies
18:01by basically committing to buy a certain number of doses
18:04before we ever have any efficacy data.
18:06So we're in the hole basically to all of these companies
18:08for a fixed number of doses right.
18:11But that motivates the companies then to scale up
18:13their manufacturing ahead of the time
18:15that efficacy data are available.
18:18And that type of agreement has been entered
18:20into with Pfizer as well.
18:21So all of these companies that OWS Operation Warp Speed
18:24is running the phase three trials for
18:27also have this manufacturing agreement.
18:29Pfizer has that manufacturing agreement as well.
18:33So what role have I played in any of this big messy thing?
18:38So I work with a great group of scientists
18:41in the COVID-19 Prevention Network.
18:43So this was a clinical trials network established
18:45by National Institute of Allergies and Infectious Disease
18:48and NIAID so that's an arm of NIH,
18:51and it's basically anyone who works in clinical trials
18:54is fairly familiar
18:55with these clinical trials networks, right?
18:57It's an amalgamation of researchers and study sites,
19:01laboratories, people who focus on recruitment and retention
19:04of trial participants, statisticians.
19:07So it's researchers who are really experts
19:09in running clinical trials,
19:10designing clinical trials
19:12and ensuring their robust conduct.
19:15So the CoVPN was formed by basically leveraging
19:19four existing clinical trials networks.
19:21One of which I was a part of,
19:22which is the HIV vaccine trials network.
19:24And so from our group, we've really brought a great group
19:27of statisticians, many of whom are at the Fred Hutch
19:30in Seattle as well as great groups of laboratories at U Dub.
19:34And so what are the roles
19:36that we're playing in these trials?
19:38So in our statistical group,
19:40there's a couple of statisticians who are designated
19:44as like CoVPN representatives
19:46for each of these phase three trials.
19:48So I sit on calls with these trials and advise
19:53on their design and analysis approaches
19:55for their efficacy monitoring, for their safety monitoring.
19:58We help them address DSMB and FDA comments
20:01and sort of that's all happening in conjunction
20:04with both government statisticians, right.
20:06Representatives of BARDA and NIAID
20:10as well as company statisticians.
20:12And so we get on these calls and, you know,
20:14nerd out over clinical trials,
20:16statistical decision-making, and it's a good old time.
20:21Another aspect that we really contribute a lot on,
20:24or that CoVPN has sort of been tasked with taking
20:26the lead on is the development of immune correlates.
20:29And so that's the part of my talk
20:31where I'll get a little bit into statistics
20:32and talking about what immune correlates are,
20:34some of the types of analytic approaches
20:36we use to study those and the idea of immune correlates
20:38just to give you a teaser
20:40so you don't, you know, sign off Zoom early.
20:42So immune correlates are really the idea there is
20:45we're looking for immune responses that are predictive
20:48of the vaccines working, right.
20:52So what we'd really like to be able to do is understand,
20:54okay, if we're able to generate this level
20:56of neutralizing antibody,
20:58then that will lead to this level of protective effect
21:00of the vaccine, right?
21:02So that's the whole goal there is identifying
21:04what are these immune responses that are
21:05responsible for providing protection?
21:08Okay so I'm gonna walk through just a few of the design
21:11and analysis questions.
21:12And so these are things that have come up
21:14as we've worked with these company statisticians,
21:16as we thought about sort of the whole OWS vaccine program,
21:20what are some of the issues that statisticians
21:22are kicking around and people who have worked
21:24on clinical trials, right,
21:25a lot of these issues aren't gonna be new
21:27and one thing that I think is sort of interesting about this
21:30whole pandemic and operating as a public health professional
21:34in this and a clinical trial statistician in particular,
21:37is that a lot of things that we take for granted
21:39as scientists are either very confusing
21:42or sort of counterintuitive for a lot of the lay public.
21:45And so it's been sort of interesting to have that laid bare.
21:48In some of these issues, some of these things
21:50that we think are no-brainers like doing interim analysis
21:53for example are kind of highly controversial
21:56and have ended up being, you know,
21:57sort of areas of huge disputes.
21:59And so I just want to run through some of these issues
22:01that I think are quite fascinating, a lot of which,
22:04you know, really don't have a correct answer
22:06and they're really just sort of food for thought
22:07the types of things that we're thinking about
22:09when we're designing these trials.
22:11So I'll start by just giving a sort of more specific idea
22:16of what these trials look like and how they're conducted
22:18and I've picked AstraZeneca because that's the one
22:20I've worked on for the longest and most closely,
22:23but all of the trials sort of follow this similar design.
22:26And so the first thing I'll note
22:27is that you can read these trial protocols.
22:29So one of the interesting things that's happened
22:31in this COVID-19 development processes
22:33is there was a huge public push led by like Eric Topol
22:36and others to have the protocols of these trials
22:39made public, which when it happened was I guess
22:43when that push started happening, you know,
22:45I emailed all my colleagues and said,
22:47really do we not usually make protocols public?
22:50And that was just sort of interesting disconnect
22:51for me as an academic who's used to sort of everything
22:54being open science and that's a no brainer right.
22:57Working in this setting, right,
22:58where these protocols are really seen as trade secrets
23:00for pharmaceutical companies.
23:02So it's really unusual that actually these protocols
23:05for clinical trials have been made public.
23:06So it's sort of neat, but one of the things that happened
23:09is all of these protocols went public and reporters
23:12got their hands on them and said, wow,
23:13these are really dense documents, right?
23:15If you've ever looked at the clinical trial protocol,
23:18it's like a hundred pages of very specific definitions
23:21and safety monitoring and what symptoms lists
23:24you're gonna use and what surveys
23:26you're gonna give to people.
23:27So they're very sort of detailed documents
23:28that are kind of hard for the public to parse.
23:32So it's been sort of a be careful what you wish for thing
23:34in terms of releasing these protocols, but that's an aside.
23:38So let's talk about actually what these trials look like.
23:40So here's a schematic, and again,
23:42this is AstraZeneca in particular,
23:44but this is basically the design of most of these trials
23:47will look something like this.
23:48So who is the population?
23:50Most of these trials are gonna be primarily in adults.
23:53I think Pfizer has now started
23:55to talk about including children.
23:57I'm not exactly sure where that's happening,
23:59but adults for the most part,
24:01these are mostly healthy individuals
24:05that don't have, you know, chronic diseases
24:07that are at risk or high risk of death.
24:10And we're really looking at targeting individuals
24:12who are at an increased risk for SARS-CoV-2 acquisition
24:15and severe COVID disease
24:17and so the idea there is number one
24:19these are the people that are bearing
24:20the brunt of the pandemic, right?
24:23So we want to be able to get a product to those people
24:26as fast as possible.
24:27But number two also, right, that means that we'll accrure
24:29from a sort of cold hearted and statistician point of view
24:32that means we'll accrue end points faster.
24:34We'll observe more cases of COVID-19 disease
24:37and potentially get an efficacy signal a little bit faster.
24:40So there's a lot of interest in sort of recruiting
24:43and retaining individuals at high risk for COVID-19.
24:45So you can go onto the COVID-19 prevention trials network
24:48and fill out a survey, right.
24:49Then we'll basically under the hood
24:51assess your risk for COVID-19
24:53and if you're found to be at high risk,
24:54we'll aggressively email you and try to get you
24:55enrolled in one of these trials.
24:57If you're at low risk,
24:57we'll say, thanks for taking the survey,
24:59we'll be in touch and likely
25:01you won't hear from us anytime soon.
25:03Okay so that's the trial population.
25:05So how does the actual trial conduct look?
25:07So there's kind of a mixture here.
25:10AstraZeneca is using a two to one randomization scheme.
25:13So you have two chances of getting the active vaccine
25:16versus one chance of getting a placebo.
25:18And in this case, it's a true placebo, just a saline dose
25:22and then most of the vaccines, most all with Janssen
25:26being the accepted are two dose vaccines.
25:28So you receive the first dose at day one
25:30and the second dose about a month later.
25:32And in the interim, we take a couple of measurements.
25:34We have a phone call to assess reactogenicity right.
25:38Does your arm hurt, or have you experienced any adverse side
25:41effects of the first dose of vaccine?
25:44And then there's also an immune response measurement
25:46that happens after a couple of days.
25:48So we get an early signal
25:49of how immunogenetic these vaccines are.
25:51And so then individuals come in for their second dose
25:53of vaccine and it's a similar story, right?
25:55Did you have any reactions?
25:57We measure your immune response and after that,
25:59that's sort of when the clock starts for active follow-ups.
26:02So this day 57, that's two weeks roughly after,
26:07am I doing that math right?
26:08Well, it looks like roughly two weeks after the second dose
26:12of the vaccine is typically when this clock
26:14is gonna start and we're gonna start counting COVID events.
26:17And then it's sort of just the standard sort of game we play
26:21in clinical trials.
26:22We wait for events to accrue.
26:23We have certain monitoring plan
26:25for when we're gonna check for efficacy
26:27and we'll talk about some of that.
26:28So, I just want to note that there's sort of two ways
26:30that we're ascertaining events
26:32that are happening here, right?
26:33The first is passive monitoring.
26:35What that means is we basically wait for individuals
26:37to present with symptoms of COVID 19, right?
26:39So you get a cough, you lose taste, right?
26:41You call the study site, right?
26:45So I am having these symptoms.
26:46They say, come on in.
26:47You get a PCR test to see whether you're infected.
26:50And in that case, you would count
26:51as a COVID-19 endpoint, right?
26:53If you check off some check boxes for symptoms
26:56with COVID-19 disease, you have a PCR positive test.
26:59You'd go down as a COVID 19 endpoint.
27:01There's also these sort of active follow-up visits.
27:04So these like day 90, day, 180 and day 360,
27:08and at those visits we'll do a serology check.
27:11And what that means is we basically take a blood draw
27:13and we measure whether you have antibodies
27:16against SARS-CoV-2, right, antibodies that are distinct
27:19from the antibodies that are generated
27:20in response to the vaccine.
27:21So we're basically able to tell whether you were infected
27:24in this sort of interim period,
27:26when you show up for these visits.
27:29So that's active follow up
27:30and so there you're gonna be able
27:31to pick up sort of asymptomatic cases, right?
27:33'Cause if you never have symptoms, you'll never come in
27:36and be captured by passive followup.
27:38So we have to wait for these set clinic visits
27:40to do the serology testing,
27:42to ascertain it asymptomatic cases.
27:44And so this is gonna actually play a role
27:46in a little bit, when I started talking about, you know,
27:48what are the end points that we're thinking about measuring?
27:50Like, what do we want to know how well
27:52the vaccine works at preventing?
27:53Is it asymptomatic infection?
27:55Is it disease?
27:56Is it severe disease and so forth?
27:57So we'll talk through some of those issues,
27:59but just want to note already that the design has started
28:02to inform some of the challenges that we might see
28:04when we want to talk about how well the vaccine works
28:07against certain forms of infection and disease.
28:10And so I think if you read the newspaper and you'll see
28:13the term vaccine efficacy tossed around a lot.
28:15So the first thing I want to talk about is right,
28:17what is the primary hypothesis
28:19that these trials are trying to test?
28:21And what is the parameter?
28:23What is the estimate, right, that they're going after
28:25in these trials and for whatever reason
28:27nobody consulted me when they decided that VE
28:29would be measured in this way.
28:32But for whatever reason, we studied this that we quantify
28:35the efficacy of a vaccine in a sort of weird way.
28:37So a vaccine efficacy, we describe as the percent reduction
28:40in relative risk comparing vaccine to placebo.
28:43So it's this one minus a risk ratio.
28:46There's a one minus a risk ratio where you take the risk
28:49in the vaccine and the numerator and the risk
28:51in the placebo and the denominator.
28:53So, I mean,
28:54we can just play a quick little intuitive game, right?
28:56How do we get a VE close
28:57to one that would be a perfect vaccine?
29:00Well, we would make the risk
29:01in the vaccine close to zero, right?
29:03So that sorta makes sense.
29:04If you have a perfectly effective vaccine,
29:06there'll be no risk of infection and or disease
29:08amongst the vaccinated.
29:09So you would get VE close to one.
29:11But on the other hand, how do we make VE zero?
29:13Well, we would take the risk in the vaccine
29:16and set it equal to the risk in the placebo, right.
29:18In which case basically saying the vaccine's not doing
29:21anything and then on the other hand,
29:23a VE is negative, right?
29:24That's indicating that there's actually higher risk
29:26in the vaccine.
29:28So just to give you sort of a few reference points, right?
29:31So that VE of one is perfect, VE of zero is nothing
29:34and what we're really hoping for with these COVID trials
29:38is a VE of at least 50%.
29:40And that's sort of the cutoff that FDA guidance
29:42has stipulated is that you need to show a point estimate
29:46of VE for your primary end point.
29:48And again, we'll talk about what these primary end points
29:50are but we need a VE against a primary end point
29:53of at least 50%
29:54and we need to definitively rule out the possibility
29:59that the vaccine efficacy is less than 30%.
30:02So basically we have to reject the null hypothesis
30:05that VE is less than 30% along with having a point
30:08estimate of VE being greater than 50%, right.
30:11And we need to do that while controlling type one error
30:13at two and a half percent.
30:16Okay and so here, just one final note,
30:19since this is a statistics talk,
30:20I'll talk a little bit more
30:21about what I mean by risk, right?
30:23So risk here can be quantified in a number of ways
30:26and it often is.
30:27So we can quantify this using hazards, for example,
30:29like you can imagine fitting a Cox model, right.
30:31A proportional hazards model, right.
30:33That only adjusts for vaccine, right.
30:35And presenting like one minus a hazard ratio
30:37from a Cox model, that's something that's commonly done.
30:39You can also think about cumulative incidents, right?
30:41So like mapping,
30:43maybe one minus a survival probability as a way
30:46of quantifying risk, incidents rate ratios.
30:49So they're all sort of used for different vaccines.
30:52And usually we like to sort of argue
30:55about which one of these is better
30:56and I've thought a lot about that in my career.
30:58And in this setting, it turns out because COVID
31:00is such a rare event that all of these ways of quantifying
31:03rates are basically the same and you end up
31:05with almost identical operating characteristics of a trial.
31:08So it's really not worth sort of losing sleep over
31:10whether we're talking about VE in terms of hazard
31:12or incidents or incidents rate and so forth.
31:16So how are folks going about estimating this VE?
31:19Here's just a quick table of the four most advanced
31:22phase three trials,
31:23the ones that have released their protocols at least.
31:25So we see for Moderna, AstraZeneca, and Janssen,
31:28they're using pretty kind of the standard approaches.
31:31Moderna a Cox model as I describe,
31:33AstraZeneca a Poisson regression model,
31:35it's like, okay, that's basically a Cox model,
31:38and then Janssen is using a sort of exact binomial test
31:41with this sequential probability ratio rest.
31:44Pfizer is a little bit of the oddball.
31:46So they have stipulated a bayesian approach
31:49wherein they're basically specifying a prior
31:53for vaccine efficacy and are using sort of
31:55a beta-binomial bayesian approach to evaluate
31:58the posterior probability of the vaccine efficacy
32:00is greater than 30% and so at the end of the day,
32:04there's four different statistical methods here.
32:06Again, if you do a simulation study with parameters
32:09that are approximately similar to what we expect to see
32:11in these COVID trials,
32:12you're really not gonna see much difference in terms
32:14of operating characteristics across these.
32:16So it's interesting to notice that assertions
32:18that there's these different approaches,
32:19but at the end of the day,
32:20we're basically talking about how many vaccinated people
32:22get infected, how many placebo people got infected,
32:25and almost all of these methods are gonna yield
32:27very similar inference.
32:29When it comes down to brass tacks,
32:31how many numbers fall into those categories?
32:33So that's a little bit about sort of
32:36how we quantify VE in these settings
32:38but one of the big things I haven't described yet
32:40is VE against what, right?
32:42What is the end point that we're measuring here?
32:43And so here's a figure from a paper we just had come out
32:47in Annals of Internal Medicine, the link's here.
32:50So this is where we were spending a lot of time,
32:52you know, earlier this summer, thinking about,
32:54you know, what's the right end point,
32:56what's the right end point for a primary analysis
32:58of the clinical trial.
32:59And it's complicated for something like SARS-CoV-2, right?
33:03Because we know we can start up here
33:04with the SARS-CoV-2 infection, right?
33:07That's sort of the base, you can become infected
33:09and then a number of things can happen, right?
33:11You can go on to be infected but develop no symptoms.
33:14So we would call that an asymptomatic infection,
33:17or you can develop symptoms, right.
33:18In which case we don't call you
33:20a SAR-CoV-2 infection anymore,
33:21we call you a COVID-19 disease endpoint.
33:25You have a clinical manifestation of your infection.
33:28But even beyond that, right,
33:30amongst people who exhibit symptoms
33:31some of them, maybe many of them are quite mild, right.
33:34So we have this kind of category of non-severe COVID,
33:38whereas others we know that are extremely adversely
33:41impacted by infection and end up with severe COVID disease.
33:45So you have all of these choices of sort of
33:49which end points you might want to talk about
33:51and so I'll kind of walk through some what I see
33:53as the positives and negatives of this and then I'll also
33:56talk about this burden of disease
33:58very briefly end point that we've put together
34:01and so that's kind of a composite end point
34:03that we've suggested that could kind of bring all
34:05of these different end points together.
34:07Okay so starting with SARS-CoV-2 infection, right?
34:09Why might we like any sort of any infection, right.
34:12Asymptomatic, symptomatic don't care,
34:14let's count any infection as an event
34:17and measure VE against preventing infection.
34:20Okay and so that's definitely relevant, right.
34:22It's relevant the context of a pandemic.
34:24We're preventing infections,
34:25we're preventing spread of the disease,
34:27we're bringing our knot down, we're impacting the pandemic.
34:30And moreover, we're going to see many more infections
34:33than we will cases of symptomatic disease.
34:36We know that many people who were infected
34:37never go on to develop symptoms
34:39so thinking about having an answer faster, right.
34:42SARS-CoV-2 infection is a nice endpoint,
34:44but then the question is,
34:46is it a clinically relevant endpoint?
34:47So it's really not describing an impact on patients at all.
34:52So we could kind of question its relevance
34:56from that perspective.
34:57The other thing, right, is that we remember going back
34:59to the study design, we're only able to ascertain
35:01asymptomatic infections sort of very coarsely in time
35:05and moreover you have this phenomenon that happens
35:09is that when you're testing many, many individuals, right.
35:12It's sort of the classic biostat
35:14one-on-one problem that we give people, right.
35:16You're testing many individuals, but the prevalence is low.
35:19So even if you have high sensitivity and high specificity,
35:22you could end up with low positive predictive value.
35:25And the effect of that when you come to the time to analyze
35:28the data is that you'll be biasing VE towards the knoll.
35:31So it's actually, while it seems like maybe a nice end point
35:35from the perspective of observing many infections,
35:37it's a very challenging endpoint to analyze quantitatively.
35:41So moving down we could talk about COVID.
35:43So again, COVID is just infection,
35:46PCR confirmed infection with clinical symptoms.
35:49So that's of course more clinically relevant, right.
35:51Because we're starting to talk about
35:53an impact, excuse me, the endpoint that impacts patients.
35:57All right so that's more clinically relevant and moreover
36:00we'll expect to have a reasonable number of cases, right.
36:03By including more mild cases, for example,
36:06in this endpoint definition.
36:09But then on the other side of that coin
36:10is it really that clinically relevant
36:12if we're just talking about mild symptoms?
36:15We're talking about a disease where you get it
36:16and you end up with a little cough for a couple of weeks
36:18and that's it.
36:19So then maybe you suggest using severe COVID right.
36:22That's the most clinically relevant one.
36:24We want to be protecting the most vulnerable individuals
36:26so we should be quantifying how well our vaccines
36:28work towards preventing those most severe end points.
36:33And so most clinically relevant,
36:35and also there's sort of a long history
36:37of vaccine development where really we see the best VE
36:40against severe cases of disease.
36:43So that's really where we expect the vaccines
36:45to have the most impact is maybe we are not preventing
36:47you from being infected but we're lessening the symptoms
36:51once you become infected.
36:52So we're not totally blocking transmission
36:55but we're making a clinical impact on disease
36:57and that's sort of been seen
36:58for a number of vaccines in the past.
37:00The downside of this end point of course
37:02is that there's very few cases expected to be observed.
37:04So amongst all infections,
37:05only a fraction have any symptoms.
37:07Amongst those with any symptoms,
37:09only a fraction develops severe symptoms.
37:10So we're really whittling away the number of end points.
37:13So we need to do larger trials
37:15or have longer follow-up to evaluate this endpoint.
37:19And so in that paper, I'm sort of pressed for time
37:21so I won't spend too much time talking about this,
37:24we also proposed this burden of disease measure
37:26where you're sort of scoring these these outcomes, right?
37:29So maybe you would get a score of zero
37:31if you're an asymptomatic infection
37:33'cause it's really no burden on you as a patient, right?
37:36You don't have any symptoms.
37:37And then we're sort of assigning arbitrarily
37:39a score of one for non severe COVID so that's like
37:42mild cases of COVID and a score of two
37:45for severe cases of COVID and this end point actually
37:48has some nice operating characteristics we think,
37:51but of course it's subject to controversy, anytime you start
37:53talking about an ordinal scoring system, right,
37:57you start to raise questions about how you're assigning
37:59the burden of disease score, right?
38:01Why should severe cases be a two
38:03versus a three versus a five and so forth?
38:06So you can kind of get bogged down
38:07in some of the specifics of that.
38:10So what has FDA said about this?
38:12So FDA guidance documents states that either
38:15the COVID end point or SARS-CoV-2 infection
38:18is an acceptable primary endpoint
38:19and then somewhat ironically OWS has been telling companies
38:22that infection alone is not acceptable
38:24as a primary end point.
38:25So we had one company that was interested in including
38:28that as co-primary and for whatever reason we told them
38:31please don't do that, and then beyond that so COVID
38:36has sort of won out as the end point of choice.
38:39But beyond that FDA guidance states that companies should
38:42consider powering efficacy trials
38:44for the severe COVID endpoint as a co-primary or at least
38:48as a key secondary endpoint in the trial.
38:51And so so far only Janssen has taken them up on that offer
38:54of making severe COVID primary.
38:56And that's why, if you look at the number of individuals
38:58that are planning to enroll in their trial,
38:59it's twice as many as any of the other OWS trials.
39:03So like AstraZeneca is planning for 30,000,
39:04Janssen is planning for 60,000 in their trial.
39:08And that's the power, to see more cases of severe disease
39:10to be sufficiently powered to detect VE against that.
39:15So this is a controversial slide.
39:17Or this is virtual topic I found,
39:20again, something that clinical trials statisticians
39:22sort of take for granted is doing interim analyses, right?
39:26If the treatment is working and we have enough evidence
39:28to claim that a treatment is working,
39:29we'd like to stop that trial early
39:31to get that treatment to patients, right.
39:33One would think that that's true here
39:35and so many of these trials
39:36were designed with aggressive sort of interim looks, right?
39:40Because we're in the middle of the pandemic
39:41and we'd like to get a vaccine to individuals
39:44as quickly as possible.
39:45So I have a table, we won't go through it all here,
39:47just sort of the planned interim analysis
39:50for these different trials.
39:52I would say Pfizer seems to be the most aggressive so far.
39:56They have five interim looks or four interim looks
40:00and a final look at their data, right?
40:02So that's fairly aggressive.
40:03OWS again, the trials that we're running,
40:07we're really encouraging companies to be a bit
40:09more conservative in the approach to this
40:11and only maybe two or three
40:13and so you see what's been adopted
40:14by Moderna and AstraZeneca
40:17and so this was really a big point of contention
40:19I think when these protocols were made public is this idea
40:22that like, can you really know that a vaccine works
40:25based on 32 data points, right?
40:27We're talking about a vaccine that's going to be given
40:30to billions of people around
40:32the world based on these results
40:33and you're gonna do that based
40:35on the results in 32 individuals?
40:37And like, so I can stare at the math and say that like, yes,
40:40that appropriately controls type one error and so forth,
40:42but it still makes me just feel a little bit uncomfortable.
40:45There's a bit of dissonance between sort of my life
40:48as a statistician and just me being a human
40:50and saying 32 data points is probably not enough
40:52to decide to vaccinate billions of people.
40:54And so a lot of people I think sort of shared
40:57that viewpoint and in response FDA has now been sort of
41:01backpedaling in a way and asking companies to provide more
41:06data in order to grant an emergency authorization
41:10for their vaccine.
41:11So this EUA mechanism that FDA has of approving vaccines.
41:14And so in addition to an efficacy signal,
41:17now companies also are gonna be required, I think,
41:20and this is sort of still a moving target so this is maybe
41:23like data news at this point but I think prior to offering
41:26an EUA, FDA has now said that companies need to have 50%
41:29of participants complete at least two months of follow-up
41:33for safety signals and that you need to have at least
41:36six COVID cases in the oldest age group.
41:39Of course, that's an age group of particular interest
41:41in terms of severe cases and at least five cases
41:44of severe COVID in the placebo group.
41:45So they want to be able to see some data,
41:48even if you're not specifying severe COVID
41:50as a primary end point,
41:51they want to be able to see some data,
41:53some signal of efficacy against that
41:55in order to grant licensure.
41:57So I'll sort of, I won't go through this slide.
42:01It's just to say that like,
42:02sort of when Pfizer released their protocol,
42:04everyone was like, ooh a bayesian analysis
42:06and got very sort of skeptical, right?
42:09Because the Pfizer CEO has been out there
42:11sort of chest thumping and saying they're gonna have
42:13a vaccine before the election and so forth
42:15and then they came out with this bayesian design
42:17that was a little atypical and so everybody was asking
42:19the question, well, are they trying to hide something?
42:21So I sort of did a quick analysis
42:23and found that really it doesn't look that different
42:25than a classic kind of post hoc monitored design.
42:28And if you want to read more about that,
42:30I have some slides up on my GitHub about it.
42:33So let's see, I'm running low on time
42:36so I'm gonna skip over sort of the question
42:39of what happens if efficacy is declared early.
42:41So I have some reasons that we should be excited, right?
42:43If one of these trials stops earlier, I can get a vaccine.
42:46There's good data that the vaccine works
42:48and that's nice.
42:50I'd like to go back to something resembling normal
42:52as I'm sure you all would,
42:54but of course there's reasons to be concerned, right?
42:56If efficacy is declared early in particular,
42:58if that means that blinded follow-up
43:01in a study stops, right?
43:02Because that means we have no way
43:03to assess how durable the vaccine is.
43:05We won't be able to assess VE
43:07and key subgroups that we care about.
43:09We might not be able to assess VE
43:11formally against severe end points.
43:13So there's real sort of concerns
43:15about stopping these trials too early,
43:17and what the implications of that
43:18are both for evaluating the vaccine in question,
43:21but as well as how it impacts
43:23the other clinical trials that are ongoing.
43:26And of course in the current political climate,
43:29everybody's very concerned about the role
43:31political pressure might play in all of this.
43:33So yeah, so it's kind of a double-edged sword in some sense
43:37in terms of what happens if efficacy is declared early,
43:42but I want to save just a few minutes
43:43to talk about vaccine correlates 'cause I promised
43:45that I would show you some math and prove to you
43:47that I'm a real statistician.
43:48So let's do a little bit of that.
43:51So again, we're kind of shifting gears here.
43:53So that's the end of sort of talking about the primary
43:55analysis of these trials,
43:56what's gonna lead to their licensure.
43:58And the correlates of protection
44:00is sort of a key secondary analysis
44:02and so why is it so important
44:04that we're able to establish correlates of protection?
44:07Well, because it's gonna speed up
44:09the whole vaccine development process.
44:12So again, a correlative protection is really just,
44:14it's an immune response and really an assay
44:18to measure that immune response that's been validated
44:20to reliably predict vaccine efficacy.
44:23So why is that so important?
44:25Well, basically what we're hoping to achieve
44:28is the establishment of a surrogate
44:29endpoint for COVID disease right?
44:32So I've sort of mentioned the numbers that we're talking
44:34about in these phase three trials,
44:36enrolling 30,000 participants, 60,000 participants
44:40and ending up with one or two years of followup, right.
44:42Just to be able to answer the primary question, right.
44:44Does the vaccine prevent infection and/or disease?
44:48So that's a huge, expensive clinical trial.
44:50It takes a long time to get an answer
44:52and so it would be very nice if all we had to do right
44:56was give people the doses of vaccine that they need,
44:59wait two weeks and measure their immune response
45:02and understand does that vaccine work or not.
45:05That would be a much faster vaccine development process
45:07than where we're currently at
45:09in having to run these enormous phase three trials.
45:11So it's valuable for establishing a surrogate endpoint.
45:14It's also valuable for accelerating approval
45:17of vaccines that have been licensed in certain populations,
45:22but not others.
45:23For example, I mentioned that these phase three trials
45:25are mostly being conducted in adults.
45:27Well, what if we want to also obtain licensure for use
45:30of this vaccine in children?
45:32Well, if we had an established immune correlate
45:34we wouldn't have to do
45:35a large randomized trial in children.
45:37We could do it just a small immunogenicity study
45:39and use the correlates results to bridge the VE
45:42that we observed from the phase three trial.
45:45That's the immune response that we've observed
45:47in these children or pregnant women for example are
45:49another key population they're being
45:51excluded from these phase three trials
45:53but we'd like to understand if these vaccines
45:55are safe and effective in those women as well.
45:59So really this is one of the key goals
46:01of this whole OWS program and the key role
46:05that we're playing in the CoVPN is developing
46:08the sampling plan and the statistical analysis plan
46:12for the immune correlate studies
46:14and so it's just a little bit of the statistical issues
46:17that we're dealing with in these trials, right,
46:20is that sort of running assays
46:22so running these immuno assays on 30,000, 60,000 individuals
46:26takes a long time, it's expensive, and as it turns out,
46:30it's really overkill in terms of statistical power.
46:34So we can actually be a little bit more
46:35clever about how we design these correlate studies in order
46:40to get answers faster and more cheaply.
46:42So the way we do this is we use a case cohort design.
46:45So we're not gonna measure immune responses
46:47in all trial participants,
46:48we're gonna measure them in a sub cohort
46:50and that sub cohort will consist
46:51of a stratified random sub cohort.
46:54So we're gonna be sampling individuals randomly
46:56based on their baseline infection status.
46:58Were you infected with SARS-CoV-2 in the past?
47:01Based on your race, ethnicity, and based on age.
47:07And so based on that, we'll take a random draw
47:09of the trial population, about 1600 individuals,
47:13excuse me and everyone so I should mention right
47:19in the trial design everybody is having their blood drawn.
47:22And right now we're talking about whose blood
47:24are we gonna use to measure these immune responses?
47:27So we're gonna measure it in a random sample
47:29and then we're gonna wait until the trial is over
47:31or until one of these interim analysis concludes efficacy
47:35and we're gonna measure immune responses
47:38in all of the end points, right?
47:39Remember that like power in these analyses is drive
47:42by the individuals in which we observe endpoints.
47:46So we're gonna make sure we get immune responses
47:47in all the end point data, as in addition
47:49to this random sub cohort and it turns out that that's about
47:53as statistically efficient as running the immune assays
47:57on all 30,000 individuals in the trial.
47:59So this is this kind of classic case cohort design
48:01that Ross Prentice has been writing about for years
48:04that Norman Breslow did some sort of pioneering work
48:06on in the 2000s and I'll just talk a little bit about sort
48:11of how this complicates our life as statisticians
48:13and then maybe we'll leave a few minutes for questions.
48:16So here's the math, we made it.
48:18Well, the moment you've all been waiting for it
48:20to see some math.
48:21So just introducing, you know,
48:23why is this sampling design challenging
48:26from a perspective of generating estimators, right?
48:29Well, we can sort of immediately see that this isn't
48:31a totally random sample of the trial population, right?
48:35In particular we've over-sampled the individuals who end up
48:38getting diseased and it's fairly obvious
48:42that those individuals have potential to be very different
48:45than a randomly selected individual in the population.
48:47So we have a bias sub sample.
48:49So we need some statistical methodology to try to back out,
48:52you know, whatever this parameter is.
48:54We want to be estimating it in the whole trial population,
48:56not just in this biased sub samples.
48:59So how do we do that?
49:00So just a quick notation here,
49:02let's call W baseline covariates,
49:04A is a binary vaccine assignment,
49:07Y is your binary COVID endpoint for example
49:11and then we'll introduce this sort of indicators.
49:13Delta is one, if you're selected into this immune response
49:18sub cohort, either because you were a case,
49:20you were an end point or because you were randomly selected
49:23into the cohort.
49:25And then we'll call S your immune response.
49:28And then we'll just say, we'll represent this as Delta S,
49:31which just means we'll arbitrarily set everybody
49:33who's not in our sub cohorts immune response to be zero,
49:36that's arbitrary doesn't really matter.
49:38So let's talk about how estimation would happen.
49:41So let's pick a very simple parameter, right?
49:43Let's just say that we want to know what's the overall
49:45immune response in the whole population,
49:47not a particularly interesting parameter
49:50for actually measuring correlates,
49:51but just to motivate the types of statistical approaches
49:54that we use in these settings.
49:56So how can we control for the bias of the sampling design?
49:59Well, one of the most straightforward ways
50:01is to use the tried and true
50:02Horvitz-Thompson or IPTW estimator, right.
50:05Where we're just taking basically a sample mean
50:08but all our observations are sort of inverse weighted
50:11by their probability of being sampled into this sub cohort.
50:16And so that's, IPTW estimator if you're in causal inference,
50:18you're very familiar with this.
50:19If you're in survey sampling,
50:21very familiar with this.
50:22Very classical way of adjusting for this selection bias.
50:27It turns out that there's ways
50:28that we can be more efficient in doing this.
50:30We can use augmented estimators, AIPTW estimators.
50:33And the key idea there is that we take the IPTW estimator
50:37and we add a little bit of something to it
50:39and the key thing is that that little bit
50:41of something involves a regression of S the immune response
50:46onto the covariates that were used to sample individuals
50:50into the sub cohort.
50:52And so what's the intuition as
50:54to why this is more efficient?
50:56Well, you can imagine what if we had a perfect predictor
50:59of S measured at baseline, right?
51:01Then this regression here is essentially imputing
51:05the correct value of S
51:06for every single person in the population.
51:09So it's kind of like we're getting more data
51:11in some sense, and the nice thing about
51:15these approaches, these AIPTW approaches
51:17is that they're double robust and so again,
51:18if you work in causal inference a very familiar idea,
51:21and it turns out because we know
51:23the sampling probability by design,
51:25this regression doesn't have to be consistently estimated
51:28in order to obtain a consistent estimate
51:30of the parameter measures.
51:31So it's this really nice sort of double robustness property
51:33that says, yeah, you might be turned off
51:35from this augmented estimator
51:36because you have to do a little bit of extra work,
51:38you have to fit a regression model say,
51:40and maybe you're worried about missspecifying
51:42that regression well it turns out that because the sampling
51:44probabilities are known by design,
51:46you don't have to concern yourself with that.
51:47So it turns out you can use any old regression estimator
51:50here and still end up with a consistent estimate
51:53of the parameter of interest.
51:54And so we're applying this
51:55to much more interesting parameters.
51:57So we had a paper come out recently
51:59in biometrics that's linked here
52:01where we're starting to study a sort of causal inference
52:03flavored parameters in this context,
52:06things that we can really use to pin down,
52:08you know, mechanisms of these vaccines working.
52:10So, in this case, we're studying sort of the effect
52:13of a stochastic intervention, we call it.
52:16So it's sort of saying what would happen
52:18if we took everybody's immune response,
52:20this particular immune response that we observed,
52:22and we shifted it up just a little bit
52:25or we shifted it down just a little bit.
52:26How would that impact the risk of disease amongst
52:30the vaccinated individuals?
52:30So that's what this big, gnarly parameter is right here.
52:34And so you ended up looking at a plot
52:35that's kind of like this.
52:36So this is from an HIV vaccine trial.
52:39So at zero we're saying that's just the observed risk
52:42of the trial and as we move left we're saying,
52:44what would the risk be if we decreased your immune response?
52:47And so we can see in this example,
52:49we found that the risk would be increasing, right.
52:52And then if we're moving to the right
52:53is what would happen if we increase your immune response.
52:57And so we're kind of getting at something
52:59that's like a controlled effects mediation type parameter
53:03with this approach and so we're working out some
53:06of the details of the correlates plan currently
53:10and so when that's done
53:11we'll have it available for public comment.
53:13And again, we're academics, right?
53:14So we'll do it all open science.
53:16And then I'll just say like two words of conclusion
53:18and I'll shut up and leave some time for questions.
53:21So there's been a big concern
53:23in the current political climate that we're gonna sneak
53:26something through, that something's gonna be approved
53:28without sort of the standard amount of evidence
53:32that would be required, right.
53:33That there's political interference at the FDA
53:36and from where I sit, you know,
53:39I can say that the science behind the vaccine
53:41development program for COVID is extremely rigorous.
53:43These are exactly the type of people who you would want
53:46in charge of this decision making process
53:48and the type of people that will raise red flags
53:51as soon as sort of the process goes off the rails.
53:54So right now I feel good about where things stand.
53:57Of course, I watch presidential debates and hear, you know,
54:00garbage science coming out and I get a little bit concerned,
54:04but from where I sit right now,
54:05everything's looking pretty good.
54:07So overall, I'd say that the increased transparency
54:09by releasing these protocols
54:11has been good for scientists and consumers.
54:13We want to bring vaccines to market,
54:15but we also want people to trust those vaccine
54:17so increasing transparency in whatever way we can is great.
54:20And then finally, the final point is that a lot of these
54:23issues that I've talked about,
54:24how do we do interim monitoring, right?
54:26What's the right end point to be studying?
54:28What's the right S demand, right?
54:30These are really hard decisions
54:32and there are no right answers.
54:34And so one of the things that's been a little bit
54:37disconcerting or disheartening to me
54:40is the extent to which in the pandemic era,
54:43academic debates have been made very much public
54:46and I'm not against academic debates.
54:49It's just that most individuals aren't used to seeing them.
54:52And so what I'm worried is happening is that people
54:55see high profile academics debating these challenging
55:00problems where there's no real right answer.
55:02And they're saying, well, these guys don't know
55:03what they're talking about.
55:05So I think as academics and public health professionals
55:08in this pandemic, one thing that we can do
55:10is just to be very careful in how we're presenting,
55:12you know, the science that we're doing
55:15and acknowledge when there's not a right answer,
55:17that you're presenting your opinion.
55:19And that there is some validity, right?
55:21That this is very gray, unfortunately,
55:23that there's nothing black and white here.
55:25So maybe that's a controversial statement to end on,
55:28but I'll end there and then thanks again to Fan
55:30for giving me the opportunity to talk
55:32and I'm happy to take questions as there's time.
55:35I don't have anything scheduled after this,
55:36so I can stay a few minutes over as would be helpful.
55:39So thanks again.
55:41- [Fan] Thank you David for this very nice talk.
55:44I think we do have three to four minutes for questions
55:47from the audience, if there's any.
55:54- [Woman] Hi David, I have a question
55:55'cause right now for COVID situation and because of the time
56:00and the faster progress of the disease
56:04and it's a hard to keep the standard method,
56:08but do you have other proofed vaccine for other disease
56:14and have a quick trial have a similar way as COVID
56:19and apply the method you're using right now
56:24and we have standard results already
56:27and then compare to see how good the current method is.
56:32So that's my question.
56:35- [David] Yeah it's an interesting question.
56:37So let me try to restate, so you're saying,
56:39are there any lessons from vaccine development
56:42that we can try to draw from here
56:44to evaluate our methodology, whether it work?
56:48- [Woman] Yes, from other vaccines.
56:52- [David] So I guess what I would say is that at this stage,
56:55in phase three vaccines, these phase three trials
56:58look completely normal.
57:00So I would say the process of getting to the phase three
57:03looked very different and much more accelerated
57:05in terms of kind of squashing together
57:07phase one and phase two in terms of the manufacturing,
57:11but in terms of what's happening in a phase three trial,
57:13this is probably the phase three trial
57:15that would be done outside of the setting of a pandemic.
57:18Maybe the interim analysis would be a little bit
57:20less aggressive for some of these companies, but really,
57:23I think the approaches that the companies are taking
57:27would be fairly standard even in any other vaccine context.
57:35- -Woman] Yeah. I mean, even though
57:37for the established vaccine,
57:41there could be some field trial
57:43and that they also went through a phase three,
57:47but you can do the similar thing to enhance,
57:51to see whether it is possible to pass the current protocol
57:57and become some sort of false positive.
58:02- [David] Yeah and, you know, I think speaking,
58:08I mean, speaking of failed vaccines,
58:09as someone who works in HIV vaccines,
58:11we're very familiar with failure and learning from that.
58:15So again, I think the people who are running these trials
58:17are sort of the right people in terms of looking out
58:20for these false positive signals and so forth.
58:24- [Woman] Thank you.
58:27- [Fan] So I think we are just about the time
58:30and I'm sure that David is happy
58:32to take your questions afterwards by email.
58:35So I'll thank David more time.
58:37Again, thank you for sharing with us
58:39and we'll see everyone again next week.
58:43- [David] Thanks everybody.