YSPH Biostatistics Seminar: “A Sequential Basket Trial Design Based on Multi-Source Exchangeability with Predictive Probability Monitoring”

November 17, 2022

Information

Dr. Alex Kaizer, Assistant Professor, Center for Innovative Design and Analysis (CIDA) Department of Biostatistics and Informatics, University of Colorado Anschutz Medical Campus

November 15, 2022

ID9130

To CiteDCA Citation Guide

00:00<v Wayne>We introduce Dr. Alex Kaizer.</v>
00:04Dr. Kaizer is an assistant professor
00:07in the Department of Biostatistics and Informatics,
00:10and he's a faculty member
00:12in the Center for Innovative Design and Analysis
00:14at the University of Colorado Medical Campus.
00:19He's passionate about translational research
00:21and the development of normal
00:24and at clinical trial designs
00:25that the more efficient that they
00:27and factually utilize available resources
00:31including past trails and past studies.
00:34And Dr. Kaizer strives to translate
00:37(indistinct) topics into understandable material
00:41that is more than just a mask
00:42and something we can appropriate
00:44and utilize in our daily lives and research.
00:46Now let's welcome Dr. Kaizer.
00:53<v Alex>Thank you Wayne.</v>
00:54So apologies for my own technical difficulties today,
00:57but I'm going to be presenting on
00:59this idea of a sequential basket trial design
01:02based on multi-source exchangeability
01:04with predictive probability monitoring.
01:06And that is admittedly quite the mouthful
01:09and I'm hoping throughout this presentation
01:11to break down each of these concepts
01:13and ideas building upon them sort of until we have this
01:17cumulative effect that represents this title today.
01:21Before jumping into everything though,
01:23I do wanna make a few acknowledgements.
01:25This paper was actually published
01:26just at the end of this past summer in PLOS ONE,
01:28and so if you're interested in more of the technical details
01:31or additional simulation examples
01:33and things beyond what I present today,
01:35I include this paper here
01:36and we'll also have it up again at the very end of my talk
01:39just for reference.
01:40Also acknowledgement to Dr. Nan Chen
01:42who helped with some of the initial coding
01:44of some of these methods and approaches.
01:50So to set the context for my seminar today,
01:52I want to think here about
01:54this move towards precision medicine generally,
01:56but especially in the context of oncology.
01:59And so within oncology, like many other disciplines,
02:02when we design research studies,
02:03we often design these for a particular,
02:05what we might call a histology
02:07or an indication or a disease.
02:09So for example, we might say,
02:11"Well, I have a treatment or intervention
02:13which I hope or think will work in lung cancer,
02:15therefore I'm going to design
02:18and enroll in the study for lung cancer."
02:21Now this represents a very standard way
02:24that we do clinical trial design where we try to
02:26really rigorously define
02:27and limitedly define what our scope is.
02:31Now within oncology,
02:32we've had some exciting scientific developments
02:34over the past few decades.
02:36So now instead of seeing cancer as just
02:38based on the site like you have a lung cancer
02:40or a prostate cancer,
02:42we actually have identified that we can partition cancers
02:45into many small molecular subtypes.
02:48And further, we've actually been able to
02:49leverage this information by being able to say that
02:52what we thought of as a holistic lung cancer
02:54isn't just one type of disease,
02:57we can actually develop therapies that we hope to
02:59target some of these differences in genetic alterations.
03:02And this really gets to that idea of precision medicine that
03:05instead of throwing a treatment at someone
03:07where we think it should work
03:08or it has worked in some people on average,
03:10hopefully we can really target the intervention
03:13based off of some signal
03:14or some indication like a biomarker or a genotype
03:17that we actually hope could respond more ideally
03:20to that intervention.
03:22Now what's really interesting about this as well
03:26is that there could be a potential for heterogeneity
03:28in this treatment benefit by indication.
03:30And what I mean by that is once we've identified
03:33that there's these different genetic alterations,
03:35we've actually discovered that these alterations
03:37aren't necessarily unique to one site of cancer.
03:41For example, we may identify a genetic alteration
03:43in the lung that also is present in the prostate, liver,
03:46and kidney in some of those types of cancer.
03:49Now the challenge here though is that
03:50even though we have the same driver hypothetically
03:53based on our clinical or scientific hypothesis
03:56of that potential benefit for a treatment we've designed
03:59to address it,
04:00there's still may be important differences
04:01that we don't know about
04:02or have yet to account for based off of each site.
04:05So what may have worked actually really well in the lung
04:07for one given mutation,
04:09even for that same mutation, let's say present in the liver,
04:12may not work as well.
04:13And that's that idea of heterogeneity and treatment benefit.
04:16That we can have different levels of response
04:18across different sites or groups of individuals.
04:23Now the cool thing I think here
04:24from the statistical perspective is that the scientific
04:27and clinical advancements
04:28have also led to the revolution and statistical
04:31and clinical design challenges and approaches.
04:34And of course that's the sweet spot that I work at.
04:36I know many of you
04:37and especially students are training
04:39and studying to work in this area
04:40to collaborate with scientific and clinical researchers
04:43and leaders to translate those results
04:45in statistically meaningful ways
04:47and to potentially design trials or studies
04:49that really target these questions and hypotheses.
04:53Now specifically in this talk today,
04:56I'm going to focus on this idea of
04:58a master protocol design or evolution.
05:01And these provide a flexible approach
05:02to the design of trials with multiple indications,
05:05but they do have their own unique challenges
05:06that I'm gonna highlight a few of here in a second.
05:09But there are a variety of master protocols out there
05:11in case you've heard some of these buzzwords.
05:13I'll be focusing on basket trials today,
05:16but you may have also heard of things like umbrella trials
05:18or even more generally platform trial designs.
05:24And so one example of what this looks like here is
05:26this is a graphic from a paper in the
05:28New England Journal by Dr. Woodcock and LaVange,
05:31Dr. Woodcock being a clinician,
05:32and Dr. Lisa LaVange being a past president of
05:35The American Statistical Association,
05:37where they actually tried to put to rest some of the
05:39confusion surrounding some of these design types
05:42because it turns out,
05:43up until 2017 when we discussed these designs
05:46across even statistical communities
05:48and with clinical researchers,
05:50we tend to use these terms fairly interchangeably
05:53even though we are really getting at
05:55very different concepts.
05:57So for example,
05:58in the top here we have this idea of an umbrella trial
06:02and this is really the context of a single disease
06:04like lung cancer,
06:05but we actually then will screen for
06:07those genetic alterations
06:08and have different therapies that we're trying to
06:10target a different biomarker or genetic alteration for.
06:14This contrasts to what we're focusing on today below
06:17of a basket trial,
06:18we actually have different diseases or indications,
06:20but they share a common target or genetic alteration
06:23which we wish to target.
06:25And in this sense we can think of it potentially
06:26as them sharing a basket
06:28or sharing a sort of that commonality there.
06:32Now, this is a fairly broad general idea of these designs.
06:35And so I think for the sake of
06:37what we're gonna talk about today
06:38and some of the statistical considerations
06:40that can be helpful to do a bit of a
06:42oversimplification of what a design might look like here.
06:46And so on the slide that I've presented,
06:48I have this kind of naive graphic of actual baskets
06:53and we're going to assume that in each column
06:54we have a different indication or site of cancer
06:56that has that common genetic alteration.
06:59So for example, basket one may represent the lung,
07:02basket two may represent the liver and so on.
07:05Now when we're in the case of designing
07:07or the design stage of a study,
07:09we tend to make oversimplifying assumptions
07:11to address these potential calculations for
07:13power, sample size,
07:15and quantities that we're usually interested in
07:17for study design.
07:19So here on this graph,
07:20we are gonna make a assumption that
07:22there's only two possible responses in this planning stage.
07:25One is that the baskets have no response or a null basket,
07:29that's the blue colored solid baskets on the screen.
07:32The other case would be a alternative response
07:35where there is some hopeful benefit to the treatment
07:37and those are the open orange colored baskets
07:41we see on the screen here.
07:43Now, one of the challenges I think with basket trial design
07:46that can be overlooked sometime,
07:48even in this design stage,
07:49is that for a standard two arm trial,
07:51we do have to make this assumption of,
07:52what is our null hypothesis or response?
07:55What's our alternative hypothesis or response?
07:57We really only have to do that for one configuration
08:00or combination because we have two arms.
08:03In the case of a single arm basket trial here,
08:05we actually see that
08:06just by having five baskets in a study
08:08and many actual trials that are implemented at
08:10far more baskets,
08:12we actually see a range of just six possible
08:14binary combinations of the basket works
08:17or it doesn't work,
08:18ranging from at the extremes a global null
08:21where unfortunately the treatment does not work
08:23in any basket down to the sort of dream scenario
08:26where the basket is actually,
08:28or the drug actually works across all baskets.
08:31There is this homogenous actually response
08:34in a positive direction for the sort of clinical outcome.
08:39More realistically,
08:40we actually will probably encounter
08:42something that we see falls in the middle here,
08:44scenarios two through five,
08:46where there's some mixture of baskets
08:48that actually do show a response
08:49and some that for whatever reason we might not know yet,
08:51it just doesn't appear to have any effect
08:53and is a null response.
08:55So this can make it challenging
08:56for some of the considerations of
08:58what analysis strategy you plan to use in practice.
09:03And so to just, at a high level,
09:05highlight some of these challenges
09:07before we jump into the methods for today's talk.
09:11In practice, each of these baskets within trial
09:13often have what we call a small
09:14and or small sample size for each of those indications.
09:18It turns out
09:18once we actually have this idea of precision medicine
09:20and we can be fairly precise
09:22for who counts for a trial,
09:23we actually have a much smaller potential sample
09:26or population to enroll.
09:27This means that even though we might have a treatment
09:29that works really well,
09:30it can be challenging to find individuals who qualify
09:33or are eligible to enroll
09:34or they may have competing trials or demands
09:36for other studies or care to consider.
09:41As I've also alluded to earlier the challenge,
09:43we also have this potential for indication
09:44or subgroup heterogeneity and that may be likely.
09:47In other words,
09:48we might not expect the same response
09:50across all those baskets.
09:50And that gets back to the previous graphic
09:52on that last slide
09:53where we might have something like two null baskets
09:56and three alternative baskets.
09:57And that can make it really challenging in the presence
09:59of a small n to determine how do we
10:02appropriately analyze that data
10:03so we capture the potentially applications baskets
10:06and can move those forward so patients benefit
10:09while not carrying forward null baskets
10:11where there is no response for those patients.
10:16Statistically speaking,
10:17we also have these ideas of operating characteristics
10:19and in the context of a trial,
10:21what we mean by that is things like power
10:22and type one error
10:24and I just have additional considerations with respect to
10:26how do we summarize these?
10:28Do we summarize them within each basket or each column
10:31on that graphic on the previous slide,
10:32essentially treating it as a bunch of
10:34standalone independent one arm trials
10:36just under one overall study design or idea?
10:40Or do we try to account for the fact that we have
10:42five baskets enrolling like on the graphic before
10:45and we might wanna consider something like a
10:47family wise type one error rate
10:48where any false positive would be a negative outcome
10:52if we're trying to correctly predict
10:53or identify associations?
10:57Now the focus of today's talk,
10:58and I could talk about these other points
11:00till the cows come home,
11:02but I'm gonna focus today on
11:04depending on that research stage we're at,
11:06if it's a phase one, two or three trial,
11:08we may wish to terminate early for some reason like
11:10efficacy or futility.
11:12And specifically for time today,
11:13I'm gonna focus on the idea of stopping for futility
11:16where we don't wanna keep enrolling baskets
11:18that are poorly performing both for ethical reasons.
11:20In other words,
11:21patients may benefit from other trials or treatments
11:24that are out there and we don't wanna subject them to
11:26treatments that have no benefit.
11:28But also from a resource consideration perspective.
11:31You can imagine that running a study or trial is expensive
11:34and can be complicated.
11:36And especially if we're doing something like a basket trial
11:38where we're having to enroll across multiple baskets,
11:40it may be ideal to be able to drop baskets early on
11:43that don't show promise
11:44so we can reallocate those resources to
11:46either different studies, research projects,
11:49or trials that we're trying to implement or run.
11:55So the motivation for today's talk
11:57building off of these ideas is that
11:58I want to demonstrate that a design that's very popular
12:01called Simon's two-stage design is
12:04generally speaking suboptimal
12:05compared to the multitude of alternative methods
12:08and designs that are out there.
12:10And then this is especially true in our context of
12:12a basket trial where within the single study
12:15we actually are simultaneously enrolling
12:17multiple one arm trials in our case today.
12:20Then the second point I'd like to highlight is
12:22we can identify when methods for sharing information
12:24across baskets could be beneficial to further improve
12:27the efficiency of our clinical trials.
12:31And so to highlight this,
12:32I wanna first just build us through
12:34and sort of illustrate or introduce these designs
12:36and the general concepts behind them
12:37because I know if you don't work in this space
12:40it may be sort of just ideas vaguely.
12:43So I wanna start with the Simon two-stage design,
12:45that comparator that people are commonly using.
12:48So Richard Simon, and this is back in 1989,
12:51introduced what he called optimal two-stage designs
12:54for phase two clinical trials.
12:56And this was specifically in the context
12:57that we're focusing on today for a one sample trial
12:59to evaluate the success of a binary outcome.
13:02So for oncology we might think of this as a yes no outcome
13:05for is there a reduction in tumor size
13:07or a survival to some predefined time point.
13:13Now specifically what Dr. Simon was motivated by
13:16was the stage-two trials
13:18as it says in the title of his paper,
13:20and just to kind of
13:22give a common lay of the land for everyone,
13:23the purpose generally speaking of a phase two trial
13:26is to identify if the intervention
13:28warrants further development
13:30while collecting additional safety data.
13:32Generally speaking,
13:33we will have already completed what we call
13:35a phase one trial where we collect preliminary safety data
13:38to make sure that the drug is not toxic
13:40or at least has expected side effects
13:44that we are willing to tolerate for that
13:45potential gain in efficacy.
13:48And then in phase two here we're actually trying to say,
13:50"You know, is there some benefit?
13:51Is it worth potentially moving this drug
13:53on either for approval
13:54or some larger confirmatory study
13:57to identify if it truly works or doesn't?"
14:01Now the motivation for Dr. Simon is that
14:03we would like to terminate studies earlier,
14:05as I mentioned before,
14:06for both ethical and resource considerations
14:08that they appear futile.
14:09In other words, it's not a great use of our resources
14:11and we should try in some
14:12rigorous statistical way to address this.
14:17If you do go back and look at Simon's 1989 paper
14:20or you just Google this
14:21and there's various calculators that people have
14:22put out there,
14:23there are two flavors of this design that exist
14:26from this original paper.
14:27One is an optimal
14:28and one is called a minimax design.
14:31Within clinical trials,
14:32once we introduce this idea of stopping early potentially
14:36or have the chance to stop early based on our data,
14:39we now have this idea that there's this expected sample size
14:42because we could enroll the entire sample size
14:44that we planned for or we could potentially stop early.
14:47And since we could stop early or go the whole way
14:49and we don't know what our choice will be
14:51until we actually collect the data and do the study,
14:53we now have sample size of the random variable,
14:56something that we can calculate an expectation
14:58or an average for.
14:59And so Simon's optimal design tries to
15:01minimize what that average sample size might be in theory.
15:06In contrast, the minimax design
15:08tries to minimize whatever that largest sample size would be
15:11if we didn't stop early.
15:13So if we kept enrolling
15:14and we never stopped at any of our interim looks,
15:16how much data would we need to collect
15:18until we choose a design that minimizes that
15:20at the expense of potentially stopping early?
15:25I think this is most helpful to see the
15:27sort of elegance of this design
15:29and why it's I think so popular
15:30by just introducing example
15:31that will also motivate our simulations
15:33here that we're gonna talk about in a minute.
15:36We're gonna consider a study where
15:37the null response rate is 10%.
15:40And we're going to consider a target
15:42for an alternative response rate of 30%.
15:44So this isn't a situation where
15:45we're looking for necessarily a curative drug,
15:48but something that does show what we think of
15:49as a clinically meaningful benefit from 10 to 30%,
15:52let's say survival or tumor response.
15:55Now if we have these two parameters
15:57and we wanna do a Simon two-stage minimax design
16:00to minimize that maximum possible sample size
16:03we would enroll,
16:04we would have to also define
16:06the type one error rate or alpha
16:08that cancels a false positive.
16:10Here we're going to set 10% for this phase two design
16:12and we also wish to target a 90% power
16:15to detect that treatment of 30% if it truly exists.
16:20So we put all of this into our calculator
16:22to Simon's framework and we turn that statistical crank.
16:25What we see is that
16:26it gives us this approach where in stage one
16:29we would enroll 16 participants
16:31and we would terminate the trial or this study arm
16:34for futility if one or fewer responses are observed.
16:37Now if we observe two or more responses,
16:41we would continue enrollment
16:43to the overall maximum sample size that we plan for
16:45of 25 in the second stage.
16:48And at this point if four or fewer responses are observed,
16:51no further investigation is warranted
16:53or we can think of this as a situation where
16:55our P value would be larger than our defined alpha 0.1.
17:00Now, the nice thing here is that it is quite simple.
17:03In fact, after we trim that statistical crank
17:05and we have this decision rule,
17:06you in theory don't even need a statistician
17:08because you can count the number of responses
17:10for your binary outcome on your hand
17:12and determine should I stop early, should I continue?
17:15And if I continue,
17:16do I have some benefit potentially
17:18that says it's worth either doing a future study
17:21or I did a statistical test,
17:23would find that the P value meets my threshold
17:25I set for significance.
17:29Now, of course,
17:31it wouldn't be a great talk if I stopped there and said,
17:33"You know, this is everything.
17:34It's perfect. There's nothing to change."
17:36There are some potential limitations
17:37and of course some solutions I think
17:39that we could address in this talk.
17:42The first thing to note is that
17:43this is extremely restrictive in when it could terminate
17:46and it may continue to the maximum sample size
17:48even if a null effect is present.
17:50And we're gonna see this come to fruition
17:52in the simulation studies,
17:54but it's worth noting here it only looks once.
17:55It's a two stage design.
17:58And depending on the criteria you plug in,
18:00it might not look for quite some time.
18:0216 out of 25 total participants enrolled
18:05is still a pretty large sample size
18:07relative to where we expect to be.
18:11One solution that we could look at
18:12and that I'm going to propose today
18:14is that we could use Bayesian methods instead
18:16for more frequent interim monitoring.
18:18And this could use quantities that we think of
18:20as the posterior or the predictive probabilities
18:23of our data.
18:26Another limitation that we wish to address as well is that
18:28in designs like a basket trial
18:30that have multiple indications
18:31or multiple arms that have the same entry criteria,
18:35Simon's two-stage design is going to
18:36fail to take advantage of the potential
18:38what we call exchange ability across baskets.
18:41In other words, if baskets appear to have the same response,
18:45whether it's let's say that null
18:46or that alternative response,
18:48it would be great if we could
18:49informatively pull them together into meta subgroups
18:52so we can increase the sample size
18:54and start to address that challenge of the small n
18:56that I mentioned earlier for these basket trial designs.
18:59And specifically today we're going to examine the use of
19:02what we call multi-source exchangeability models
19:05to share information across baskets when appropriate.
19:08And I'll walk through a very high level sort of
19:10conceptual idea of what these models
19:12and how they work and what they look like.
19:17Before we get into that though,
19:18I wanna just briefly mention the idea of posterior
19:20and predictive probabilities
19:22and give some definitions here
19:23so we can conceptually envision what we mean
19:25and especially if you haven't had the chance
19:27to work with a lot of patient methods,
19:29this can help give us an idea
19:31of some of the analogs to maybe a frequentist approach
19:33or what we're trying to do here
19:34that you may be familiar with.
19:36Now I will mention,
19:37I'm not the first person to propose looking at
19:39Bayesian interim stopping rules.
19:41I have a couple citations here by Dmitrienko
19:43and Wang and Saville et all
19:45and they do a lot of extensive work in addition to
19:47hundreds of other papers considering
19:49Bayesian interim monitoring.
19:51But specifically to motivate this
19:53we have these two concepts that commonly come up
19:56in Bayesian analysis,
19:57a posterior probability or a predictive probability.
20:01The posterior probability
20:03is very much analogous to kinda like a P value
20:05in a frequent significance.
20:06It says, "Based on the posterior distribution
20:09we arrive at through a Bayesian analysis,
20:11we're gonna calculate the probability
20:13that our proportion exceeds the null response rate
20:15we wish to beat."
20:16So in our case, we're basically saying,
20:18"What's the probability based on our data
20:20and a prior we've given that the response is 10% or higher."
20:25So this covers a lot of ground
20:26'cause anything you know from 10.1 up to 100%
20:29would meet this criteria being better than 10%.
20:32But it does quantify,
20:34based on the evidence we've observed so far,
20:37how the data suggests the
20:40benefit may be with respect to that null.
20:42So in the case of let's say
20:44an interim look for futility at the data, we could say,
20:47if we just use Simon's two-stage design as our motivating
20:51ground to consider, we might say,
20:53"Okay, we have 16 people so far,
20:55what's the probability based on these 16 people
20:58that I could actually say
20:59there's no chance or limited chance
21:00I'm going to detect something in the trial here
21:03based on the data I've seen so far?"
21:05Now the challenge here is that
21:07it is based on off the data we've seen so far
21:09and it doesn't take into account the fact that we still have
21:12another nine potential participants to enroll
21:15to get to that maximum sample size of 25.
21:18That's where this idea of what we call a
21:20predictive probability comes in.
21:22We're considering our accumulated data
21:24and the priors we've specified in our Bayesian context,
21:27it's the probability that we will have observed
21:30a significant result if we've met
21:32and enrolled up to our maximum sample size.
21:36In other words, I think it's a very natural place to be
21:38for interim monitoring
21:39because it says based on the data I've seen so far,
21:41i.e the posterior probability,
21:43if I use that to help identify what are likely futures
21:46to observe or likely sample sizes
21:48I will continue enrolling to get to that maximum of 25,
21:51what's the probability at the end of the day
21:53when I do hit that sample size of 25,
21:56I will have a significant conclusion?
21:58And if it's a really low predictive probability,
22:00if I say there's only a 5% chance
22:02of you actually declaring significance if you
22:04keep enrolling participants,
22:06that can be really informative both statistically
22:08and for clinical partners to say
22:10it doesn't seem very likely that we're gonna hit our target.
22:13That being said,
22:15a lot of people are very happy to continue trials going
22:17with low chances or low probability
22:19because you're saying there's still a chance
22:21I may detect something that could be
22:23significant enough worth.
22:25So we'll see that across a range of these thresholds,
22:28the performance of these models may change.
22:32Now this brings us to a brief recap
22:34of sort of our motivation.
22:35I just spent a few minutes
22:37introducing that popular Simon two-stage design,
22:39the idea behind it,
22:40what it might look like in practice,
22:42as well as some alternatives with the Bayesian flare.
22:45The next part I wanna briefly address is that
22:47we can also now look at this idea
22:50of sharing information across baskets
22:52to further improve that trial efficiency
22:54'cause so far both Simon's design
22:56and the just using a posterior predictive probability
22:59for an interim monitoring will still treat each basket
23:02as its own little one arm trial.
23:07Now specifically today I'm gonna focus on this idea
23:10we call multi-source exchangeability models or MEMs.
23:13This is a general Bayesian framework
23:15to enable the incorporation of independent sources
23:18of supplemental information
23:20and its original work that I developed
23:22during my dissertation at the University of Minnesota.
23:25In this case,
23:26the amount of borrowing is determined by
23:27the exchange ability of our data,
23:29which in our context is really,
23:31how equivalent are the response rates?
23:33If two baskets have the exact same response rate,
23:35we may think that there's a higher probability
23:38that the true underlying population
23:40we are trying to estimate are truly exchangeable.
23:42We wish to combine that data as much as we possibly can.
23:46First is if again we see something that is like a
23:4810% response rate for one basket
23:50and a 30% response rate for another basket,
23:53we likely don't want to combine that data because
23:55those are not very equivalent response rates.
23:57In fact, we seem to have identified
23:59two different subgroups
24:00and performances in those two baskets.
24:04One of the advantages of MEMs relative to
24:07a host of other statistical methods that are out there
24:09that include things like power priors, commensurate priors,
24:12meta analytic priors, and so forth,
24:15is that we've been able to demonstrate that
24:16in their most basic iteration without
24:18any extra bells or whistles,
24:20MEMs are able to actually account for this heterogeneity
24:23across different potential response rates
24:26and appropriately down weight non-changeable sources.
24:29Whereas we show through simulation
24:30and earlier work some of these other methods without
24:33newer advancements to them
24:35actually either naively pull everything together
24:38even if there's non-changeable groups
24:41or they're afraid of the sort of presence of
24:43non-change ability and if anything seems amiss,
24:45they quickly go to an independence analysis
24:48that doesn't leverage this potential sharing
24:51of information across meta subgroups that are exchangeable.
24:56Now again, I don't wanna get too much into the weeds
24:59of the math behind the MEMs,
25:00but I will have a few formulas in a couple slides
25:02but I do think it's helpful to
25:03conceptualize it with graphics.
25:05And so here I just want to illustrate a very simplified case
25:08where we're gonna assume that we have a three basket trial
25:11and for the sake of doing an analysis with MEMs,
25:14I think it's helpful to also think of it as
25:16we're looking at the perspective of the analysis
25:18from one particular basket.
25:20So here on this slide here we see that we have this
25:24theta P circle in the middle
25:25and that's the parameter or parameters of interest
25:28we wish to estimate.
25:29In our case, that would be that
25:31binary outcome in each basket.
25:34Now, for this graphic we're using each of these circles here
25:37to represent a different data source.
25:40We're gonna say Y sub P is that primary basket
25:42that we're interested in or the perspective
25:44we're looking at for this example
25:46and Y sub one and Y sub two
25:48are two of the other baskets enrolled within the trial.
25:51Now a standard analysis
25:53without any information sharing across baskets
25:55would only have a data pooled from the observed data.
26:00I mean this is sort of the unexciting
26:01or unsurprising analysis
26:03where we basically are analyzing the data we have
26:05for the one basket that actually represents that group.
26:10However, we could imagine if we wish
26:11to pool together data from these other sources,
26:14we have different ways we could add arrows to this figure
26:17to represent different combinations of these groups.
26:21And this brings us to
26:22that multi-source exchangeability framework.
26:25So we see here on this slide,
26:26I now of a graphic showing four different combinations
26:29of exchangeability when we have these two other baskets
26:32that compare to our one basket of interest right now.
26:36And from top left to the bottom left
26:38in sort of a clockwise fashion,
26:39we see that making different assumptions from
26:42that standard analysis with no borrowing
26:44in the top right here where I'm drawing that arrow.
26:46So it is possible that
26:48none of our data sources are exchangeable
26:49and we should be doing an analysis that
26:51doesn't share information.
26:53On the right hand side that we might envision that
26:55well maybe the first basket or Y1 is exchangeable.
26:58So we wanna pull that with Y2 or excuse me with Yp,
27:01but Y2 is not.
27:03In the bottom right, this capital omega two,
27:05we actually assume that Y2 is exchangeable
27:07but Y1 is not.
27:09And in the bottom left we assume in this case
27:10that all the data is exchangeable
27:12and we should just pool it all together.
27:15So at this stage we've actually
27:17proposed all the configurations we can pairwise
27:20of combining these different data sources with Y sub P.
27:23And we know that these are fitting four now different models
27:26based off of the data
27:27because for example in the top left, that standard analysis,
27:30there is no extra information from those other baskets
27:33versus like in the bottom left,
27:35we basically have combined everything
27:36and we think there's some common effect.
27:39Now this leads to two challenges on its own
27:40if we just stopped here with the framework.
27:43One would be that we'd have this idea of maybe
27:44cherry picking or trying to pick whichever combination
27:47best suits your prior hypotheses clinically.
27:50And so that would be a big no-go.
27:51We don't like cherry picking
27:52or fishing for things like P values
27:54or significance in our statistical analyses.
27:57The other challenge also is that
27:59all of these configurations are just assumptions
28:01of how we could combine data
28:03but we know underlying everything in the population is that
28:05true assumption of exchange ability of
28:07are these baskets or groups truly combinable or not?
28:11And we're just approximating that with our sample.
28:13And so right now if we have four separate models
28:15and potentially four separate conclusions,
28:18we need some way of combining these models
28:20to make inference.
28:21And in this case we propose
28:23leveraging a Bayesian model averaging framework
28:26where we calculate in this case
28:28and in our formulas here,
28:29the queues represent a posterior distribution
28:32where I've drawn this little arrow
28:33and I'm underlining right now,
28:35that reflects each square's configuration of
28:39exchange ability for our estimates.
28:41And through this process
28:42we estimate these lower case omega model weights
28:45that tries to estimate the appropriateness
28:47of exchangeability with the ultimate goal of
28:50having a average posterior that we can use
28:53for statistical inference
28:54to draw a conclusion about the potential efficacy
28:57or lack thereof of a treatment.
29:02Now very briefly,
29:03because this is a Bayesian model averaging framework,
29:06just one of the few formulas I have in the presentation,
29:08we just see over here that we have
29:10the way we calculate these posterior model weights
29:13as the prior on each model
29:15multiplied by an integrated marginal likelihood.
29:18Essentially, we can think of that as saying
29:20based off of that square we saw on the previous slide
29:22and combining those different data sources,
29:25what is that estimate of the effect
29:26with those different combinations?
29:29One unique thing about the MEM framework
29:31that differs from Bayesian model averaging though is that
29:34we actually specify priors with respect to these sources.
29:37And in the case of this example
29:39with only two supplemental like sources for our graphic,
29:42it's not a great cost savings,
29:45but we can imagine that if we have more and more sources,
29:47there's actually two to the P if P's the number of sources,
29:50combinations of exchange ability
29:52that we have to consider and model.
29:54And that quickly can become overwhelming if we have
29:56multiple sources that we have to define
29:58for each one of those squares,
29:59what's my prior that each combination of exchangeability
30:02is potentially true.
30:04Versus if we define it with respect to the source,
30:06we now go from two to the P priors to just P priors
30:09we have to specify for exchangeability.
30:14A few more notes about this idea here
30:17and just really zooming in on
30:19what we're gonna focus on for today's presentation.
30:21We have developed both fully
30:23and empirically Bayesian prior approaches here,
30:25fully Bayesian meaning that it is defined a priori
30:29and is agnostic to the data you've collected,
30:31empirically Bayesian meaning
30:32we leverage the data we've collected
30:34to help inform that prior for what we've observed.
30:38Specifically there is a what we call a
30:40non constrained, or naive,
30:42empirically based prior
30:43where we would look through all of those growths we had
30:45and we would say, "Whichever one of these
30:47maximizes the integrated marginal likelihood
30:49that's the correct configuration
30:51and we're gonna put all of our eggs into that basket."
30:53Or 100% of the probability there
30:55and that's the only model we use for analysis.
30:59We know, generally speaking,
31:00since we went to all the work to defining
31:02all of these different combinations of exchangeability
31:04and that it's based off of samples,
31:06potentially small samples,
31:07that this can be a very strong assumption.
31:10And so we can also modify this prior
31:12to what we call a constrained EB prior,
31:15where instead of just giving everyone of those model
31:18sources in that MEM that
31:20maximizes the likelihood 100% weight,
31:23we instead give it a weight of what we're calling just B.
31:25This is our hyper prior value here
31:28where if it's a value of zero or up to one,
31:31it'll control the amount of borrowing
31:32and allow other nested models of exchangeability
31:36to also be potentially considered for analysis.
31:39So for example,
31:40if we do set a value of one
31:42that actually replicates the non constrained EB prior
31:44and really aggressively borrows from one specific model.
31:48At the other extreme here, if we set a value of zero,
31:50we essentially recreate an independent analysis
31:53like assign a two stage design or just using those
31:55Bayesian methods for futility monitoring
31:57that doesn't share information.
31:59And then any value in between
32:00gives a little more granularity or control
32:03over the amount of borrowing.
32:06So with that background behind us,
32:09I'm gonna introduce the simulation stuff
32:11and then present results for a couple
32:13key operating characteristics for our trial.
32:16In this case, we're going to assume for our simulations
32:18that we have a basket trial
32:19with 10 different baskets or indications.
32:21So again, that's 10 different types of cancer
32:23that we have enrolled that all have
32:25the same genetic mutation that we think is targeted
32:28by the therapy of interest.
32:31Like we had before,
32:32we're going to assume a null response P knot of 0.1 or 10%.
32:36And an alternative response rate of 30% or P1 here.
32:41We are gonna compare then three different designs
32:43that we just spent some time introducing and outlining.
32:46The first is a Simon minimax two-stage design
32:49using that exact set up that we had before
32:52where we will enroll 16 people,
32:54determine if we have one or fewer observations of success.
32:56If so, stop the trial.
32:58If not, continue on.
33:00In the second case,
33:01we're going to implement a Bayesian design
33:03that uses predictive probability monitoring
33:05but we don't use any information sharing
33:07just to illustrate that we can at least
33:09potentially improve upon the frequency
33:12in use of a interim monitoring above a single look
33:14from the Simon minimax design.
33:17And then the third design
33:18will add another layer of complexity
33:20where we will try to share information across baskets
33:23that have what we estimate to be exchangeable subgroups.
33:28One thing to note here is that
33:29we are setting this hyper parameter value B at 0.1.
33:32This is a fairly conservative value
33:34and admittedly for this design
33:36we actually did not calibrate specifically
33:39for the amount of borrowing to be 0.1.
33:40This is actually based off of
33:41some other prior work we've done
33:43and published on basket trials that just showed that
33:45in the case of an empirically Bayesian prior for MEMs,
33:49this actually allows information sharing
33:51in cases where there's a high degree of exchangeability
33:53and low heterogeneity
33:55and down leap it in cases where we might be
33:57a little more uncertain,
33:58so it's a little more conservative
33:59but we'll see in the simulation results
34:01there are some potential benefits.
34:05For each of the scenarios we're gonna look at today,
34:08we will generate a thousand trials
34:10with a maximum sample size of 25 per basket.
34:14We're gonna look at two cases,
34:16there's a few other in the paper
34:17but we're gonna focus on first the global scenario
34:20where all the baskets are either null
34:21or all 10 baskets have some meaningful effect.
34:24And this is the setting where
34:25information sharing methods like meds
34:27really should outperform anything else
34:29because everything is truly exchangeable
34:31and everything could naively be pooled together
34:34because we're simulating them to have the same response.
34:37We'll then look at what happens if we actually have
34:39a mixed scenario,
34:40which I think is actually more indicative
34:42of what's happened in practice
34:43with some of the published basket trials
34:45and clinically what we've seen from applications
34:47of these types of designs.
34:49Specifically here, we're gonna look at the case where
34:52there are eight null baskets and two alternative baskets.
34:57A few other points just to highlight here.
34:59We're going to assume a beta 0.5 0.5 prior
35:02for our Bayesian models.
35:04This essentially for a binary outcome can be thought of as
35:06adding half of a response
35:08and half of a lack of a response to our observed data.
35:12We're going to look at the most extreme dream Bayesian case
35:16of doing utility monitoring
35:18or any type of interim monitoring continually.
35:20So after every single participant's enrolled
35:22we will do a calculation
35:24and determine if we should stop the trial.
35:27We will then look at the effect of this choice
35:29across a range of predictive probability thresholds
35:34ranging from 0%,
35:35meaning we wouldn't stop early at all,
35:37up to 50% saying if there's anything less
35:39than a 50% chance I'll find success,
35:41I wanna stop that trial.
35:44And then finally it's worth noting
35:45we're actually also completely disregarding calibration
35:49for this interim monitoring.
35:51And so what we're gonna do is
35:52we're gonna calibrate our decision rules
35:54for the posterior probability at the end of the trial
35:57based off of a global scenario where
35:59we think it's ideal to share information
36:02and we're all not gonna account for the fact that
36:03we're doing interim looks at the data.
36:05Part of the question here was
36:07if we truly do all these assumptions
36:08and we do sort of the most naive thing,
36:11how badly do we actually do?
36:13Like is there enough reason to fear the results
36:16if we don't correctly calibrate for everything here?
36:21So I'm gonna paint some pictures here building from the
36:24simpler Simon design to our more complex Bayesian designs
36:27and then with information sharing
36:29just to illustrate three different properties.
36:31I'm gonna go fairly quickly
36:33'cause I know that you all have to vacate the classroom
36:35in about 10 minutes.
36:37So for the global scenario that we're looking at here,
36:41the like rate lines are going to represent
36:44the alternative basket scenario.
36:46So all, in this case, all 10 null baskets.
36:49Here we see we plan for 90% power
36:51Simon's design appropriately achieved
36:53that rejection rate of 90%.
36:55Likewise, the lines at the bottom here,
36:58these black lines,
36:59are going to represent the results of null baskets.
37:01Here are the global null scenario
37:03and we see that it achieves a 10% rejection rate.
37:06Now, this is a flat line here
37:08because again Simon's design is agnostic to things like
37:11the predictive probability.
37:13Now if we do frequent Bayesian monitoring,
37:16we see two interesting things here with these new lines.
37:19We see that at the top
37:21and the bottom, here I add these circles
37:23where the predictive probability threshold is 0%.
37:25This does represent the actual design
37:27that would correspond to the actual calibration we did
37:29without interim monitoring.
37:31And we see that it is possible with Bayesian approaches
37:34to achieve the same frequent operating characteristics
37:37that we would achieve with something like the Simon design.
37:40We can see though that if we want to do interim monitoring
37:43but we didn't calibrate
37:44or think of that in our calculations,
37:46we do see this trade off where we have our
37:49alternative baskets having a decreasing power
37:51or rejection rate as the aggressiveness of the
37:53predictive probability threshold increases.
37:56And likewise the type one error rate or the
37:59rejection rate of the marginal baskets also decreases.
38:02Now if we add information sharing to this design,
38:06we actually see some encouraging results
38:08in this global scenario.
38:09First, it's worth noting that in the case
38:11where we actually calibrated for,
38:12we actually see an increase in power from 90% to about 97%.
38:17And even when we actually have a
38:1910% predictive probability threshold for interim monitoring,
38:23we see that we actually still achieve 90% power
38:26with a corresponding reduction in that type one error rate.
38:30Of course, this is with the caveat that
38:32this is the ideal setting for sharing information
38:35because all of the baskets are truly exchangeable.
38:38Now the rejection rate correlates to something we call
38:41that expected sample size.
38:42What is the average sample size we might enroll
38:44for each basket of our 10 baskets in the trial?
38:47We see here that in the case of a null basket
38:50the Simon design is about 20.
38:53If we do interim monitoring with Bayesian approaches
38:56and no information sharing,
38:57obviously if we don't do any interim looks at the data,
38:59we have a 0% threshold,
39:01we're gonna have a sample size of 25 every single time.
39:05I think what's encouraging though is that
39:06by looking fairly aggressively we see that our sample size,
39:09even with a very marginal
39:11or low 5% threshold for futility monitoring,
39:14drops from 20 in the assignment design to about 15
39:18in the Bayesian design,
39:19the trade-off of course being because we didn't calibrate.
39:22We also see a reduction in the sample size
39:24for the alternative baskets.
39:28And if we add that layer of information sharing,
39:30we actually see that we do slightly better than
39:32the design without information sharing
39:34while attenuating at the top here the effect
39:37our solid gray line has for the alternative baskets.
39:42Now, briefly tying this together then to the stopping rate,
39:45which we can kind of infer from those past results,
39:47we do see that on average the Simon two-stage design
39:50for the null baskets stopping for futility
39:52is only taking place a little over 50% of the time
39:55in this simulation.
39:57The advantage here though is that it is
39:58very rarely stopping for the alternative baskets.
40:02In our Bayesian approaches,
40:03we see that there is an over 80%
40:06of these low thresholds probability of stopping
40:09if it's a null effect.
40:10And this is ideal because we have 10 baskets.
40:12And so these potential savings or effects
40:14can compound themselves across these multiple baskets.
40:18We then see that the design adding these solid lines
40:21for information sharing do very similarly
40:23where again the the consequence of not calibrating
40:26are attenuated in this circumstance.
40:30Now the thing to note here that
40:31everything I presented on these few graphics
40:34were with respect to the global scenario,
40:36that ideal scenario that I actually don't think
40:38is super realistic in practice.
40:41So we see here, if we do a mixed scenario where
40:44we now have calibrated for the global scenarios,
40:46we've miscalibrated with respect to that.
40:48We've also not calibrated for interim looks at the data.
40:51We can actually see that the results for
40:53the Simon two-stage in the Bayesian design
40:55without information sharing are very similar
40:58to what we saw before.
40:59That's because they don't share information.
41:00And so in this case with eight null baskets
41:02into alternative baskets,
41:04they have very similar responses.
41:06This contrasts of course with the MEM approach
41:09or the information sharing approach
41:10where we actually see now
41:12many of these results are actually overlapping
41:15for information sharing and no information sharing.
41:18What this tells us is that even though we miscalibrated
41:21up and down the design,
41:23we are actually able with this more conservative prior
41:26to down weight borrowing
41:27and effectuate similar results
41:30that at lower thresholds for utility monitoring for example
41:34at 5% can still show potential gains in efficiency relative
41:38to the Simon design that could likely further be improved
41:41with actual calibration.
41:44So just as a reminder,
41:45we demonstrated today
41:46and introduced the idea of Simon's two-stage design
41:48and some alternative methods to compete with them.
41:51And some just brief discussion and concluding points.
41:53There is no free lunch
41:55and this is true regardless of where we are in statistics
41:57that for example in our designs,
41:59besides the fact that we miscalibrated
42:01and made it a bit harder of a comparison for our methods,
42:04we did try to replicate
42:05what people might be doing in practice
42:07or the challenge of
42:07calibrating these designs into actuality.
42:10Simon's two-stage design does have a lot of benefits
42:13from it's ideal characteristics
42:15that are easy to implement,
42:17but it is limited in how often it may stop.
42:20Our Bayesian designs,
42:21with or without information sharing,
42:22can lead to reductions in the expected sample size
42:24in the null basket
42:25and further could be improved
42:27if we actually incorporate calibration,
42:29which we further explored
42:30in a statistical methods of medical research paper
42:33published in 2020.
42:35And so that I have some sources here
42:36and I thank you for your attention
42:37and welcome any questions or discussion at this point.
42:56<v Man>Thank you so much. Any questions from the room?</v>
43:12<v Student>Okay, so yeah, I have questions.</v>
43:14So in the example you just showed,
43:18all the like the task becomes so, can be achievable, right?
43:22So if the baskets,
43:25they are expected to have different benefits (indistinct),
43:28and say the 10 basket (indistinct)
43:33some other basket MEMs would allow a bigger benefit,
43:37how will the (indistinct)
43:45scenarios?
43:48<v Alex>Yeah, well, I think,</v>
43:49if I understood your question correctly
43:51and I misheard through the phone, let me know,
43:54but if we have different sample sizes for baskets,
43:56which actually really corresponds
43:58to what we've seen in practice for real basket trials
44:01where they have fairly
44:02wide range of sample sizes in each basket.
44:06I think what we would see,
44:07and let me see if I can pop back quickly to the
44:09mixed scenario results here just to illustrate some ideas.
44:13One of the concepts here that,
44:14so we did explicitly look at that to say like,
44:16"Well, what if one basket never gets beyond seven
44:18of the 25," let's say.
44:20But what we can infer is that
44:21if a basket stopped early for futility,
44:23it essentially has a smaller sample size to contribute
44:26to any analysis whether or not it was a
44:29falsely stopped basket that had a 30% effect
44:32or it was truly a null basket.
44:34And so we do see in this case that the method
44:36averaging over those ideas of differential sample sizes
44:39based off of soft baskets
44:41does seem to be borrowing,
44:43appropriately depending on the context.
44:45So like the mixed scenario results here suggests
44:47limited borrowing in the presence of that uncertainty
44:50from the global scenario
44:51because we didn't calibrate for anything else
44:53it does show more of a benefit of the stopping rate
44:56and other properties incorporating that data
44:58even in small sample sizes.
45:00And there's also been some other work
45:02and illustrations done by Dr. Emily Zebra
45:04at the Cleveland Clinic with who I work
45:06about some of the re-analysis of oncology trials
45:09that do show even in small basket sizes,
45:11we can move that significance evaluation
45:14into a more clinically meaningful realm.
45:26<v Wayne>Thanks, so do we have other questions?</v>
45:57Okay, so (indistinct) that's (indistinct).
46:01Okay, so since there are no questions let's stop here.
46:07(indistinct)
46:16<v Alex>Yeah. Thank you all.</v>