Learn-As-You-Go (LAGO) to Adapt the Intervention in an Ongoing Trial to Prevent Trial Failure

Transcript

• 00:00Okay.
• 00:06This meeting is being recorded.
• 00:07So if you make comments,
• 00:08they would be here forever.
• 00:11So
• 00:12they are
• 00:14they'll study among statisticians
• 00:17and they're increasingly also applied
• 00:19in in public health and
• 00:21medical
• 00:23science. So the goal of
• 00:26of an adaptive design is
• 00:28to be able to modify
• 00:29and sometimes also stop a
• 00:31clinical trial
• 00:33after
• 00:34not in unplanned
• 00:36yeah even planned but unplanned.
• 00:38So the rules the rules
• 00:40to
• 00:41carry out in a depth
• 00:42of variable trial,
• 00:44they have to be specified
• 00:46in advance. The FDA is
• 00:47very clear about that if
• 00:48you won't register anything of
• 00:50your medicine
• 00:51with the FDA, you have
• 00:53to prepend everything.
• 00:55That doesn't mean that you
• 00:56have to keep the sample
• 00:58sizes the same and everything
• 00:59the same. It's just that
• 01:01you have to say how
• 01:02you will adapt it and
• 01:03you also, in those cases,
• 01:05you will probably have to,
• 01:07have a conversation with the
• 01:08FDA in advance
• 01:10of starting your study.
• 01:12So everything has to be
• 01:13pre planned when we adapt
• 01:15a clinical trial.
• 01:17Examples are sample size adjustments,
• 01:19removing treatment arms and changing
• 01:22the target population.
• 01:24But in public health intervention
• 01:25trials, sometimes there there there
• 01:27is more that people want
• 01:29to adapt.
• 01:31Let's try this out the
• 01:33notice.
• 01:47We have too much too
• 01:48much on the screen. And,
• 01:50so there is more that
• 01:52people want to want to
• 01:53adapt because in these large
• 01:55scale implementation trials,
• 01:57there's often more than one
• 01:59intervention. There could be
• 02:01a roll out of the
• 02:03intervention
• 02:03that may take one or
• 02:05may take three days. There
• 02:06could be coaching visits that
• 02:08that take place during over
• 02:10the course of time. And
• 02:11all these intervention
• 02:13components,
• 02:14if you would have to
• 02:15specify exactly how many coaching
• 02:17days, exactly all these things
• 02:19and the of these intervention
• 02:20components,
• 02:21that might just be too
• 02:23much to ask for. In
• 02:24in most of the trial,
• 02:25people typically have an idea
• 02:27about what would be the
• 02:28optimal intervention
• 02:29package,
• 02:30but then this this center
• 02:32does something else. This other
• 02:33thing sets about something else.
• 02:35And maybe over the course
• 02:36of time, it seems like
• 02:37one of the intervention
• 02:39components is more
• 02:41useful and people focus more
• 02:43on that. So what statisticians
• 02:45have often done is say,
• 02:47okay. If you're in an
• 02:47ethnic intervention about the trials
• 02:49ongoing,
• 02:50we don't wanna
• 02:51touch the data because that's
• 02:53not planned and there could
• 02:54be all kinds of issues
• 02:56with that.
• 02:57And Donna found out through
• 02:59her,
• 03:00sounding board people that it
• 03:03would be a good idea
• 03:04to also figure out if
• 03:06we can formalize
• 03:07this learning from prior data
• 03:10because that's what, LAGO is
• 03:12about. It's a at least
• 03:14two stage trial
• 03:16where
• 03:17the data from the first
• 03:18stage are in are analyzed,
• 03:21and they are used to
• 03:22figure out which would be
• 03:23a good intervention to do
• 03:25in the next step.
• 03:28So the intervention could be
• 03:29a treatment, a device, a
• 03:31new way to organize care
• 03:33or a combination of all
• 03:34that.
• 03:35And
• 03:37the Lago trial, an inherent
• 03:38part of a Lago trial
• 03:40is to optimize and prevent
• 03:42it. And that means maybe
• 03:43then you want to search
• 03:44an outcome or maybe you
• 03:45want a probability of success
• 03:47of at least ninety percent
• 03:49or maybe forty percent,
• 03:51Whatever but some kind of
• 03:52goal or maybe you want
• 03:54a mean blood pressure in
• 03:56those who have suffered from
• 03:59who have who have suffered
• 04:00from high blood pressure
• 04:01of say eighty
• 04:03or maybe nine
• 04:05depending on your aim. So
• 04:07maybe you have an idea
• 04:08about how you want to
• 04:09have the outcome
• 04:10and that an
• 04:11important part of a lab
• 04:13coat trial is to figure
• 04:14out what is the optimal
• 04:16way to achieve this goal
• 04:18where we not only look
• 04:20at efficacy, we also look
• 04:22at cost.
• 04:24Because,
• 04:24of course, if you look
• 04:25at efficacy, then you will
• 04:27just max out all the
• 04:28effect effective components,
• 04:30but then you may add
• 04:31end up with a implementation
• 04:33package
• 04:34that is very expensive
• 04:36and maybe also not very
• 04:37practical.
• 04:38So we also look at
• 04:40first
• 04:41to, to optimize here the
• 04:42batch.
• 04:47In the beta burst study,
• 04:50the intervention
• 04:51was the encouragement of the
• 04:52use of a checklist to
• 04:54improve
• 04:55safety of mother and the
• 04:57newborn
• 04:58child, and it goes to
• 05:00a WHO
• 05:01checklist of a lot of
• 05:02interventions
• 05:03that are known to help,
• 05:06prevent
• 05:07complications during birth, complications of
• 05:09the child and complications of
• 05:12the mother.
• 05:13And how they
• 05:15implemented this this this, enforcement
• 05:18of the checklist changed over
• 05:20a course of time. They
• 05:21first had a trial with
• 05:23with limited number of centers
• 05:24then then the next phase
• 05:25where they had they had
• 05:26another number of centers and
• 05:28then the big trial where
• 05:29they actually randomized the intervention.
• 05:33The VEDAMAR study took place
• 05:34in Uttar Pradesh in India
• 05:37and the outcomes,
• 05:39are the outcomes of a
• 05:40coaching based WHO
• 05:42safe burn safe childbirth
• 05:45checklist
• 05:46program
• 05:47in India.
• 05:48And this,
• 05:49this was published in the
• 05:50New England Journal of Medicine.
• 05:53One of the aspects of
• 05:54the better bird trial, it
• 05:55did provide useful information on
• 05:58how to improve
• 06:00process outcomes,
• 06:01but it failed to show
• 06:03a significant result on the
• 06:05on the primary outcome.
• 06:07So, technically, it was a
• 06:09failed trial, although still,
• 06:11they found effects on process
• 06:13outcomes.
• 06:15Perfect.
• 06:16So just a nine to
• 06:17twenty graph, did this check
• 06:18this is for,
• 06:20practitioners that are involved with
• 06:22Yeah. Well, I think it's
• 06:23just wash hands. One of
• 06:25the things is,
• 06:27make sure that the mother
• 06:29gets oxytocin
• 06:31during delivery. That's during the
• 06:33Very,
• 06:34very specific things that happened
• 06:36around
• 06:37In and around childhood. There's,
• 06:39like, I think twenty seven
• 06:40different
• 06:41recommended things that should be
• 06:42done
• 06:44before, during, and right after
• 06:46your childbirth, and most of
• 06:47them are very low tech.
• 06:50That works in hand.
• 06:52That's simple.
• 06:54But the third time not
• 06:55everybody does that.
• 06:59We looked at package components
• 07:01intervention, launch units, the number
• 07:03of coaching visits, and a
• 07:05data feedback form
• 07:07And, so what we are
• 07:09going to do is figure
• 07:10out how we could have
• 07:11used Laval in order to
• 07:13improve
• 07:14the outcomes for the better
• 07:16start.
• 07:20Judith, some of the people
• 07:21online said that if you
• 07:23move away from, I guess,
• 07:24the computers where the mic
• 07:26is, they're having trouble hearing
• 07:27you. Oh, okay. So it's
• 07:29a little bit weird.
• 07:31And I think the AV
• 07:32guys coming back can help
• 07:34too. There's also a coating
• 07:35that you'd be more comfortable
• 07:36with. Would that be easier
• 07:37for you? Yes. Let's do
• 07:39that. Let me load. Sure.
• 07:45While while you're doing that,
• 07:47before you even talk about
• 07:49including the outcomes,
• 07:50what were the outcomes?
• 07:53So they looked at
• 07:56The primary outcome of the
• 07:58study was a combined outcome
• 08:00of maternal mortality and neonatal
• 08:02mortality.
• 08:04And it's a huge trial,
• 08:05like, you know, something like
• 08:06over a hundred and fifty
• 08:08thousand
• 08:08births or something like that.
• 08:10Yeah. We've heard six hundred.
• 08:12Based on the way they've
• 08:13done it done the trial?
• 08:15Yeah. Yeah. Exactly. Exactly.
• 08:18These colleagues are all evidence
• 08:19based interventions
• 08:21that are routinely all done
• 08:23in high income countries where
• 08:24we have very, very low,
• 08:27maternal mortality rates and neonatal
• 08:29mortality rates. And those are
• 08:31all not in the US.
• 08:32We have high rates.
• 08:35So it couldn't be down
• 08:36the US because we have
• 08:37high rates. We have high
• 08:38rates. Yeah. We have high
• 08:39rates higher than in the
• 08:40Dallas. Very common. Excuse me.
• 08:42That's right. But it is
• 08:43very it's very this this
• 08:45terrible. Somebody like me is
• 08:46not at risk because I
• 08:47go to Brigham and Women's
• 08:48Hospital.
• 08:49Other people,
• 08:51oh, yeah. Other other
• 08:53people,
• 08:55are not affected.
• 08:56You can get lucky you
• 08:57can get notified.
• 09:00It's a problem.
• 09:04So this randomized leopovirus had
• 09:06thirty six or more patients
• 09:08and still on the back
• 09:09primary outcome, they didn't have
• 09:10a significant result.
• 09:13So,
• 09:14the Golub,
• 09:15Lago study is now how
• 09:17the probability of success. For
• 09:19example, the percentage of
• 09:21developed birth checklist is used
• 09:23depends on the intervention package
• 09:25x. Let's say x consists
• 09:27of both the launch duration
• 09:28and the number of coaching
• 09:30visits. This will be, considered
• 09:32in the middle of study
• 09:33because there were more intervention
• 09:35components,
• 09:36but they were highly coordinated.
• 09:38So we looked only at
• 09:39these two.
• 09:41For example, finding the intervention
• 09:42package xopt
• 09:44so that, it solves that
• 09:46the probability of success
• 09:48is at least zero point
• 09:49nine
• 09:50by minimizing the cost of
• 09:52x
• 09:53and subject to natural constraints
• 09:55on x. When you look
• 09:56at launch duration, you can
• 09:58probably imagine that it's probably
• 09:59not a good idea to
• 10:00have a launch duration of
• 10:02more than three or four
• 10:03days.
• 10:06And there's also a minimum
• 10:07because we cannot be we
• 10:08we we're assuming that at
• 10:10least the intervention needs to
• 10:11be rolled out so it
• 10:12would be considered a launch
• 10:14duration also of at least
• 10:15one day.
• 10:18In March, Exoq minimizes the
• 10:20cost of the intervention package
• 10:21while keeping the probability of
• 10:23success to be at least
• 10:24ninety percent.
• 10:26In
• 10:28the present of center effects,
• 10:30one could look at the
• 10:31average log or the centers
• 10:33of this probability of success
• 10:34and that's what we have
• 10:35been doing in the POLISA
• 10:37study.
• 10:38So the POLISA study is
• 10:39currently ongoing. I cannot show
• 10:41you results. That's why I
• 10:42don't have it on the
• 10:43slides. But the POLISA study
• 10:45is one where we are
• 10:46currently doing level optimization.
• 10:48So we are looking at
• 10:49this, at this particular
• 10:52outcome goal, we call it.
• 10:55So in MERGE, actual minimizes
• 10:57the cost of the intervention
• 10:58package while keeping the average
• 11:00probability of success to be
• 11:01at least ninety percent.
• 11:06So the level, the learn
• 11:07as you go aspect, we
• 11:08will start with two stages.
• 11:10Like, the level can also
• 11:12be done for more than
• 11:13two stages,
• 11:14But the basic things of
• 11:15the how we roll it
• 11:17out and what are the
• 11:18things that we do is
• 11:20better explained to the only
• 11:21two stages.
• 11:23And we consider the stage
• 11:25the situation where the stage
• 11:27one attaches
• 11:28determining amount because what capability
• 11:30you have to play your
• 11:31trial. And then what happens
• 11:33in the first stage happens.
• 11:35In the second stage, we
• 11:36will calculate a re a
• 11:38recommended intervention
• 11:40to use in the next
• 11:41stage of the trial. And
• 11:43that recommended
• 11:44intervention typically
• 11:45depends on the stage one
• 11:47data. So what you get
• 11:48is a dependence between the
• 11:50stage one data and the
• 11:51stage two data
• 11:53because the recommended intervention is
• 11:55learned
• 11:56from the stage one data.
• 11:59And the typical reason for
• 12:00wanting to learn as you
• 12:02go is, okay, we are
• 12:03in this trial and if
• 12:05it's broken, let's try to
• 12:07fix it. So Donna found
• 12:09out through her study group
• 12:10that it is also actually
• 12:12happening in the practice
• 12:13that people do want to
• 12:14learn. They see that their
• 12:16trial is not working and
• 12:17they do want to adapt
• 12:19the intervention to keep to
• 12:20be able to keep going,
• 12:22hoping for trial success.
• 12:26So when we learn as
• 12:28we go and we have
• 12:30a stage two intervention that
• 12:32depends on the stage one
• 12:34data, then we lose one
• 12:36important
• 12:37tool that we have in
• 12:38statistics
• 12:39and that is the independence
• 12:41of data across centers, the
• 12:43independence
• 12:44of data,
• 12:46between stages.
• 12:48Because
• 12:49the outcomes in the second
• 12:50stage are going to depend
• 12:52on the outcomes in the
• 12:53first stage,
• 12:55especially if the treatment has
• 12:56any effect
• 12:58because the learning
• 12:59of the recommended intervention
• 13:02depends on the first stage.
• 13:04So the usual techniques that
• 13:06you use to prove consistency
• 13:08and asymptotic normality if your
• 13:10estimators feel
• 13:12miserably in this case because
• 13:14there's dependence between the stages.
• 13:16And that means that also
• 13:17between the different centers, even
• 13:19if they
• 13:20if they're
• 13:22having independent patients,
• 13:24their dates are going to
• 13:25be dependent because their interventions
• 13:27are going to be dependent.
• 13:32Here we have
• 13:34a long list of things.
• 13:36So we do two stages
• 13:37for now and no early
• 13:38stopping that we can also
• 13:39relax.
• 13:40Penny will be the notation
• 13:42for stage.
• 13:44We are assuming the same
• 13:45number of patients in both
• 13:46stages for now. This can
• 13:48be relaxed. It's really not
• 13:49big deal.
• 13:50X one is the recommended
• 13:52intervention
• 13:53at stage one. It's free
• 13:55trial, so I denote it
• 13:56with a little x. And
• 13:58x two n is the
• 13:59recommended multivariate intervention at stage
• 14:02two. And we are assuming
• 14:03that it's a bounded random
• 14:05variable that is learned from
• 14:06the stage one data. So
• 14:08that's also the reason that
• 14:09it's x two n. So
• 14:11x one doesn't need an
• 14:12n because it's free trial,
• 14:14but x two n depends
• 14:15on how many patients are
• 14:17there in stage one and
• 14:19who is in stage one
• 14:20and what were their outcomes.
• 14:21So in order to, stress
• 14:23that, we have x two
• 14:25m.
• 14:27Then we are assuming that
• 14:28maybe in these trials, not
• 14:31everybody does ex every center
• 14:33does exactly as the recommended
• 14:35intervention is recommended.
• 14:37So we have a different
• 14:38notation, a j k n,
• 14:40for the actual intervention in
• 14:42center or side j of
• 14:43stage k.
• 14:45And we are assuming that
• 14:47it depends on maybe on
• 14:48the recommended intervention, hopefully on
• 14:50the recommended intervention,
• 14:52but not further on prior
• 14:54data. And that means that
• 14:55learning is done
• 14:57through the recommended intervention and
• 14:59that's it, not also through
• 15:01other means. This is a
• 15:03restriction that we need to
• 15:04ensure the validity
• 15:06of the conference intervals and
• 15:08the estimators.
• 15:11Then we have y for
• 15:13outcomes. Y k n is
• 15:15the outcomes of the n
• 15:16participants
• 15:17in stage k.
• 15:19And their binary outcomes in
• 15:21this case, I can do
• 15:22it also for continuous, but
• 15:24let's set the the stage
• 15:25for binary outcomes.
• 15:26And,
• 15:27y I k n is
• 15:28the outcome of participant I
• 15:31in stage k.
• 15:34That means that y I
• 15:35one n and y I
• 15:36two n denote outcomes of
• 15:38two different participants.
• 15:40This comes from stage one,
• 15:42and this is a participant
• 15:43in stage two.
• 15:46The n superscript is needed
• 15:47in stages after stage one
• 15:49because the intervention
• 15:50in those stages depends on
• 15:52the prior outcomes from stage
• 15:54one,
• 15:57under the councilman. Luke. Yeah.
• 15:58Just wanna,
• 15:59clarification.
• 16:01Stop if this is a
• 16:02silly question. But you said
• 16:03that we're assuming that we're
• 16:04only learning from the intervention,
• 16:07I think you meant, in
• 16:08this study. But I think
• 16:09that, often we might be
• 16:11learning
• 16:12at the same time from
• 16:13other studies that are not
• 16:14involved. Is that, a problem?
• 16:17So this might be a
• 16:17problem for the FDA
• 16:19because they want things preplanned.
• 16:22It might not be as
• 16:23much of a problem in
• 16:25other settings where the the
• 16:27important thing with LAGO is
• 16:29that we can somehow assume
• 16:31that the intervention,
• 16:33in the second stage
• 16:35converges in probability.
• 16:38So
• 16:41if we look at the
• 16:42stage one
• 16:43outcomes, we can learn from
• 16:45an MLE. We can learn
• 16:47from averages. We can learn
• 16:48from average purchase outcomes. Doesn't
• 16:50need to be the bias.
• 16:51It can be something else,
• 16:53but it needs to converge
• 16:54in probability.
• 16:55So if you have these
• 16:56outside things happening,
• 16:58we also have to at
• 16:59least conceptualize
• 17:00that they could converge to
• 17:01something.
• 17:10So the recommended intervention x
• 17:11two n is random determined
• 17:13by some function of the
• 17:14prior data. It depends on
• 17:16what researchers decide upon looking
• 17:18at the stage one results.
• 17:20And if at all possible,
• 17:21it should be preplanned.
• 17:23And that means in terms
• 17:24of random variables,
• 17:26x two n is not
• 17:28independent of the stage one
• 17:29data, and so outcomes in
• 17:31different stages are going to
• 17:33be dependent.
• 17:36We are now going to
• 17:37assume that x two and
• 17:38the recommended intervention depends deterministically
• 17:41on the stage one data
• 17:42in a prespecified
• 17:44rule,
• 17:45in order to not jeopardize
• 17:46the validity of the trial.
• 17:48For example, the decision goal
• 17:50could be based based, as
• 17:51I said, about means and
• 17:53averages,
• 17:54in the entire sample
• 17:56in a smooth and differentiable
• 17:58way. The decision rule should
• 18:00be based on an MLE
• 18:01calculated from the stage one
• 18:03data in a smooth
• 18:04differentiable data would be the
• 18:05other option.
• 18:09We are looking at asymptotic
• 18:11inference. In the meta birth
• 18:12study, that is not a
• 18:13big deal. In the POLISA
• 18:14trial, that is also not
• 18:15a big deal because in
• 18:16typically, these are large scale
• 18:18intervention trials,
• 18:20implementation trials.
• 18:22And we assume that the
• 18:23number of observations in each
• 18:25of the stages gets large
• 18:26at the same rate.
• 18:30So the main,
• 18:32the main assumption in LAGO
• 18:33is that given the recommended
• 18:35intervention, the stage two data
• 18:37are independent of the stage
• 18:38one data. So conditional on
• 18:40the recommended state intervention for
• 18:42stage two,
• 18:43the treatment and the outcomes
• 18:45in stage
• 18:47two are independent of the
• 18:48stage one data, both the
• 18:50treatment and the outcomes.
• 18:53So learning is only through
• 18:55the recommended intervention.
• 18:59And then the second assumption
• 19:01is that the recommended intervention
• 19:02converges in probability to a
• 19:04constant as the number of
• 19:05observations
• 19:06in stage one goes to
• 19:08infinity.
• 19:11The actual intervention
• 19:13is assumed to be a
• 19:15fixed
• 19:16function,
• 19:17maybe center specific,
• 19:19of the recommended intervention,
• 19:21and this makes sure that
• 19:24this function is also deterministically
• 19:26continuous and bounded
• 19:28that the actual intervention in
• 19:29the center also converges in
• 19:32probability to something fixed. And
• 19:34that's something that we really
• 19:35use in their proofs.
• 19:39The
• 19:40binary outcome label typically assumes
• 19:43a logistic regression for the
• 19:44outcome given the intervention.
• 19:47So if this is the
• 19:48intervention in center j,
• 19:50then an outcome of a
• 19:51patient in center j follows
• 19:53a logistic regression model with
• 19:56parameters that depend that that
• 19:58determine how
• 20:00the
• 20:01probability of success
• 20:02depends on the different intervention
• 20:04components.
• 20:06And then this case, the
• 20:08outcome goal would be that
• 20:10the,
• 20:12average probability of success,
• 20:15that is is,
• 20:17at least
• 20:18point nine where we look
• 20:20at the probability of set
• 20:22of success
• 20:23under the estimated beta, where
• 20:25beta is estimated based on
• 20:27the stage one data that
• 20:28are known
• 20:29at the time that this
• 20:30level optimization
• 20:31takes place.
• 20:33And this while minimizing the
• 20:35cost. If it was not
• 20:36minimizing the cost, we would
• 20:38not find a unique solution.
• 20:39We would just max out
• 20:40all the effective components, and
• 20:42that is not, believable.
• 20:47So stage one, we have
• 20:48a recommending settling intervention, collects
• 20:51outcomes, recommend intervention. We we
• 20:53calculate that. That's what we
• 20:55are currently doing for POLISA
• 20:57and then collects outcomes.
• 20:59And then we are going
• 21:00to estimate
• 21:02the parameters in this logistic
• 21:04regression model
• 21:06in order to figure out
• 21:07at the end of the
• 21:08study what is a good
• 21:11intervention or the optimal intervention,
• 21:13so the one that has
• 21:14success probability of at least
• 21:16ninety
• 21:17percent by minimizing costs.
• 21:20And we would do that
• 21:21final estimate of beta or
• 21:24the and and the final
• 21:25optimization
• 21:26based on the data from
• 21:27all stages combined.
• 21:29And this is different from
• 21:30a design that is called
• 21:32MOST,
• 21:33where they optimize the intervention
• 21:34in the first stage
• 21:36and then
• 21:37implement it in the second
• 21:38stage and then only the
• 21:39estimate use the second
• 21:42stage data to look at
• 21:44the effect of the intervention.
• 21:45In Lago,
• 21:46the idea is to use
• 21:47all data
• 21:48combined
• 21:49at the final analysis to
• 21:51estimate this logistic regression model
• 21:53and also to determine the
• 21:55optimal intervention.
• 22:00Here, I'm going to go
• 22:01fast
• 22:02because I think that is
• 22:03more for tomorrow.
• 22:04We have a likelihood here.
• 22:05So we have the probability
• 22:07of the first
• 22:09data by one, second stage,
• 22:12outcomes by two, and then
• 22:14the both interventions.
• 22:16And this is a fixed,
• 22:19intervention because it's pretrial determined.
• 22:21And this is the beta
• 22:22this is the beta that
• 22:23comes from.
• 22:26Where is
• 22:27where is my beta? Here's
• 22:29my beta.
• 22:30That beta.
• 22:34So first, the actual interventions
• 22:36come based on x one,
• 22:38and we're assuming that, it's
• 22:40not depending on beta, so
• 22:41this this this goes out.
• 22:44And then we have the
• 22:45first stage outcomes. They depend
• 22:47on beta.
• 22:49And then we have the
• 22:52the second stage interventions.
• 22:55They depend on the prior
• 22:56outcomes, but they typically don't
• 22:58depend on beta. And then
• 23:00you get the second stage
• 23:01outcomes and those do the.
• 23:04If you look at the
• 23:05MLE or whatever you wanna
• 23:07call this, then it will
• 23:08solve the same score equations
• 23:11that you would solve
• 23:12had the intervention not been
• 23:14learned.
• 23:15So this would be maybe
• 23:16the end of story
• 23:17if it wasn't the case
• 23:19that, okay, this tells the
• 23:20same score equations, that's great.
• 23:23But these score equations
• 23:24are highly
• 23:25unusual
• 23:27because what would usually happen
• 23:29is that these two terms
• 23:30are independent. They involve different
• 23:32patients who are treated in
• 23:34different ways, so why bother?
• 23:36But we bother because these
• 23:38two terms are actually dependent
• 23:40and this term is highly
• 23:42dependent on what happens here
• 23:44through the learning of the
• 23:45intervention.
• 23:50So, we can show that
• 23:51they're unbiased estimating equations. That's
• 23:53not a big deal. But
• 23:55then,
• 23:56consistently
• 23:57consistency doesn't immediately follow
• 24:00from theory about,
• 24:02estimating equations because that's mostly
• 24:04on IID observations.
• 24:06So we had to use
• 24:07tricks.
• 24:08If you wanna know those
• 24:09tricks, you have to come
• 24:11to the talk
• 24:12of the of the of
• 24:14Tuna's course tomorrow because then
• 24:15I will show how that
• 24:16works.
• 24:19As I told you, normality,
• 24:21so
• 24:21I think you have seen
• 24:22things like this if you
• 24:23are a little bit, familiar
• 24:25with
• 24:26the estimating equations.
• 24:28You would solve the score
• 24:29equations. So you set the
• 24:31derivative of the log likelihood
• 24:32to zero. So you solve
• 24:34the score equations, and then
• 24:35you do this standard Taylor
• 24:37expansion
• 24:37around beta star.
• 24:40And you'll find that, okay.
• 24:41This is great. So I
• 24:42can put this to the
• 24:43other side, and then I
• 24:45divide by the inverse of
• 24:46this thing, and it will
• 24:47all be fine.
• 24:48The problem is going to
• 24:50be that even though this
• 24:52could be fine,
• 24:54this is not the sum
• 24:55of IID terms.
• 24:57So even here, we are
• 24:59not at home yet because
• 25:01typically you would say this
• 25:02goes nicely to a normal
• 25:03random variable because it's IID
• 25:05terms. No. They're not IID
• 25:08terms. There's the two stages.
• 25:10So what we did
• 25:11was do a trick.
• 25:14We have been doing we
• 25:15have been replacing the outcomes
• 25:18in the second stage
• 25:20by outcomes that are coupled
• 25:22with the fixed design.
• 25:24And the fixed design is
• 25:26where the limiting you have
• 25:28this this this
• 25:29stage two interventions.
• 25:31We have been assuming through
• 25:33different stages
• 25:34that the this this,
• 25:37this intervention in the second
• 25:39stage conversion probability
• 25:41to a constant intervention. Let's
• 25:43call it little a.
• 25:44Under the little a intervention,
• 25:47we have that's a fixed
• 25:48intervention, so we lose all
• 25:50these dependencies.
• 25:52But we don't have data
• 25:53from the limiting intervention. We
• 25:55have from the actual intervention.
• 25:56So what we don't really
• 25:58couple the outcomes.
• 25:59And,
• 26:00you have all probably, if
• 26:02you have that one simulation,
• 26:03you have probably a couple
• 26:04of outcomes.
• 26:05Because what you do is
• 26:07you have these these outcomes
• 26:09that have similar probabilities.
• 26:11You can say, okay. The
• 26:12probability is a little bit
• 26:13higher, a little bit lower,
• 26:14but not too much difference.
• 26:16We can make these variables
• 26:18almost the same
• 26:20by using the same uniform
• 26:23to decide on success versus
• 26:24failure
• 26:25because you can have a
• 26:26success probability that depends on
• 26:29a uniform. If you have
• 26:30p is, p is point
• 26:31six, you draw a uniform,
• 26:33you say, okay, if it's
• 26:34less than point six, it's
• 26:35a success. If it's over
• 26:36point six, it's a it's
• 26:38a failure. Right?
• 26:39Now you have a different
• 26:40probability,
• 26:42but it's similar. If you
• 26:43use the same uniform,
• 26:45these outcomes are going to
• 26:46be almost the same.
• 26:47And the beauty of asymptotic
• 26:49normality is that you don't
• 26:51need to show
• 26:52that the original converges
• 26:54except because
• 26:56asymptotic normality is a distributional
• 26:58result.
• 26:59So you can replace
• 27:01these outcomes that you have
• 27:03under the actual design by
• 27:06outcomes that are coupled
• 27:08with the limiting
• 27:09design without changing the distribution
• 27:12of what you have.
• 27:14It's a trick, just a
• 27:15dirty trick.
• 27:18So
• 27:19you change them you the
• 27:20you change
• 27:22the
• 27:23things that you have by
• 27:24something that has the same
• 27:26distribution, and that doesn't affect
• 27:28the limiting results. So that's
• 27:30what we did. If you
• 27:30wanna know more about that,
• 27:32again, you can come tomorrow.
• 27:35K.
• 27:36Twenty nine.
• 27:40Because of the asymptotic normality,
• 27:42Michelle, the Volta likelihood ratio
• 27:44test for beta one is
• 27:45beta one star are asymptotically
• 27:47valid for any beta one
• 27:49star.
• 27:50And now typically you are
• 27:52interested not so much in
• 27:53any beta one star,
• 27:55because typically what you want
• 27:57to do is test the
• 27:58null hypothesis,
• 27:59maybe of no effect of
• 28:01any of the intervention
• 28:03components.
• 28:04So if you want to
• 28:05test the null hypothesis of
• 28:06none of the intervention components,
• 28:09there is a trick again.
• 28:11Because if none of the
• 28:13components
• 28:14has any effect on the
• 28:15outcome,
• 28:16all this learning is pretty
• 28:18irrelevant.
• 28:19And so what we did
• 28:20is figuring out, okay, we
• 28:22want to test the null
• 28:24and it has to remain
• 28:25a valid test under the
• 28:27null.
• 28:28What it does on the
• 28:29alternative,
• 28:30we hope it goes well,
• 28:31but we don't need to
• 28:32prove anything.
• 28:34We just have to make
• 28:35sure that under the now,
• 28:36this is a valid test
• 28:38with the right correct type
• 28:39one error.
• 28:41Now if there's none of
• 28:42the intervention components has any
• 28:44effect,
• 28:45it doesn't matter how you
• 28:46learn
• 28:47because you won't have an
• 28:48effect anyway. So you can
• 28:50learn, and you don't even
• 28:52need to necessarily restrict to
• 28:54this learning on averages and
• 28:55MLEs. You could even learn
• 28:57from donors' results in the
• 28:59trial.
• 29:00Yes. This may be a
• 29:01really dumb question.
• 29:03Totally. But
• 29:04but it it it it
• 29:05seems to be the assumption
• 29:06you're making is that each
• 29:07component of the intervention is
• 29:09independent of each other.
• 29:12No. I'm not making that
• 29:13assumption. I just the null
• 29:15hypothesis that I'm considering
• 29:17is that none of the
• 29:18intervention components has any effect.
• 29:20So it is a very
• 29:21sharp null.
• 29:23Not sharp, but but it's
• 29:25a very You're doing a
• 29:26trial of all the components
• 29:27together and then knowing each
• 29:29individual component
• 29:31isn't, for instance, one in
• 29:32one positive, one is it
• 29:33canceling out a negative?
• 29:35So that could happen,
• 29:37and, that's not our null
• 29:38hypothesis. Our null hypothesis is
• 29:40really that none of the
• 29:41intervention components has an effect
• 29:42and not that they cancel
• 29:43out. The null hypothesis that
• 29:45we're looking at is really
• 29:46none of the components has
• 29:47an effect.
• 29:48But you can't can't learn
• 29:50that from the data.
• 29:52No. But you can test
• 29:53it.
• 29:55You cannot learn it from
• 29:56the data. You can get
• 29:57an idea about whether it
• 29:58happens because not only are
• 30:00we testing, we are also
• 30:01estimating the effect of all
• 30:03the intervention components.
• 30:07But
• 30:08the the it it is
• 30:09a restriction that we are
• 30:11looking at the null of
• 30:12no effect of any of
• 30:13the intervention components because this
• 30:15might not be true or
• 30:17it's maybe they can come
• 30:18out. So it's a limited
• 30:19case. It's a limited case.
• 30:21Yes.
• 30:25But under the law on
• 30:26under this now, you can
• 30:27learn as you want, not
• 30:29learn.
• 30:30Because because none of none
• 30:31of what you're doing is
• 30:32has an effect on the
• 30:33distribution of the outcome. So
• 30:35it may affect the outcome,
• 30:36but not the distribution of
• 30:37the outcome.
• 30:42They have been lately also
• 30:44focusing on not only an
• 30:46outcome goal, but also a
• 30:48power goal.
• 30:49And a power goal is
• 30:50where after stage one, we
• 30:52sit around and we try
• 30:54to figure out, okay, what
• 30:55do we think? How do
• 30:56we think the trial is
• 30:57going?
• 30:58Are we achieve achieving the
• 31:00power
• 31:01that,
• 31:02that we hoped for?
• 31:04So there's two ways to
• 31:05do this. We can look
• 31:06at conditional or unconditional
• 31:08power.
• 31:09At first, I thought we
• 31:10should just be looking at
• 31:12unconditional power. But in conversations
• 31:14with Donna, we decided we
• 31:15also have to look at
• 31:16unconditional
• 31:17power. And this turns out
• 31:19to be important, especially
• 31:21in cases
• 31:22where the where the power
• 31:23is not very high.
• 31:25If the power is is
• 31:27is is pretty high, when
• 31:29the conditional power works full,
• 31:32if the
• 31:33overall power is not very
• 31:35high, it may be better
• 31:36to look at unconditional power.
• 31:38So what is the difference?
• 31:40Unconditional
• 31:40power
• 31:41ignores that we have already,
• 31:45collected the stage one data,
• 31:46and those will be the
• 31:48data that they'll be using
• 31:50at the test.
• 31:51So they consider more variation
• 31:53in the numerator of the
• 31:54test statistic. Because think of
• 31:56this, if you look at
• 31:57at at, at
• 32:00the test at the end
• 32:01of the study, they compare
• 32:02the treated and the untreated.
• 32:03And maybe just that the
• 32:05two sample t test with
• 32:06a variant studies like p
• 32:08one one minus p one
• 32:09p zero one minus p
• 32:10dot p zero with some
• 32:12n's and square roots and
• 32:13stuff on the board.
• 32:14So when you look at
• 32:16this, you have this p
• 32:17one hat minus p zero
• 32:19hats in the numerator.
• 32:20And part of this p
• 32:22one hats and part of
• 32:23this p zero hats at
• 32:24the end of stage two
• 32:25we have,
• 32:26we have the number of
• 32:28successes in the stage one
• 32:29trial in the stage one,
• 32:31both in the intervention and
• 32:33the control group.
• 32:34When we look at unconditional
• 32:36power, we still consider variation
• 32:39in stage one and stage
• 32:40two. When we look at
• 32:42conditional power, we only consider
• 32:44the stage two variation.
• 32:46So this indicates that if
• 32:48we look at unconditional
• 32:50power, we recommend
• 32:52a more effective intervention
• 32:55than if we look at
• 32:56an at conditional power where
• 32:58we can do with a
• 32:59little bit less
• 33:01effective intervention.
• 33:03So you see that if
• 33:04there is cases where we
• 33:05cannot really reach the uncondition
• 33:07reach the conditional power in
• 33:09many of the in many
• 33:10of the trials,
• 33:12then it's good to have
• 33:13the unconditional
• 33:14power.
• 33:15So based on the uncondition
• 33:16based on the
• 33:18power constraints,
• 33:19we can estimate what is
• 33:21the power of the test
• 33:22at the end of the
• 33:23study.
• 33:24So at at stage one,
• 33:25you could do two things.
• 33:26You can estimate you can
• 33:28work with the
• 33:30outcome goal,
• 33:31and you can work with
• 33:32a power goal.
• 33:34So you can try to
• 33:35have them both be okay.
• 33:37So maybe a power of
• 33:38eighty percent
• 33:40and maybe an outcome
• 33:42goal of ninety percent. And
• 33:43these are two constraints,
• 33:45and you can,
• 33:47minimize
• 33:48the cost
• 33:49by the
• 33:51by the recommended
• 33:53intervention.
• 33:55So,
• 33:56adheres to both constraints, so
• 33:58the power goal and the
• 34:00outcome goal.
• 34:01And with these two goals,
• 34:03you're still adapting only the
• 34:05intervention, not the sample size?
• 34:07No. Not the sample size.
• 34:08We're for now, we're working
• 34:09with sample size fix. It's
• 34:11actually an interesting idea to
• 34:12also work on sample size
• 34:14adjustments for for a level.
• 34:16But I think they're not
• 34:17there yet because we have,
• 34:19repeated outcomes to work on
• 34:21and confounding by indications work.
• 34:23Right. So, essentially, you're just
• 34:25an intervention,
• 34:26not only to reach that
• 34:27outcome goal, but also there'll
• 34:29be different interventions that have
• 34:31more like, you could achieve
• 34:32more or less power, so
• 34:34you're also
• 34:35changing the Too long strange.
• 34:37Okay. We we we want
• 34:38an intervention
• 34:40that hopefully
• 34:41gets us the outcome goal
• 34:43and the power goal.
• 34:45And And all the models.
• 34:47On those among those two
• 34:49goals, the intervention that satisfy
• 34:52both goals
• 34:53estimated based on the stage
• 34:54found data. Among those, we
• 34:56choose the one that is
• 34:57least cost.
• 35:00Yes. Thank you.
• 35:04Simulation study. How about they
• 35:06stick that, Dona?
• 35:07That sounds like a good
• 35:08idea. We can go over
• 35:09tomorrow in class. Yeah. Which
• 35:11by the way,
• 35:12I told people on chat,
• 35:14but Judith will give a
• 35:15more detailed technical
• 35:17lecture tomorrow, two hour lecture
• 35:19in my course
• 35:21in
• 35:22room one zero two at
• 35:23sixty Public Street.
• 35:25And you're all welcome to
• 35:26join that as well if
• 35:27you wanna hear more about
• 35:28Mago and learn a bit
• 35:30more in in fact the
• 35:30mathematical details.
• 35:33And I can get you
• 35:34that information if you email
• 35:35me on me at j
• 35:36j dot com.
• 35:38Or we can and you
• 35:39can ask anyone who also
• 35:40knows. Sally or me.
• 35:47Oh, cool.
• 35:48Yeah. So,
• 35:50what we have we can
• 35:52do, we can use the
• 35:53resulting estimates and confidence intervals
• 35:55for the beta using the
• 35:56coupling argument. We can find
• 35:58asymptotic normality.
• 35:59We can also find formulas
• 36:01for the confidence intervals.
• 36:03We can use that to,
• 36:04create estimates and confidence regions
• 36:07for the optimal intervention
• 36:09x opt.
• 36:10And then we could, maybe
• 36:12keep only package configurations
• 36:14for which the confidence interval
• 36:14for the
• 36:17success probability
• 36:18contains zero point nine to
• 36:20create a confidence set
• 36:22for x points.
• 36:25They can also,
• 36:26do a confidence band for
• 36:28the probability
• 36:29of success based on beta
• 36:32hat, since Jaffe's,
• 36:34theorem.
• 36:35This is something that, Daniel
• 36:36was able to figure out.
• 36:37Once we have a subdoted
• 36:39normality, we can use standard
• 36:40results,
• 36:41and that gives us an
• 36:42option to create confidence bands
• 36:45for the probability of success
• 36:47under any possible
• 36:49configuration of the inter implementation
• 36:52package.
• 36:52And then we can also
• 36:53get a no confidence interval
• 36:55for success probability on the
• 36:57of x opt
• 36:58that we estimated because we
• 37:00can just look at the
• 37:01confidence bands at x opt.
• 37:04It's a little bit conservative,
• 37:05but it works.
• 37:10Okay.
• 37:12Thirty nine.
• 37:13No. Not thirty nine. Ah,
• 37:14here. Yeah.
• 37:16The better birth starts. We
• 37:18looked at one outcome. It
• 37:19is a process outcome. We
• 37:20looked at our satoshian administration
• 37:23immediately
• 37:23after delivery.
• 37:25This is recommended by the
• 37:26WHO.
• 37:31The intervention components we considered
• 37:33were the number of coaching
• 37:34visits and the launch duration,
• 37:36and we used the center
• 37:37characteristics. I didn't this would
• 37:39be one that we also
• 37:41included in the logistic regression
• 37:43model. It was the monthly
• 37:44birth birth volume because it
• 37:46was highly,
• 37:48highly variable among the centers
• 37:50in bed of birth.
• 37:52And then you can get
• 37:54the, confidence intervals and the
• 37:56estimate decides this is the
• 37:58beta zero hat, beta one
• 38:00hat, beta two hat.
• 38:02After stage one, you could
• 38:03get an estimate of the
• 38:05optimal intervention,
• 38:07and and the recommended intervention
• 38:09in this case would have
• 38:09been
• 38:10one,
• 38:11long day of lunch duration
• 38:13and five coaching visits.
• 38:16And then after stage two,
• 38:17the recommended intervention would be
• 38:19three lunch duration
• 38:21and only one coaching visit.
• 38:23And after the study,
• 38:26one could estimate all the
• 38:28in the effect of all
• 38:29the of of the both
• 38:30intervention components
• 38:32and also the recommended optimal
• 38:34intervention.
• 38:40The ninety five percent confidence
• 38:42set for the optimal intervention
• 38:43over the grid of x,
• 38:45we looked at,
• 38:46all possible numbers of coaching
• 38:48visits that can only be
• 38:49half because you have a
• 38:50coaching visit or you don't
• 38:51have it. Launchuation, you could
• 38:53think of having it one
• 38:54or one point five days.
• 38:56And of the three hundred
• 38:57and sixty potential intervention package,
• 39:00we included only ten point
• 39:02five percent in the confidence
• 39:04set.
• 39:06And they had different,
• 39:08different costs.
• 39:13We also created,
• 39:17I think it's right here.
• 39:19Sorry.
• 39:20We also created these conference
• 39:22bands for the probability of
• 39:24success under the different intervention
• 39:26component compositions. So you see
• 39:28the first component here and
• 39:30the second component here, and
• 39:32you can visualize this. This
• 39:33is pictures that Daniel never
• 39:35was able to create in
• 39:37art.
• 39:42And now I'm going.
• 39:44So the adaptive learn as
• 39:45you go, Labo design aims
• 39:47to find the optimal intervention
• 39:49composition
• 39:50and its effect.
• 39:52There's a dependence between stage
• 39:53one and stage two data,
• 39:55which makes it really nice
• 39:56for me to work on
• 39:57this because I am also
• 39:58a medical statistic from time
• 40:00to time. I love doing
• 40:01those math things. We use
• 40:03the coupling argument to provide
• 40:05theoretical
• 40:06justification for using existing software
• 40:08for estimation and inference because
• 40:10these estimating equations that I
• 40:12showed you are actually really
• 40:14just the usual estimating equations
• 40:16that logistic regression solves if
• 40:18you run
• 40:19like a PROC LOGISTIC or,
• 40:22or or,
• 40:23geom with a logic.
• 40:25It provided a confidence set
• 40:27for the optimal intervention and
• 40:29confidence band for the probabilities
• 40:31under any,
• 40:33package composition.
• 40:36Extensions, these are things we
• 40:38have done and I haven't
• 40:39shown.
• 40:40Replace n is n one
• 40:42and n two with n
• 40:44one and n two that
• 40:45go, at the same rate.
• 40:48Include center specific covariates
• 40:50in the regression in the
• 40:51logistic regression model,
• 40:53including more than two stages.
• 40:56And, also, we worked on
• 40:57LaVu for continuous outcomes. This
• 40:59is work with my PhD
• 41:01student, advisee, Amta Bing. I
• 41:03think he's home live if
• 41:04you want to ask him
• 41:05about details.
• 41:06So we looked at continuous
• 41:08outcomes,
• 41:09and we looked not only
• 41:10at a linear regression. So
• 41:12with a linear link, we
• 41:13also looked at different link
• 41:15functions.
• 41:16What we did assume so
• 41:17far is that all the
• 41:18errors are identically
• 41:20distributed.
• 41:22And the way he did
• 41:23this was to look at
• 41:25replacing
• 41:26the errors under the
• 41:28actual intervention
• 41:30by errors under the under
• 41:32the limiting
• 41:33intervention
• 41:34without changing the distribution. So
• 41:36it's a little bit like
• 41:36coupling,
• 41:37except for coupling, you cannot
• 41:39really copy them over because
• 41:40they have different
• 41:42different success probabilities.
• 41:43But here, we are showing
• 41:44that the errors are iid,
• 41:45so we can re replace
• 41:47one by the other. The
• 41:49other thing that he needed
• 41:50to do was, use, theory
• 41:52from,
• 41:53from empirical purposes.
• 42:00No. But I do.
• 42:04I'm not fully loading this
• 42:05morning.
• 42:06Let me go fast.
• 42:21Okay.
• 42:23Then another thing,
• 42:25the power constraints, this is
• 42:27still really work in progress.
• 42:31We're almost there. I expect
• 42:32that we can
• 42:34we can send the one
• 42:36liter power constraints
• 42:38within a month or so.
• 42:39Okay?
• 42:42There's future research.
• 42:45Xin Yu here is still
• 42:46is working on,
• 42:48including random center effects. This
• 42:50is joint work, Jun Jin
• 42:51Yu Yu, Donna Spiegelman and
• 42:53Xin Xu Zhou. I'm also
• 42:56involved. I should have put
• 42:57that. Oh, that's your joint
• 42:59work business. Xin Yu is
• 43:00leading this.
• 43:02Allow centers to contribute to
• 43:04more than one stage. I
• 43:05think that we are there
• 43:06now.
• 43:07Study design, consider help additional
• 43:10ways to choose the recommended
• 43:12intervention,
• 43:13but he left. So
• 43:15I don't remember his name,
• 43:16but I also would look
• 43:18look at.
• 43:19So
• 43:21far, we have done worked
• 43:22with a power goal and
• 43:24an outcome sorry, power goal
• 43:26and outcome goal, but there
• 43:28are other ways to do
• 43:29it. It could be based
• 43:29on process outcomes. I would
• 43:31be curious to see that
• 43:33implemented in practice.
• 43:35Replace the logistic regression models
• 43:37for the outcome with a
• 43:38more flexible model. One can
• 43:39think of a probate model.
• 43:41We didn't work with that.
• 43:42Probably that's not a big
• 43:43deal, but it would be
• 43:44interesting.
• 43:45Create an R package for
• 43:46design. This is joint work
• 43:48with Hunter Bing. He is
• 43:49at Boston University and on
• 43:51the Spiegelman here.
• 43:53Hunter has a package working
• 43:55for the better birth, both
• 43:56for continuous outcome and for,
• 43:59for a binary outcome, both
• 44:01with a logic link, but
• 44:02he's, extending it so that
• 44:04hopefully we can also use
• 44:06it for POLISA in the
• 44:07future and then other people
• 44:09can use it for different
• 44:10studies.
• 44:12Consider confounding by indication. This
• 44:14is joint work with Minh
• 44:16Bui,
• 44:17who is currently also online.
• 44:19So she is in, at
• 44:20Boston University.
• 44:22What if a center adopts
• 44:24the intervention?
• 44:25How they do that correlates
• 44:27with their potential outcomes? Maybe
• 44:29a very successful center Yeah.
• 44:31Will have a a higher
• 44:32intervention component on physician
• 44:35than a not as successful
• 44:36center. So we are considering
• 44:38this now.
• 44:39It seems like
• 44:41it's a permitting approach to
• 44:42include fixed center effects
• 44:45because the fixed center effects
• 44:46could pick up the effects
• 44:49of this successful center.
• 44:54And,
• 44:55I will
• 44:56answer questions.