YSPH Biostatistics Seminar: “Versatile Deep Learning Provider Profiling: A Design-Based Approach”

November 02, 2023

Information

Wenbo Wu, PhD, Assistant Professor, Department of Population Health, New York University Grossman School of Medicine

October 31, 2023

ID10936

To CiteDCA Citation Guide

00:00<v Host>Assistant professor</v>
00:01in the Department of Population Health
00:03and in the Department of Medicine at New York University,
00:06Dr. Wu's research synthesizes state-of-the-art methods
00:09from statistics, machine learning, optimization,
00:11and computational science to address critical
00:14and far reaching issues in health services,
00:16research and clinical practice.
00:18Leveraging large scale data
00:20from national disease registries, administrative databases,
00:24electronic health records, and randomized control trials.
00:27Let's give a warm welcome to Dr. Wu.
00:31<v Dr. Wu>Thank you for the nice introduction.</v>
00:34And it's a great honor to be here with all of you.
00:39And so I'm Wenbo, I am from New York.
00:45I joined NYU just a bit over a year ago.
00:53So I think, 'cause we have so many people here,
00:55I think it would be good to run a promotion first.
00:58(Dr. Wu laughs)
00:59So this is our group.
01:01So at NYU we have,
01:03I mean it's a tremendously growing group
01:07and we have like 24 faculty
01:08and we're about to welcome our newest,
01:12like the 25th faculty member into our divisions.
01:16And we have 7 staff.
01:19We have a small PhD program,
01:22we have 20 PhD students and 10 postdocs.
01:26And we have a team of 25 research scientists.
01:30And part of the reason I wanna do this is
01:33because I wanna encourage you guys
01:36to apply to our PhD programs.
01:38So if you're interested,
01:40scan this QR code and you apply, okay?
01:43All right,
01:45so I have been doing things in provider profiling
01:50for the past for five years
01:53and so this is the overview of what it is.
01:59So provider profiling is basically the assessment,
02:03the evaluation of the performance of healthcare providers.
02:09So I listed here,
02:10could be say acute-care hospitals.
02:14(Wu speaks indistinctly)
02:17This acute-care hospitals, kidney dialysis facilities,
02:21I have been working on other evaluations
02:25like organ procurement organizations,
02:27which is a type of organizations
02:31that are responsible for procuring organs
02:34for patients who are in great need
02:36of organ transplant patients.
02:39And the transplant centers, of course, physician, surgeons.
02:42So you can see,
02:44this includes so many different types
02:46of healthcare providers and stakeholders include,
02:50say, insurance companies, regulation, government,
02:55federal agencies.
02:56They're all interested in provider profiling,
02:59I will tell you why.
03:00Providers is basically who are doing profile evaluations
03:06and of course patients.
03:08So because they are interested in the information,
03:11interested in the profiling results
03:14so they can make care seeking decisions.
03:17Okay?
03:18And so I listed here a few outcomes,
03:22like emergency department encounters,
03:25unplanned re-hospitalizations,
03:29which is hospital readmissions.
03:31And I will jump into the details later
03:34and post-discharge deaths and you can,
03:36I mean there are so many different types of outcomes
03:39to consider in provider profiling.
03:41And one of the goals was
03:43to basically identify those providers
03:47with very bad performance in terms
03:49of patient-centered outcomes.
03:50And they can get penalization,
03:55like they can have payment reductions
03:58from government agencies.
04:01Okay?
04:02And as you can see here, this is very important.
04:04This is a very important business,
04:07and profiling can actually help
04:10improve evidence-based accountability
04:13for those providers and how facility targeted interventions
04:18that aimed at improving the care quality.
04:24Alright, so,
04:36so,
04:40this is a slide of a few example papers
04:43that are about evaluating hospitals across the nations.
04:47So they're mostly from the program called,
04:53Hospital Re-admission Reduction Program,
04:55which is a very important national level program
04:59that I will explain later.
05:01But there are just so many papers in this field.
05:05I mean, these are just,
05:07like there are publications in top,
05:10medical journals, analysts of internal medicine,
05:13and New England Journal of Medicine.
05:24Okay?
05:27So, this is another type of profiling stuff.
05:30So it's called physician profiling.
05:31Basically they wanna evaluate physicians.
05:36So this is, as you can see, it's a report,
05:39it's called the health report
05:41from Massachusetts Medical Society,
05:44which is the publisher
05:46of The New England Journal of Medicine.
05:47Okay?
05:48So they prepared this principles
05:50for profiling physician performance, I think many years ago.
05:56So this is a list of exemplar profiling programs
06:01and they are still existing.
06:04So the first one is an interesting state level program
06:08which is arguably one of the first programs.
06:12So it is still administered
06:19by the New York State of Department of Health.
06:22Basically they're interested in evaluating hospitals
06:25that do coronary artery bypass graft surgeries,
06:31and also PCIs and the program have been running
06:35for at least 20 years or so.
06:39And the second one is another important program,
06:41which was launched I think in 2003.
06:46And it is,
06:49I think it is from the one of the Federal Level Act.
06:52And it is currently administered
06:54by the US Centers for Medicare and Medicaid Services.
06:58And their interest in outcomes for, again,
07:0430-day readmissions and mortality for a AMIs
07:09and the heart failure, et cetera.
07:11And the next one is another federal level readmission,
07:17federal level profiling program,
07:19which is also established by Affordable Care Act,
07:24which is Obama care.
07:25You guys probably know that, in 2012.
07:29And so, yeah, they're also interested in,
07:33evaluating hospitals and they will punish those hospitals
07:37with very bad performance in terms of payment reductions.
07:40Okay?
07:41The last one is an interesting program,
07:43which is kind of my focus.
07:46I have been working on evaluating kidney dialysis facilities
07:54for patients with kidney failure.
07:56And there are actually over 7,000 dialysis facilities
08:01across the nation, believe it or not.
08:04But this is the first to pay for performance program
08:09in contrast to other pay for service programs.
08:14Okay.
08:15And the program is called ESRD.
08:17ESRD is short for End Stage Renal Disease.
08:21Basically the patients with kidney failure,
08:24a quality incentive program, okay?
08:26Alright.
08:27So as you can see, there are so many programs,
08:30so many initiatives across the nation about profiling.
08:36And one natural question is about the,
08:41how the landscape of the statistical landscape
08:44of profiling looks like.
08:46And because of the importance of profiling
08:49and here I said,
08:53there are many far reaching implications
08:55because providers can get penalizations
08:58and it's high stakes.
09:02So it's important
09:03that we have principles statistical methods
09:05to evaluate them, right?
09:08So this is like two examples.
09:11The first,
09:13it's a paper published on analysts of internal medicine,
09:17but it is written by two statisticians.
09:21They are calling for the improvement
09:25of statistical approach in this field.
09:28And also the second one,
09:30this one is even more important
09:32because it is a white paper issued
09:35by the Committee of Presidents of Statistical Society.
09:40You probably know about COPS.
09:42So one of the most important words in the statistic field,
09:47it's the COPS presence of work, right?
09:49So this is a white paper by COPS
09:53and also a group of people from the CMS.
09:58So this is also an important work.
10:00It's about the statistical issues
10:02and assessing hospital performance.
10:05So as you can see,
10:07there are many people are interested
10:10in improving the statistical landscape for profiling.
10:14Alright,
10:15so this is a slight briefly introducing the existing methods
10:23of provider profiling.
10:25There are a few.
10:26I grouped them into like roughly four categories.
10:31So the first group,
10:34is hierarchical random-effects models,
10:38there are many papers in this group,
10:42but I just highlighted one paper in,
10:45I think in 1997 was published on Jassa
10:50by Dr. Sharon Lee Norman at Harvard Medical School.
10:54So it's about hierarchical random-effects models
10:58which is still being used in many settings.
11:02Especially, I mean,
11:04not sure whether you guys know
11:05that there is a group at Yale called Yale Core,
11:08I think Center for Outcomes Research and,
11:14Something. <v ->Evaluation.</v>
11:16<v Dr. Wu>Okay, great, thank you.</v>
11:17So they have been using hierarchical random-effects model
11:21for over 30 years, I guess.
11:24And the second stream of approach is fixed-effects models,
11:31as you can tell from the names,
11:36people are using like a fixed effects in the models.
11:40And this is one example paper,
11:44actually was published in 2013 by my advisors.
11:49And the next one is,
11:53I mean these groups of papers,
11:56they're not mutually exclusive because,
12:00for example, this one,
12:01competing risks or semi-competing risks.
12:04I mean there are some papers
12:06that use higher hierarch random-effects model
12:08or they're also papers using fixed-effects models.
12:12But they are just kind of,
12:13they're handling like different types of outcomes.
12:16So I listened here.
12:18And also for recurring events,
12:20if you take a class in survival analysis,
12:23you probably know that, for example,
12:26patient can have multiple hospitalizations in a year.
12:29So they are considered as recurring events.
12:31Okay.
12:32And then the last one is,
12:35some people are using causal inference
12:37and some clustering approaches to handle profiling issues.
12:44But these papers are relatively new,
12:47and this is one paper here.
12:50It was by all statistics, I think.
12:54Alright, so I wanna discuss a few limitations
12:59of the current landscape,
13:01the current statistical in profiling.
13:05So the first limitation is, people have been, I think,
13:10intensely using models with a linear predictor.
13:15So the limitation is this may not be true
13:19when we have very complex outcome
13:23and the factor associations.
13:25So this is an example.
13:28This figure.
13:30This is in my one of my papers.
13:36So the background,
13:38I'll give you a bit of background information.
13:41So this is about, okay,
13:42evaluating the effect of covid
13:46and the outcome is a 30 day unplanned hospital readmissions.
13:51So this, on the left is the surface plot.
13:54On the right is the conquer plot.
13:56As you can see,
13:59we are interested in the variation
14:02of the covid effect across, this might be too small,
14:06but across post discharge time,
14:09post discharge days and also across calendar days
14:13because we used data in 2020.
14:15So we set time zero at, I think mid-March or,
14:21yeah, mid-March.
14:22So this is April the 1st.
14:25And then May 1st until I think mid-October.
14:30So as you can see there's a lot of variation going on here.
14:36So the covid effect is definitely not constant here.
14:38So basically it means that we cannot use the linear model
14:44to do this.
14:44It's just not valid, right?
14:48So the second methodological limitation is existing methods
14:54have been historically driven by cost effective spending.
14:57Like,
15:01I think in the very first program,
15:03in those first early programs,
15:06people are interested in how to reduce costs
15:10by, of course they wanna improve,
15:13they wanna improve care quality
15:15but cost effectiveness is a very important factor.
15:20So,
15:22and these analysis,
15:22they basically combine all racial ethnic groups together
15:25without accounting for their heterogeneity.
15:31So this is an another example.
15:33So we basically look at the performance
15:37of Organ Procurement Organizations, OPOs.
15:42So we are interested
15:44in organization level transplantation rates.
15:49And we have data in 2020.
15:53So these are,
15:55so on the y-axis we have the normalized OPO IDs,
16:03and this is just like a three panels of caterpillar plots.
16:07And if we focus on a certain OPO, then,
16:12for example, in this panel,
16:13this is a panel for white patients.
16:16And if you look at this is,
16:19I know this is a little bit small,
16:21but this is OPO 30 and this,
16:23the conference interval is above the national rate
16:27for white patients.
16:29So it's significantly better than the national average.
16:32But if you look at the this panel,
16:37this is also OPO 30
16:40and we have the confidence interval being lower
16:44than the national average for black patients.
16:46And this is a panel for Asian Americans
16:52and Pacific Islanders.
16:53We also have the same issue going on here for OPO 30.
16:58So as you can see, there's definitely racial disparity here,
17:04but this was never examined in those early programs.
17:11So this is an limitation of course.
17:15And the last one is,
17:17there is a lack of a unifying framework
17:20to accommodate different provider profiling objectives
17:24and the different performance benchmarks.
17:27I will give you like four different examples.
17:31The first one,
17:34I tried to make the notation very easy.
17:37So say we have a random-effects model here.
17:42We just consider a binary outcome.
17:45Y can be zero or one.
17:47Okay?
17:48And we basically use the logistic regression, here.
17:52So this gamma i, it's a sum of two things.
17:56The first one is mu as the mean effect.
17:58And the second one is ID normally distributed,
18:05a random variable, okay?
18:07And we can construct a type of,
18:08we call it standardized measure.
18:10It's Oi divided by Ei,
18:13O is just a sum of all those YIJs.
18:17And the Ei is the,
18:19basically the sig y function transformation
18:23of mu plus beta.
18:25Okay?
18:27So here, if you look at the model,
18:30we have gamma I here,
18:31but when we calculate the expected number of events
18:35or outcomes, we replace this with the mean.
18:40Okay?
18:41So this is the first example
18:44of course using random effects models.
18:46But if we look at the fixed effects model,
18:49we have the similar formulation here,
18:52but here because this is a fixed-effects model,
18:54gamma I is just unknown fixed effect, okay?
18:58And if we define gamma,
19:02start to be the median of gamma, this is a vector actually.
19:05So it's a vector of vault fixed-effects.
19:08Then this is basically the median of vault provider effects
19:12or fixed effects.
19:14And so we can also construct this standardized measure,
19:17but this time, this E is defined as this,
19:22and this is gamma star.
19:26So we basically use the median of all fixed effects
19:30to construct the standardized measure.
19:33Okay?
19:34So now we have two cases.
19:36One is, okay, we use the, oops,
19:39we use mu, which is the mean of all provider effects,
19:44although it's a random effects model.
19:46And,
19:48here we have median of all fixed provider effects, okay?
19:54So these are two cases,
19:55basically two types of models that have been used before.
19:58And next one is, and some causal papers,
20:04they can use a selected set of provider,
20:09it could be a single provider,
20:11let's say, I'm a a hospital administrator,
20:14I wanna see, okay,
20:15whether my hospital is performing better or worse
20:19than another hospital,
20:21then of course I can use my hospital as the benchmark,
20:25as the reference and compare all other hospital
20:29with my hospital, okay?
20:30So this is the first case.
20:32We can just choose a single hospital or provider
20:36as the benchmark.
20:37And the second case is we can group a few providers,
20:42hospitals in the specific geographic region together
20:45and to form a benchmark, this is also doable, okay?
20:49And it is actually used in the paper.
20:53The last one is, we can basically treat all hospitals,
20:58you can group all hospitals together
21:00into a large super hospital, of course,
21:02this is a hypothetical one but we can do that.
21:06And that is kind of like a national average thing, right?
21:10These are all reasonable ways to define a benchmark.
21:17And there is the last one.
21:19So the last one is kind of more like equity driven thing.
21:23So we can form a benchmark such that say,
21:26okay, say,
21:27from the regulator's perspective,
21:29we really wanna push hospitals to improve their performance
21:34for minority patients.
21:36So say, we can set the benchmark to be something like,
21:41okay, for within the minority groups,
21:43we can intentionally select patients with better outcomes.
21:48We can make the proportion to be very large
21:51so that in the benchmark group,
21:54we can have a very good performance for minority patients.
21:59And then black non-Hispanic patients.
22:03So this is kind of a equity driven thing.
22:06So as you can see, I give you like,
22:11at least the four examples.
22:12But these are scattered in the literature
22:15and there is no unifying framework
22:18to accommodate all of these cases.
22:20But we actually can develop a general framework
22:25to accommodate all.
22:26I will give you the details later.
22:30So, all right,
22:34so the framework
22:36that we proposed is what we termed,
22:40a versatile deep learning provider profiling.
22:43So we proposed a versatile or probabilistic framework
22:50based on the, so-called provider comparators,
22:52which is, you can name it as you know, provider comparator,
22:56hypothetical provider performance benchmark
22:58or population norm.
22:59These are all the same interchangeable terms.
23:03Okay?
23:04Here versatile means, okay,
23:06we can use the framework to do a lot of different things.
23:10So they are adaptable to different profiling objectives
23:14and contexts, okay?
23:15It's why we use the term versatile
23:18and here provider comparator,
23:21which is defined to be a hypothetical reference provider
23:28that is corresponding to your profiling objective.
23:30So if you have a certain objective,
23:32of course you can define your own hypothetical provider.
23:37And if you have a different objective,
23:39you can define another one, okay?
23:42And the deep learning thing comes
23:45into play because it is nice that,
23:49generally it relaxed the linearity assumption
23:51in most existing portfolio models
23:55that relies heavily on linear this assumption.
23:58Okay?
24:00Alright, so this is slide of the basic setup
24:07of this new approach.
24:09So let's say we have a ID random sample
24:13with Y as the outcome,
24:17and the Fi star is the provider identifier,
24:22and Zi is simply a vector of variants,
24:27and they are one from a population Y, F star, Z.
24:38And we have the following assumptions
24:40that these two assumptions, one and two,
24:46so F star.
24:47So basically this script F star is the support
24:52of this provider identifier, F star.
24:56Okay?
24:58So we require that this report for any value
25:05that this F star can pay,
25:07we assume that the probability of F star equal
25:11to F is positive,
25:13which means that in the dataset,
25:15you can at least observe one patient from that provider.
25:19Okay?
25:20Say if this is zero, then basically it means,
25:24okay, we do not observe any patient from that provider,
25:27which is useless, right?
25:31So the second assumption is simply,
25:34okay, so this script F star includes all possible providers,
25:41we wanna evaluate.
25:42So basically this F star has to fall
25:45into this set of values, okay?
25:49So that's why it's the probability as equal to one.
25:52Okay?
25:54So we have two important assumptions,
25:58regarding data generating mechanism.
26:00So the first one is basically the distribution
26:03of this F star.
26:05The provider identifier depends on covariate.
26:10And this is like, okay, so for a patient,
26:14say, I'm a patient, I wanna choose my provider,
26:17I wanna choose my hospital,
26:19my decision will largely based on,
26:21okay, what conditions I have,
26:23and what insurance I have, right?
26:27And say what is the possible feasible set
26:31of hospitals I can choose from?
26:34Okay?
26:34So these are all covariates
26:36that we can include in the model.
26:37So basically the F star is the distribution
26:41of a star depends on all those covariates
26:45which is reasonable assumption.
26:48The second one,
26:49the distribution of the outcome Y
26:51as a function of Z and F star,
26:54which means that, okay, the outcome,
26:57if I go to the hospital and say I have a certain disease
27:03and I got a treatment and whether I feel better
27:08or not really depends on, okay,
27:10of course, depends on my conditions,
27:12and also depends on which hospitals I went to, right?
27:17So the distribution is denoted
27:20as pi, y, given Z and F star.
27:25Okay?
27:26So basically these two assumptions gives us the,
27:31basically the basic setting for a patient who is looking
27:35for care to improve their conditions.
27:42So the main idea in this new framework is reclassification.
27:48So basically,
27:49we wanna construct a hypothetical provider comparator
27:54as a performance benchmark
27:56that is corresponding to our specific profiling objective.
28:01Okay?
28:02So reclassification here means that we wanna,
28:06we reclassify subjects from existing providers
28:11into a hypothetical one
28:13following a certain probability distribution.
28:15Okay?
28:16To do this, we introduced a random indicator,
28:19it's just a 0, 1.
28:21Which we termed reclassifier.
28:24This reclassifier is equal to 0.
28:26Here it is kind of different.
28:28So reclassifier is equal to zero.
28:31When the subject is reclassified
28:33into the hypothetical provider,
28:35if it is equal to one, then the subject is not reclassified.
28:39So the patient stays in their original provider, okay?
28:47And with this reclassified redefined, F,
28:51so F is different from F star.
28:54So F is defined as the product of R,
28:57basically R times F star.
28:59And we basically add a singleton to this F script F star.
29:06So now we can see, okay,
29:09so whatever providers we have originally,
29:13now we add a single hypothetical provider
29:16and we provide the provider indicator,
29:21we fix that as zero.
29:23So zero is the hypothetical one.
29:26So now this F can take values,
29:29importantly, it can take whatever values
29:31from the original script F
29:34but now it can also take values
29:37to take the value zero, right?
29:40So basically this R is used
29:43to manipulate a subject's provider membership.
29:46So, a subject from a provider F star equal to F.
29:54So here in this case,
29:55because it's F star, it cannot be equal to zero, right?
29:59So we wanna reclassify patients
30:01from a certain existing real provider
30:04to that hypothetical provider.
30:07You know, this F is equal to zero.
30:10So this is a new provider membership for that patient, okay?
30:14But if R is equal to zero,
30:16then the patient stays in that original hospital.
30:20Okay?
30:22Alright.
30:23We have additional two assumptions
30:25regarding this reclassification thing.
30:29So the first one is for any provider, real provider,
30:35we have this probability, being less than one.
30:38This means that, okay,
30:40so given a set of covariates and given
30:44that the patient is in a certain provider,
30:49then the patient being reclassified
30:53into the new hypothetical provider,
30:56the probability is less than one,
30:58which means that we should keep at least a few patients
31:03in their original provider
31:05so that we can still evaluate the outcome distributions
31:10of the original provider, okay?
31:13And this actually,
31:15if you do some, a simple algebra,
31:20we can show that basically this implies that,
31:23I mean this, we can basically drop this condition
31:26because if you do the sum
31:28of the conditional probability thing,
31:31you can basically drop this condition
31:33and this actually holds.
31:35So it's like, okay, no matter which hospital,
31:38no matter which provider the patient is in currently,
31:42the probability that the patient
31:44will be reclassified is less than one.
31:46So not all patients will be reclassified, right?
31:50And this is the second condition.
31:52So combining these two, basically, okay,
31:58so basically not all patients can be reclassified
32:02or also all patients cannot be living
32:08in their original providers.
32:10Basically we require that, okay, each patient can,
32:15so we should have
32:17at least a few patients who are remaining
32:20in their original hospitals so that we can evaluate
32:22their original outcome distributions.
32:25And also we need a, of course characterize the distribution,
32:28that hypothetical reference provider.
32:31Okay?
32:33Alright.
32:34Then the last assumption is,
32:38this is kind of an interesting setting.
32:40So rather than observing the original data,
32:44Y, F star, Z, we can only observe this set.
32:52So it's R, Y, F, Z, this tuple.
32:59So the big difference between these two is,
33:02for this one, we know exactly for every patient,
33:05we know exactly where they're from,
33:08which provider they are in.
33:11But for the this one, say if R is equal to 0,
33:17F is automatically 0
33:18because F is defined as R times F star.
33:22So for those patients,
33:23we actually don't know where they come from, right?
33:28But here we assume
33:31that we can only observe post-reclassification data.
33:35And this actually is nice,
33:37I mean this is not always necessary in the practice,
33:42but this assumption actually helps,
33:45facilitates the implementation
33:50of some certain privacy preserving protocols
33:53and data security protocols.
33:54If say, okay, we don't want the,
33:57because of certain powerful influential providers
34:01can actually have a strong influence
34:05in policy making.
34:07So, because this is capped like confidential,
34:11so they actually don't know how we design,
34:14how we choose the re-classification scheme.
34:18So it can help reduce some unwarranted inference
34:25from those very powerful stakeholders.
34:29So this is a nice setting,
34:32but it doesn't have to be like this in reality.
34:36Alright, so now we have four assumptions,
34:41important assumptions
34:42to regarding the data generating mechanism
34:44and to regarding the reclassification scheme.
34:48So, the ultimate goals of profiling is
34:54to first to evaluate all providers,
34:57and then we wanna identify goals,
35:00especially with very bad performance
35:03and we can take additional actions
35:07and so we can, you know,
35:10improve their performance in certain way.
35:12Okay?
35:13But yeah,
35:14so this quantitatively or mathematically,
35:20we have the two overarching goals.
35:22The first one is to harness,
35:24to use the post reclassification data,
35:29to contrast the distribution of each existing
35:35or real provider.
35:37F star was the newly defined reference group.
35:42So we wanna compare,
35:44basically, compare the distribution of these two groups.
35:47I mean each of them
35:48because we have so many real providers,
35:51and we only have a single hypothetical provider, okay?
35:54We wanna compare them, we wanna do contrasts.
35:57And of course the second goal is
35:59to identify those providers with very bad performance.
36:06All right,
36:08so, this actually,
36:14because we introduced this hypothetical provider,
36:17this is really nice actually.
36:19But there is a difficult issue here
36:24because we introduced this hypothetical provider,
36:30we actually have to account for
36:32or address reclassification dues to bias.
36:35So the details are in this proposition.
36:39So let's assume that those four assumptions hold
36:43and the distribution of the outcome given Z and this F,
36:50F is the newly defined provider indicator.
36:54We can actually write the outcome distribution,
36:58like in two cases.
36:59So when F is equal to 0,
37:01this is corresponding to the reference,
37:05the hypothetical provider.
37:07So this is actually the average,
37:14you can consider as the distribution of the outcome
37:21basically for all patient.
37:22If you group all patients together into a single group,
37:25this is basically the distribution of that group.
37:28Okay?
37:29But we have this term here,
37:31and this is not necessarily equal to 1,
37:34F is equal to 1 then it's very simple,
37:38but it could be unequal to 1.
37:43And also in the second case when F is not equal to 0,
37:48which means that okay, for those existing providers,
37:53their distribution also changes because you basically,
37:56you move a few patients to the new provider.
37:59So the original distribution changes, right?
38:02And because we cannot observe this by assumption.
38:06So this is basically the observed outcome distribution
38:10for existing providers.
38:11But according, as you can see here,
38:14it's a bias distribution.
38:15It's no longer the original one, right?
38:17Because this ratio, again,
38:18it is not necessarily equal to 1, okay?
38:22Right?
38:23So as I said,
38:26you can consider this as the average distribution,
38:29basically as the outcome distribution
38:31of the whole patient population, okay?
38:33So of course you can write it as a sum of the,
38:39you know, weighted probabilities.
38:42So the weight being the probability provider membership,
38:46and this is basically, okay, within this certain provider,
38:50what does the outcome distribution look like?
38:53Okay. All right.
38:58So a few things.
39:03This proposition basically outlines a,
39:06what we call design based approach
39:08to provider profiling, basically, okay.
39:12So,
39:14I actually, I mentioned this early,
39:17in profiling there are a few different parties.
39:20The first one is regulars
39:23who initiated the profiling process
39:25because they are interested
39:26in the performance of these providers.
39:28And also we have profilers,
39:31which basically evaluates the performance,
39:33but they don't have to be the same as regulators.
39:36And also we have of course,
39:38providers who are the subject of evaluation
39:42and we also have patient who need the information
39:44to make their decision, okay?
39:46So the design-based approach
39:47basically tell us that, okay, so, for regulators,
39:51they can basically lead the development
39:53of a reclassification scene because in this framework,
39:57we never say what the distribution,
39:59say, what this looks like, where, right?
40:02So this is a very general specification
40:05and we only made that four assumptions,
40:08but we don't have any distributional assumption here.
40:12So we can make it very general.
40:15And so in this framework,
40:19regulators will get more involved in this process.
40:24So that's why they can
40:25basically design the reclassification scheme
40:30based on their specific objectives, okay?
40:35Alright.
40:35So, and given a specific reclassification scheme,
40:41of course they can design their own reference group,
40:45their hypothetical providers
40:47and having defined this hypothetical provider,
40:52profilers of course can use post the reclassification data
41:00and also the dependence.
41:01Because here, as you can see here,
41:03this R actually depends on Y,
41:05depends on the outcome covariate
41:06and the provider identification.
41:10So using this information
41:16and also the post reclassification data,
41:20profilers that can actually do the profiling
41:23and we can use the framework
41:26to estimate the probabilities reclassification,
41:29which is also the propensity scores actually.
41:33So the next step would be
41:39to use the estimated propensity scores
41:41to correct for reclassification induced bias.
41:45And then we can basically construct the distribution
41:51of the hypothetical provider with the distribution
41:55of the existing provider, okay?
42:00Alright.
42:01So as sketched in the previous slide,
42:05there are a few important things
42:07or advantages of the design-based approach.
42:11So this approach actually,
42:13in this framework,
42:14providers can be more involved in this framework.
42:21And,
42:24so we can use the profiling result,
42:28from this new approach can be more relevant
42:31to what people are interested
42:33in the care decision making process, okay?
42:38So, I think I'm a bit over time,
42:43but I wanna quickly skim through a few examples.
42:48But these examples are basically,
42:51we need a few assumptions like
42:54whether the reclassifier is depending on the outcome,
43:00so in this example, it's very simple.
43:02Basically the reclassifier is independent of everything.
43:08So,
43:09actually this reduces to the most simple case.
43:12So nothing changes actually after reclassification,
43:15but this is an example about the setting.
43:21And we also have like a few non-dependent settings.
43:27This R can depend on F star and given F star,
43:32it can be independent with Y.
43:35And we also have some examples,
43:37this is called equal rate representation.
43:39We also have singular representation,
43:42basically the setting
43:43where we only choose a single provider
43:47and we also have the case
43:49where R actually depends on Y, the outcome.
43:54So we can basically choose the outcome,
43:56sorry, we can choose patients based on the outcome.
44:02And I also give an example,
44:04this is actually an interesting example,
44:06but seems like we don't have enough time today.
44:10So this is the most general case where R is allowed
44:13to depend on F and also Y.
44:18So we don't have independence anymore,
44:20but unfortunately this case,
44:22we have the unidentifiability issue.
44:26So this case won't work
44:27under the post-reclassification data assumption.
44:31So we actually developed a framework,
44:38we looked at the deep learning methods
44:41and the singular representation case.
44:44And this is a relatively simple framework.
44:47We only consider exponential distribution.
44:50I mean the outcome
44:51involves the exponential family distribution
44:56and we construct a neural network model.
45:01So we have the input layer
45:02and the fully connected hidden layers and the outcome layer,
45:05and we use stratify sampling based optimization algorithm.
45:11Here, I will skip the detail.
45:14And we developed a exact test based outcome distribution,
45:19exact test based approach to identify outlined performers.
45:25Okay?
45:27And this is basically the motivation
45:29why we need deep learning here, because simply speaking,
45:32the covid effect is not constant over calendar time
45:36and we have to easily account for that
45:39while doing profiling,
45:40but the effect itself is not of interest.
45:48Basically a visualization of the profile results.
45:53So here we construct the,
45:57we construct what we call the funnel plot here.
46:01So the benchmark, the reference, the indicator,
46:06we use is again Oi divided by Ei
46:09and Ei and defined where this one is the median.
46:14And this is actually the neural network part.
46:18And we have the funnel plots.
46:22So those dots represent providers, okay?
46:25So because this, I mean,
46:29the higher, the worse the performance,
46:31the lower, the better the performance.
46:32So these blue dots here are actually better performers.
46:38So as you can see, if you add these two supporters up,
46:42this is like over 20%,
46:47what does not make practical sense
46:48because in practice you cannot identify outliers
46:52with over 20%, you know, this is too much.
46:57So we have to somehow account
46:59for provider level unmeasured confounding.
47:03And I didn't include the technical details here.
47:08But after the adjustment,
47:11as you can see the proportion of a better
47:14and the worse performers are much lower than before.
47:18And I think I only have one more slide.
47:23So some takeaways.
47:25So profiling is very important
47:27as a major societal undertaking in the United States.
47:30And we have so many applications,
47:33important implications and important consequences as well.
47:38And the new framework actually
47:41increased the regulators engagement in this process.
47:45And it's called versatile
47:47because we can handle different profiling objectives
47:49and it is compatible
47:50with many different model specifications,
47:53machine learning models, data science models.
47:55And here we use deep learning
47:58because it relaxes the linearity assumption
48:01and it is often a good idea to account
48:05for provider level measure confounding
48:08when we do this profiling stuff.
48:11And that's all for today.
48:15Thank you so much for.
48:20I know we only have like two-
48:22<v Host>Yeah, We have two minutes.</v>
48:23Thank you very much Dr. Wu for your presentation.
48:26Any questions in the audience?
48:36Anyone online?
48:37Just giving everyone a chance.
48:40No, I'll ask a question.
48:41So I think it's really cool to be able
48:45to identify providers who are doing really well
48:48or doing bad.
48:49What do you do with that?
48:51Now that you have that result?
48:52Like do you tell the profiler
48:54or the patient get to give it to say,
48:56"Oh, I don't wanna go to them, they're bad."
48:58<v Dr. Wu>Yeah, that's a good question.</v>
48:59So actually CMS,
49:02they have many programs say, one is for dialysis patients,
49:07they have dialysis facility compare,
49:10which is an online program.
49:12So patient can have access to different types
49:15of information like whether diet facility is good or bad
49:20and many other different fields
49:24of information they have online.
49:26So they can choose their favorite providers.
49:30Yeah, that's possible.
49:32And it's something that is going on, yeah.
49:36<v Host>Oh, I think we have questions.</v>
49:37<v ->Yep.</v> <v ->Just very briefly,</v>
49:39because I know we're out of time but.
49:43To what extent do you feel that,
49:45if this is true, I guess, and doesn't matter,
49:47the patients don't necessarily have meetings.
49:51So for example, like I grew up in a rural county,
49:55we had one hospital, you were going to a hospital,
49:57you were going there.
49:58Even in New Haven,
50:00there are two campuses of Yale New Haven Hospital,
50:02but there's only one hospital in metro area.
50:06So, I mean, choice is kind of not a real thing.
50:11How does that affect?
50:13<v Dr. Wu>Right, that's a very good point, so-</v>
50:17<v Questioner>We are actually in city,</v>
50:18I understand there's more than one.
50:19(Host laughs) Right, there are so many.
50:21<v Dr. Wu>Yeah, but that's a very good point</v>
50:23because we are actually considering another framework
50:27which is also clustering framework,
50:30which basically gives you
50:32under certain conditions you can choose,
50:34there's a feasible set of providers
50:36that you can choose from,
50:37of course, under certain strengths,
50:39say your insurance, your location, many other conditions.
50:45But I mean, in this framework,
50:49maybe we can address that issue
50:53in the set of areas that we included here.
50:57But yeah, I mean, you know, very important issue.
51:06<v Host>Unfortunately, that's time.</v>
51:06So let's thank Dr. Wu again.
51:12If you haven't signed in, please sign in before you speak.
51:15You are registered.
51:17Oh no, it's good, I don't know.
51:19(indistinct chattering)