# YSPH Biostatistics Seminar: “Versatile Deep Learning Provider Profiling: A Design-Based Approach”

November 02, 2023## Information

Wenbo Wu, PhD, Assistant Professor, Department of Population Health, New York University Grossman School of Medicine

October 31, 2023

ID10936

To CiteDCA Citation Guide

- 00:00<v Host>Assistant professor</v>
- 00:01in the Department of Population Health
- 00:03and in the Department of Medicine at New York University,
- 00:06Dr. Wu's research synthesizes state-of-the-art methods
- 00:09from statistics, machine learning, optimization,
- 00:11and computational science to address critical
- 00:14and far reaching issues in health services,
- 00:16research and clinical practice.
- 00:18Leveraging large scale data
- 00:20from national disease registries, administrative databases,
- 00:24electronic health records, and randomized control trials.
- 00:27Let's give a warm welcome to Dr. Wu.
- 00:31<v Dr. Wu>Thank you for the nice introduction.</v>
- 00:34And it's a great honor to be here with all of you.
- 00:39And so I'm Wenbo, I am from New York.
- 00:45I joined NYU just a bit over a year ago.
- 00:53So I think, 'cause we have so many people here,
- 00:55I think it would be good to run a promotion first.
- 00:58(Dr. Wu laughs)
- 00:59So this is our group.
- 01:01So at NYU we have,
- 01:03I mean it's a tremendously growing group
- 01:07and we have like 24 faculty
- 01:08and we're about to welcome our newest,
- 01:12like the 25th faculty member into our divisions.
- 01:16And we have 7 staff.
- 01:19We have a small PhD program,
- 01:22we have 20 PhD students and 10 postdocs.
- 01:26And we have a team of 25 research scientists.
- 01:30And part of the reason I wanna do this is
- 01:33because I wanna encourage you guys
- 01:36to apply to our PhD programs.
- 01:38So if you're interested,
- 01:40scan this QR code and you apply, okay?
- 01:43All right,
- 01:45so I have been doing things in provider profiling
- 01:50for the past for five years
- 01:53and so this is the overview of what it is.
- 01:59So provider profiling is basically the assessment,
- 02:03the evaluation of the performance of healthcare providers.
- 02:09So I listed here,
- 02:10could be say acute-care hospitals.
- 02:14(Wu speaks indistinctly)
- 02:17This acute-care hospitals, kidney dialysis facilities,
- 02:21I have been working on other evaluations
- 02:25like organ procurement organizations,
- 02:27which is a type of organizations
- 02:31that are responsible for procuring organs
- 02:34for patients who are in great need
- 02:36of organ transplant patients.
- 02:39And the transplant centers, of course, physician, surgeons.
- 02:42So you can see,
- 02:44this includes so many different types
- 02:46of healthcare providers and stakeholders include,
- 02:50say, insurance companies, regulation, government,
- 02:55federal agencies.
- 02:56They're all interested in provider profiling,
- 02:59I will tell you why.
- 03:00Providers is basically who are doing profile evaluations
- 03:06and of course patients.
- 03:08So because they are interested in the information,
- 03:11interested in the profiling results
- 03:14so they can make care seeking decisions.
- 03:17Okay?
- 03:18And so I listed here a few outcomes,
- 03:22like emergency department encounters,
- 03:25unplanned re-hospitalizations,
- 03:29which is hospital readmissions.
- 03:31And I will jump into the details later
- 03:34and post-discharge deaths and you can,
- 03:36I mean there are so many different types of outcomes
- 03:39to consider in provider profiling.
- 03:41And one of the goals was
- 03:43to basically identify those providers
- 03:47with very bad performance in terms
- 03:49of patient-centered outcomes.
- 03:50And they can get penalization,
- 03:55like they can have payment reductions
- 03:58from government agencies.
- 04:01Okay?
- 04:02And as you can see here, this is very important.
- 04:04This is a very important business,
- 04:07and profiling can actually help
- 04:10improve evidence-based accountability
- 04:13for those providers and how facility targeted interventions
- 04:18that aimed at improving the care quality.
- 04:24Alright, so,
- 04:36so,
- 04:40this is a slide of a few example papers
- 04:43that are about evaluating hospitals across the nations.
- 04:47So they're mostly from the program called,
- 04:53Hospital Re-admission Reduction Program,
- 04:55which is a very important national level program
- 04:59that I will explain later.
- 05:01But there are just so many papers in this field.
- 05:05I mean, these are just,
- 05:07like there are publications in top,
- 05:10medical journals, analysts of internal medicine,
- 05:13and New England Journal of Medicine.
- 05:24Okay?
- 05:27So, this is another type of profiling stuff.
- 05:30So it's called physician profiling.
- 05:31Basically they wanna evaluate physicians.
- 05:36So this is, as you can see, it's a report,
- 05:39it's called the health report
- 05:41from Massachusetts Medical Society,
- 05:44which is the publisher
- 05:46of The New England Journal of Medicine.
- 05:47Okay?
- 05:48So they prepared this principles
- 05:50for profiling physician performance, I think many years ago.
- 05:56So this is a list of exemplar profiling programs
- 06:01and they are still existing.
- 06:04So the first one is an interesting state level program
- 06:08which is arguably one of the first programs.
- 06:12So it is still administered
- 06:19by the New York State of Department of Health.
- 06:22Basically they're interested in evaluating hospitals
- 06:25that do coronary artery bypass graft surgeries,
- 06:31and also PCIs and the program have been running
- 06:35for at least 20 years or so.
- 06:39And the second one is another important program,
- 06:41which was launched I think in 2003.
- 06:46And it is,
- 06:49I think it is from the one of the Federal Level Act.
- 06:52And it is currently administered
- 06:54by the US Centers for Medicare and Medicaid Services.
- 06:58And their interest in outcomes for, again,
- 07:0430-day readmissions and mortality for a AMIs
- 07:09and the heart failure, et cetera.
- 07:11And the next one is another federal level readmission,
- 07:17federal level profiling program,
- 07:19which is also established by Affordable Care Act,
- 07:24which is Obama care.
- 07:25You guys probably know that, in 2012.
- 07:29And so, yeah, they're also interested in,
- 07:33evaluating hospitals and they will punish those hospitals
- 07:37with very bad performance in terms of payment reductions.
- 07:40Okay?
- 07:41The last one is an interesting program,
- 07:43which is kind of my focus.
- 07:46I have been working on evaluating kidney dialysis facilities
- 07:54for patients with kidney failure.
- 07:56And there are actually over 7,000 dialysis facilities
- 08:01across the nation, believe it or not.
- 08:04But this is the first to pay for performance program
- 08:09in contrast to other pay for service programs.
- 08:14Okay.
- 08:15And the program is called ESRD.
- 08:17ESRD is short for End Stage Renal Disease.
- 08:21Basically the patients with kidney failure,
- 08:24a quality incentive program, okay?
- 08:26Alright.
- 08:27So as you can see, there are so many programs,
- 08:30so many initiatives across the nation about profiling.
- 08:36And one natural question is about the,
- 08:41how the landscape of the statistical landscape
- 08:44of profiling looks like.
- 08:46And because of the importance of profiling
- 08:49and here I said,
- 08:53there are many far reaching implications
- 08:55because providers can get penalizations
- 08:58and it's high stakes.
- 09:02So it's important
- 09:03that we have principles statistical methods
- 09:05to evaluate them, right?
- 09:08So this is like two examples.
- 09:11The first,
- 09:13it's a paper published on analysts of internal medicine,
- 09:17but it is written by two statisticians.
- 09:21They are calling for the improvement
- 09:25of statistical approach in this field.
- 09:28And also the second one,
- 09:30this one is even more important
- 09:32because it is a white paper issued
- 09:35by the Committee of Presidents of Statistical Society.
- 09:40You probably know about COPS.
- 09:42So one of the most important words in the statistic field,
- 09:47it's the COPS presence of work, right?
- 09:49So this is a white paper by COPS
- 09:53and also a group of people from the CMS.
- 09:58So this is also an important work.
- 10:00It's about the statistical issues
- 10:02and assessing hospital performance.
- 10:05So as you can see,
- 10:07there are many people are interested
- 10:10in improving the statistical landscape for profiling.
- 10:14Alright,
- 10:15so this is a slight briefly introducing the existing methods
- 10:23of provider profiling.
- 10:25There are a few.
- 10:26I grouped them into like roughly four categories.
- 10:31So the first group,
- 10:34is hierarchical random-effects models,
- 10:38there are many papers in this group,
- 10:42but I just highlighted one paper in,
- 10:45I think in 1997 was published on Jassa
- 10:50by Dr. Sharon Lee Norman at Harvard Medical School.
- 10:54So it's about hierarchical random-effects models
- 10:58which is still being used in many settings.
- 11:02Especially, I mean,
- 11:04not sure whether you guys know
- 11:05that there is a group at Yale called Yale Core,
- 11:08I think Center for Outcomes Research and,
- 11:14Something. <v ->Evaluation.</v>
- 11:16<v Dr. Wu>Okay, great, thank you.</v>
- 11:17So they have been using hierarchical random-effects model
- 11:21for over 30 years, I guess.
- 11:24And the second stream of approach is fixed-effects models,
- 11:31as you can tell from the names,
- 11:36people are using like a fixed effects in the models.
- 11:40And this is one example paper,
- 11:44actually was published in 2013 by my advisors.
- 11:49And the next one is,
- 11:53I mean these groups of papers,
- 11:56they're not mutually exclusive because,
- 12:00for example, this one,
- 12:01competing risks or semi-competing risks.
- 12:04I mean there are some papers
- 12:06that use higher hierarch random-effects model
- 12:08or they're also papers using fixed-effects models.
- 12:12But they are just kind of,
- 12:13they're handling like different types of outcomes.
- 12:16So I listened here.
- 12:18And also for recurring events,
- 12:20if you take a class in survival analysis,
- 12:23you probably know that, for example,
- 12:26patient can have multiple hospitalizations in a year.
- 12:29So they are considered as recurring events.
- 12:31Okay.
- 12:32And then the last one is,
- 12:35some people are using causal inference
- 12:37and some clustering approaches to handle profiling issues.
- 12:44But these papers are relatively new,
- 12:47and this is one paper here.
- 12:50It was by all statistics, I think.
- 12:54Alright, so I wanna discuss a few limitations
- 12:59of the current landscape,
- 13:01the current statistical in profiling.
- 13:05So the first limitation is, people have been, I think,
- 13:10intensely using models with a linear predictor.
- 13:15So the limitation is this may not be true
- 13:19when we have very complex outcome
- 13:23and the factor associations.
- 13:25So this is an example.
- 13:28This figure.
- 13:30This is in my one of my papers.
- 13:36So the background,
- 13:38I'll give you a bit of background information.
- 13:41So this is about, okay,
- 13:42evaluating the effect of covid
- 13:46and the outcome is a 30 day unplanned hospital readmissions.
- 13:51So this, on the left is the surface plot.
- 13:54On the right is the conquer plot.
- 13:56As you can see,
- 13:59we are interested in the variation
- 14:02of the covid effect across, this might be too small,
- 14:06but across post discharge time,
- 14:09post discharge days and also across calendar days
- 14:13because we used data in 2020.
- 14:15So we set time zero at, I think mid-March or,
- 14:21yeah, mid-March.
- 14:22So this is April the 1st.
- 14:25And then May 1st until I think mid-October.
- 14:30So as you can see there's a lot of variation going on here.
- 14:36So the covid effect is definitely not constant here.
- 14:38So basically it means that we cannot use the linear model
- 14:44to do this.
- 14:44It's just not valid, right?
- 14:48So the second methodological limitation is existing methods
- 14:54have been historically driven by cost effective spending.
- 14:57Like,
- 15:01I think in the very first program,
- 15:03in those first early programs,
- 15:06people are interested in how to reduce costs
- 15:10by, of course they wanna improve,
- 15:13they wanna improve care quality
- 15:15but cost effectiveness is a very important factor.
- 15:20So,
- 15:22and these analysis,
- 15:22they basically combine all racial ethnic groups together
- 15:25without accounting for their heterogeneity.
- 15:31So this is an another example.
- 15:33So we basically look at the performance
- 15:37of Organ Procurement Organizations, OPOs.
- 15:42So we are interested
- 15:44in organization level transplantation rates.
- 15:49And we have data in 2020.
- 15:53So these are,
- 15:55so on the y-axis we have the normalized OPO IDs,
- 16:03and this is just like a three panels of caterpillar plots.
- 16:07And if we focus on a certain OPO, then,
- 16:12for example, in this panel,
- 16:13this is a panel for white patients.
- 16:16And if you look at this is,
- 16:19I know this is a little bit small,
- 16:21but this is OPO 30 and this,
- 16:23the conference interval is above the national rate
- 16:27for white patients.
- 16:29So it's significantly better than the national average.
- 16:32But if you look at the this panel,
- 16:37this is also OPO 30
- 16:40and we have the confidence interval being lower
- 16:44than the national average for black patients.
- 16:46And this is a panel for Asian Americans
- 16:52and Pacific Islanders.
- 16:53We also have the same issue going on here for OPO 30.
- 16:58So as you can see, there's definitely racial disparity here,
- 17:04but this was never examined in those early programs.
- 17:11So this is an limitation of course.
- 17:15And the last one is,
- 17:17there is a lack of a unifying framework
- 17:20to accommodate different provider profiling objectives
- 17:24and the different performance benchmarks.
- 17:27I will give you like four different examples.
- 17:31The first one,
- 17:34I tried to make the notation very easy.
- 17:37So say we have a random-effects model here.
- 17:42We just consider a binary outcome.
- 17:45Y can be zero or one.
- 17:47Okay?
- 17:48And we basically use the logistic regression, here.
- 17:52So this gamma i, it's a sum of two things.
- 17:56The first one is mu as the mean effect.
- 17:58And the second one is ID normally distributed,
- 18:05a random variable, okay?
- 18:07And we can construct a type of,
- 18:08we call it standardized measure.
- 18:10It's Oi divided by Ei,
- 18:13O is just a sum of all those YIJs.
- 18:17And the Ei is the,
- 18:19basically the sig y function transformation
- 18:23of mu plus beta.
- 18:25Okay?
- 18:27So here, if you look at the model,
- 18:30we have gamma I here,
- 18:31but when we calculate the expected number of events
- 18:35or outcomes, we replace this with the mean.
- 18:40Okay?
- 18:41So this is the first example
- 18:44of course using random effects models.
- 18:46But if we look at the fixed effects model,
- 18:49we have the similar formulation here,
- 18:52but here because this is a fixed-effects model,
- 18:54gamma I is just unknown fixed effect, okay?
- 18:58And if we define gamma,
- 19:02start to be the median of gamma, this is a vector actually.
- 19:05So it's a vector of vault fixed-effects.
- 19:08Then this is basically the median of vault provider effects
- 19:12or fixed effects.
- 19:14And so we can also construct this standardized measure,
- 19:17but this time, this E is defined as this,
- 19:22and this is gamma star.
- 19:26So we basically use the median of all fixed effects
- 19:30to construct the standardized measure.
- 19:33Okay?
- 19:34So now we have two cases.
- 19:36One is, okay, we use the, oops,
- 19:39we use mu, which is the mean of all provider effects,
- 19:44although it's a random effects model.
- 19:46And,
- 19:48here we have median of all fixed provider effects, okay?
- 19:54So these are two cases,
- 19:55basically two types of models that have been used before.
- 19:58And next one is, and some causal papers,
- 20:04they can use a selected set of provider,
- 20:09it could be a single provider,
- 20:11let's say, I'm a a hospital administrator,
- 20:14I wanna see, okay,
- 20:15whether my hospital is performing better or worse
- 20:19than another hospital,
- 20:21then of course I can use my hospital as the benchmark,
- 20:25as the reference and compare all other hospital
- 20:29with my hospital, okay?
- 20:30So this is the first case.
- 20:32We can just choose a single hospital or provider
- 20:36as the benchmark.
- 20:37And the second case is we can group a few providers,
- 20:42hospitals in the specific geographic region together
- 20:45and to form a benchmark, this is also doable, okay?
- 20:49And it is actually used in the paper.
- 20:53The last one is, we can basically treat all hospitals,
- 20:58you can group all hospitals together
- 21:00into a large super hospital, of course,
- 21:02this is a hypothetical one but we can do that.
- 21:06And that is kind of like a national average thing, right?
- 21:10These are all reasonable ways to define a benchmark.
- 21:17And there is the last one.
- 21:19So the last one is kind of more like equity driven thing.
- 21:23So we can form a benchmark such that say,
- 21:26okay, say,
- 21:27from the regulator's perspective,
- 21:29we really wanna push hospitals to improve their performance
- 21:34for minority patients.
- 21:36So say, we can set the benchmark to be something like,
- 21:41okay, for within the minority groups,
- 21:43we can intentionally select patients with better outcomes.
- 21:48We can make the proportion to be very large
- 21:51so that in the benchmark group,
- 21:54we can have a very good performance for minority patients.
- 21:59And then black non-Hispanic patients.
- 22:03So this is kind of a equity driven thing.
- 22:06So as you can see, I give you like,
- 22:11at least the four examples.
- 22:12But these are scattered in the literature
- 22:15and there is no unifying framework
- 22:18to accommodate all of these cases.
- 22:20But we actually can develop a general framework
- 22:25to accommodate all.
- 22:26I will give you the details later.
- 22:30So, all right,
- 22:34so the framework
- 22:36that we proposed is what we termed,
- 22:40a versatile deep learning provider profiling.
- 22:43So we proposed a versatile or probabilistic framework
- 22:50based on the, so-called provider comparators,
- 22:52which is, you can name it as you know, provider comparator,
- 22:56hypothetical provider performance benchmark
- 22:58or population norm.
- 22:59These are all the same interchangeable terms.
- 23:03Okay?
- 23:04Here versatile means, okay,
- 23:06we can use the framework to do a lot of different things.
- 23:10So they are adaptable to different profiling objectives
- 23:14and contexts, okay?
- 23:15It's why we use the term versatile
- 23:18and here provider comparator,
- 23:21which is defined to be a hypothetical reference provider
- 23:28that is corresponding to your profiling objective.
- 23:30So if you have a certain objective,
- 23:32of course you can define your own hypothetical provider.
- 23:37And if you have a different objective,
- 23:39you can define another one, okay?
- 23:42And the deep learning thing comes
- 23:45into play because it is nice that,
- 23:49generally it relaxed the linearity assumption
- 23:51in most existing portfolio models
- 23:55that relies heavily on linear this assumption.
- 23:58Okay?
- 24:00Alright, so this is slide of the basic setup
- 24:07of this new approach.
- 24:09So let's say we have a ID random sample
- 24:13with Y as the outcome,
- 24:17and the Fi star is the provider identifier,
- 24:22and Zi is simply a vector of variants,
- 24:27and they are one from a population Y, F star, Z.
- 24:38And we have the following assumptions
- 24:40that these two assumptions, one and two,
- 24:46so F star.
- 24:47So basically this script F star is the support
- 24:52of this provider identifier, F star.
- 24:56Okay?
- 24:58So we require that this report for any value
- 25:05that this F star can pay,
- 25:07we assume that the probability of F star equal
- 25:11to F is positive,
- 25:13which means that in the dataset,
- 25:15you can at least observe one patient from that provider.
- 25:19Okay?
- 25:20Say if this is zero, then basically it means,
- 25:24okay, we do not observe any patient from that provider,
- 25:27which is useless, right?
- 25:31So the second assumption is simply,
- 25:34okay, so this script F star includes all possible providers,
- 25:41we wanna evaluate.
- 25:42So basically this F star has to fall
- 25:45into this set of values, okay?
- 25:49So that's why it's the probability as equal to one.
- 25:52Okay?
- 25:54So we have two important assumptions,
- 25:58regarding data generating mechanism.
- 26:00So the first one is basically the distribution
- 26:03of this F star.
- 26:05The provider identifier depends on covariate.
- 26:10And this is like, okay, so for a patient,
- 26:14say, I'm a patient, I wanna choose my provider,
- 26:17I wanna choose my hospital,
- 26:19my decision will largely based on,
- 26:21okay, what conditions I have,
- 26:23and what insurance I have, right?
- 26:27And say what is the possible feasible set
- 26:31of hospitals I can choose from?
- 26:34Okay?
- 26:34So these are all covariates
- 26:36that we can include in the model.
- 26:37So basically the F star is the distribution
- 26:41of a star depends on all those covariates
- 26:45which is reasonable assumption.
- 26:48The second one,
- 26:49the distribution of the outcome Y
- 26:51as a function of Z and F star,
- 26:54which means that, okay, the outcome,
- 26:57if I go to the hospital and say I have a certain disease
- 27:03and I got a treatment and whether I feel better
- 27:08or not really depends on, okay,
- 27:10of course, depends on my conditions,
- 27:12and also depends on which hospitals I went to, right?
- 27:17So the distribution is denoted
- 27:20as pi, y, given Z and F star.
- 27:25Okay?
- 27:26So basically these two assumptions gives us the,
- 27:31basically the basic setting for a patient who is looking
- 27:35for care to improve their conditions.
- 27:42So the main idea in this new framework is reclassification.
- 27:48So basically,
- 27:49we wanna construct a hypothetical provider comparator
- 27:54as a performance benchmark
- 27:56that is corresponding to our specific profiling objective.
- 28:01Okay?
- 28:02So reclassification here means that we wanna,
- 28:06we reclassify subjects from existing providers
- 28:11into a hypothetical one
- 28:13following a certain probability distribution.
- 28:15Okay?
- 28:16To do this, we introduced a random indicator,
- 28:19it's just a 0, 1.
- 28:21Which we termed reclassifier.
- 28:24This reclassifier is equal to 0.
- 28:26Here it is kind of different.
- 28:28So reclassifier is equal to zero.
- 28:31When the subject is reclassified
- 28:33into the hypothetical provider,
- 28:35if it is equal to one, then the subject is not reclassified.
- 28:39So the patient stays in their original provider, okay?
- 28:47And with this reclassified redefined, F,
- 28:51so F is different from F star.
- 28:54So F is defined as the product of R,
- 28:57basically R times F star.
- 28:59And we basically add a singleton to this F script F star.
- 29:06So now we can see, okay,
- 29:09so whatever providers we have originally,
- 29:13now we add a single hypothetical provider
- 29:16and we provide the provider indicator,
- 29:21we fix that as zero.
- 29:23So zero is the hypothetical one.
- 29:26So now this F can take values,
- 29:29importantly, it can take whatever values
- 29:31from the original script F
- 29:34but now it can also take values
- 29:37to take the value zero, right?
- 29:40So basically this R is used
- 29:43to manipulate a subject's provider membership.
- 29:46So, a subject from a provider F star equal to F.
- 29:54So here in this case,
- 29:55because it's F star, it cannot be equal to zero, right?
- 29:59So we wanna reclassify patients
- 30:01from a certain existing real provider
- 30:04to that hypothetical provider.
- 30:07You know, this F is equal to zero.
- 30:10So this is a new provider membership for that patient, okay?
- 30:14But if R is equal to zero,
- 30:16then the patient stays in that original hospital.
- 30:20Okay?
- 30:22Alright.
- 30:23We have additional two assumptions
- 30:25regarding this reclassification thing.
- 30:29So the first one is for any provider, real provider,
- 30:35we have this probability, being less than one.
- 30:38This means that, okay,
- 30:40so given a set of covariates and given
- 30:44that the patient is in a certain provider,
- 30:49then the patient being reclassified
- 30:53into the new hypothetical provider,
- 30:56the probability is less than one,
- 30:58which means that we should keep at least a few patients
- 31:03in their original provider
- 31:05so that we can still evaluate the outcome distributions
- 31:10of the original provider, okay?
- 31:13And this actually,
- 31:15if you do some, a simple algebra,
- 31:20we can show that basically this implies that,
- 31:23I mean this, we can basically drop this condition
- 31:26because if you do the sum
- 31:28of the conditional probability thing,
- 31:31you can basically drop this condition
- 31:33and this actually holds.
- 31:35So it's like, okay, no matter which hospital,
- 31:38no matter which provider the patient is in currently,
- 31:42the probability that the patient
- 31:44will be reclassified is less than one.
- 31:46So not all patients will be reclassified, right?
- 31:50And this is the second condition.
- 31:52So combining these two, basically, okay,
- 31:58so basically not all patients can be reclassified
- 32:02or also all patients cannot be living
- 32:08in their original providers.
- 32:10Basically we require that, okay, each patient can,
- 32:15so we should have
- 32:17at least a few patients who are remaining
- 32:20in their original hospitals so that we can evaluate
- 32:22their original outcome distributions.
- 32:25And also we need a, of course characterize the distribution,
- 32:28that hypothetical reference provider.
- 32:31Okay?
- 32:33Alright.
- 32:34Then the last assumption is,
- 32:38this is kind of an interesting setting.
- 32:40So rather than observing the original data,
- 32:44Y, F star, Z, we can only observe this set.
- 32:52So it's R, Y, F, Z, this tuple.
- 32:59So the big difference between these two is,
- 33:02for this one, we know exactly for every patient,
- 33:05we know exactly where they're from,
- 33:08which provider they are in.
- 33:11But for the this one, say if R is equal to 0,
- 33:17F is automatically 0
- 33:18because F is defined as R times F star.
- 33:22So for those patients,
- 33:23we actually don't know where they come from, right?
- 33:28But here we assume
- 33:31that we can only observe post-reclassification data.
- 33:35And this actually is nice,
- 33:37I mean this is not always necessary in the practice,
- 33:42but this assumption actually helps,
- 33:45facilitates the implementation
- 33:50of some certain privacy preserving protocols
- 33:53and data security protocols.
- 33:54If say, okay, we don't want the,
- 33:57because of certain powerful influential providers
- 34:01can actually have a strong influence
- 34:05in policy making.
- 34:07So, because this is capped like confidential,
- 34:11so they actually don't know how we design,
- 34:14how we choose the re-classification scheme.
- 34:18So it can help reduce some unwarranted inference
- 34:25from those very powerful stakeholders.
- 34:29So this is a nice setting,
- 34:32but it doesn't have to be like this in reality.
- 34:36Alright, so now we have four assumptions,
- 34:41important assumptions
- 34:42to regarding the data generating mechanism
- 34:44and to regarding the reclassification scheme.
- 34:48So, the ultimate goals of profiling is
- 34:54to first to evaluate all providers,
- 34:57and then we wanna identify goals,
- 35:00especially with very bad performance
- 35:03and we can take additional actions
- 35:07and so we can, you know,
- 35:10improve their performance in certain way.
- 35:12Okay?
- 35:13But yeah,
- 35:14so this quantitatively or mathematically,
- 35:20we have the two overarching goals.
- 35:22The first one is to harness,
- 35:24to use the post reclassification data,
- 35:29to contrast the distribution of each existing
- 35:35or real provider.
- 35:37F star was the newly defined reference group.
- 35:42So we wanna compare,
- 35:44basically, compare the distribution of these two groups.
- 35:47I mean each of them
- 35:48because we have so many real providers,
- 35:51and we only have a single hypothetical provider, okay?
- 35:54We wanna compare them, we wanna do contrasts.
- 35:57And of course the second goal is
- 35:59to identify those providers with very bad performance.
- 36:06All right,
- 36:08so, this actually,
- 36:14because we introduced this hypothetical provider,
- 36:17this is really nice actually.
- 36:19But there is a difficult issue here
- 36:24because we introduced this hypothetical provider,
- 36:30we actually have to account for
- 36:32or address reclassification dues to bias.
- 36:35So the details are in this proposition.
- 36:39So let's assume that those four assumptions hold
- 36:43and the distribution of the outcome given Z and this F,
- 36:50F is the newly defined provider indicator.
- 36:54We can actually write the outcome distribution,
- 36:58like in two cases.
- 36:59So when F is equal to 0,
- 37:01this is corresponding to the reference,
- 37:05the hypothetical provider.
- 37:07So this is actually the average,
- 37:14you can consider as the distribution of the outcome
- 37:21basically for all patient.
- 37:22If you group all patients together into a single group,
- 37:25this is basically the distribution of that group.
- 37:28Okay?
- 37:29But we have this term here,
- 37:31and this is not necessarily equal to 1,
- 37:34F is equal to 1 then it's very simple,
- 37:38but it could be unequal to 1.
- 37:43And also in the second case when F is not equal to 0,
- 37:48which means that okay, for those existing providers,
- 37:53their distribution also changes because you basically,
- 37:56you move a few patients to the new provider.
- 37:59So the original distribution changes, right?
- 38:02And because we cannot observe this by assumption.
- 38:06So this is basically the observed outcome distribution
- 38:10for existing providers.
- 38:11But according, as you can see here,
- 38:14it's a bias distribution.
- 38:15It's no longer the original one, right?
- 38:17Because this ratio, again,
- 38:18it is not necessarily equal to 1, okay?
- 38:22Right?
- 38:23So as I said,
- 38:26you can consider this as the average distribution,
- 38:29basically as the outcome distribution
- 38:31of the whole patient population, okay?
- 38:33So of course you can write it as a sum of the,
- 38:39you know, weighted probabilities.
- 38:42So the weight being the probability provider membership,
- 38:46and this is basically, okay, within this certain provider,
- 38:50what does the outcome distribution look like?
- 38:53Okay. All right.
- 38:58So a few things.
- 39:03This proposition basically outlines a,
- 39:06what we call design based approach
- 39:08to provider profiling, basically, okay.
- 39:12So,
- 39:14I actually, I mentioned this early,
- 39:17in profiling there are a few different parties.
- 39:20The first one is regulars
- 39:23who initiated the profiling process
- 39:25because they are interested
- 39:26in the performance of these providers.
- 39:28And also we have profilers,
- 39:31which basically evaluates the performance,
- 39:33but they don't have to be the same as regulators.
- 39:36And also we have of course,
- 39:38providers who are the subject of evaluation
- 39:42and we also have patient who need the information
- 39:44to make their decision, okay?
- 39:46So the design-based approach
- 39:47basically tell us that, okay, so, for regulators,
- 39:51they can basically lead the development
- 39:53of a reclassification scene because in this framework,
- 39:57we never say what the distribution,
- 39:59say, what this looks like, where, right?
- 40:02So this is a very general specification
- 40:05and we only made that four assumptions,
- 40:08but we don't have any distributional assumption here.
- 40:12So we can make it very general.
- 40:15And so in this framework,
- 40:19regulators will get more involved in this process.
- 40:24So that's why they can
- 40:25basically design the reclassification scheme
- 40:30based on their specific objectives, okay?
- 40:35Alright.
- 40:35So, and given a specific reclassification scheme,
- 40:41of course they can design their own reference group,
- 40:45their hypothetical providers
- 40:47and having defined this hypothetical provider,
- 40:52profilers of course can use post the reclassification data
- 41:00and also the dependence.
- 41:01Because here, as you can see here,
- 41:03this R actually depends on Y,
- 41:05depends on the outcome covariate
- 41:06and the provider identification.
- 41:10So using this information
- 41:16and also the post reclassification data,
- 41:20profilers that can actually do the profiling
- 41:23and we can use the framework
- 41:26to estimate the probabilities reclassification,
- 41:29which is also the propensity scores actually.
- 41:33So the next step would be
- 41:39to use the estimated propensity scores
- 41:41to correct for reclassification induced bias.
- 41:45And then we can basically construct the distribution
- 41:51of the hypothetical provider with the distribution
- 41:55of the existing provider, okay?
- 42:00Alright.
- 42:01So as sketched in the previous slide,
- 42:05there are a few important things
- 42:07or advantages of the design-based approach.
- 42:11So this approach actually,
- 42:13in this framework,
- 42:14providers can be more involved in this framework.
- 42:21And,
- 42:24so we can use the profiling result,
- 42:28from this new approach can be more relevant
- 42:31to what people are interested
- 42:33in the care decision making process, okay?
- 42:38So, I think I'm a bit over time,
- 42:43but I wanna quickly skim through a few examples.
- 42:48But these examples are basically,
- 42:51we need a few assumptions like
- 42:54whether the reclassifier is depending on the outcome,
- 43:00so in this example, it's very simple.
- 43:02Basically the reclassifier is independent of everything.
- 43:08So,
- 43:09actually this reduces to the most simple case.
- 43:12So nothing changes actually after reclassification,
- 43:15but this is an example about the setting.
- 43:21And we also have like a few non-dependent settings.
- 43:27This R can depend on F star and given F star,
- 43:32it can be independent with Y.
- 43:35And we also have some examples,
- 43:37this is called equal rate representation.
- 43:39We also have singular representation,
- 43:42basically the setting
- 43:43where we only choose a single provider
- 43:47and we also have the case
- 43:49where R actually depends on Y, the outcome.
- 43:54So we can basically choose the outcome,
- 43:56sorry, we can choose patients based on the outcome.
- 44:02And I also give an example,
- 44:04this is actually an interesting example,
- 44:06but seems like we don't have enough time today.
- 44:10So this is the most general case where R is allowed
- 44:13to depend on F and also Y.
- 44:18So we don't have independence anymore,
- 44:20but unfortunately this case,
- 44:22we have the unidentifiability issue.
- 44:26So this case won't work
- 44:27under the post-reclassification data assumption.
- 44:31So we actually developed a framework,
- 44:38we looked at the deep learning methods
- 44:41and the singular representation case.
- 44:44And this is a relatively simple framework.
- 44:47We only consider exponential distribution.
- 44:50I mean the outcome
- 44:51involves the exponential family distribution
- 44:56and we construct a neural network model.
- 45:01So we have the input layer
- 45:02and the fully connected hidden layers and the outcome layer,
- 45:05and we use stratify sampling based optimization algorithm.
- 45:11Here, I will skip the detail.
- 45:14And we developed a exact test based outcome distribution,
- 45:19exact test based approach to identify outlined performers.
- 45:25Okay?
- 45:27And this is basically the motivation
- 45:29why we need deep learning here, because simply speaking,
- 45:32the covid effect is not constant over calendar time
- 45:36and we have to easily account for that
- 45:39while doing profiling,
- 45:40but the effect itself is not of interest.
- 45:48Basically a visualization of the profile results.
- 45:53So here we construct the,
- 45:57we construct what we call the funnel plot here.
- 46:01So the benchmark, the reference, the indicator,
- 46:06we use is again Oi divided by Ei
- 46:09and Ei and defined where this one is the median.
- 46:14And this is actually the neural network part.
- 46:18And we have the funnel plots.
- 46:22So those dots represent providers, okay?
- 46:25So because this, I mean,
- 46:29the higher, the worse the performance,
- 46:31the lower, the better the performance.
- 46:32So these blue dots here are actually better performers.
- 46:38So as you can see, if you add these two supporters up,
- 46:42this is like over 20%,
- 46:47what does not make practical sense
- 46:48because in practice you cannot identify outliers
- 46:52with over 20%, you know, this is too much.
- 46:57So we have to somehow account
- 46:59for provider level unmeasured confounding.
- 47:03And I didn't include the technical details here.
- 47:08But after the adjustment,
- 47:11as you can see the proportion of a better
- 47:14and the worse performers are much lower than before.
- 47:18And I think I only have one more slide.
- 47:23So some takeaways.
- 47:25So profiling is very important
- 47:27as a major societal undertaking in the United States.
- 47:30And we have so many applications,
- 47:33important implications and important consequences as well.
- 47:38And the new framework actually
- 47:41increased the regulators engagement in this process.
- 47:45And it's called versatile
- 47:47because we can handle different profiling objectives
- 47:49and it is compatible
- 47:50with many different model specifications,
- 47:53machine learning models, data science models.
- 47:55And here we use deep learning
- 47:58because it relaxes the linearity assumption
- 48:01and it is often a good idea to account
- 48:05for provider level measure confounding
- 48:08when we do this profiling stuff.
- 48:11And that's all for today.
- 48:15Thank you so much for.
- 48:20I know we only have like two-
- 48:22<v Host>Yeah, We have two minutes.</v>
- 48:23Thank you very much Dr. Wu for your presentation.
- 48:26Any questions in the audience?
- 48:36Anyone online?
- 48:37Just giving everyone a chance.
- 48:40No, I'll ask a question.
- 48:41So I think it's really cool to be able
- 48:45to identify providers who are doing really well
- 48:48or doing bad.
- 48:49What do you do with that?
- 48:51Now that you have that result?
- 48:52Like do you tell the profiler
- 48:54or the patient get to give it to say,
- 48:56"Oh, I don't wanna go to them, they're bad."
- 48:58<v Dr. Wu>Yeah, that's a good question.</v>
- 48:59So actually CMS,
- 49:02they have many programs say, one is for dialysis patients,
- 49:07they have dialysis facility compare,
- 49:10which is an online program.
- 49:12So patient can have access to different types
- 49:15of information like whether diet facility is good or bad
- 49:20and many other different fields
- 49:24of information they have online.
- 49:26So they can choose their favorite providers.
- 49:30Yeah, that's possible.
- 49:32And it's something that is going on, yeah.
- 49:36<v Host>Oh, I think we have questions.</v>
- 49:37<v ->Yep.</v> <v ->Just very briefly,</v>
- 49:39because I know we're out of time but.
- 49:43To what extent do you feel that,
- 49:45if this is true, I guess, and doesn't matter,
- 49:47the patients don't necessarily have meetings.
- 49:51So for example, like I grew up in a rural county,
- 49:55we had one hospital, you were going to a hospital,
- 49:57you were going there.
- 49:58Even in New Haven,
- 50:00there are two campuses of Yale New Haven Hospital,
- 50:02but there's only one hospital in metro area.
- 50:06So, I mean, choice is kind of not a real thing.
- 50:11How does that affect?
- 50:13<v Dr. Wu>Right, that's a very good point, so-</v>
- 50:17<v Questioner>We are actually in city,</v>
- 50:18I understand there's more than one.
- 50:19(Host laughs) Right, there are so many.
- 50:21<v Dr. Wu>Yeah, but that's a very good point</v>
- 50:23because we are actually considering another framework
- 50:27which is also clustering framework,
- 50:30which basically gives you
- 50:32under certain conditions you can choose,
- 50:34there's a feasible set of providers
- 50:36that you can choose from,
- 50:37of course, under certain strengths,
- 50:39say your insurance, your location, many other conditions.
- 50:45But I mean, in this framework,
- 50:49maybe we can address that issue
- 50:53in the set of areas that we included here.
- 50:57But yeah, I mean, you know, very important issue.
- 51:06<v Host>Unfortunately, that's time.</v>
- 51:06So let's thank Dr. Wu again.
- 51:12If you haven't signed in, please sign in before you speak.
- 51:15You are registered.
- 51:17Oh no, it's good, I don't know.
- 51:19(indistinct chattering)