Bo Kim CMIPS Seminar

Transcript

• 00:02Okay. Hi. Hi, everyone.
• 00:05Is
• 00:06there
• 00:08Yes. It's me. Oh, okay.
• 00:10Okay.
• 00:11I didn't even put that
• 00:12slide in. Okay. Alright. Hi,
• 00:14everyone.
• 00:15I'm Ashley Hageman,
• 00:19a core faculty at our
• 00:20Center for Methods and Implementation
• 00:22and Prevention Science.
• 00:24And it's my privilege,
• 00:26and our center is honored
• 00:27to have here with us
• 00:28today Professor Bo Kim.
• 00:29Before I introduce her, I
• 00:31wanna take an opportunity just
• 00:32to sit share a few
• 00:34center updates.
• 00:35So the Center for Methods
• 00:36in Implementation
• 00:38and Prevention Science, which we
• 00:39affectionately call CMIPs,
• 00:41is dedicated to methodologically
• 00:43advancing our field.
• 00:45And the program that I
• 00:46help lead here, the Qualitative
• 00:48Methods Innovation Program, specifically focuses
• 00:51on thinking through how qualitative
• 00:53methods can better better integrate
• 00:55with quantitative
• 00:56advancements,
• 00:58and be responsive to large
• 01:00scale
• 01:00multi level
• 01:02dynamic and diverse sighted,
• 01:04interventions.
• 01:06And then definitely better attend
• 01:07to the workers and the
• 01:08systems and the communities
• 01:10that
• 01:11our implementation interventions are engaging
• 01:13with. And so one way
• 01:14that we do that is
• 01:15through these qualitative methods seminars
• 01:17where we bring in leaders
• 01:18in the field like Doctor.
• 01:19Kim, and also developing and
• 01:21testing new methods within the
• 01:22studies that our center is
• 01:24involved with.
• 01:25And so if you want
• 01:26to be more involved just
• 01:27absolutely send me an email
• 01:29and be in touch. Many
• 01:30of the people working here
• 01:31have
• 01:32massive qualitative datasets that are
• 01:34also integrating into quantitative datasets.
• 01:36And so we'd be we'd
• 01:38be thrilled just to talk
• 01:39to you more about it
• 01:40and to get you involved.
• 01:42And so now I'm eager
• 01:43to hand over the rest
• 01:44of the hour to our
• 01:45guest scholar, Professor Bo Khin.
• 01:47She's an assistant professor of
• 01:48psychiatry at the Harvard Medical
• 01:50School and an investigator with
• 01:51the VA Center for Healthcare
• 01:53Organization and Implementation
• 01:55Research.
• 01:57She's an interdisciplinary
• 01:58scholar that has a really
• 01:59unique and interdisciplinary
• 02:01background,
• 02:02which maybe I'll talk about.
• 02:04She holds a PhD in
• 02:05electrical engineering and computer science
• 02:07with expertise in systems engineering
• 02:09that she's since applied
• 02:11to health systems engineering, which
• 02:13is really a foundation of
• 02:14of implementation practice and and
• 02:16implementation science.
• 02:18She's has more than a
• 02:19decade of experience and expertise
• 02:21with health services research
• 02:23particularly anchored in her complex
• 02:24work for the Veterans Administration.
• 02:27And her current grants
• 02:29are focused on implementing
• 02:30for example, a peer intervention,
• 02:33for veterans that are leaving
• 02:34incarceration
• 02:36to improve reentry support,
• 02:38multiple high level mental health
• 02:39care systems improvement projects to
• 02:41enhance quality and uptake of
• 02:42mental health services,
• 02:45EHR,
• 02:46electronic health record systems improvements,
• 02:48youth drug use prevention, and
• 02:49many, many more.
• 02:51And she has diverse expertise
• 02:53that's really thoughtfully engaged,
• 02:55really big qualitative
• 02:57and quantitative datasets,
• 02:59which is why we have
• 03:00her today.
• 03:01And so we look forward
• 03:02to her talk titled matrix
• 03:04multiple case study, a systematic
• 03:06mixed methods approach
• 03:08to examine
• 03:09context and mechanisms of action
• 03:11that influence implementation
• 03:12outcomes. So thank you again
• 03:14and welcome.
• 03:19Thank you so much for
• 03:20that wonderful introduction.
• 03:23I feel that all the
• 03:24projects and things like that
• 03:26that you've mentioned are only
• 03:27possible through great collaborations that
• 03:29I've been able to have
• 03:29with colleagues,
• 03:31who are leading those important
• 03:32projects.
• 03:33So I look forward to
• 03:35having this screen start to
• 03:36ways in which we might
• 03:37be able to look for
• 03:38collaborations in the future as
• 03:40well.
• 03:40So thank you again for
• 03:41the introduction.
• 03:43So as we are getting
• 03:44started here see whether I
• 03:46can move the slide.
• 03:51Wanted to say a special
• 03:52thanks to the codevelopers
• 03:54of the NMCS method that
• 03:56I'll talk about today,
• 03:57especially at the VA's
• 03:59behavioral health quality enhancement research
• 04:01initiative or query program. And
• 04:04I also, of course, wanted
• 04:05to thank colleagues to the
• 04:06society with society for implementation
• 04:09research collaboration,
• 04:10implementation research institute,
• 04:12as well as various query
• 04:14networks without which really does
• 04:16work. And anything else that
• 04:17I do in implementation science
• 04:19would not be flexible.
• 04:21Then just before we get
• 04:23started, one more slide. In
• 04:24terms of I have no
• 04:25conflicts of of interest to
• 04:27report, and, of course, the
• 04:29views to be shared today
• 04:30are all my own.
• 04:33So I would like to
• 04:35see a raise of hands
• 04:36actually for those of you
• 04:37in the room.
• 04:38When you introduce yourself, how
• 04:40many of you say that
• 04:40you're an implementation scientist?
• 04:44There are quite a few
• 04:44hands here, for
• 04:46those of you online just
• 04:48reporting out. So it looks
• 04:50like it's very possible that
• 04:51many of the things that
• 04:52I may be discussing are
• 04:53familiar to you already and
• 04:55maybe topics that you think
• 04:56about a lot. So I'd
• 04:57really love to hear your
• 04:59your thoughts on what is
• 05:00to be shared. If we
• 05:01can leave some time for
• 05:02discussion, that'll be really great
• 05:03too. And that, of course,
• 05:04goes to those of you
• 05:06joining online as well. I
• 05:08will try to leave time
• 05:09for being able to check,
• 05:11Chad or other ways in
• 05:12which you might be able
• 05:13to connect with the room
• 05:13as well.
• 05:15So for implementation science
• 05:17to be useful, I think
• 05:18there are many,
• 05:20stories out there as to
• 05:21how to make that happen.
• 05:22But when it comes down
• 05:23to it, I think it
• 05:24is about making sure that
• 05:25the science is advancing knowledge
• 05:28that can make
• 05:29implementation
• 05:30go well. So I think,
• 05:31ultimately, that's it. And I
• 05:33think, the questions that our
• 05:35teams that I'm working with
• 05:36are interested in answering are,
• 05:38So in order to do
• 05:39that, what works? And then
• 05:41taking one step beyond that,
• 05:43what works and for whom
• 05:44and how since the same
• 05:45thing may not necessarily work
• 05:48in all different cases.
• 05:50So that really is about
• 05:52asking the question of what
• 05:54happens, but more than just
• 05:55on average. So because there
• 05:58may be so much variability
• 05:59when it comes to resources
• 06:01or organizational
• 06:02culture,
• 06:03population needs, etcetera.
• 06:04It, once again, is unlikely
• 06:06that the same kind of
• 06:07ways of going about innovation
• 06:09is going to work in
• 06:10all places. So thinking about
• 06:12what works and having analytic
• 06:14methods to be able to
• 06:15look into
• 06:16what works for whom and
• 06:17how is a big interest
• 06:18of mine and look forward
• 06:19to discussing that with you
• 06:21today.
• 06:24So just a quick glance.
• 06:25We'll we'll actually go through
• 06:27the steps of the MMCS,
• 06:28together today. But what MMCS
• 06:31can help explore are things
• 06:32like what are the factors
• 06:33that can drive implementation success
• 06:35and challenges,
• 06:36and how do implementation outcomes
• 06:38differ across different contexts. So
• 06:41this method can be helpful
• 06:42for looking at these types
• 06:43of questions.
• 06:44And in introducing this method,
• 06:46I think at the core
• 06:47of it is
• 06:49a process where it helps
• 06:50to systematically
• 06:51analyze multiple sources of data,
• 06:54multiple contexts of data, and
• 06:56then it uses, as a
• 06:58structure, matrices to be able
• 07:00to identify patterns, put them
• 07:02together,
• 07:03the ability to increase the
• 07:04dimensions that we look at,
• 07:05etcetera.
• 07:08So in terms of starting
• 07:09with MMCs, we sort of
• 07:11were talking about it a
• 07:12minute ago.
• 07:13There's a lot of variability.
• 07:15We think oftentimes we refer
• 07:17to an example like, oh,
• 07:18implementing an innovation
• 07:20in a rural clinic versus
• 07:22implementing it in a large
• 07:23urban hospital might be very,
• 07:25very different with the context
• 07:26being different. And so there
• 07:28may be very different challenges
• 07:29you can imagine as well
• 07:30as different drivers of success
• 07:32as at each of those
• 07:33sites.
• 07:33What might be interesting to
• 07:35think about, and this is
• 07:35just a thought question for
• 07:37all of us, is actually
• 07:38we think it will be
• 07:39different, but there may be
• 07:40cases in which it may
• 07:41not be actually that different
• 07:42either. And we actually don't
• 07:44know until we look depending
• 07:45on the type of innovation
• 07:46or depending on type of
• 07:48resources, type of people involved.
• 07:50Maybe there isn't as very
• 07:51much variability as expected. So
• 07:53what's expected? How do you
• 07:55look at it? What's unexpected?
• 07:57But how can we uncover
• 07:58that? Those are all part
• 07:59of things that MMCs is
• 08:01interested.
• 08:03So one gap that MMCs,
• 08:05I think, tries to fill.
• 08:06So, of course, there's a
• 08:08lot of multisite
• 08:09studies going on. Looking at
• 08:11variation
• 08:12in context in and of
• 08:13itself may not be that
• 08:15rare. Right? So we try
• 08:16to understand how context
• 08:18differ. I think in a
• 08:19way, what might be a
• 08:20little bit less common
• 08:22is how those different contexts
• 08:24may be directly linked in
• 08:26a traceable way to specific
• 08:28sort of phenomena of interest
• 08:30when it comes to implementation.
• 08:32So, for example, let's say
• 08:33implementation success is what we're
• 08:35interested in. There may be
• 08:36varying levels of it. How
• 08:38do we actually do this
• 08:39investigation to really link all
• 08:41the beta pieces with many
• 08:42different pieces together in a
• 08:44visible, traceable way,
• 08:46in a way that can
• 08:47perhaps be navigatable between different
• 08:50projects as well? So that's
• 08:51sort of what MCS is
• 08:52trying to do in this
• 08:53case.
• 08:54In doing so, MCS tries
• 08:56to also pull on different
• 08:57sources of data as much
• 08:59as possible.
• 09:00So, examples of this may
• 09:02be between our platform
• 09:03interviews.
• 09:04They can come from different
• 09:06documents, different metrics,
• 09:07maybe quantitative as well. And
• 09:09then observational data too. So
• 09:11this is not an exhaustive
• 09:12list by any means, but
• 09:13it has the ability to
• 09:14be able to work with
• 09:15different types of data. So
• 09:17then that allows us to
• 09:19look at the process of
• 09:20implementation in a pretty comprehensive
• 09:22way. And once again, it
• 09:24is about making visible all
• 09:25the different players or influencing
• 09:28factors on being able to
• 09:29do that in a structured
• 09:31way is the goal.
• 09:35So another thing that NOCS
• 09:37can be helpful with is
• 09:38to mention different players, right,
• 09:40or different influencing factors. What
• 09:42might be the interactions between
• 09:43them? When is it that
• 09:45both teams need to be
• 09:46together to work well? Whereas
• 09:48depending on the context, maybe
• 09:50one is enough or maybe
• 09:51even if both exist, it
• 09:52doesn't work. So for example,
• 09:54factors like leadership support for
• 09:55an implementation
• 09:57exists, or there is enough
• 09:58patient engagement that may suffice
• 10:00in some cases and may
• 10:01not. So being able to
• 10:03sort of understand what are
• 10:04those cases so that, eventually,
• 10:06the goal could be that
• 10:07you can tailor strategies in
• 10:09a way that's meaningful for
• 10:11different contexts if we know
• 10:13if we have the data
• 10:14and the evidence to know
• 10:15that different types of emphases
• 10:17work better in different situations.
• 10:20So overview of MMCs. So
• 10:22as I, in recent years,
• 10:24had opportunities to be able
• 10:25to discuss this with different
• 10:27groups and being able to
• 10:28share our teams now, I've
• 10:30tried out different ways of
• 10:31doing this. First of all,
• 10:32saying all the steps first
• 10:33and then putting an example
• 10:35trying to put an example
• 10:37together.
• 10:38So today, I'm going to
• 10:39do the latter where we'll
• 10:40walk through the step using
• 10:42kind of an illustrative example
• 10:43of a project. So you
• 10:45all can please tell me
• 10:47if that does not work.
• 10:49I'll try to hone it
• 10:50for next time. But today,
• 10:51we'll give that one a
• 10:52try and see how it
• 10:53goes.
• 10:54So the particular evidence based
• 10:56practice that I'll use for
• 10:58this example is a collaborative
• 10:59chronic care model.
• 11:01So we'll use this as
• 11:02an example. And this care
• 11:03modeling is is an evidence
• 11:05based care model that has
• 11:06six different elements that you
• 11:08see up here. So it
• 11:09has this full workflow redesign
• 11:12where it is really about
• 11:14redesigning work goals of an
• 11:15interdisciplinary
• 11:16team so that they can
• 11:17better deliver anticipatory care, coordinated
• 11:20care that is really meeting
• 11:22the needs of the patients.
• 11:23There's another element of veteran
• 11:25or patient more widely, self
• 11:28management support, and that is
• 11:29so that patients can be
• 11:31striving for their well-being even
• 11:33if it's it's not during
• 11:34their health care appointments, let's
• 11:36say. And then there's an
• 11:37element of provider decision support,
• 11:39making sure that they have
• 11:40access to clinical guidelines, specialty
• 11:42expertise, etcetera.
• 11:44And then being able to
• 11:45really leverage information management systems
• 11:47as well so that outcomes
• 11:49can be tracked not just
• 11:50individually, but at registry levels.
• 11:53And, also, measurement based care
• 11:54can be an accurate as
• 11:55well. And then there's community
• 11:57engagement.
• 11:58No particular
• 11:59single setting of care can
• 12:01possibly meet
• 12:02all the needs of an
• 12:04individual. So understanding that how
• 12:06do you make sure that
• 12:07the care can be linked
• 12:08to other resources that are
• 12:09outside? So that's the element.
• 12:11And underlying all of that,
• 12:12as you see on the
• 12:13bottom, is the sixth element.
• 12:15But we numb we number
• 12:16it one because we probably
• 12:17think it's the most important
• 12:19is to have, of course,
• 12:20organizational and leadership support. So
• 12:22that is just like a
• 12:24quick overview of what this
• 12:26care model evidence based care
• 12:28model that this example was
• 12:29trying to implement was. So
• 12:31just in one slide,
• 12:33it summarizes the trial, in
• 12:35terms of how we worked
• 12:37together with the VA's office
• 12:39of mental health and suicide
• 12:40prevention. Now they are actually
• 12:41two separate offices. So this
• 12:43was back then, they were
• 12:44the OMHSP,
• 12:45and, we worked at nine
• 12:47different
• 12:48VA medical centers. They're specifically
• 12:50their, general mental health clinics
• 12:52to implement an interdisciplinary
• 12:54team based care model that
• 12:55is based on the CCM
• 12:57or the collaborative chronic care
• 12:58model.
• 12:59The design that we used
• 13:00then was a step twenty.
• 13:01We randomized the nine different
• 13:03sites to three different start
• 13:04times and then the implementation
• 13:07strategy. So the thing we
• 13:08did to be able to
• 13:10implement the CCM was external
• 13:12internal implementation facilitation.
• 13:15Some of you may be
• 13:15very familiar with that already.
• 13:17I understand that there are
• 13:18some students in the room
• 13:19too. So just very briefly
• 13:24so very briefly,
• 13:25we had,
• 13:26external
• 13:27facilitators with knowledge about the
• 13:28CCM as well as sort
• 13:30of process improvement
• 13:32expertise
• 13:33come and join together with
• 13:34an internal facilitator
• 13:36at the site, knowledgeable about
• 13:38the site's culture,
• 13:39about the way in which
• 13:40things are done, their norms,
• 13:41etcetera. So together, they would
• 13:43support the implementation
• 13:44of this missing model.
• 13:47So they received this as
• 13:49sort of the intervention
• 13:50of the implementation, and then
• 13:51the technical assistance,
• 13:53was given to them as
• 13:54the waiting condition.
• 13:56So that was just one
• 13:58slide on that trial, and
• 13:59this is just one slide
• 14:00on what we found because
• 14:01we will come back to
• 14:02it if needed, but for
• 14:03now. So we saw some
• 14:04aspects of team functioning improve.
• 14:06So the interdisciplinary
• 14:08eighteen. They had better role
• 14:09clarity. They were able to
• 14:10have better team primacy in
• 14:12terms of putting teams' goals
• 14:13ahead of their individual goals.
• 14:15We also saw some increase
• 14:17in the several of the
• 14:18CCM elements, the six different
• 14:20elements in terms of the
• 14:21workload redesign,
• 14:22self management support, and so
• 14:24we were able to see
• 14:25that. It's interesting and very
• 14:27exciting to see reduction in
• 14:28mental health hospitalizations
• 14:30actually of the patients that
• 14:31were being treated by the
• 14:32teams that we were working
• 14:34with versus teams at actually
• 14:36the same medical center who
• 14:37were not undergoing the intervention
• 14:39from our end. And then
• 14:40it was also, of course,
• 14:41exciting to see all cause
• 14:43mortality
• 14:44decrease as well. Now the
• 14:46bold faced,
• 14:47I just pulled over here.
• 14:49So,
• 14:50there was no evidence that
• 14:52across the board at all
• 14:53the different sites, the gains
• 14:55were maintained.
• 14:56And that's why we decided
• 14:58to put this as the
• 14:59implementation
• 15:00phenomenon of interest to us.
• 15:02So sustainability.
• 15:03Right? So
• 15:04why and how sustainably differed
• 15:07across the different sites. And
• 15:09so we use this as
• 15:09the MFCS. Can you just
• 15:11give us a quick sense
• 15:12of timeline? So how long
• 15:13were you facilitating
• 15:15facilitating and then how long
• 15:17would did you consider sustainment?
• 15:19Yes. So in this trial's
• 15:21case, we had one year
• 15:23of, facilitation, active facilitation.
• 15:26The first six months of
• 15:27it was definitely
• 15:28a much more structured,
• 15:30facilitation, and then the second
• 15:32half tapered,
• 15:33to meet the site's needs.
• 15:35And then when we looked
• 15:36at how long it had
• 15:37lasted, it was two, three
• 15:38years down the road to
• 15:40see whether the CCM practices
• 15:42was still going on.
• 15:44And so that's when we
• 15:45realized that yes and no
• 15:47and sometimes. That's that's big.
• 15:48Yeah. Thank you.
• 15:51Right. So we wanted to
• 15:52see sort of what were
• 15:53the different factors that played
• 15:55a role here. And then,
• 15:56importantly, we wanted to know
• 15:57this because we were getting
• 15:59ready specifically for the subsequent
• 16:01CCM trial, which I'll talk
• 16:03to you about, which we
• 16:04just finished wrapping up the
• 16:06data collection for the main
• 16:07outcomes pieces
• 16:09yesterday.
• 16:10So
• 16:11we're very excited. Well, still
• 16:12a lot to do since
• 16:13it's looking at sustainability. It'll
• 16:14keep going on, but, so
• 16:16we had to design this
• 16:17trial. So we wanted to
• 16:18know if we're designing a
• 16:19trial so that we can
• 16:20be doing better at sustainability,
• 16:22we should really learn about
• 16:23what is mattering when it
• 16:25comes to sustainability.
• 16:27So here's the nine step,
• 16:29process. So first, we
• 16:32established the evaluation goal. Be
• 16:33very explicit about this. This
• 16:35was to identify how and
• 16:36why sustainability
• 16:37differed at the different sites.
• 16:39The second step is that
• 16:41we wanted to define what
• 16:42are we going to consider
• 16:43to be sustained
• 16:45CCM. So this is about
• 16:45continued existence of CCM practices
• 16:48practices along the lines of
• 16:49those six different
• 16:51elements that I mentioned earlier.
• 16:53And then the third step
• 16:54was to select
• 16:55relevant domains of factors to
• 16:57look at.
• 16:58There can be so many
• 17:00different things that matter. And
• 17:02I think when it comes
• 17:03to thinking about theories or
• 17:05models or frameworks and choosing
• 17:06one to be able to
• 17:07help with this is for
• 17:09that very reason. Now we
• 17:10keep put some boundaries, at
• 17:11least, around where we're getting
• 17:13started in looking at the
• 17:14relevant factors.
• 17:24Implementation
• 17:25of an innovation
• 17:26to be used by the
• 17:28recipients within a context
• 17:31is activated by the facilitation.
• 17:32And this facilitation being both
• 17:34facilitators, like the people who
• 17:36are doing the implementation support
• 17:38Mhmm. As well as sort
• 17:39of process
• 17:40of facilitating and providing support
• 17:42for the implementation.
• 17:43So that's sort of the
• 17:44different domains of factors we
• 17:46wanted to look at for
• 17:47this.
• 17:48That's one through three.
• 17:49Now step four, we wanted
• 17:51to gather data, of course,
• 17:53on sustainability.
• 17:54In this particular case,
• 17:56we focused specifically on interview
• 17:58data with the different providers,
• 18:00the general mental health care
• 18:01providers at the different sites.
• 18:03We were asking about CCM
• 18:05practices that were still going
• 18:07on as well as different
• 18:08factors that may be impacted.
• 18:10So we then analyzed that
• 18:12information
• 18:13qualitatively to understand sort of
• 18:14the extent of CCM practices
• 18:16that are still remaining as
• 18:18well as factors under the
• 18:19different IFRS domains that seemed
• 18:21relevant.
• 18:23Yes?
• 18:25So I know there's at
• 18:26least one, like, validated
• 18:28sustainability
• 18:29or sustainment
• 18:30tool. Do you know what
• 18:31I'm talking about? Yes. I'm
• 18:32wondering if you considered using
• 18:34that or why you didn't
• 18:35use it.
• 18:36Yes. At that point, we
• 18:38did not use it.
• 18:39I think that was not
• 18:40part of sort of the
• 18:41data collection that had been
• 18:43planned at that point in
• 18:44time. So the timing of
• 18:46this also was when we
• 18:47were, I think, beginning to
• 18:49pay more attention to kind
• 18:51of measurable ways of going
• 18:52forward with sustainment. So I
• 18:54think right after this ended,
• 18:56we realized, and we mentioned
• 18:57in the paper associated with
• 18:59this as well, well, one
• 19:00of the ways in which
• 19:00we would would do this
• 19:02in a different way or
• 19:03for the future would be
• 19:04to have quantitative
• 19:05measures accompany what we are
• 19:07seeing qualitatively for sustainability.
• 19:10Thank you for that question.
• 19:13So,
• 19:16in terms of, the next
• 19:18step, we wanted to assess
• 19:19the extent of sustainability. Therefore,
• 19:21again, in this case, it
• 19:22was done in terms of
• 19:24based on the the qualitative
• 19:25information, but definitely doable using
• 19:28different measures here, hence, kind
• 19:30of the ways in which
• 19:30different, types of metrics can
• 19:32be used for MMCs.
• 19:34And then in this case,
• 19:35it turned out, and this
• 19:36is part of the results
• 19:37too, we turned out that
• 19:39three sites each
• 19:41were high, medium, or low
• 19:42levels
• 19:43of sustainability
• 19:44in that case. And that's
• 19:45sort of the difference that
• 19:46we wanted to see in
• 19:47terms of kind of the
• 19:48phenomenon that we're focusing on,
• 19:50along which we would want
• 19:51to look at the factors
• 19:52that differed. When When we
• 19:54went then ahead to step
• 19:55six of identifying those relevant
• 19:57factors that could be driving
• 19:59this, we found the twenty
• 20:01yes. How did you code
• 20:03in step five? Isn't that
• 20:04sort of nontrivial to take
• 20:06all that qualitative data and
• 20:08sum it up in terms
• 20:09of high, medium, low?
• 20:11It is not trivial at
• 20:12all. So the because the
• 20:14resource intensity is definitely something
• 20:16we'll get to for sure.
• 20:17Yeah.
• 20:19One thing that was helpful,
• 20:20and I'll mention this again
• 20:21later too, is that MMCS
• 20:23is able to take advantage
• 20:25of, you know, already planned
• 20:27qualitative,
• 20:28analyses or quantitative analyses
• 20:31and pull together
• 20:32into a comprehensive
• 20:34matrix and be able to
• 20:35look at what is going.
• 20:36So it's not necessarily that
• 20:38teams that even outside of
• 20:39this project that I've been
• 20:40working with or have been
• 20:41consulting with, it's not that
• 20:43they,
• 20:44have to collect necessarily
• 20:46additional data just for the
• 20:47MCS part. But let's say
• 20:49they were planning a mixed
• 20:50method study already, planning qualitative
• 20:52analysis, which is not trivial
• 20:55at all. Already, we're able
• 20:57to pull that information together
• 20:59in infrastructure format to be
• 21:00able to look at it
• 21:01in a comprehensive way. So
• 21:02thank you for that question.
• 21:04So nontrivial
• 21:05step five, although it's half
• 21:07of one slide,
• 21:09and then definitely a nontrivial
• 21:11six too because that was
• 21:12a qualitative look at the
• 21:13different factors as well. So
• 21:15we had these relevant factors
• 21:16that showed up across the
• 21:17different sites that we were
• 21:18looking at, and that's where
• 21:20we kind of bring in
• 21:21the matrix. So I'm going
• 21:22to not sure how best
• 21:24to point on the screen
• 21:25here for our online,
• 21:26members.
• 21:27But as you can see
• 21:28here, we can think about
• 21:30one site where we are
• 21:31doing this work being one
• 21:33sort of, in this case,
• 21:34a sheet, a spreadsheet.
• 21:36And then we would align
• 21:37kind of each influencing factor
• 21:39with exactly which data source
• 21:41they are coming from and
• 21:42be able to lay out
• 21:44exactly where the data is,
• 21:46what we're looking at, with
• 21:47whom, and for whom. And
• 21:48that way, sometimes there's going
• 21:50to be missing data, which
• 21:51I think often times is
• 21:53difficult to trace back to.
• 21:54This is one way in
• 21:55which we can really understand
• 21:56what are the things that
• 21:57are going into deciding when
• 21:59we say,
• 22:01a factor was relevant in
• 22:03enabling of implementation or hindering
• 22:05the implementation, etcetera. So we
• 22:07organize the data in this
• 22:08way. You can kind of
• 22:09see the different sites making
• 22:11up sort of the into
• 22:12the board, dimension of the
• 22:14matrix here.
• 22:15And then what we do
• 22:16using that Can I ask
• 22:17a question? Yes.
• 22:19So I'm just trying to
• 22:20think about it with the
• 22:21data filled in. So we
• 22:22do sort let's say the
• 22:23influence factor was
• 22:25number of staff. Would you
• 22:26show x's for the data
• 22:28source, or would you put
• 22:29the actual data source in
• 22:30the browser? Yeah. So to
• 22:32start, we would put the
• 22:33actual data.
• 22:35But that's sort of the
• 22:36those are the steps that
• 22:37the research team decides Okay.
• 22:39How to abstract the data.
• 22:40Right? So at what level
• 22:42do we sort of,
• 22:43reduce it down to what
• 22:44we want to be looking
• 22:45at? So if it was
• 22:46a quantitative measure of, let's
• 22:48say,
• 22:49satisfaction provider satisfaction, then that
• 22:52quantitative value can be what
• 22:54goes into that particular,
• 22:56cell from that data source
• 22:58where we gathered that information.
• 22:59If it's qualitative information, then
• 23:01it's where the choice is.
• 23:03Either,
• 23:03we're able to put in,
• 23:05what the data is indicating
• 23:07as to that satisfaction,
• 23:08or if there is one
• 23:10step to be additionally taken,
• 23:11we will put a summary,
• 23:12let's say, across what we're
• 23:13learning into that or a
• 23:15particular indicator of what that
• 23:17might be that the team
• 23:18comes up with. So kind
• 23:19of the app the number
• 23:20of sort of abstraction level,
• 23:22I guess that's a big
• 23:23point of discussion and team
• 23:25sort of decision that that
• 23:26needs to come to. So
• 23:27the way you did it
• 23:28is the data source, would
• 23:29that be the participant you
• 23:30interviewed? And then you put
• 23:31the sort of quote back.
• 23:33So
• 23:34in this study's case, yes.
• 23:36So because we had different,
• 23:37people that we were interviewing,
• 23:39for the different sites. Now
• 23:40there are multiple data source,
• 23:42multiple types of data source
• 23:43studies that we've used this
• 23:45for too. So they can
• 23:46be, for example, all employee
• 23:48survey data
• 23:49might be indicating
• 23:50that the average percent of
• 23:52satisfaction might be. Then that
• 23:53would be a data source
• 23:54that put in there as
• 23:55well. Oh, gosh. It's flexible.
• 23:57Exactly. Exactly. It's flexible. Another
• 23:59point I hope to return
• 24:00to because I would love
• 24:01to discuss with you all
• 24:03how to
• 24:03better use that point as
• 24:05well moving forward.
• 24:06So using this kind of,
• 24:07data,
• 24:09step eight is to do
• 24:10first looking at sort of
• 24:12within site,
• 24:14information.
• 24:14So because there are multiple
• 24:16data sources, ideally, from which
• 24:18we're drawing, we wanna focus
• 24:19on,
• 24:20what within data the different
• 24:22data sources are saying about
• 24:23sort of the way in
• 24:24which that factor might be
• 24:25hindering
• 24:26or enabling,
• 24:28implementation.
• 24:29And then once we designate
• 24:30that, then we are ready
• 24:32with that field to be
• 24:33able to then look
• 24:35cross site wise as to
• 24:37if there are sites at
• 24:38which implementation, in this case,
• 24:40sustainability
• 24:41has gone successfully, less successfully,
• 24:44etcetera.
• 24:44How do their sort of
• 24:46influencing factors and the extent
• 24:47to which they were existent
• 24:49differ? So going back briefly
• 24:51to this slide, we would
• 24:53first look kind of horizontally,
• 24:55just kind of visualizing,
• 24:56right, to look at it
• 24:57site wise. And then once
• 24:59we're able to determine what
• 25:00site wise analysis looks like,
• 25:02we're then able to have
• 25:03a structure that's set up
• 25:04so that you can look
• 25:05in a way systematically
• 25:07across the sites as well.
• 25:09And once again, it's about
• 25:10kind of understanding where the
• 25:11cells are, how they are
• 25:13filled, which ones are actually
• 25:14linking versus not. So that's
• 25:15the advantage of being able
• 25:17to structure in this way.
• 25:18Once again, traceability,
• 25:20documentation,
• 25:21and being able to trace
• 25:22back.
• 25:25So we're working on kind
• 25:26of different ways in which
• 25:27we wanna visualize what we
• 25:28eventually come to. And this
• 25:30sort of goes back to
• 25:30what you were saying. So
• 25:31this was a quite raw
• 25:32matrix that we dealt with
• 25:34here. So how we actually
• 25:35put that together into a
• 25:37visualization that actually makes sense
• 25:38is something that we're continuing
• 25:40to work on. For our
• 25:41publication on this particular,
• 25:43project, this is kind of
• 25:45what we came to in
• 25:46terms of wanting to sort
• 25:47of organize by, the framework
• 25:49that we used, the IPRIS
• 25:51domain, the different influencing factors
• 25:53that seem to pop out
• 25:54more than others, and then
• 25:56grouping sort of by how
• 25:57the low sites did, medium
• 25:59sites did, high sustainability sites
• 26:01did, and being able to
• 26:03pictorially,
• 26:04if possible, indicate sort of
• 26:05the direction in which or
• 26:06the how strongly a certain
• 26:08factor was indicated.
• 26:09This is an evolution,
• 26:11I must say. So I
• 26:12think part of what I
• 26:13would love to hear everyone's
• 26:14thoughts on and going forward
• 26:15after today too is to
• 26:16think together about visualization and
• 26:18ways to make this really
• 26:19more useful too. But this
• 26:20is what is in our
• 26:21paper for this particular study.
• 26:24Some of the takeaways are
• 26:26So could would you mind
• 26:27going back? Oh, sure. So
• 26:29when I saw that matrix,
• 26:30it seemed to me I
• 26:31thought, oh, this is a
• 26:32way of turning the qualitative
• 26:34data into
• 26:35quantitative data so you can
• 26:36actually run a regression
• 26:38where the dependent variable could
• 26:41be this three level sustainability,
• 26:43and then the independent variables
• 26:45are all the different factors
• 26:46in the rows. And then
• 26:48you have, you know, it's
• 26:49Yeah. You have a random
• 26:50effect for Yes. Facility
• 26:52or a fixed effect for
• 26:54facility, and,
• 26:55you could do all of
• 26:56that rather than this, which
• 26:58is essentially a univariate
• 26:59analysis.
• 27:01So I think there are.
• 27:02That is very true. And
• 27:03I I wanted to actually
• 27:04get to talk about sort
• 27:06of the complementarity
• 27:07of MMCs with existing methods
• 27:09too. Because I think some
• 27:10of the structuring, some of
• 27:12the readying the data, some
• 27:14of, again, putting it in
• 27:15this format, I think can
• 27:16be very complimentary
• 27:17with many different ways in
• 27:19which that next analysis is
• 27:21going to be done to
• 27:22make sense of the data.
• 27:23In this case, we did
• 27:25it in this particular way
• 27:26because we were trying to
• 27:27stay true to kind of
• 27:28the qualitative,
• 27:30school of thought of being
• 27:31very careful with quantifying some
• 27:33of the data to be
• 27:34able to claim that it
• 27:35indeed is one variable being
• 27:37able to represent something in
• 27:39a rep in a statistically
• 27:41representative way. So that's why
• 27:43in this case, we decided
• 27:44to really go with being,
• 27:46explanatory, being able to find
• 27:47examples of what these cases
• 27:49could be, but not claiming
• 27:51that this may be a
• 27:52frequent or a statistically representative,
• 27:55way of being able to
• 27:56show. So this was a
• 27:57show of possibilities, I guess,
• 27:58in that sense. So yes.
• 28:00But I I love what
• 28:01you mentioned. Yes. That's exactly
• 28:02kind of what we want
• 28:03want to be able to
• 28:04do. We want to see
• 28:05where and which parts of
• 28:06these steps of MMCs can
• 28:08be really helpful for other
• 28:09analytical steps too in terms
• 28:11of preparation
• 28:12and structure, etcetera.
• 28:15So some of the things
• 28:16that came through are things
• 28:18like,
• 28:19staff and leadership turnover. They
• 28:20were present everywhere. I mean,
• 28:22it was kind of a
• 28:23ubiquitous issue,
• 28:24and hindering about to be
• 28:26hindering. This collaborativeness
• 28:27and teamwork piece, it was
• 28:29present present and enabling at
• 28:31a lot of the high
• 28:31and medium sustainability sites, and
• 28:33it was not enabling. It
• 28:35was more present and just
• 28:36neutral.
• 28:37So at the low sustainability
• 28:39sites.
• 28:40Also, consistent and strong, specifically
• 28:42internal facilitator
• 28:44came out to be important.
• 28:45And it was really interesting
• 28:47to see that at the
• 28:47high sustainability
• 28:48sites, they really found this
• 28:50internal facilitator, somebody there to
• 28:52be able to really be
• 28:53the expert and someone that
• 28:55can carry the implementation to
• 28:56be a big deal.
• 28:57You may recall me mentioning
• 28:58a few minutes ago that
• 29:00all this was so that
• 29:01we could better prepare for
• 29:03this current new trial of
• 29:04this. And here's sort of
• 29:06the ways in which we
• 29:07specifically use this information. So
• 29:08we are still using,
• 29:10external internal facilitation. And we
• 29:12made changes to that in
• 29:14the sense that we knew
• 29:15that we needed to focus
• 29:16a little more deeply on
• 29:17knowledge retention, specifically during transition.
• 29:20So we try to think
• 29:21about ways in which as
• 29:22we work on implementation support
• 29:24processes,
• 29:25how we would translate knowledge
• 29:27to and keeping knowledge documented
• 29:29at the site.
• 29:30We had the danger of
• 29:32being seen as, oh, we
• 29:33are external consultants coming in
• 29:35who are running the show
• 29:36and once we leave,
• 29:37back to usual. They're normal.
• 29:39So that was one of
• 29:40the things we'd want wanted
• 29:41to make sure. Collaboration wise,
• 29:43because it was important,
• 29:45we also wanted to make
• 29:46sure that regular CCM related
• 29:48information exchange was happening both
• 29:51in terms of the interdisciplinary
• 29:52team members of the general
• 29:54mental health as well as
• 29:55with different levels of leadership.
• 29:56So this was really building
• 29:58into kind of our planned
• 29:59implementation and ways in which
• 30:01you're guiding the internal facilitator
• 30:03and other side personnel as
• 30:04to specific email templates that
• 30:06can be used or ways
• 30:07in which there are sequences
• 30:08to make sure, for example,
• 30:10if you're,
• 30:11communicating with somebody up the
• 30:12chain, you include everybody between
• 30:14yourself and the chain for
• 30:14that communication types of things
• 30:14that may not be necessarily
• 30:15implementation,
• 30:25really, really helpful for ongoing
• 30:27sort of capacity building at
• 30:29the different sites as well.
• 30:30And this piece of still
• 30:31internal facilitators,
• 30:33we very deliberately,
• 30:35instead of us being seen
• 30:37as the facilitators of the
• 30:38work, made sure that the
• 30:39locus of control was more
• 30:41with the internal facilitator.
• 30:43We also
• 30:44made a change in terms
• 30:45of not working with a
• 30:46single,
• 30:47interdisciplinary team at the sites
• 30:49this time around, but at
• 30:50the service level where they
• 30:52had more
• 30:53has control, a little more
• 30:54say, a little more decision
• 30:56making power in being able
• 30:57to make these processes the
• 30:59focus of the dissertation.
• 31:01So we directly used what
• 31:03we had found to be
• 31:04able to better design,
• 31:06sustainability
• 31:07focused implementation trial this time
• 31:09around.
• 31:11So I mentioned this one
• 31:13sort of example as a
• 31:14way to just walk through
• 31:15the different steps. There are
• 31:17I just wanted to share
• 31:18a couple other ways in
• 31:19which it has been used
• 31:20more than anything because I
• 31:22wanted to also share with
• 31:23you share with share with
• 31:24you some, you know,
• 31:25resources or ways in which
• 31:27they have done some of
• 31:28these visualizations that could be
• 31:29helpful too.
• 31:31So one, project,
• 31:33that I was fortunate to
• 31:34be involved in is this,
• 31:36idea of being able to
• 31:38safely graduate
• 31:40stable mental health patients from
• 31:41specialty mental health back to
• 31:43primary care so that we
• 31:44can help with mental health
• 31:45care access at the specialty
• 31:47care level, etcetera. So the
• 31:48innovation there was a use
• 31:49of a dashboard that's able
• 31:51to really indicate which patients
• 31:53may be appropriate for transition.
• 31:54Of course, the decision lies
• 31:56with the patient and together
• 31:57with the provider, etcetera. Of
• 31:58course, their expertise is ultimate,
• 32:00but this is one way
• 32:01in which they can be
• 32:02informed that maybe it's the
• 32:03time to have that conversation
• 32:05if that is okay. So
• 32:06this was also a a
• 32:08nine VA medical center study.
• 32:09We're looking at outcomes like
• 32:11number of veterans to whom
• 32:12this option,
• 32:14was offered and then the
• 32:16adoption that the providers had
• 32:17of this as well. This
• 32:19case, we used, qualitative
• 32:21administrative
• 32:22data to be able to
• 32:23look at things like what
• 32:23we're talking about, like turnover
• 32:25rates or ways in which,
• 32:27the satisfaction came through, etcetera.
• 32:29And
• 32:30this is similar, right, because
• 32:32we were doing the study
• 32:33in a similar time period.
• 32:35We used similar layouts. But
• 32:37the study, we decided to
• 32:38actually separate out,
• 32:40the types of, factors that
• 32:42were seeming to be more
• 32:43trending
• 32:44with,
• 32:46sort of the low through
• 32:47low, medium, and high. So
• 32:49you can see that the
• 32:50presence was much stronger, for
• 32:51example, for the high sites
• 32:53of high kind of reach
• 32:54and therefore implementation outcome sites.
• 32:56When we separated that out
• 32:57with others, which all or
• 33:00important things that came up
• 33:01in our data too, but
• 33:02they didn't necessarily serve as
• 33:04items that we found to
• 33:06be differentiating
• 33:07between the different sites. So
• 33:09once again, this is just
• 33:10one way of visualizing, and
• 33:12we are looking for other
• 33:13ways to do this. We
• 33:14wanted to put this as
• 33:15an example of a side
• 33:16by side kind of different
• 33:17way of looking at trending
• 33:18versus less trending types of
• 33:20factors this way. Also, I
• 33:21wanted to point out,
• 33:23sort of the way in
• 33:24which remember the step where
• 33:26we brought in the IPRIS
• 33:27as the framework that would
• 33:28guide us? It's not a
• 33:30requirement that it, by any
• 33:31means, has to be IPRIS.
• 33:33It can be any sort
• 33:34of guiding framework
• 33:35just because it is helpful
• 33:36for us to kind of
• 33:37have a structure by which
• 33:38to organize the different factors
• 33:40that we're thinking about. In
• 33:41this particular case, we were
• 33:43using c for the consolidated
• 33:44framework for implementation research. And
• 33:47as you can see, some
• 33:47of the factors did not
• 33:49necessarily belong as,
• 33:51constructs under that. Some of
• 33:52them came directly from quantitative
• 33:53metrics too. So that's why
• 33:55you see some of those
• 33:55additional lines there too. So
• 33:57just wanted to show a
• 33:57little bit of flexibility there
• 33:59by sharing this.
• 34:01And the second example,
• 34:04is for screening for, intimate
• 34:06partner violence.
• 34:07So I wasn't directly involved
• 34:09in this project, actually. I
• 34:10was only consulting on the
• 34:11method a little bit.
• 34:13But if you can see
• 34:15here, once again, it was
• 34:16a multi site study. Their
• 34:17outcome in this case was
• 34:19intimate partner violence screening rate
• 34:21and whether there was an
• 34:22increase in that rate as
• 34:23a result of implementing this.
• 34:25And then they used a
• 34:26variety of key data sources
• 34:27in this case as well.
• 34:28Everything from medical records to
• 34:30surveys
• 34:31and interviews
• 34:32of the involved personnel. And
• 34:34then the reason why I
• 34:35wanted to share this example
• 34:37was because the first author
• 34:38of the, the paper that
• 34:40reported on this,
• 34:42Ajugno and Al, I
• 34:44did a really wonderful job.
• 34:46Not that you should be
• 34:46reading this. I think did
• 34:48a one really wonderful job
• 34:49of sort of totally
• 34:51being able to lay out
• 34:53the nine steps that I
• 34:54just talked about a few
• 34:55minutes ago. So I've been
• 34:57really actively pointing people to
• 34:59actually this article,
• 35:02to be able to look
• 35:03at it because I think
• 35:04the the color is a
• 35:05little bit,
• 35:06I mean, saturated here,
• 35:08the screen. But as you
• 35:09can see, like, they made
• 35:10very clear
• 35:11where the site implementation
• 35:13success analysis was happening, like,
• 35:15which boxes were associated with
• 35:17kind of the influencing factor
• 35:18analysis,
• 35:19and then how you will
• 35:20put them together for the,
• 35:22within site and cross site
• 35:23analysis as well. So I
• 35:24was really happy to see
• 35:25kind of a visualization
• 35:27of the process itself. As
• 35:28I mentioned earlier, it's one
• 35:29I'm struggling with to figure
• 35:31out how best to really
• 35:32deliver and discuss and being
• 35:34able to talk about
• 35:35MNCS.
• 35:37So
• 35:38the next part is about
• 35:42sort of MMCS
• 35:44developments underway. I think we
• 35:45already talked a little bit
• 35:46about ways in which we're
• 35:47excited to think about complementarity
• 35:49with other,
• 35:50kind of methods.
• 35:52The way in which we're
• 35:53thinking about it right now,
• 35:54I just wanted to share
• 35:55a couple examples and to
• 35:56see whether there may be
• 35:57interest in and if there
• 35:59are things that you are
• 36:00working on that could be,
• 36:02meaningful,
• 36:03for this as well.
• 36:04One of my projects right
• 36:05now is, trying to figure
• 36:08out and better understand how
• 36:09VA is implementing a new
• 36:11congressionally mandated grant program
• 36:14in which the VA funds
• 36:15community organizations
• 36:16to deliver legal services to
• 36:18veterans.
• 36:19Oftentimes, legal services
• 36:21unavailability,
• 36:23leads to them not being
• 36:24able to access health care
• 36:26or other services that are
• 36:27really important for their well-being.
• 36:28So VA is able to
• 36:30now grant funds to the
• 36:31external organizations. So it's a
• 36:32new program. We're only in
• 36:33our second year of grantees.
• 36:36Seventy nine grantees in the
• 36:37first year from all across
• 36:39the United States and a
• 36:40hundred fourteen in the second
• 36:42current year
• 36:43of different organizations.
• 36:44So we have a lot
• 36:45of, as you can probably
• 36:46imagine, variability in terms of
• 36:48the number of veterans that
• 36:49they're being able to deliver
• 36:51legal services to, etcetera. So
• 36:52we have a lot of
• 36:53data sources, both things that
• 36:55we're gathering directly from grantees
• 36:57of their experiences,
• 36:58veterans, their experiences, but also,
• 37:01these organizations
• 37:02have to also submit to
• 37:04the national program sort of,
• 37:06quarterly data
• 37:08on services delivered, number of
• 37:10veterans that they deliver services
• 37:11to costs incurred, etcetera. So
• 37:13we have a pretty rich
• 37:14data base here to work
• 37:16with.
• 37:17And so outcomes vary. How
• 37:19and why they vary
• 37:20are maybe there are expectations
• 37:22as to why that may
• 37:23be the case given the
• 37:24variability and where they're located,
• 37:26the populations that they're serving.
• 37:28But it really is not
• 37:29known as to what really
• 37:31is driving this. So we
• 37:33thought this is an
• 37:35opportunity
• 37:36to think about MMCs
• 37:37and how it might be
• 37:39integratable with realist evaluation.
• 37:42Realist evaluation, some of you
• 37:43may be familiar with it
• 37:44already, is really well suited
• 37:45for answering these how or
• 37:47why types of questions, and
• 37:49it really tries to get
• 37:50at at its crux, it
• 37:51really is about identifying
• 37:53context
• 37:54and mechanisms
• 37:55that come together to be
• 37:57able to lead to outcomes.
• 37:59So identifying these configurations of
• 38:01CMO
• 38:02kind of are at the
• 38:02center of our realist evaluation
• 38:04is about. But isn't that
• 38:05what you're trying to do
• 38:06in on
• 38:07CMOs also? Yes. So we
• 38:09should be doing it in
• 38:10different ways. We're trying to
• 38:12learn from kind of the
• 38:13established realist evaluation,
• 38:15thinking and framework and the
• 38:17way in which their evaluation
• 38:19the robust way in which
• 38:20they can look at things
• 38:21causally,
• 38:22I think, is what we're
• 38:22trying to get at using
• 38:23this integration. So what we're
• 38:25thinking about right now and
• 38:26the work that
• 38:28keeping fingers crossed for funding
• 38:30is to
• 38:31be able to, get at
• 38:33this.
• 38:34Right before we get there,
• 38:35some of you may be
• 38:35really familiar. Again, I I
• 38:37know many of you raised
• 38:38your hands, so maybe familiar
• 38:39with what we mean by
• 38:40mechanisms, but I'm still trying
• 38:42to figure this out. And
• 38:44I'm thinking about it as
• 38:45sort of processes
• 38:47through which interventions
• 38:49produce outcome. So, like, one
• 38:50example that I always try
• 38:51to think of is, like,
• 38:53let's say there's, like, an
• 38:54audit and feedback type of
• 38:55strategy
• 38:56that is put in place
• 38:57to, carry out implementation,
• 38:59then that might enhance clinician's
• 39:00sort of awareness
• 39:02of sort of what they
• 39:02are doing or what their
• 39:03their peers are doing, what's
• 39:04being expected of it, etcetera.
• 39:06And that awareness
• 39:07can act as that mechanism
• 39:09that then
• 39:10leads to the outcome of
• 39:12improved
• 39:13adherence to care guidelines.
• 39:15So it's not by any
• 39:17means any, the best example
• 39:18or anything, but it helps
• 39:19me kinda think about where
• 39:21the strategies lie, where are
• 39:23kind of mechanisms in this
• 39:24process, and how does it
• 39:25lead to outcomes.
• 39:27So with that said and
• 39:28getting to your question yes.
• 39:30Oh, so in biostatistics,
• 39:33we would call these processes
• 39:34mediators.
• 39:36And there's a very well
• 39:37developed literature in biostatistics
• 39:40on maybe causal mediation analysis.
• 39:42And a number of us
• 39:43here in our center and
• 39:44other faculty here are doing
• 39:47statistical methods work in mediation
• 39:49analysis. Indeed. Indeed. And one
• 39:51one,
• 39:52kind of area or direction
• 39:54of complement clarity that I
• 39:55would like to talk about
• 39:56is ways in which, again,
• 39:58Canada data preparation,
• 39:59ways in which we're able
• 40:00to see the links
• 40:02so that that can inform
• 40:04different types of analysis when
• 40:05it comes to wanting to
• 40:06understand how these are linked,
• 40:08whether it's in this realist
• 40:09evaluation way or it's using
• 40:11other, as you mentioned, established
• 40:13methods on the more quantitative
• 40:14and mediation analysis, etcetera. What's
• 40:16the information that we're gathering
• 40:18to be able to inform
• 40:19those structures? Mhmm. Hopefully.
• 40:22MFS has a place and
• 40:23that is our hope.
• 40:26So,
• 40:27let's see.
• 40:29So in terms of configuration,
• 40:31exact I just wanted to
• 40:32put this up here as
• 40:33an example. I know the
• 40:34writing is a little small.
• 40:35But let's say we are
• 40:36looking for a kind of
• 40:37outcome, the outcome that veterans
• 40:38are actually served by this
• 40:40legal services for veterans program.
• 40:42Then something that's really necessary
• 40:44for that outcome to happen
• 40:46could be that veterans have
• 40:47to be aware that the
• 40:49is issue that they're experiencing
• 40:50is a legal issue. This
• 40:52actually is not necessarily the
• 40:53case. They might not think
• 40:54of disputes or difficulty with
• 40:57their housing landlord situation to
• 40:58be a legal matter necessarily.
• 41:00That might not be the
• 41:01first thought, but their awareness
• 41:02is actually important for them
• 41:03to then seek and receive
• 41:05help legally. So if that's
• 41:07what needs to happen, then
• 41:08some guidance
• 41:09existing to be able to
• 41:11let them know that this
• 41:12is a legal issue might
• 41:13be an important sort of
• 41:14context to have in place.
• 41:16As you can see, that's
• 41:17just one sort of c,
• 41:18m, and o.
• 41:20But as you can imagine,
• 41:21it probably is not the
• 41:22only configuration that exists in
• 41:24any programmatic scenario.
• 41:26Oftentimes, they're not just simply
• 41:28configurations. They are
• 41:30interlinked.
• 41:31They are networks, actually. Right?
• 41:33There may be ways in
• 41:34which the same context links
• 41:35to different mechanisms. There are
• 41:36ways in which same mechanisms
• 41:38also lead to different outcomes.
• 41:40So ways in which we're
• 41:41able to sort of think
• 41:43about these networks, this is
• 41:44where I'm hoping
• 41:46that being able to really
• 41:47put the data together and
• 41:49structure it in a
• 41:51traceable way can be helpful.
• 41:53Specifically speaking, what we are
• 41:55thinking about, and we're still
• 41:56much in development, so this
• 41:57is not out anywhere,
• 42:00is kind of in this
• 42:01kind of a way. It's
• 42:01similar to what you saw
• 42:03earlier. But let's say now
• 42:04we're thinking about specific configurations
• 42:06of these context mechanism and
• 42:08outcomes.
• 42:10If we're able to think
• 42:11about that, we can see
• 42:12for which grantees what types
• 42:14of configurations
• 42:15might exist. And there may
• 42:17be different combinations of context
• 42:19one being associated with mechanism
• 42:20two and outcome two. It
• 42:22may be one with both
• 42:24mechanisms put putting a role
• 42:26and then with, certain other
• 42:27outcome, etcetera. So by being
• 42:29able to put it in
• 42:30this way, we're able to
• 42:31see where the overlaps might
• 42:33be.
• 42:34And, again, we can look
• 42:35at it in a site
• 42:36specific way first, but once
• 42:38we know what are, existing
• 42:39at the site level, we
• 42:41can look at it across
• 42:42the different, in this case,
• 42:43grantees with different locations or
• 42:45settings at which implementation is
• 42:47being been done. And sort
• 42:48of circling back to your
• 42:49point as well, oftentimes,
• 42:51the, parameterization
• 42:53or the understanding
• 42:54or the context in which
• 42:55I think the more the
• 42:56other models you are talking
• 42:57about are built need the
• 42:59contextual
• 43:00sort of knowledge and expertise
• 43:01of where that might come
• 43:02from. Some of that, I
• 43:04think, can come from whether
• 43:06it's information directly from the
• 43:08data or in realist evaluation
• 43:10or realist sense of this
• 43:11synthesis that looks at the
• 43:12literature, etcetera, is also a
• 43:14big part of that realist
• 43:15evaluation to to come up
• 43:17with kind of a theory
• 43:18of how programs work. So
• 43:19that can also lead to
• 43:21identifying these CMO networks
• 43:24or configurations that can inform
• 43:26whether they may be more
• 43:27quantitative
• 43:28or qualitative ways of being
• 43:30able to model how these
• 43:32mechanisms play a role in
• 43:33leading to outcomes when a
• 43:35certain context is possible. So
• 43:36we're trying to think about
• 43:38ways in which we can
• 43:40integrate and learn from as
• 43:42well as contribute to ways
• 43:43in which realist evaluation can
• 43:45move forward to once again
• 43:46get up to how and
• 43:47why.
• 43:50So this next thing is
• 43:51also not out of it
• 43:52anywhere.
• 43:54So I feel very vulnerable
• 43:55talking about all these pieces.
• 43:57And, again, I have no,
• 43:59publication to point to or
• 44:00anything for any of these
• 44:01later slides.
• 44:03So this is what I
• 44:04just showed you. Right? Now
• 44:05let's say we did this
• 44:07and understood this for this
• 44:08program that I'm evaluating.
• 44:10So this study, we're able
• 44:11to understand what are the
• 44:13CMO
• 44:13configurations or networks that matter.
• 44:17In our minds, let's, like,
• 44:18collapse that,
• 44:19right,
• 44:21where that becomes information about
• 44:23one study.
• 44:24So for that study, we
• 44:26can understand what are the
• 44:27configurations, what are the networks.
• 44:29And then what might be
• 44:30really interesting to be able
• 44:32to do is to see
• 44:33whether some of those configurations
• 44:35hold when we compare them
• 44:37with other studies.
• 44:39I think first place where
• 44:40my mind goes are things
• 44:42like there are, multiple now
• 44:44programs
• 44:45of research,
• 44:46where there are related
• 44:48studies that look at similar
• 44:49outcomes
• 44:50or are different, let's say,
• 44:52implementation projects but have a
• 44:53similar audience, etcetera. So for
• 44:55those, are we able to
• 44:57put studies
• 44:58together? And, again, because of
• 45:00the matrix structure and the
• 45:01way in which dimensions can
• 45:02be increased, this is possible.
• 45:03Right? And so that's one
• 45:05way to think about it.
• 45:06And then really
• 45:08exploratory,
• 45:09I guess, if we are
• 45:10trying to see, okay, they're
• 45:11not all program,
• 45:13they're not all projects under
• 45:14one program that is expected
• 45:16to be similar
• 45:17in what the,
• 45:18configurations might be.
• 45:20But if we are able
• 45:21to compare them because we
• 45:22have approached in a way
• 45:23of putting the data together
• 45:25like this across different types
• 45:26of studies, different types of
• 45:28interventions even. And are there
• 45:30sort of slightly higher level
• 45:32takeaways
• 45:33that are helpful for us
• 45:34to come away with and
• 45:35help advance some of the
• 45:36knowledge that we have in
• 45:37the field? So, again, feeling
• 45:39very vulnerable, sharing.
• 45:42Just kind of, open thoughts
• 45:43here towards the end. But
• 45:44I did want to kind
• 45:45of use this opportunity to
• 45:46not just talk about what
• 45:47we have done, but really
• 45:48where can we go from
• 45:50here. And I really appreciate
• 45:51it in the comments about,
• 45:53okay. Oh, can't that be,
• 45:54you know, prepared to use
• 45:55it for this or use
• 45:56it for that? I think
• 45:57that's really what we wanna
• 45:58do. We want to make
• 45:58sure that this is something
• 45:59that's useful. Principles that we're
• 46:01bringing here in terms of
• 46:02organization structure, etcetera, of the
• 46:04data, systematic approach to things
• 46:06can really be
• 46:08applicable beyond this one narrow
• 46:09way of doing things. So
• 46:11that's sort of, I hope,
• 46:13the way in which we
• 46:14can move this forward beyond
• 46:15this.
• 46:17Considerations
• 46:17for incorporating
• 46:19MNCS into your projects, hopefully.
• 46:21We can talk more, of
• 46:22course.
• 46:23Examples of data, as I
• 46:25mentioned, can be quite broad.
• 46:26So I think in terms
• 46:27of making sure that multidisciplinary
• 46:30collaboration exists within the group
• 46:31to understand how best to
• 46:33really understand
• 46:34and make sure that especially,
• 46:35if you have two different,
• 46:37let's say, measures, qualitative and
• 46:38quantitative of, let's say, staff
• 46:40turnover,
• 46:41how do you kind of
• 46:41reconcile that? How do you
• 46:43decide that which one is
• 46:44indeed the primary if there
• 46:46needs to be a primary
• 46:47or other ways in which
• 46:48you might bring them together.
• 46:49So I think that is
• 46:50really important to keep in
• 46:51mind in terms of combining
• 46:52different data. So strength, but
• 46:54definitely needs thought. And then
• 46:56the documentation of steps and
• 46:58decisions, I think this goes
• 46:59back to the word traceability
• 47:00I've been using many times
• 47:01today. So I think, once
• 47:03again, what NMCs has to
• 47:04offer is definitely,
• 47:06formalized protocolized
• 47:08sequence
• 47:09of you starting always with
• 47:11kind of defining that question.
• 47:13So always with thinking about
• 47:14what it is that you
• 47:15data that you want to
• 47:16use and deciding that together
• 47:19and making sure that studies
• 47:20to study study to study
• 47:21may differ in how that
• 47:22decision is made, but let's
• 47:24make that explicit is the
• 47:25goal here. So our training
• 47:27team members, they're forming consistent
• 47:29protocols when it comes to
• 47:30this. So that's, of course,
• 47:31important.
• 47:32Yes. This is the time
• 47:33intensity part I'm going to
• 47:35go to.
• 47:36So, indeed, none of those
• 47:38steps that I mentioned are
• 47:39like, oh, yeah. You just,
• 47:41do it and you get
• 47:42this, coded
• 47:43data. No. Of course. And
• 47:45we know from having done
• 47:47analyses,
• 47:48that it's very, very,
• 47:49resource intense. Therefore, I think
• 47:51this is the point I
• 47:52made briefly earlier too. I
• 47:54think where, there could be
• 47:55an advantage is that it
• 47:56can leverage
• 47:58analysis that's done
• 48:00for
• 48:01a different purpose of the
• 48:02study or a subsection of
• 48:04the study that looked specifically
• 48:06to do qualitative information
• 48:08analysis, and then, therefore, you're
• 48:09able to bring it into
• 48:10this. You can then combine
• 48:11it with quantitative analysis that
• 48:13have been done. So it's
• 48:14a way of really bringing
• 48:15together different it's different mixed
• 48:18data pieces. And I think
• 48:19if you are proposing to
• 48:21do so anyway because there
• 48:22are the more established traditional
• 48:24ways of going about this,
• 48:25this may be one way
• 48:26to, push this forward.
• 48:29So complementarity,
• 48:30I think I talked about
• 48:31that.
• 48:32I'm really looking forward to,
• 48:34I think, Beyonce too, thinking
• 48:36about ways in which this
• 48:38can be a tailorable, easily
• 48:39usable tool. Like, right now,
• 48:41it's in a spreadsheet format,
• 48:43and we're able to kind
• 48:44of use it and I
• 48:45am able to discuss with
• 48:46teams as to how this
• 48:47might be done. But if
• 48:48there's a way in which
• 48:49this can be made user
• 48:50friendly, I think that's one
• 48:51way in which people can
• 48:53try it out more
• 48:54too, in a easier way.
• 48:55So thinking about analysis, visualization,
• 48:58etcetera, I'd love your thoughts
• 48:59on that. And then yeah.
• 49:00I I think we're all
• 49:01cautiously optimistic a bit about,
• 49:05you know, leveraging,
• 49:07developing AI methods, and other
• 49:08things into traditional ways of
• 49:10doing work. So I think
• 49:11it could play a big
• 49:12role when it comes to
• 49:13especially that resource intensity piece.
• 49:15So I think there are
• 49:16ways in which this can
• 49:18move MMCS
• 49:19forward too. Yep. Comparison across
• 49:21studies, all three different procedures,
• 49:24trying to make variations
• 49:25and studies of them very
• 49:27traceable.
• 49:28Formalized sequence of steps is
• 49:29what this is about, concrete
• 49:31data structure around which everything
• 49:32is built. And this is
• 49:34the main sort of,
• 49:35original paper that had put
• 49:38out this,
• 49:39method. And since then, again,
• 49:41I think other papers have
• 49:42done a great job, for
• 49:44example, showing the visual of
• 49:45the methods process and things
• 49:47like that that were done
• 49:48in this original
• 49:50piece. So,
• 49:51that, I think, brings me
• 49:52to the end of my
• 49:53prepared slides.
• 49:56Thank you.
• 50:01So
• 50:02I don't have a way
• 50:03of seeing questions from online.
• 50:05Can you I do. I'm
• 50:06looking at it. So people,
• 50:08can either raise their hands
• 50:10if they'd like or,
• 50:11they