WEBVTT
1
00:00:00.180 --> 00:00:01.863
Reductions in criteria.
2
00:00:04.620 --> 00:00:06.330
Attempts and ideation,
3
00:00:06.330 --> 00:00:08.523
including a one and two year follow up.
4
00:00:11.670 --> 00:00:13.500
And here's just an example.
5
00:00:13.500 --> 00:00:16.110
Oh, I'm sorry, Tracy. This is Donna Spiegelman.
6
00:00:16.110 --> 00:00:18.120
I raised my hand-
Oh.
7
00:00:18.120 --> 00:00:19.500
Through the screen.
8
00:00:19.500 --> 00:00:21.570
But I'm not sure how that's monitored,
9
00:00:21.570 --> 00:00:25.590
or if you can see whether people are raising their hands.
10
00:00:25.590 --> 00:00:26.787
But I had a question,
11
00:00:26.787 --> 00:00:29.133
if you could go back to the previous slide.
12
00:00:30.180 --> 00:00:31.140
Absolutely.
Just starting with
13
00:00:31.140 --> 00:00:32.970
sending patients who are suicidal,
14
00:00:32.970 --> 00:00:35.715
and you mentioned something about screening for them.
15
00:00:35.715 --> 00:00:38.473
Could you say a little bit about how that's done?
16
00:00:38.473 --> 00:00:42.720
'Cause that can be a very big part of a project like this,
17
00:00:42.720 --> 00:00:44.823
especially when it's, like, scaled up.
18
00:00:45.870 --> 00:00:49.200
Absolutely, so in the VA, they have implemented
19
00:00:49.200 --> 00:00:52.860
the Columbia Screener in emergency departments.
20
00:00:52.860 --> 00:00:56.430
And this is being implemented in the ED, by the way.
21
00:00:56.430 --> 00:00:59.910
So they are screened when they present
22
00:00:59.910 --> 00:01:02.940
to emergency departments using the Columbia.
23
00:01:02.940 --> 00:01:04.713
And that's already been in place,
24
00:01:05.850 --> 00:01:07.893
and we are building off of that.
25
00:01:11.190 --> 00:01:14.392
So every person who goes to a VA emergency room
26
00:01:14.392 --> 00:01:16.770
as part of the standard of care,
27
00:01:16.770 --> 00:01:19.353
they're getting this screener?
28
00:01:20.190 --> 00:01:22.380
Absolutely.
And then its result
29
00:01:22.380 --> 00:01:26.193
is entered into the computer?
Absolutely. Yep.
30
00:01:27.540 --> 00:01:29.160
Lots of screening at the VA.
31
00:01:29.160 --> 00:01:31.757
They get screened for all sorts of things.
32
00:01:34.878 --> 00:01:36.980
Okay. Thank you.
I can't see if anyone
33
00:01:36.980 --> 00:01:38.940
raises their hand.
34
00:01:38.940 --> 00:01:42.120
Just throw something at me if that happens again.
35
00:01:42.120 --> 00:01:46.901
Okay, so these are examples of Caring Contacts
36
00:01:46.901 --> 00:01:51.630
that we adapted
37
00:01:51.630 --> 00:01:54.150
for veterans, of course
38
00:01:54.150 --> 00:01:57.783
with feedback from veterans.
39
00:01:58.920 --> 00:02:00.720
There is no Ms. Squirrel, of course.
40
00:02:00.720 --> 00:02:02.040
That's just the example.
41
00:02:02.040 --> 00:02:04.500
So you can see very non-demanding.
42
00:02:04.500 --> 00:02:07.110
We value your health and are honored to serve you,
43
00:02:07.110 --> 00:02:09.933
and it should say in the Emergency Department.
44
00:02:10.980 --> 00:02:13.260
And then they also get birthday cards,
45
00:02:13.260 --> 00:02:15.963
and they get another card on Veteran's Day.
46
00:02:20.220 --> 00:02:22.710
Of course, it's always helpful
47
00:02:22.710 --> 00:02:26.130
to have a framework borrowed from implementation science
48
00:02:26.130 --> 00:02:29.310
if you want to successfully implement something.
49
00:02:29.310 --> 00:02:32.073
And our framework is the i-PARIHS Framework.
50
00:02:33.300 --> 00:02:36.305
So the i-PARIHS has these different dimensions, right?
51
00:02:36.305 --> 00:02:39.600
And it hypothesizes it.
52
00:02:39.600 --> 00:02:43.290
It's this magical combination of context,
53
00:02:43.290 --> 00:02:48.290
innovation and recipient characteristics and qualities
54
00:02:48.480 --> 00:02:52.830
when they are joined with facilitation
55
00:02:52.830 --> 00:02:55.260
that results in successful implementation
56
00:02:55.260 --> 00:02:57.783
of a practice in a clinical context.
57
00:03:00.750 --> 00:03:04.020
And in keeping with that theoretical approach,
58
00:03:04.020 --> 00:03:08.430
we are using facilitation, the implementation strategy
59
00:03:08.430 --> 00:03:11.340
of external facilitation actually,
60
00:03:11.340 --> 00:03:13.953
to help implement Caring Contacts in the ED.
61
00:03:15.210 --> 00:03:16.950
And this is our study design.
62
00:03:16.950 --> 00:03:19.860
We bring the sites on in waves.
63
00:03:19.860 --> 00:03:23.673
Obviously you don't want to bring all 28 sites on at once.
64
00:03:24.720 --> 00:03:29.490
So well, what was intended, what was planned is
65
00:03:29.490 --> 00:03:33.810
that they would get six months of facilitation support
66
00:03:33.810 --> 00:03:36.690
and we would bring them in four waves.
67
00:03:36.690 --> 00:03:38.580
COVID hit.
68
00:03:38.580 --> 00:03:40.470
Taking a bit longer for some sites
69
00:03:40.470 --> 00:03:42.840
to implement than we thought it would.
70
00:03:42.840 --> 00:03:45.413
And we're actually moving into a fifth wave
71
00:03:45.413 --> 00:03:47.790
I think fairly soon.
72
00:03:47.790 --> 00:03:50.610
So things have changed as they often do
73
00:03:50.610 --> 00:03:52.293
in research in the real world.
74
00:03:55.890 --> 00:03:58.050
It's a mixed methods evaluation.
75
00:03:58.050 --> 00:04:00.240
I won't cover the quantitative part,
76
00:04:00.240 --> 00:04:02.670
because frankly we could spend all day just talking
77
00:04:02.670 --> 00:04:06.240
about the entire approach to getting this project
78
00:04:06.240 --> 00:04:07.410
off the ground.
79
00:04:07.410 --> 00:04:09.560
We'll just talk about the qualitative eval.
80
00:04:11.190 --> 00:04:13.890
And the overarching aim is really to identify
81
00:04:13.890 --> 00:04:16.710
the contextual recipient and innovation factors
82
00:04:16.710 --> 00:04:20.853
that impact implementation, and that's pretty easy.
83
00:04:22.410 --> 00:04:26.958
The challenge that was put to me was understand how barriers
84
00:04:26.958 --> 00:04:31.958
and facilitators impact implementation as a process.
85
00:04:32.730 --> 00:04:37.110
And that was reiterated to me multiple times,
86
00:04:37.110 --> 00:04:42.110
implementation as a process over and over again.
87
00:04:43.530 --> 00:04:46.710
And so I had to think of how exactly would I go
88
00:04:46.710 --> 00:04:50.755
about understanding how barriers
89
00:04:50.755 --> 00:04:54.797
and facilitators impact this process.
90
00:04:54.797 --> 00:04:58.713
'Cause of course we collect data at discreet time points,
91
00:04:59.730 --> 00:05:02.553
and what you have is a window into one time point.
92
00:05:03.420 --> 00:05:07.053
So how do you move that into a process?
93
00:05:10.320 --> 00:05:15.033
For the part of the evaluation that I'll be talking about,
94
00:05:16.110 --> 00:05:19.560
what we're doing is leading 30 minute dyadic debriefs
95
00:05:19.560 --> 00:05:22.890
with a team of external facilitators.
96
00:05:22.890 --> 00:05:27.890
They're biweekly until the first Caring Contact is sent.
97
00:05:28.530 --> 00:05:32.040
And then once the site moves into sustainment,
98
00:05:32.040 --> 00:05:34.833
we switch to monthly debriefs.
99
00:05:35.790 --> 00:05:39.210
As of today, we've led 100, as of September,
100
00:05:39.210 --> 00:05:42.931
I should say September 10, we had about 169.
101
00:05:42.931 --> 00:05:47.553
I think we're close to about 180 at this point.
102
00:05:48.450 --> 00:05:50.130
And then we're also leading interviews
103
00:05:50.130 --> 00:05:52.140
with stakeholders and veterans,
104
00:05:52.140 --> 00:05:54.843
but I won't be talking about that moving forward.
105
00:05:55.950 --> 00:05:57.780
These are what the debriefs look like,
106
00:05:57.780 --> 00:06:02.400
or at least a snapshot of part of a debrief.
107
00:06:02.400 --> 00:06:05.250
It's the typical sort of probing around,
108
00:06:05.250 --> 00:06:08.130
what are the barriers and facilitators.
109
00:06:08.130 --> 00:06:12.360
And of course as we move forward because they're biweekly,
110
00:06:12.360 --> 00:06:15.700
they change, and they also change significantly
111
00:06:16.920 --> 00:06:18.960
depending on the time point.
112
00:06:18.960 --> 00:06:22.438
So we have a certain debrief that we used to collect data
113
00:06:22.438 --> 00:06:25.650
while they're actively implementing.
114
00:06:25.650 --> 00:06:27.300
And then we have another one when
115
00:06:27.300 --> 00:06:32.300
the implementation plan has been finalized.
116
00:06:32.460 --> 00:06:35.010
And then we have another one that we bring in
117
00:06:35.010 --> 00:06:38.100
after the first Caring Contacts have been sent
118
00:06:38.100 --> 00:06:40.060
to understand what is going on
119
00:06:41.910 --> 00:06:44.823
in these clinics while they're trying to implement.
120
00:06:50.130 --> 00:06:52.320
To meet this challenge
121
00:06:52.320 --> 00:06:55.830
of approaching implementation
122
00:06:55.830 --> 00:07:00.830
as more of a process and as discreet time points,
123
00:07:02.292 --> 00:07:06.450
I have chosen really a two-part approach
124
00:07:06.450 --> 00:07:08.970
to analyzing our data.
125
00:07:08.970 --> 00:07:10.663
And that's first we are
126
00:07:14.100 --> 00:07:17.460
the qualitative researchers
127
00:07:17.460 --> 00:07:19.083
that are leading the debriefs.
128
00:07:20.160 --> 00:07:25.160
Very quickly record key implementation events
129
00:07:25.710 --> 00:07:27.840
and other important things that are happening
130
00:07:27.840 --> 00:07:31.860
during implementation in these brief narrative summaries,
131
00:07:31.860 --> 00:07:35.850
which I translate into data visualizations
132
00:07:35.850 --> 00:07:39.390
that help us understand where the process
133
00:07:39.390 --> 00:07:43.353
of implementation progressed and where it stalled.
134
00:07:45.450 --> 00:07:50.370
Secondly, we are templating each debrief
135
00:07:50.370 --> 00:07:52.770
using individual templates
136
00:07:52.770 --> 00:07:55.833
which essentially organize and reduce our data.
137
00:07:58.290 --> 00:08:01.710
Then we are synthesizing individual templates
138
00:08:01.710 --> 00:08:06.710
into site matrix displays that further reduce the data
139
00:08:07.290 --> 00:08:09.300
and help me make comparisons
140
00:08:09.300 --> 00:08:13.350
across time points and across sites.
141
00:08:13.350 --> 00:08:16.560
And these I'm translating into case studies.
142
00:08:16.560 --> 00:08:19.410
That's how I'm presenting the findings
143
00:08:19.410 --> 00:08:21.633
that come from the matrices.
144
00:08:24.360 --> 00:08:27.600
I don't wanna go through the analytic approach
145
00:08:27.600 --> 00:08:31.710
one by one because I'm worried
146
00:08:31.710 --> 00:08:35.190
that we won't get through the whole presentation in time.
147
00:08:35.190 --> 00:08:39.630
But what I really want to emphasize to everyone on here
148
00:08:39.630 --> 00:08:44.430
is that time was actually the least important reason
149
00:08:44.430 --> 00:08:49.020
for me to choose rapid analytic approaches for this project.
150
00:08:49.020 --> 00:08:52.920
I knew that we would have enough time to analyze the data.
151
00:08:52.920 --> 00:08:56.170
It was really more about the research questions, the goals
152
00:08:57.270 --> 00:09:01.500
and the challenges collaborating across sites.
153
00:09:01.500 --> 00:09:06.420
Two of my analysts are located,
154
00:09:06.420 --> 00:09:08.730
sorry, one of my, no, two of them.
155
00:09:08.730 --> 00:09:11.940
One is in Iowa City and one is in California.
156
00:09:11.940 --> 00:09:16.260
And sharing analytic programs in the VA
157
00:09:16.260 --> 00:09:17.883
can get really problematic.
158
00:09:18.900 --> 00:09:22.950
What templating allows us to do is get around those problems
159
00:09:22.950 --> 00:09:26.370
because templates are developed in Word documents
160
00:09:26.370 --> 00:09:29.550
and that's much easier in the VA.
161
00:09:29.550 --> 00:09:32.520
So that was really honestly my primary concern
162
00:09:32.520 --> 00:09:36.660
is if this thing stalls because we run into problems
163
00:09:36.660 --> 00:09:39.990
with the program and sharing across sites,
164
00:09:39.990 --> 00:09:42.420
we are in a lot of trouble.
165
00:09:42.420 --> 00:09:45.270
That was the primary consideration
166
00:09:45.270 --> 00:09:48.843
rather than the timeframe for this study.
167
00:09:50.070 --> 00:09:55.070
And to emphasize that this is a team effort,
168
00:09:55.110 --> 00:09:59.370
this is how we manage every step in the analytic process.
169
00:09:59.370 --> 00:10:02.700
You can see the debrief on the left.
170
00:10:02.700 --> 00:10:05.883
This is for one site, 695,
171
00:10:07.080 --> 00:10:08.880
and then each person along the way
172
00:10:08.880 --> 00:10:10.200
has their own role, right?
173
00:10:10.200 --> 00:10:14.250
It's audio recorded by the person leading the debrief,
174
00:10:14.250 --> 00:10:18.060
then transcribed, and then there are these different steps
175
00:10:18.060 --> 00:10:19.410
in the analytic process.
176
00:10:19.410 --> 00:10:21.510
It's first templated.
177
00:10:21.510 --> 00:10:24.963
So each debrief is analyzed independently.
178
00:10:26.250 --> 00:10:29.520
And then I begin bringing those data
179
00:10:29.520 --> 00:10:32.520
into the matrix to synthesize it.
180
00:10:32.520 --> 00:10:37.260
And what often happens is I have questions, I have comments
181
00:10:37.260 --> 00:10:39.180
and using the track changes
182
00:10:39.180 --> 00:10:41.460
and the comments feature of order,
183
00:10:41.460 --> 00:10:44.767
I can create this dialogue with that primary analyst
184
00:10:44.767 --> 00:10:47.640
to help me understand what's going on in that template
185
00:10:47.640 --> 00:10:50.130
before I move it into the matrix.
186
00:10:50.130 --> 00:10:52.683
And then once we've gotten through that process,
187
00:10:54.060 --> 00:10:56.670
then I can indicate that the matrix is complete.
188
00:10:56.670 --> 00:11:00.450
But it is a process and it takes a heavy lift,
189
00:11:00.450 --> 00:11:03.843
a lot of collaboration, a lot of coordination.
190
00:11:07.890 --> 00:11:11.760
Earlier I spoke of these ongoing brief narrative summaries.
191
00:11:11.760 --> 00:11:15.480
These are what the researchers immediately
192
00:11:15.480 --> 00:11:17.133
after they've led a debrief,
193
00:11:18.660 --> 00:11:21.960
they record their observations and they need these actually
194
00:11:21.960 --> 00:11:24.060
not for analysis so much
195
00:11:24.060 --> 00:11:27.360
except for the key implementation events,
196
00:11:27.360 --> 00:11:28.703
which I bolded in this for you.
197
00:11:30.582 --> 00:11:32.160
What they really need it for
198
00:11:32.160 --> 00:11:35.130
is to remind the external facilitators
199
00:11:35.130 --> 00:11:39.060
of what they talked about during the prior debrief,
200
00:11:39.060 --> 00:11:41.755
because they're doing so much work across so many sites
201
00:11:41.755 --> 00:11:44.070
that people just can't always remember
202
00:11:44.070 --> 00:11:46.230
what they said the time before.
203
00:11:46.230 --> 00:11:48.540
So it serves a dual purpose
204
00:11:48.540 --> 00:11:52.140
of recording these key implementation events
205
00:11:52.140 --> 00:11:54.960
while helping team qual
206
00:11:54.960 --> 00:11:58.263
to keep the facilitators on track.
207
00:12:01.680 --> 00:12:03.630
And then we have templates.
208
00:12:03.630 --> 00:12:07.500
For those of you who haven't heard of template analysis,
209
00:12:07.500 --> 00:12:10.080
again, they're just Word documents that you use
210
00:12:10.080 --> 00:12:13.514
to summarize and organize content
211
00:12:13.514 --> 00:12:18.153
from individual interviews or focus group discussions.
212
00:12:19.530 --> 00:12:21.390
They can be theoretically informed
213
00:12:21.390 --> 00:12:24.390
as the one I'm going to show you has been.
214
00:12:24.390 --> 00:12:28.410
Or they can be goal-oriented, meaning, you know,
215
00:12:28.410 --> 00:12:31.300
just let's say you want to adapt
216
00:12:34.806 --> 00:12:38.477
a program that is delivered via internet
217
00:12:39.990 --> 00:12:43.560
to veterans and you need to collect data
218
00:12:43.560 --> 00:12:46.380
that will help you adapt that program.
219
00:12:46.380 --> 00:12:50.160
So you might just organize your template instead
220
00:12:50.160 --> 00:12:54.360
of using theoretical domains or categories.
221
00:12:54.360 --> 00:12:58.920
You can say look and feel changes or recommendations
222
00:12:58.920 --> 00:13:02.250
or language recommendations if they don't
223
00:13:02.250 --> 00:13:04.233
like the language that's used.
224
00:13:05.130 --> 00:13:06.630
So they focus analysis.
225
00:13:06.630 --> 00:13:10.350
You generally start with some deductive domains
226
00:13:10.350 --> 00:13:13.620
and categories while permitting discovery.
227
00:13:13.620 --> 00:13:14.730
And you'll understand that
228
00:13:14.730 --> 00:13:16.580
a little bit better in just a second.
229
00:13:17.850 --> 00:13:21.180
Most importantly from my perspective
230
00:13:21.180 --> 00:13:23.340
is that when you're an anthropologist working
231
00:13:23.340 --> 00:13:27.210
in implementation science, templates can provide
232
00:13:27.210 --> 00:13:31.383
this common language between you
233
00:13:31.383 --> 00:13:34.080
and the other people on your team
234
00:13:34.080 --> 00:13:37.170
that can help you communicate with them
235
00:13:37.170 --> 00:13:39.330
and they can help you translate your findings
236
00:13:39.330 --> 00:13:41.700
much more efficiently down the road,
237
00:13:41.700 --> 00:13:45.150
whether it's in presentations or publications
238
00:13:45.150 --> 00:13:47.223
or in improving a process.
239
00:13:48.930 --> 00:13:52.890
So this is an example of what I call a master template.
240
00:13:52.890 --> 00:13:56.356
And it's just a template that everyone goes into
241
00:13:56.356 --> 00:14:01.356
and makes a copy of before they begin analysis.
242
00:14:02.010 --> 00:14:07.010
This one has some deductive categories in black,
243
00:14:07.382 --> 00:14:11.190
and you can see the inductive categories,
244
00:14:11.190 --> 00:14:13.817
domains, here you go, domains from the i-PARIHS,
245
00:14:14.850 --> 00:14:17.373
categories and subcategories over here.
246
00:14:18.210 --> 00:14:21.780
The ones that are in blue are subcategories
247
00:14:21.780 --> 00:14:24.690
that we have developed inductively
248
00:14:24.690 --> 00:14:26.733
while analysis is ongoing.
249
00:14:27.600 --> 00:14:31.240
So you begin deductive and you build in
250
00:14:32.550 --> 00:14:36.720
your inductive categories and subcategories as you proceed.
251
00:14:36.720 --> 00:14:39.300
You bring them into the master.
252
00:14:39.300 --> 00:14:44.190
And then when it's time for the next episode,
253
00:14:44.190 --> 00:14:46.710
if you would, data collection episode
254
00:14:46.710 --> 00:14:49.620
to be templated, they have it here.
255
00:14:49.620 --> 00:14:52.503
They make a copy of the master and away they go.
256
00:14:53.970 --> 00:14:55.830
And this is what a template looks like
257
00:14:55.830 --> 00:14:57.453
after it's been completed.
258
00:14:59.490 --> 00:15:02.253
You have your domains from the i-PARIHS, again,
259
00:15:05.370 --> 00:15:08.580
some more categories and subcategories.
260
00:15:08.580 --> 00:15:11.100
For this one, you can see they don't have as many
261
00:15:11.100 --> 00:15:16.100
categories and subcategories, 'cause when you don't use them
262
00:15:16.440 --> 00:15:19.083
for any particular template, you delete them off.
263
00:15:21.120 --> 00:15:23.640
You only keep the ones that emerged
264
00:15:23.640 --> 00:15:26.043
during that data collection episode.
265
00:15:27.060 --> 00:15:29.760
And then you have your content from your debriefs
266
00:15:29.760 --> 00:15:32.070
or your interviews or your focus groups.
267
00:15:32.070 --> 00:15:35.910
However it is that you are collecting data.
268
00:15:35.910 --> 00:15:38.460
So you go into your transcript
269
00:15:38.460 --> 00:15:42.870
and literally just copy and paste, and often as you can,
270
00:15:42.870 --> 00:15:44.841
I think you can probably see my pointer.
271
00:15:44.841 --> 00:15:47.283
Can you? Yes.
272
00:15:48.780 --> 00:15:50.730
You need to get a little bit creative, right?
273
00:15:50.730 --> 00:15:52.262
Yes, we can see the pointer.
274
00:15:52.262 --> 00:15:54.020
Great, thank you.
275
00:15:54.020 --> 00:15:56.820
We all know what qual interviews are like.
276
00:15:56.820 --> 00:16:00.330
If it's a good qualitative interview, you ask a question
277
00:16:00.330 --> 00:16:02.030
and people talk and talk and talk.
278
00:16:03.771 --> 00:16:06.840
And so you have to reduce those data.
279
00:16:06.840 --> 00:16:09.641
If you're putting, you know, oceans of words
280
00:16:09.641 --> 00:16:13.920
on your template, your template can't tell you a story,
281
00:16:13.920 --> 00:16:16.803
because you'll be drowning in those words.
282
00:16:18.390 --> 00:16:19.833
Right, the beauty of a template is
283
00:16:19.833 --> 00:16:22.590
that it reduces those data for you.
284
00:16:22.590 --> 00:16:25.110
And what you see, look at this,
285
00:16:25.110 --> 00:16:28.470
what you see is this beautiful coherent story
286
00:16:28.470 --> 00:16:32.008
of one data collection episode of what was happening
287
00:16:32.008 --> 00:16:35.550
at that site, at that point in time.
288
00:16:35.550 --> 00:16:38.820
The barriers and facilitators that either helped
289
00:16:38.820 --> 00:16:42.810
or hindered implementation of this particular practice,
290
00:16:42.810 --> 00:16:46.503
in this particular clinic, in this particular point in time.
291
00:16:48.690 --> 00:16:51.780
So delete off everything that didn't apply,
292
00:16:51.780 --> 00:16:56.110
keep everything that did, reduce it down to its bare essence
293
00:16:59.340 --> 00:17:02.163
and it tells you this beautiful coherent story.
294
00:17:03.834 --> 00:17:06.270
In the hands of a skilled analyst, of course.
295
00:17:06.270 --> 00:17:09.900
It does take time to pick up these skills.
296
00:17:09.900 --> 00:17:13.230
And then after this, what do you do, right? Okay, great.
297
00:17:13.230 --> 00:17:15.930
So this is what happened at this clinic
298
00:17:15.930 --> 00:17:18.240
at this particular point in time.
299
00:17:18.240 --> 00:17:19.590
Now what do you do with it?
300
00:17:20.970 --> 00:17:22.410
Well, there are a few different things
301
00:17:22.410 --> 00:17:23.280
that you can do with it.
302
00:17:23.280 --> 00:17:26.760
And what I chose to do was synthesize and further reduce it
303
00:17:26.760 --> 00:17:28.653
in what's called a site matrix.
304
00:17:30.210 --> 00:17:33.810
And what makes our site matrices for this project
305
00:17:33.810 --> 00:17:38.280
a little bit different is that generally matrices tend
306
00:17:38.280 --> 00:17:42.690
to be organized by participants and by category,
307
00:17:42.690 --> 00:17:45.873
and you compare across participants.
308
00:17:47.670 --> 00:17:50.490
The way that I organized these was by time point,
309
00:17:50.490 --> 00:17:52.050
you'll see this in just a second,
310
00:17:52.050 --> 00:17:55.075
to allow me to get this processual insight
311
00:17:55.075 --> 00:17:58.263
into what's happening over time.
312
00:17:59.310 --> 00:18:03.243
And matrices give you a very broad overview,
313
00:18:04.197 --> 00:18:07.530
at least this particular matrix does,
314
00:18:07.530 --> 00:18:10.170
into when factors come into play,
315
00:18:10.170 --> 00:18:12.330
which factors are coming into play,
316
00:18:12.330 --> 00:18:15.120
and for how long they continue
317
00:18:15.120 --> 00:18:17.793
to impact implementation over time.
318
00:18:19.140 --> 00:18:22.623
And then on another tab,
319
00:18:24.390 --> 00:18:26.103
this is what a matrix looks like.
320
00:18:27.120 --> 00:18:30.390
On another tab, that is where the magic happens.
321
00:18:30.390 --> 00:18:33.093
That's where you put all your brilliant insights.
322
00:18:35.100 --> 00:18:39.390
All the, what I call qualitatively significant factors
323
00:18:39.390 --> 00:18:42.840
that impacted implementation.
324
00:18:42.840 --> 00:18:44.820
Those go on a separate tab. So here we go.
325
00:18:44.820 --> 00:18:45.900
It's in Excel form.
326
00:18:45.900 --> 00:18:48.810
So it's just one. Is it one site?
327
00:18:48.810 --> 00:18:51.510
This is just one site, yeah.
328
00:18:51.510 --> 00:18:56.460
One site, I organized it by time points across the x-axis.
329
00:18:57.360 --> 00:19:00.510
You can see there's been 18, at least for this one,
330
00:19:00.510 --> 00:19:02.492
at the time that I took this snapshot,
331
00:19:02.492 --> 00:19:07.492
18 different debriefs with facilitators from this site.
332
00:19:09.660 --> 00:19:12.060
And here are all those categories
333
00:19:12.060 --> 00:19:13.713
that you saw on the template.
334
00:19:17.610 --> 00:19:19.653
And then in these fields,
335
00:19:21.210 --> 00:19:23.920
I literally, again, just copy and paste
336
00:19:24.900 --> 00:19:28.380
the subcategories, the barriers and facilitators,
337
00:19:28.380 --> 00:19:30.870
and as much of the excerpts
338
00:19:30.870 --> 00:19:34.533
as I think I need or want.
339
00:19:35.490 --> 00:19:37.736
And another factor that has to come into play
340
00:19:37.736 --> 00:19:42.736
when you're populating matrices
341
00:19:43.175 --> 00:19:46.007
is how much data can you manage
342
00:19:46.007 --> 00:19:48.813
before you're going to get overwhelmed.
343
00:19:49.680 --> 00:19:53.820
I can wrap my head around a relatively large volume
344
00:19:53.820 --> 00:19:56.040
of words of qualitative data.
345
00:19:56.040 --> 00:19:58.830
So I tend to have matrices
346
00:19:58.830 --> 00:20:01.513
that have a lot of words in them.
347
00:20:01.513 --> 00:20:05.070
And I think other people not so much.
348
00:20:05.070 --> 00:20:05.970
So here we go.
349
00:20:05.970 --> 00:20:09.333
Here's an excerpt again, straight from the template.
350
00:20:14.730 --> 00:20:16.683
And there's the magical tab.
351
00:20:17.700 --> 00:20:19.620
That's actually, I put analysis there,
352
00:20:19.620 --> 00:20:21.693
but it's actually the results tab.
353
00:20:23.070 --> 00:20:26.490
So everything, what I do is scroll through that matrix,
354
00:20:26.490 --> 00:20:27.993
and look over time.
355
00:20:29.340 --> 00:20:31.293
I scroll up and I scroll down.
356
00:20:33.060 --> 00:20:36.240
And I take a good hard look at what is happening.
357
00:20:36.240 --> 00:20:38.844
What are the qualitatively significant
358
00:20:38.844 --> 00:20:41.340
barriers and facilitators?
359
00:20:41.340 --> 00:20:45.480
So not just what's happening, but what holds things up?
360
00:20:45.480 --> 00:20:48.840
That's what I mean by qualitatively significant.
361
00:20:48.840 --> 00:20:51.839
There are always all sorts of things going on in the clinic,
362
00:20:51.839 --> 00:20:55.162
but what is really holding things up
363
00:20:55.162 --> 00:20:57.453
or speeding them along?
364
00:20:59.938 --> 00:21:03.570
That's where you put your observations, if you would,
365
00:21:03.570 --> 00:21:07.143
of what's happening in the data right on this tab.
366
00:21:08.160 --> 00:21:12.330
And then I'm doing even, to make my job even harder,
367
00:21:12.330 --> 00:21:16.140
I've decided that I'm going to take data from stakeholders
368
00:21:16.140 --> 00:21:19.080
and compare what the facilitators are saying
369
00:21:19.080 --> 00:21:20.790
and see what I come up with,
370
00:21:20.790 --> 00:21:22.193
because I just, I don't know.
371
00:21:22.193 --> 00:21:26.250
I must have a masochistic streak in me or something.
372
00:21:26.250 --> 00:21:28.020
I just, nothing is ever good enough.
373
00:21:28.020 --> 00:21:30.220
I have to take everything to the next level.
374
00:21:31.740 --> 00:21:36.030
So what do you get then? What can you do with this?
375
00:21:36.030 --> 00:21:39.450
Well, one thing that you can do is build case studies.
376
00:21:39.450 --> 00:21:44.103
You can tell a story about implementation. Okay?
377
00:21:46.260 --> 00:21:47.490
I'm not going to go through
378
00:21:47.490 --> 00:21:49.620
all three of these case studies.
379
00:21:49.620 --> 00:21:53.670
I'll just show you one. We don't have time for all three.
380
00:21:53.670 --> 00:21:55.519
But early on in this project,
381
00:21:55.519 --> 00:21:57.630
I thought that a good idea would be
382
00:21:57.630 --> 00:22:01.380
to characterize implementation at each site.
383
00:22:01.380 --> 00:22:05.220
And so I called Case Study One, Rapid Implementation.
384
00:22:05.220 --> 00:22:07.680
It launched after only three months.
385
00:22:07.680 --> 00:22:10.233
Remember they were on a six month timeline.
386
00:22:11.430 --> 00:22:14.400
Case Study Two, I called Delayed,
387
00:22:14.400 --> 00:22:19.400
because they completed their formative evaluation on 5/29,
388
00:22:20.730 --> 00:22:24.360
but Caring Contacts didn't launch for another six months.
389
00:22:24.360 --> 00:22:27.030
So they experienced some delays.
390
00:22:27.030 --> 00:22:31.230
And this one I called Interruption, because it was delayed
391
00:22:31.230 --> 00:22:35.676
and then later delayed indefinitely at the site.
392
00:22:35.676 --> 00:22:37.920
So let's jump into...
393
00:22:37.920 --> 00:22:40.170
I'm no longer characterizing them by the way,
394
00:22:40.170 --> 00:22:42.210
because I've now discovered how
395
00:22:42.210 --> 00:22:44.490
to use these cool data visualizations,
396
00:22:44.490 --> 00:22:47.400
which we'll get to in a second.
397
00:22:47.400 --> 00:22:50.340
That's so much more informative about the process
398
00:22:50.340 --> 00:22:54.063
of implementation than characterizing them in this way.
399
00:22:55.470 --> 00:22:57.450
So Interrupted Implementation.
400
00:22:57.450 --> 00:23:01.293
What the heck happened at this site in this case study?
401
00:23:02.370 --> 00:23:04.440
Well, you know, honestly, this was a site
402
00:23:04.440 --> 00:23:06.897
where the facilitators anticipated
403
00:23:06.897 --> 00:23:09.570
implementation would be really easy,
404
00:23:09.570 --> 00:23:11.580
because they had a lot more facilitators
405
00:23:11.580 --> 00:23:13.620
than they had barriers.
406
00:23:13.620 --> 00:23:17.730
It was this frontier site, and of course, you expect,
407
00:23:17.730 --> 00:23:19.986
you know, these way out in the boonies,
408
00:23:19.986 --> 00:23:22.740
very rural sites to run into problems.
409
00:23:22.740 --> 00:23:25.685
But they had this incredibly cohesive clinic culture,
410
00:23:25.685 --> 00:23:27.960
and they had an influential
411
00:23:27.960 --> 00:23:31.920
and really motivated champion on site.
412
00:23:31.920 --> 00:23:35.820
And some key players, as in leaders,
413
00:23:35.820 --> 00:23:39.210
within the clinic were at the planning meetings.
414
00:23:39.210 --> 00:23:41.160
And this was the initial thoughts
415
00:23:41.160 --> 00:23:44.160
of the facilitators when we interviewed them.
416
00:23:44.160 --> 00:23:47.025
They're used to just sending each other things and tasks.
417
00:23:47.025 --> 00:23:50.100
And even though it's spread out, it meaning the clinic,
418
00:23:50.100 --> 00:23:51.960
it's in different locations,
419
00:23:51.960 --> 00:23:53.880
they really, truly work together.
420
00:23:53.880 --> 00:23:56.820
But they are quick, and they are cohesive,
421
00:23:56.820 --> 00:23:58.950
and they are really well integrated,
422
00:23:58.950 --> 00:24:00.870
I think, given their setting.
423
00:24:00.870 --> 00:24:05.670
So they had these incredible facilitators there
424
00:24:05.670 --> 00:24:08.043
that everyone thought would really help them.
425
00:24:09.450 --> 00:24:13.050
But then unfortunately, they ran into some hitches here.
426
00:24:13.050 --> 00:24:14.460
They were having a really hard time
427
00:24:14.460 --> 00:24:17.279
using the SPED dashboard, which is how you identify
428
00:24:17.279 --> 00:24:22.279
veterans who have been screened for suicidal ideation.
429
00:24:22.890 --> 00:24:25.758
They had a really hard time learning how to use it
430
00:24:25.758 --> 00:24:29.100
to identify the veterans that they needed
431
00:24:29.100 --> 00:24:30.600
to be sending the cards to.
432
00:24:30.600 --> 00:24:34.679
And then of course, COVID hit, and people were pulled
433
00:24:34.679 --> 00:24:37.140
and reassigned to other places in the clinic,
434
00:24:37.140 --> 00:24:39.330
and they needed to be retrained.
435
00:24:39.330 --> 00:24:41.377
And at this point, one of the facilitators said,
436
00:24:41.377 --> 00:24:43.380
"I'm not sure for how long we will be
437
00:24:43.380 --> 00:24:44.728
in the implementation phase,
438
00:24:44.728 --> 00:24:47.580
'cause we can't move forward 'til they're able
439
00:24:47.580 --> 00:24:50.580
to fix the health factor link to the Columbia."
440
00:24:50.580 --> 00:24:52.320
So they really just stalled,
441
00:24:52.320 --> 00:24:56.280
despite having every advantage it seemed.
442
00:24:56.280 --> 00:24:59.850
They stalled because of this issue with the SPED dashboard,
443
00:24:59.850 --> 00:25:01.293
and then of course, COVID.
444
00:25:02.520 --> 00:25:04.740
So that's one thing that you can do,
445
00:25:04.740 --> 00:25:09.480
is these beautiful case studies that help you tell a story.
446
00:25:09.480 --> 00:25:13.170
And we know that our brains, our human brains like stories.
447
00:25:13.170 --> 00:25:16.560
So this is a really powerful way to communicate
448
00:25:16.560 --> 00:25:19.290
what's happening to the rest of your team
449
00:25:19.290 --> 00:25:21.063
and make course corrections.
450
00:25:23.730 --> 00:25:27.180
But insofar as really understanding
451
00:25:27.180 --> 00:25:32.100
implementation as a process, these data visualizations
452
00:25:32.100 --> 00:25:33.510
are really what get you there.
453
00:25:33.510 --> 00:25:35.970
It's not these case studies.
454
00:25:35.970 --> 00:25:37.890
And what we're visualizing
455
00:25:37.890 --> 00:25:40.260
are five key implementation events
456
00:25:40.260 --> 00:25:42.150
that are plotted along a line graph.
457
00:25:42.150 --> 00:25:46.443
And these are not complicated visualizations.
458
00:25:48.390 --> 00:25:49.863
This is what we end up with.
459
00:25:52.950 --> 00:25:56.850
Remember those key facilitation implementation events
460
00:25:56.850 --> 00:26:00.180
are kept in those brief narrative summaries, right?
461
00:26:00.180 --> 00:26:02.520
So you just go back to those,
462
00:26:02.520 --> 00:26:05.823
look at the dates and plot them.
463
00:26:06.960 --> 00:26:10.890
And what we've done is use a color for each phase
464
00:26:10.890 --> 00:26:15.890
of implementation and one dot is for each month.
465
00:26:15.900 --> 00:26:18.490
And you can much more intuitively grasp
466
00:26:19.404 --> 00:26:23.070
for how long implementation was ongoing, right?
467
00:26:23.070 --> 00:26:27.063
This site has fewer dots than does this site.
468
00:26:28.290 --> 00:26:31.230
What that means is this site took longer
469
00:26:31.230 --> 00:26:34.200
to implement and reach sustainment.
470
00:26:34.200 --> 00:26:38.490
Not only that, but in using colors, you can see
471
00:26:38.490 --> 00:26:41.433
for how long these sites were in each phase.
472
00:26:42.690 --> 00:26:45.060
So one one dot equals one month.
473
00:26:45.060 --> 00:26:46.770
You can see this moved along
474
00:26:46.770 --> 00:26:50.583
fairly, fairly quickly to sustainment.
475
00:26:52.410 --> 00:26:55.020
And you can see right here, right,
476
00:26:55.020 --> 00:26:57.900
this is where they were hung up at this site.
477
00:26:57.900 --> 00:26:59.700
This is where the process stalled
478
00:26:59.700 --> 00:27:01.890
between this formative eval
479
00:27:01.890 --> 00:27:04.173
and the implementation planning meeting.
480
00:27:05.490 --> 00:27:07.560
And that's because they ran into these problems
481
00:27:07.560 --> 00:27:08.800
with the SPED dashboard
482
00:27:09.870 --> 00:27:11.973
and then with staffing difficulties.
483
00:27:13.530 --> 00:27:15.150
And then they very quickly,
484
00:27:15.150 --> 00:27:19.563
once they got roll in here, they very quickly moved on.
485
00:27:21.720 --> 00:27:24.780
So this is what's really allowing us insights
486
00:27:24.780 --> 00:27:26.850
into implementation as a process.
487
00:27:26.850 --> 00:27:30.960
And I think that I might take this one step up
488
00:27:30.960 --> 00:27:32.970
and actually start plotting the barriers
489
00:27:32.970 --> 00:27:34.743
and facilitators on here.
490
00:27:35.760 --> 00:27:37.350
I think that could be really helpful.
491
00:27:37.350 --> 00:27:40.620
The really qualitatively important ones.
492
00:27:40.620 --> 00:27:43.710
And then imagine how that's going to enable us
493
00:27:43.710 --> 00:27:47.080
to look across sites and see what's happening
494
00:27:48.000 --> 00:27:49.500
across these sites and see if we can
495
00:27:49.500 --> 00:27:51.153
pick up on any patterns.
496
00:27:55.860 --> 00:27:59.280
So what are we doing? What are we doing with these data?
497
00:27:59.280 --> 00:28:01.470
Of course it's fun just to play with the data, right?
498
00:28:01.470 --> 00:28:03.450
I mean, I'm more than happy just to play with data,
499
00:28:03.450 --> 00:28:05.397
if that's what you want me to do.
500
00:28:05.397 --> 00:28:09.420
But on a project like this, we want to use qualitative data
501
00:28:09.420 --> 00:28:14.420
to inform facilitation/implementation.
502
00:28:14.550 --> 00:28:18.090
And some of the recommendations that we fed back
503
00:28:18.090 --> 00:28:20.940
to the larger team is that they need
504
00:28:20.940 --> 00:28:24.420
to initiate contact with the leaders there on site
505
00:28:24.420 --> 00:28:26.640
early in the process of implementation.
506
00:28:26.640 --> 00:28:31.470
Because in the VA where you have this incredible hierarchy
507
00:28:31.470 --> 00:28:34.380
and you have your Caring Contact specialists,
508
00:28:34.380 --> 00:28:38.128
who are the ones identifying the veterans,
509
00:28:38.128 --> 00:28:41.730
getting the cards printed, signing the cards,
510
00:28:41.730 --> 00:28:45.450
and mailing the cards, they are really low down
511
00:28:45.450 --> 00:28:48.960
on this hierarchy, and they actually sometimes don't know
512
00:28:48.960 --> 00:28:51.750
who to even talk to, to get what they need.
513
00:28:51.750 --> 00:28:55.560
And they do not often have direct contact with leaders.
514
00:28:55.560 --> 00:28:58.110
So you need to involve leaders in very early on
515
00:28:58.110 --> 00:28:59.814
in the process, because they're the ones
516
00:28:59.814 --> 00:29:02.883
that help you get over these barriers.
517
00:29:04.500 --> 00:29:06.850
Stakeholders were saying that they weren't
518
00:29:08.379 --> 00:29:11.370
really clear on what the roles were,
519
00:29:11.370 --> 00:29:14.580
what was expected of them, how much time it would take
520
00:29:14.580 --> 00:29:17.373
to implement and what the costs would be.
521
00:29:18.330 --> 00:29:23.330
And so the feedback was to communicate more clearly
522
00:29:23.400 --> 00:29:25.650
with stakeholders upfront
523
00:29:25.650 --> 00:29:28.233
about these aspects of implementation.
524
00:29:29.070 --> 00:29:30.510
Ensure that the site leaders
525
00:29:30.510 --> 00:29:33.123
are both influential and engaged.
526
00:29:34.590 --> 00:29:37.560
So they have to be at the implementation planning meetings
527
00:29:37.560 --> 00:29:39.390
and they have to be talking with the people
528
00:29:39.390 --> 00:29:42.840
who are implementing the interventions
529
00:29:42.840 --> 00:29:45.030
and empower stakeholders with knowledge.
530
00:29:45.030 --> 00:29:46.590
Who do I talk to?
531
00:29:46.590 --> 00:29:50.433
Who do I go to if I need to get something printed?
532
00:29:54.030 --> 00:29:57.660
Some unexpected insights that speak really,
533
00:29:57.660 --> 00:29:59.920
to implementation in general
534
00:30:01.050 --> 00:30:04.540
is, you know, what constitutes a barrier at one site
535
00:30:05.940 --> 00:30:09.090
isn't necessarily a barrier at another.
536
00:30:09.090 --> 00:30:12.810
And what we sometimes think of as facilitators
537
00:30:12.810 --> 00:30:15.120
aren't always facilitators.
538
00:30:15.120 --> 00:30:17.910
So leadership involvement can mean many different things.
539
00:30:17.910 --> 00:30:21.330
It can mean helping and empowering stakeholders
540
00:30:21.330 --> 00:30:24.085
to implement something, or it can mean pressuring them
541
00:30:24.085 --> 00:30:26.520
to implement something.
542
00:30:26.520 --> 00:30:28.560
And you want leaders involved in the right way.
543
00:30:28.560 --> 00:30:31.290
Virtual facilitation, it works for some sites
544
00:30:31.290 --> 00:30:32.433
and not for others.
545
00:30:33.390 --> 00:30:37.140
Even sites with every advantage experience delays,
546
00:30:37.140 --> 00:30:38.820
as I said earlier.
547
00:30:38.820 --> 00:30:42.510
Implementation readiness, particularly in a time of COVID,
548
00:30:42.510 --> 00:30:44.550
it fluctuates over time.
549
00:30:44.550 --> 00:30:46.921
A site that's perfectly ready in one moment
550
00:30:46.921 --> 00:30:49.893
might not be a month down the road.
551
00:30:50.910 --> 00:30:53.370
I know there's a move in implementation science
552
00:30:53.370 --> 00:30:57.210
to try to measure implementation readiness.
553
00:30:57.210 --> 00:31:00.150
And this is a real challenge to that movement,
554
00:31:00.150 --> 00:31:02.160
because a site that's ready now
555
00:31:02.160 --> 00:31:04.383
won't necessarily be ready tomorrow.
556
00:31:06.210 --> 00:31:09.213
And what's a barrier or a facilitator,
557
00:31:10.050 --> 00:31:12.841
you know, the definition of that changes from site to site.
558
00:31:12.841 --> 00:31:15.033
So it's really complicated.
559
00:31:18.900 --> 00:31:21.693
And how do we establish rigor in this process?
560
00:31:24.390 --> 00:31:26.427
One thing that my colleagues really respect me
561
00:31:26.427 --> 00:31:30.480
for is my rigorous approach to qualitative research.
562
00:31:30.480 --> 00:31:33.480
And we establish rigor at multiple levels
563
00:31:33.480 --> 00:31:35.910
always during the process of analysis.
564
00:31:35.910 --> 00:31:39.077
First we, for this study, we independently templated
565
00:31:39.077 --> 00:31:41.860
the first three debriefs together
566
00:31:43.410 --> 00:31:46.825
and compared our debriefs to make sure our templates,
567
00:31:46.825 --> 00:31:49.680
to make sure we were all on the same page
568
00:31:49.680 --> 00:31:54.303
and everyone was interpreting the categories the same way.
569
00:31:55.350 --> 00:31:58.080
At that point, I felt like we were good to go.
570
00:31:58.080 --> 00:32:01.140
Every fourth template still
571
00:32:01.140 --> 00:32:04.053
is audited by a secondary analyst.
572
00:32:06.090 --> 00:32:09.840
When I'm working in the template or in the matrices,
573
00:32:09.840 --> 00:32:14.790
I can see if a template has been consistently,
574
00:32:14.790 --> 00:32:18.720
if content has been consistently organized
575
00:32:18.720 --> 00:32:20.010
in the template compared
576
00:32:20.010 --> 00:32:22.380
to what they were in the past, right?
577
00:32:22.380 --> 00:32:25.637
Because I have prior content
578
00:32:27.570 --> 00:32:31.110
from prior templates in that matrix.
579
00:32:31.110 --> 00:32:34.860
And so if I see a discrepancy in how people are
580
00:32:34.860 --> 00:32:38.700
defining those categories, I kick it back to them,
581
00:32:38.700 --> 00:32:39.990
and I explain to them,
582
00:32:39.990 --> 00:32:42.750
I think this is how we're interpreting this.
583
00:32:42.750 --> 00:32:44.610
What do you think?
584
00:32:44.610 --> 00:32:47.580
And then I use the Word's comments feature
585
00:32:47.580 --> 00:32:51.750
to initiate a dialogue with those primary analysts,
586
00:32:51.750 --> 00:32:55.380
not only to ensure that we're being consistent
587
00:32:55.380 --> 00:32:59.700
in templating, but also to verify my own insights
588
00:32:59.700 --> 00:33:01.290
into the data.
589
00:33:01.290 --> 00:33:03.840
Sometimes those come just looking at the templates
590
00:33:03.840 --> 00:33:06.090
rather than looking at the matrices.
591
00:33:06.090 --> 00:33:08.640
And I'll show you what that looks like in a second.
592
00:33:09.690 --> 00:33:14.690
This is a template after I've looked at it (laughing)
593
00:33:17.130 --> 00:33:20.040
and before I've moved it into a matrix,
594
00:33:20.040 --> 00:33:23.809
and they aren't always quite this hacked into,
595
00:33:23.809 --> 00:33:26.430
but you can see where I have
596
00:33:26.430 --> 00:33:28.230
further reduced the data on here.
597
00:33:28.230 --> 00:33:30.813
I don't need people's names. That's a lot of noise.
598
00:33:32.520 --> 00:33:35.220
I think I picked this up and moved it somewhere else.
599
00:33:35.220 --> 00:33:37.920
You can see I've done quite a bit in here.
600
00:33:37.920 --> 00:33:41.070
And for every edit, for every question,
601
00:33:41.070 --> 00:33:44.640
I leave a comment here for the primary analyst
602
00:33:44.640 --> 00:33:47.070
who was also the one who led the debrief,
603
00:33:47.070 --> 00:33:49.170
just to make sure that I'm understanding
604
00:33:49.170 --> 00:33:50.340
things correctly, right?
605
00:33:50.340 --> 00:33:52.485
Because I am now three people removed
606
00:33:52.485 --> 00:33:54.357
from the sources of data.
607
00:33:54.357 --> 00:33:57.990
And so this dialogue is so important,
608
00:33:57.990 --> 00:34:00.591
this constant dialogue with my team
609
00:34:00.591 --> 00:34:02.910
to help keep me on track.
610
00:34:02.910 --> 00:34:06.660
Not just them, but keep me on track
611
00:34:06.660 --> 00:34:08.730
that my insights are really valid.
612
00:34:08.730 --> 00:34:10.980
And of course, I occasionally kick things back
613
00:34:10.980 --> 00:34:15.273
to the larger team and the facilitators to verify with them.
614
00:34:20.499 --> 00:34:21.780
You know, I wanna be really forthright
615
00:34:21.780 --> 00:34:23.550
about these rapid methods.
616
00:34:23.550 --> 00:34:26.310
I know there's a lot of curiosity about them,
617
00:34:26.310 --> 00:34:29.433
and there's advantages and challenges to using them.
618
00:34:30.690 --> 00:34:34.800
With templates, like I said, they provide this beautiful,
619
00:34:34.800 --> 00:34:38.970
coherent story of a single data collection episode.
620
00:34:38.970 --> 00:34:42.360
It isn't like coding where you sort of fragment your data
621
00:34:42.360 --> 00:34:45.213
and then you have all these codes floating around.
622
00:34:46.500 --> 00:34:48.753
It's right there in a Word document.
623
00:34:49.590 --> 00:34:53.220
You can use them if you're using, you know,
624
00:34:53.220 --> 00:34:55.860
like, a framework from implementation science
625
00:34:55.860 --> 00:34:57.240
to structure your template.
626
00:34:57.240 --> 00:35:00.018
It creates this transdisciplinary language
627
00:35:00.018 --> 00:35:02.400
that an anthropologist can use
628
00:35:02.400 --> 00:35:05.220
to talk with other scientists.
629
00:35:05.220 --> 00:35:09.270
And your results are translated much more efficiently.
630
00:35:09.270 --> 00:35:11.070
They're translated by your template.
631
00:35:11.070 --> 00:35:13.650
If you're skilled at template analysis,
632
00:35:13.650 --> 00:35:17.193
your template will translate your data for you.
633
00:35:18.360 --> 00:35:20.970
The challenge really with individual templates is
634
00:35:20.970 --> 00:35:23.468
that they look very simple.
635
00:35:23.468 --> 00:35:25.830
People look at a completed template
636
00:35:25.830 --> 00:35:27.990
and they think, "Oh, I can do that.
637
00:35:27.990 --> 00:35:29.340
That looks easy."
638
00:35:29.340 --> 00:35:32.304
But it takes a remarkable amount of work
639
00:35:32.304 --> 00:35:35.070
to get it to that point, right?
640
00:35:35.070 --> 00:35:36.420
There's reducing the data.
641
00:35:36.420 --> 00:35:40.383
What do I cut out and what don't I cut out? That is a skill.
642
00:35:41.520 --> 00:35:44.100
There's what I call sinking in and reading deeper.
643
00:35:44.100 --> 00:35:46.620
You can't just stay at the surface.
644
00:35:46.620 --> 00:35:49.650
Sometimes you have to unpack what people are saying,
645
00:35:49.650 --> 00:35:52.890
and you still have to do that with template analysis.
646
00:35:52.890 --> 00:35:56.250
You can't get in copy and paste mode and just go.
647
00:35:56.250 --> 00:35:59.850
You have to continually sink in and dig deeper.
648
00:35:59.850 --> 00:36:01.263
And you know, the same challenge
649
00:36:01.263 --> 00:36:03.420
that you get with coding, right?
650
00:36:03.420 --> 00:36:05.580
Controlling the codes.
651
00:36:05.580 --> 00:36:07.316
You can get to the point where you have,
652
00:36:07.316 --> 00:36:10.470
if you let yourself run away with it,
653
00:36:10.470 --> 00:36:12.930
far too many subcategories
654
00:36:12.930 --> 00:36:16.140
to be really helpful to you at all.
655
00:36:16.140 --> 00:36:18.413
So you have to really control the proliferation
656
00:36:18.413 --> 00:36:20.940
of categories and subcategories.
657
00:36:20.940 --> 00:36:23.460
And of course, maintaining consistency
658
00:36:23.460 --> 00:36:25.863
across templates is always a challenge.
659
00:36:27.720 --> 00:36:29.667
Matrices, of course, they do reduce your data,
660
00:36:29.667 --> 00:36:31.500
and that's helpful when you're dealing
661
00:36:31.500 --> 00:36:33.570
with such a large data set.
662
00:36:33.570 --> 00:36:37.353
And it helps to ensure that consistency across templates.
663
00:36:39.060 --> 00:36:41.010
And you know, as you've seen it permits
664
00:36:41.010 --> 00:36:44.313
longitudinal comparisons and assists
665
00:36:44.313 --> 00:36:47.403
in the development of these case summaries.
666
00:36:49.290 --> 00:36:51.783
But those matrices have a lot on them.
667
00:36:52.980 --> 00:36:54.990
And you have to find your way through
668
00:36:54.990 --> 00:36:57.450
all those words, words, words, right?
669
00:36:57.450 --> 00:36:59.460
I mean, it's still just a lot to deal with.
670
00:36:59.460 --> 00:37:02.727
And the matrices don't analyze the data for you.
671
00:37:02.727 --> 00:37:04.860
You don't stick them in the matrix
672
00:37:04.860 --> 00:37:07.470
and suddenly the magic happens.
673
00:37:07.470 --> 00:37:10.770
Your brain does that. It's still qualitative research.
674
00:37:10.770 --> 00:37:12.180
You have to make sense out
675
00:37:12.180 --> 00:37:14.103
of what the matrix is telling you.
676
00:37:15.030 --> 00:37:16.983
So that's challenging.
677
00:37:18.540 --> 00:37:20.940
And then finally, the overall approach,
678
00:37:20.940 --> 00:37:24.000
I mean, it really is allowing us to see implementation
679
00:37:24.000 --> 00:37:27.420
as this dynamic process that shifts over times,
680
00:37:27.420 --> 00:37:30.183
and sometimes it moves backwards actually,
681
00:37:31.290 --> 00:37:34.110
so that we can use those findings to, you know,
682
00:37:34.110 --> 00:37:37.470
inform these course corrections, if you would.
683
00:37:37.470 --> 00:37:39.420
It's allowing us to share across sites
684
00:37:39.420 --> 00:37:43.320
without really delaying our project.
685
00:37:43.320 --> 00:37:45.774
We've experienced no delays whatsoever
686
00:37:45.774 --> 00:37:50.520
using Word documents instead of computer software.
687
00:37:50.520 --> 00:37:52.650
And it's very rigorous when you put
688
00:37:52.650 --> 00:37:54.513
all these techniques together.
689
00:37:55.620 --> 00:37:57.840
You know, the challenge is, for me,
690
00:37:57.840 --> 00:38:00.990
being three levels removed from the sources of data,
691
00:38:00.990 --> 00:38:03.120
that's very challenging.
692
00:38:03.120 --> 00:38:07.470
I have to continually kick things back to my team
693
00:38:07.470 --> 00:38:10.160
and to not only team qual, but the larger team
694
00:38:10.160 --> 00:38:13.770
to ensure that my insights are valid.
695
00:38:13.770 --> 00:38:17.130
And it really requires consistent engagement.
696
00:38:17.130 --> 00:38:22.090
This is not a collection of techniques
697
00:38:24.690 --> 00:38:28.050
that will allow a qualitative team lead
698
00:38:28.050 --> 00:38:32.400
to just sit back and show up at weekly meetings
699
00:38:32.400 --> 00:38:34.257
and say, "So what's going on?"
700
00:38:35.910 --> 00:38:38.820
You have to consistently engage with your data,
701
00:38:38.820 --> 00:38:41.370
you have to keep your team motivated,
702
00:38:41.370 --> 00:38:44.940
and project management is absolutely vital.
703
00:38:44.940 --> 00:38:46.997
Everyone has to be on top of things,
704
00:38:46.997 --> 00:38:50.970
because we all build, right, one after another.
705
00:38:50.970 --> 00:38:54.390
So if somebody drops the ball, somebody else has to wait
706
00:38:54.390 --> 00:38:56.090
for them to pick it back up again.
707
00:38:57.960 --> 00:38:59.997
Stop and see if anyone has questions?
708
00:38:59.997 --> 00:39:01.847
I'm done.
Awesome.
709
00:39:04.110 --> 00:39:04.943
Yeah...
710
00:39:10.579 --> 00:39:12.162
That was all right.
711
00:39:13.274 --> 00:39:15.690
(participant speaking indistinctly)
712
00:39:15.690 --> 00:39:17.610
Ashley, are you available?
713
00:39:17.610 --> 00:39:18.443
Yep, I'm here.
714
00:39:18.443 --> 00:39:20.866
Yeah, so if anyone has any questions, feel free.
715
00:39:20.866 --> 00:39:24.390
We have just five minutes, but yeah,
716
00:39:24.390 --> 00:39:27.330
we'd love to have you ask some questions
717
00:39:27.330 --> 00:39:28.630
to Tracy if you have them.
718
00:39:30.360 --> 00:39:31.650
I have a question.
I...
719
00:39:31.650 --> 00:39:32.483
Oh, go ahead.
720
00:39:32.483 --> 00:39:33.565
This is not (indistinct).
721
00:39:33.565 --> 00:39:38.340
You mentioned, Tracy, that time wasn't the primary reason
722
00:39:38.340 --> 00:39:42.330
for adopting rapid qualitative analysis.
723
00:39:42.330 --> 00:39:45.120
But for me, I've thought, like, especially
724
00:39:45.120 --> 00:39:49.020
in implementation science, where you need this information
725
00:39:49.020 --> 00:39:51.060
at the stage and the formative stage
726
00:39:51.060 --> 00:39:53.370
in terms of refining the intervention,
727
00:39:53.370 --> 00:39:56.070
and you might need it for course correction,
728
00:39:56.070 --> 00:39:57.930
that the rapid aspect
729
00:39:57.930 --> 00:40:00.960
of the qualitative analysis is very important.
730
00:40:00.960 --> 00:40:04.890
And I've worked in the past with other qualitative analysts
731
00:40:04.890 --> 00:40:08.430
in studies where when they use the traditional methods,
732
00:40:08.430 --> 00:40:11.490
they're kind of on their own timeframe
733
00:40:11.490 --> 00:40:14.310
and may take, like, three or more years
734
00:40:14.310 --> 00:40:17.340
to fully process the data and write something up.
735
00:40:17.340 --> 00:40:21.093
And by that time, the actual study might even be over.
736
00:40:23.220 --> 00:40:25.830
Absolutely, well, especially if you take, you know,
737
00:40:25.830 --> 00:40:30.830
what I call slow-mo longstanding sort of approaches to it.
738
00:40:32.430 --> 00:40:36.900
For me, what I really like
739
00:40:36.900 --> 00:40:39.990
about these methods is the translation,
740
00:40:39.990 --> 00:40:44.898
the ease with which you move from analysis to translating
741
00:40:44.898 --> 00:40:48.420
your findings into something that's meaningful.
742
00:40:48.420 --> 00:40:53.310
And that's always why I keep going back to rapid methods
743
00:40:53.310 --> 00:40:57.120
and because I find them more challenging, honestly.
744
00:40:57.120 --> 00:40:59.793
And I really like that challenge.
745
00:41:05.139 --> 00:41:08.513
Did you wanna ask your question?
746
00:41:08.513 --> 00:41:10.515
May I just say something?
747
00:41:10.515 --> 00:41:14.760
I want to add a little bit more
748
00:41:14.760 --> 00:41:16.050
to this response.
749
00:41:16.050 --> 00:41:19.983
And thank you very much for this wonderful presentation.
750
00:41:21.060 --> 00:41:23.940
I mean, I'm doing this on a lot of projects,
751
00:41:23.940 --> 00:41:26.100
and sometimes what, you know,
752
00:41:26.100 --> 00:41:29.197
what you said was very striking that everybody thinks,
753
00:41:29.197 --> 00:41:31.050
"Oh, this is simple, I can do it."
754
00:41:31.050 --> 00:41:35.310
But what goes into it is, you know,
755
00:41:35.310 --> 00:41:38.580
a lot, and that rigor essentially
756
00:41:38.580 --> 00:41:41.550
is what makes those insights useful
757
00:41:41.550 --> 00:41:43.320
for the implementation process.
758
00:41:43.320 --> 00:41:46.830
So what I wanted to add to the use of rapid analysis
759
00:41:46.830 --> 00:41:50.670
and Donna's question that it is time bound,
760
00:41:50.670 --> 00:41:55.020
I think yes, it's extremely useful,
761
00:41:55.020 --> 00:41:56.370
and that's why, like you said,
762
00:41:56.370 --> 00:41:58.710
you keep going back to these methods.
763
00:41:58.710 --> 00:42:01.510
What I have experienced in my work
764
00:42:02.580 --> 00:42:06.570
is that sometimes it's the data size as well.
765
00:42:06.570 --> 00:42:09.900
It might not be urgent, but the data size
766
00:42:09.900 --> 00:42:13.380
can also dictate whether you want to use the rapid methods
767
00:42:13.380 --> 00:42:16.410
or if you want to go in much more detail.
768
00:42:16.410 --> 00:42:18.840
So just a small thing.
769
00:42:18.840 --> 00:42:22.080
But I think that also is one of the reasons.
770
00:42:22.080 --> 00:42:23.760
By the way, my name is Fauzia Malik
771
00:42:23.760 --> 00:42:26.310
and I'm a medical anthropologist part of-
772
00:42:26.310 --> 00:42:28.008
Oh, hi. (laughing)
773
00:42:28.008 --> 00:42:31.893
(laughing) Part of Yale School of Public Health,
774
00:42:32.910 --> 00:42:35.100
Health Policy and Management Department.
775
00:42:35.100 --> 00:42:39.870
And I absolutely loved the way you presented
776
00:42:39.870 --> 00:42:44.870
the use of data and very, very important points
777
00:42:44.910 --> 00:42:48.103
that you all brought together to make sense of this,
778
00:42:48.103 --> 00:42:51.090
you know, application of rapid analysis
779
00:42:51.090 --> 00:42:52.740
in implementation science.
780
00:42:52.740 --> 00:42:55.334
Thank you so much for that.
Oh, thank you.
781
00:42:55.334 --> 00:42:56.520
And thank you, Donna and Ashley,
782
00:42:56.520 --> 00:42:58.296
for organizing this.
783
00:42:58.296 --> 00:43:01.173
This was a pleasure to hear.
784
00:43:02.490 --> 00:43:05.280
Great. Hopefully this will be the first of many.
785
00:43:05.280 --> 00:43:08.280
Ashley, there's a question in the chat if there's time,
786
00:43:08.280 --> 00:43:09.270
but I think there might have been
787
00:43:09.270 --> 00:43:11.463
some other people speaking up as well.
788
00:43:15.352 --> 00:43:17.550
If you wanna take some, we have one minute,
789
00:43:17.550 --> 00:43:20.280
if you wanna take a timed question or-
790
00:43:20.280 --> 00:43:21.993
We can't even hear you, Ashley.
791
00:43:24.161 --> 00:43:29.161
So for the debriefs, they are really, you know,
792
00:43:29.250 --> 00:43:31.350
as the name suggests, they're very brief.
793
00:43:31.350 --> 00:43:34.860
They're 30 minutes long and they're bi-weekly.
794
00:43:34.860 --> 00:43:36.810
If you're talking about the time it takes
795
00:43:36.810 --> 00:43:39.630
for a team to do this, and this is something
796
00:43:39.630 --> 00:43:43.170
that people ask quite often, it's a heavy lift.
797
00:43:43.170 --> 00:43:47.640
We have a team of five of us.
798
00:43:47.640 --> 00:43:51.820
I am 30%, the other team members are
799
00:43:53.280 --> 00:43:55.440
30, 50 and 60%.
800
00:43:55.440 --> 00:43:57.640
It is a lot of time
801
00:43:58.560 --> 00:44:01.380
to collect and analyze all these data.
802
00:44:01.380 --> 00:44:03.120
It's not just from the debriefs, right?
803
00:44:03.120 --> 00:44:05.880
It's also the interviews with the stakeholders.
804
00:44:05.880 --> 00:44:09.540
And we have interviews with veterans coming up next week.
805
00:44:09.540 --> 00:44:12.960
So it's pricey, that's for sure.
806
00:44:12.960 --> 00:44:15.540
Especially if you get an expensive
807
00:44:15.540 --> 00:44:20.283
investigator level anthropologist on your team. (laughing)
808
00:44:22.350 --> 00:44:25.050
So everyone, thank you so much.
809
00:44:25.050 --> 00:44:27.355
It was lovely to have so many people log in,
810
00:44:27.355 --> 00:44:30.630
and we really look forward to, you know,
811
00:44:30.630 --> 00:44:31.890
more sessions in the future.
812
00:44:31.890 --> 00:44:33.780
And so I hope everyone will sort of join me
813
00:44:33.780 --> 00:44:36.925
in thanking Dr. Abraham for her talk.
814
00:44:36.925 --> 00:44:39.123
Yeah, and take care.
815
00:44:41.690 --> 00:44:43.213
Bye.
(voices overlapping)
816
00:44:43.213 --> 00:44:48.213
Thank you so much.
817
00:44:53.599 --> 00:44:58.182
(participants chattering indistinctly)