Costing and Economic Evaluations in Implementation Science

July 07, 2023

Information

Speaker: Drew Cameron

Thursday, 2022, April 7 at 12 pm.

ID10113

To CiteDCA Citation Guide

00:02<v ->Prior to joining the faculty at YSPH,</v>
00:06Dr. Cameron was a member
00:08of the Global Health Cost Consortium,
00:10an initiative funded
00:11by the Bill and Melinda Gates Foundation
00:14to provide decision makers
00:16with improved resources
00:17to estimate the cost
00:19of HIV and tuberculosis programs,
00:21and to improve the availability, quality, timing,
00:27and policy relevance of cost information.
00:30He holds an MA in international development
00:33from American University and a PhD in health policy
00:37from the University of California Berkeley
00:39School of Public Health.
00:41The title of his talk is
00:42"Costing and Economic Evaluations
00:45in Implementation Science."
00:46He'll speak for about 45 minutes,
00:48and then we'll open the floor for questions and discussion.
00:52There you go, drew.
00:54<v ->Thanks very much for that introduction, Debbie,</v>
00:56and thank you to those who were kind enough
00:59to join the call today.
01:02I won't introduce myself any further
01:04than to say as a member
01:06of the Center for Methods
01:08and Implementation and Prevention Science,
01:10this is a pretty sort of exciting opportunity
01:13to share some of the work
01:14that I've been involved with in the past.
01:16This talk though, I'm gonna focus largely on, you know,
01:21what might be for many of you a review of topics
01:25as they relate to economic evaluations and costing.
01:28So please do bear with me on that front.
01:31I'll try to get to some novel research findings
01:34towards the end of the talk,
01:36but this is sort of designed as a, anyone can participate.
01:40You don't have to have much of any background
01:42in economic evaluation or costing.
01:44So without much more,
01:47let me get into the topics here.
01:51Okay, so this talk is divided
01:53into three main sections.
01:54First, I'm gonna discuss some key economics concepts
01:57that kind of lay the groundwork for the rest of the talk.
01:59I'll briefly cover some different types
02:01of economic evaluation and decision analyses
02:05and how costing
02:07in particular is an important
02:08and often overlooked component of implementation science.
02:12Second, we'll get into the nitty gritty
02:14of costs and costing.
02:15So what are they, who uses them?
02:17How do you do it?
02:19And what are some of the key considerations
02:20that you have to make along the way
02:22when you are conducting a costing study?
02:27If you have a great deal of experience with costing already,
02:29some of you may find some of this to be familiar,
02:31so I'll apologize in advance,
02:33but I'm still eager for your feedback.
02:35And then finally,
02:36I'm gonna briefly discuss the efforts
02:39of the Global Health Cost Consortium,
02:40which includes a website
02:41with a ton of great costing resources.
02:43So if you're more interested
02:44in costing and economic evaluation,
02:47I can direct you towards those at the end of the talk.
02:50And as part of that effort,
02:51I'll share some of the results
02:52of a somewhat recent publication,
02:54a systematic review of costing literature on HIV
02:57in Sub-Saharan Africa,
02:58and another study that's in progress.
03:02So let me move ahead here.
03:05So there are four main takeaways.
03:06You know, if you take nothing else from this talk,
03:10and I wouldn't blame you if you didn't.
03:13Four things that I'd like you to walk away from this talk,
03:17considering like number one is perspective.
03:20I'll get into what perspective is, and timeframe.
03:24These are two concepts that matter,
03:26I think the absolute most in terms of economic evaluations
03:30and costing and how they're conducted and the findings,
03:33you know, that you were able to generalize
03:35from these studies.
03:37I wanna also leave you with the impression
03:40that although expenditures are real,
03:42they're real things,
03:43things you can quantify,
03:45costs really aren't.
03:47They're conceptual in nature,
03:49and they depend on a lot of different assumptions.
03:52So, you know, we'll get into the nitty gritty of that.
03:55I'd also like to leave you with the idea
03:57that perfect is often the enemy of the good.
04:00I think that's a concept that's very familiar
04:01to anybody who's ever published a paper, for example.
04:04But on the flip side,
04:06little details in your decision making and costing
04:10and data collection can matter a lot.
04:13So there's a balance that we try to strike
04:16when we do these types of studies.
04:18And then finally,
04:19reporting standards and quality are two areas
04:22that have a lot of room to improve in this field.
04:24And I'll show you some examples of why.
04:27So what is economic evaluation?
04:29Economic evaluation is an often sort of misunderstood term.
04:33You've probably heard of monitoring and evaluation,
04:35which typically refers to monitoring processes
04:37for a program's implementation and ongoing operation.
04:41If you work in the social sciences,
04:42you've probably heard of impact evaluation,
04:44which typically refers to evaluating, say, health, social,
04:48or economic impacts of the program.
04:50These are all separate but related fields and methods.
04:54Economic evaluation is sort of an umbrella term.
04:58This includes a broad range of comparative methods
05:00to help to evaluate
05:01choices or trade-offs between costs and benefits.
05:05Another familiar term might be decision analysis.
05:08At the heart though of economic evaluation
05:10and decision analysis is a concept that, you know,
05:13we spoke about or that I'll speak about a little further
05:18into the talk.
05:18And that is opportunity costs.
05:20So when we talk about economic evaluation,
05:23what we typically wanna understand
05:24is what's the best value for money?
05:26And in the most basic sense,
05:28cost is what you pay and value is what you get.
05:31It's the benefit.
05:33So it won't always indicate a clear choice,
05:36but it will help to evaluate options quantitatively
05:40based on a defined set of decisions.
05:43These processes, they aim to capture opportunity costs
05:47of doing one thing over another.
05:48So for example,
05:49do we invest in home visits
05:52or do we make mass media platforms
05:55to disseminate behavior change communication
05:57for improved, say infant and young child feeding practices,
06:02or do we promote home gardens or vitamin A capsule
06:05supplementations to address micronutrient deficiencies?
06:11So lastly, I'll leave you
06:12with the thought that economic evaluations
06:14are comparative by nature.
06:15So not only do they compare costs and benefits,
06:18but a comprehensive evaluation
06:20will compare two or more different options.
06:23So let's look at a couple of different types
06:25of economic evaluations.
06:26So you may have come across these in the past.
06:30There's really a large family
06:31of economic evaluation methods,
06:33but it's important to note that all of these different types
06:36of analyses and tools at their core
06:39are measuring costs in dollars.
06:40So that that column down the middle.
06:43And that's the main focus of the rest of this talk.
06:46But a key difference across these different types
06:50of evaluations is gonna be whose costs it is
06:54that we're taking into account
06:55as well as how we're measuring benefits.
06:58So all decision analyses are comparative,
07:00as I mentioned,
07:01we're gonna compare one or more options
07:03in relation to another.
07:04So that other option could just be
07:06what the current standard
07:07of care is or the status quo.
07:09It could be doing nothing
07:11or it could be another active intervention.
07:13So one of the first things that we need to do
07:15in any such analysis
07:16is to understand where a comparator is
07:18and the most basic type of economic evaluation
07:21is the one on the top there.
07:23We'll start with cost efficiency analysis,
07:25which is sort of a partial form of economic appraisal
07:28'cause it looks only at the costs
07:29of the program and it doesn't
07:31really provide any information on let's say agriculture,
07:34nutrition, or health outcome of interest.
07:37This typically reports out on the cost per output achieved.
07:40So another example of this might be
07:41like a budget impact analysis where you're looking
07:44at the cost per output, per patient served,
07:47or per vaccine administered.
07:51Now the more common type of economic evaluation
07:54for evaluating health interventions
07:56is cost effectiveness analysis.
07:58And CEAs, they compare the cost and outcomes of two or more
08:02alternatives or compare a new intervention
08:06or treatment with the status quo.
08:07CEAs relate the cost associated
08:10with the health outcome such as cost per disease avoided,
08:14cost per death avoided
08:15or additional expected life year for example.
08:19And then there's some other types of analysis
08:21that we won't get into too much.
08:23Cost utility analysis,
08:25you know, includes analyses with say disability
08:29or quality adjusted life years
08:32or other standardized measures of preference.
08:37And then there's cost benefit analysis,
08:39which is a method in which both the costs and the benefits
08:42of the intervention are expressed in monetary terms.
08:45So in agriculture for example,
08:47this might be the value of increased crop production.
08:51While in health it would be the dollar value associated
08:54with the life years that were gained
08:56because of an intervention.
08:58There's several other types of decision analysis
09:01that I haven't included in this table
09:03and they exist, so sorry.
09:06So those include return on investment analysis,
09:10cost of illness analysis and others.
09:13So as I've suggested,
09:14different sectors kind of use different of these approaches.
09:18So for example,
09:19global health and education disciplines
09:21have promoted the use of cost effectiveness analysis.
09:25Implementation science probably naturally falls
09:28under this category
09:30or in the cost efficiency analysis category
09:33with a little bit more emphasis
09:34on perhaps cost efficiency
09:36depending on the scope of the evaluation taking place.
09:39And I'll get into that in a moment.
09:42Meanwhile, you might see agriculture
09:45or large scale investment
09:46in infrastructure being evaluated using
09:49say a cost benefit analysis.
09:53So zooming out to sort of the level that we often interact
10:00with these types of evaluations,
10:02say an aim three of your NIH proposal,
10:05depending of course on your desired evaluation type,
10:09you'll often wanna arrive
10:11at really a simple division problem.
10:13The cost of a program or policy or intervention
10:16divided by the outcome in question.
10:18For cost effectiveness analysis,
10:20the outcome is some measure of effectiveness,
10:23particularly a health outcome.
10:27For budget impact analysis and cost efficiency.
10:29This might be an intermediate outcome
10:32per person vaccinated or per patients served.
10:36But at its core,
10:37all the economic evaluations boil down
10:39to this simple division problem.
10:41And I wanna mention one more
10:43slightly more complicated
10:45but still simple division problem
10:47that you may have come across at some point, you know,
10:51in the process of of putting together a proposal.
10:54And that is one that builds very simply on the previous,
10:58and that's for cost effectiveness analysis
11:01that compares the cost effectiveness
11:03of the intervention and question
11:05to some particular alternative.
11:06So that's, that's the ICER.
11:08And this stands for incremental cost effectiveness ratio.
11:12So usually the comparator here is a status quo
11:15or a standard of care,
11:17but it might be some new alternative you're considering too.
11:21<v Person>That is very useful for developers.</v>
11:24<v ->I dunno if I'm getting a question here or not,</v>
11:27but I'll just continue about that.
11:30You can also use ICERs to compare
11:32between multiple different
11:33alternative interventions or configurations.
11:36You might end up with like a list of different interventions
11:38that you want to choose between
11:40using a process like a shopping spree scenario.
11:45I won't get into the details of that,
11:46but there's lots of different ways
11:47that you can use this tool.
11:52Lemme move on.
11:53In this talk though,
11:54I'm mostly concerned with the numerator.
11:56So how do we choose what counts and what doesn't count?
11:59What goes into the process of collecting data
12:01to inform these numbers
12:03and how might we begin to distinguish
12:05between studies that do that well or do that poorly?
12:08So I wanna get through a couple of key concepts here
12:10before we get started.
12:12And those are scarcity, opportunity costs,
12:15types of efficiency and perspective.
12:18And I'll linger on perspective a bit longer
12:20than the others.
12:21Scarcity is one of the most important concepts in economics
12:24and that's simply stated, resources are limited.
12:27So whether we have relatively more resources
12:30as within the US or say other OECD countries,
12:33or relatively fewer resources
12:35as many low and middle income countries find,
12:39there's always trade offs to be made in terms of investments
12:42and competing programs and activities.
12:44So it's a major consideration.
12:47The most powerful implication of the concept of scarcity
12:50is of course opportunity costs.
12:52So this refers to the simple truth that the use of resources
12:55for any one purpose precludes their use for another.
12:58And I have to say that opportunity cost is is a really great
13:01concept for focusing the mind by counteracting
13:05our own personal tendencies towards inertia,
13:08which is to say the tendency to say unthinkingly continue
13:12this year, what worked or what we did last year,
13:15or for that matter,
13:16what didn't work particularly well last year.
13:19So thinking about opportunity costs stimulates
13:22some creativity about the best use of resources
13:25given the needs and alternatives that exist.
13:29And the problem of resource allocation
13:30then becomes where can we get
13:33the greatest benefit for a given expenditure.
13:36And cost effectiveness analysis provides
13:38the sort of conceptual framework and methods
13:41to help make that determination.
13:43Efficiency.
13:44I'll just touch on briefly,
13:46there's two major types.
13:47Allocative efficiency,
13:48which con concerns the division of funds
13:51among different interventions.
13:53So HIV prevention versus treatment.
13:56This often involves complex choices,
13:59even if the health gains achievable in one area are greater
14:02than in another because costs also differ.
14:05And then there's technical efficiency,
14:07which is about how we implement an intervention
14:08that we've decided upon.
14:10So for example,
14:11what's the best program designed
14:14to deliver ARV therapy?
14:17Both kinds of efficiency require good cost data
14:20in order to properly assess them.
14:22However, technical efficiency is only about costs,
14:24whereas allocative efficiency
14:26concerns health impacts as well.
14:29And then finally, perspective,
14:30and I'll spend a little more time here
14:31'cause I think this is a really key thing
14:34that we all need to be on the same level with.
14:37This is perhaps the most important concept.
14:41So different stakeholders
14:42have different experiences of cost.
14:45There's the societal perspective,
14:46which is what we consider the most inclusive
14:49and is often recommended for most policy analyses
14:54in low and middle income countries.
14:57In this case,
14:58all costs are considered regardless of who incurs them
15:01within a society.
15:03However, considering the perspective
15:06of specific stakeholders is particularly useful
15:09in helping to address those stakeholder concerns.
15:13So will a payer or an insurance provider lose money
15:17even though a society gains in net savings?
15:20Or should we align financial incentives
15:22and how do we go about accomplishing that?
15:26So here's another sort of useful graphic
15:29for thinking about perspectives
15:30and in the context of a cost effectiveness analysis,
15:33these are the areas in which you might consider your choice
15:35of inputs for costs and benefits
15:37for a programmer intervention.
15:40So you'll see that in the rows of the table here.
15:43So many decision analyses are gonna be taking a societal
15:46perspective to start,
15:49but let's drill down a little bit more
15:52where we're talking about the hospital perspective
15:55or a payer perspective.
15:57So this is sort of the narrowest definition.
16:01They'll have their own particular interests
16:03and we can also think about this being done
16:06from the perspective of an employer,
16:08how we might think about the healthcare system as a whole.
16:12The next level out would be the payer perspective,
16:19and that captures the insurer
16:22and that that may be a government entity.
16:25So Medicare, Medicaid,
16:27and then zooming out even further
16:30the societal perspective.
16:32And that captures, you know, this broader range of costs.
16:36Okay?
16:40I wanna mention then the question arises,
16:44which of these perspectives should we choose?
16:47So in implementation science,
16:51it often depends on the priorities
16:57of the those on the ground partners
17:01and the institutions that you're working with
17:02when you roll out one of these studies.
17:05So in low and middle income country settings,
17:08the guidance from traditional reference case examples,
17:13and I'll touch on those in a bit,
17:15tend to encourage including
17:17this broader set of perspectives.
17:20So the societal
17:21or the health system perspective.
17:24However, narrower perspectives
17:25are usually these are more directed at the interests
17:29of key stakeholders,
17:30and that may be the number one priority
17:33in any one of these examples.
17:37So it really, you know,
17:39in short it really depends on the priorities
17:41of your partner agencies
17:45and those who are rolling out
17:47whatever this intervention happens to be.
17:51You know, there's a really great quote
17:53from this paper
17:54by Eisman and colleagues
17:55that just came out last year,
17:57and really it's a pragmatic answer.
17:59"The perspective of the stakeholder and the decision makers
18:01who will be informed by the analysis
18:04should probably be prioritized."
18:06But you can also conduct analysis at multiple levels,
18:10you know, as long as you're able
18:11to distinguish between those different levels
18:14when you write things up.
18:15So just keep in mind in this process,
18:19the organizations that are adopting evidence-based policies,
18:22they really need to know what it's gonna cost them
18:24in their setting, given their resources.
18:28But then there's a trade off
18:29and that is whose reality counts
18:32and whose being missed
18:33by these types of analyses?
18:35I don't know if those on the call have ever read anything
18:39by Robert Chambers, but you know,
18:41I think a lot about
18:44one of his early books, "Whose Reality Counts,"
18:48when it comes to development interventions
18:49when I think about this idea of perspective.
18:55Another consideration that's closely related
18:58to perspective is timeframe.
19:00And so that is not just what costs and benefits
19:03should we collect, but like when and where
19:05in the project life cycle itself
19:07is it appropriate to collect 'em?
19:09So one option, you know, and we'll start
19:12with the first in that list there
19:13is to focus solely on the short term healthcare costs,
19:16which could include the costs of the intervention,
19:19costs of downstream stream healthcare.
19:22This approach is consistent with,
19:24I think, you know,
19:25most budget impact analyses, frameworks,
19:28particularly those mentioned
19:30in the CHEERS guidelines,
19:33which I'll provide some resources for later.
19:36And in the context of costing,
19:38this is usually on the order of about six months
19:41after a program's been implemented.
19:43So it's once that program is reached maturity,
19:46you know, what does it cost to continue to implement?
19:50But that ignores another set of questions,
19:52which is, you know, number two here,
19:55the intervention startup.
19:57And it might be extremely valuable,
19:58the organizations and key stakeholders
20:00who are carrying out
20:01that evidence-based program
20:04to include these costs
20:05in order to capture some sense
20:07of what it would take for a system
20:09to start to integrate new practices,
20:10how long it might take
20:11for those systems to reach maturity.
20:15Another option,
20:16the third in that list
20:18is to include additional research costs,
20:21research costs included in the design and development
20:24of implementing a new evidence-based practice.
20:28So that might be adapting that evidence-based practice
20:33to a local setting or an intervention.
20:35And it could also include some of the costs of say,
20:41monitoring and getting feedback
20:42from participants in order
20:44to adapt and to improve,
20:49I guess the implementation of that process.
20:52And then finally,
20:53there's this question
20:53of future downstream costs and benefits.
20:55And that's particularly to patients
20:57and to program beneficiaries.
20:58So that's number four in the image that you see.
21:01And this is often included, you know,
21:03as I mentioned before,
21:04in perspectives that are taken
21:05from the societal level,
21:08but all too often they fall outside
21:12of the scope of interest
21:14for implementing partners
21:15and particularly for payers.
21:17You know, and I'll just mention
21:18that this is one of the main major downsides to many,
21:21if not most economic evaluations
21:24that are conducted domestically.
21:25So when it, you know,
21:26this is a setting where the payer
21:28is often a private health insurer who has no monetary stake
21:31in the future health state
21:32of their clients because
21:34in most cases those customers might age out
21:39and move into a new health insurer,
21:40so Medicare, for example, when you turn 65.
21:44So I'll just mention that an analyst may choose
21:46to include any number of these different components
21:49of an intervention or these different parts
21:51of the timeframe.
21:52But in any case,
21:53it's really essential to be able to distinguish
21:55between these different sets of costs and benefits
21:57and to be explicit about the scope,
22:00particularly for generalizability
22:02and understanding limitations.
22:05So lemme go into costs and costing,
22:07I've kinda droned on for a long time here,
22:10but let's talk about what costs are.
22:15So there's definitely,
22:16there's many different sources of costs
22:19from donors and government providers,
22:21international and local NGOs,
22:23participant costs from households and individuals.
22:27There's also many different ways to describe
22:29and categorize costs,
22:31say by the type of costs.
22:33Are they real world costs or are they costs
22:35of a randomized control trial?
22:37Are they full or incremental?
22:41Or by cost categories,
22:43so different types of inputs
22:45or organized by activities.
22:48Or, you know, as I mentioned even by the timing of costs.
22:52If you remember my first major takeaway
22:53from the presentation,
22:55you should be starting to get an idea
22:56of why I say
22:57that there was my second takeaway, rather,
23:01you start to get an idea here
23:03of the notion that expenditures exist,
23:06but that costs,
23:07they're a little bit fuzzier.
23:09They're frequently abstract
23:10and they depend on a lot of different factors.
23:14So two different kinds of costs to be able
23:17to distinguish between are economic and financial costs.
23:22So financial costs are those appearing potentially
23:25in the program expenditures documents.
23:30Meanwhile, economic costs are costs
23:33that are included or that include
23:35the value of all resources,
23:37regardless of who's paying.
23:38So the difference here is resources contributed
23:42by other entities.
23:43So for example, the local hospital,
23:46the clinical staff,
23:47volunteers or subsidized costs,
23:51such as from an international agency paying for commodities.
23:56So note that when you see an economic costing exercise
24:01referred to in the literature,
24:03it typically subsumes financial costing
24:06when it's, you know, thought of in this Venn diagram.
24:13So we have a sense of what costs are
24:16and why they're important,
24:17but what exactly is costing.
24:19So we use this term costing as a shorthand
24:22to describe the estimation of the costs
24:25of producing health services, for example.
24:29So costing places of value on the total resources used
24:32to produce a good or service.
24:34It requires measurement of the amount of each resource
24:37as well as information about the price
24:39of each of those resources.
24:42So costs can vary by context.
24:45So when one is costing,
24:48it's important to gather information about location,
24:50about time period, population, and a host of other factors.
24:55Some of the different types of resources
24:57that are used to produce health services
24:59might include human resources,
25:03drugs and supplies,
25:04medical or other non-medical equipment
25:08or even patient costs.
25:10So what exactly do we use costing for?
25:13Well, costing is used
25:14in a range of research evaluations, programs,
25:18and in planning,
25:20primarily it's in the areas of financial planning
25:23and budgeting and priority setting
25:25that cost data is the most sort of visible we'll say.
25:30So for example,
25:31cost data are used to evaluate the effectiveness
25:34of an intervention or to understand
25:36the efficiency of health delivery,
25:39but what are the cost drivers
25:41and how do these vary over time?
25:43So in in ART,
25:44which is the example in the slide here,
25:48this figure is from a systematic review
25:51of the costing literature.
25:52This is one that I'll touch on later
25:54for antiretroviral treatment
25:56in low and middle income countries,
25:59it shows the different drivers
26:01of the average unit costs across five different settings.
26:06And this helps to show
26:07which cost inputs contribute most to the overall unit cost.
26:12So this figure I think does
26:13a really great job of showing the absolute difference
26:16in unit costs between settings.
26:18And in this case it's it's by country
26:20and it includes a lot of different settings
26:22within countries and doing so over time.
26:28Oops, excuse me.
26:34So cost data are also used, you know,
26:36as I mentioned in efficiency analyses.
26:39So for example,
26:39this might be to examine how costs vary
26:43with different levels of service delivery.
26:45So understanding unit costs at different coverage levels
26:49is really important to predicting costs as programs tend
26:52to scale up or to grow.
26:54So scale is a really important component here.
26:57Now these dynamics can be explored
27:00through the use of cost functions.
27:02So here the figure shows
27:04this sort of interesting dynamic relationship
27:06between scale and unit cost for antiretroviral treatments
27:10across a large number of sites
27:12in sub-Saharan African countries.
27:14So you'll note that the relationship here isn't linear.
27:18It's a little noisy,
27:19but that makes it important to estimate cost functions
27:22and then to utilize different costs
27:24for different levels of coverage.
27:26So in this case we can see
27:28some suggestive evidence of economies of scale,
27:32which is to say that per patient ART services
27:36within facilities with larger numbers
27:39of annual patients tend
27:40to be slightly cheaper than facilities
27:42with fewer annual patients.
27:46So one thing to remember here
27:48is that although there's a wide range
27:50of different analyses that need cost data,
27:52they don't always use or need the same types of data.
27:58The purpose of the analysis
27:59in this case sort of dictates
28:01the type of cost that is required.
28:03And so it's really important
28:06that the purpose be identified at the outset.
28:10So, you know,
28:11we have to consider budgeting, forecasting,
28:14efficiency analysis, priority setting, among others.
28:21Move forward.
28:22So before we go into costs further,
28:25I think it's also important to get some terminology right.
28:29In addition to the types of costs,
28:32there's also different measures of cost.
28:34So being clear about what each means, I think,
28:37is pretty central to understanding
28:41and interpreting cost information better as a field.
28:44So here are four different types
28:46of costing terms that you're probably familiar with
28:49or, you know, you may have encountered some,
28:52but what do they mean?
28:53So the first is total cost,
28:55which is as you'd imagine the total cost
28:58of producing a service.
29:00And in healthcare this is often presented as an annual cost.
29:03For example, the total annual cost of VMMC services
29:07at a clinic in a particular setting.
29:11The second term is average unit cost.
29:14And so this is the total cost per unit of output.
29:17And you'll notice that the units of output
29:19can be measured in different ways,
29:21even within the same service.
29:23So for example,
29:24there's a cost per person contacted
29:26or a cost per person treated.
29:28And I'll return to this a little later.
29:31The third term is marginal cost,
29:32which is a concept that's frequently used in economics
29:35and it's really critical to efficiency analysis.
29:38And this is the additional cost of producing
29:40just one more unit of output.
29:42So for example,
29:44the cost of testing one more person
29:46or vaccinating one additional patient.
29:49And then the final term here is incremental cost,
29:53which also examines change.
29:55In this case, however, the focus is on the cost
29:58of adding a completely new level
29:59or type of service rather
30:00than adding one more unit of output
30:03to an existing service.
30:04So for example,
30:05the additional cost of adding counseling
30:08and linkage to care
30:09to current HIV testing.
30:12Most costing exercises though that you'll encounter tend to
30:16refer to either average or to unit costs.
30:21I wanna give you just a quick example here
30:23of what a program costing table might look like.
30:27So this slide comes from a presentation by Dr. Jim Kahn,
30:31who is a mentor of mine at Cal.
30:36The example here is a small rural HIV clinic
30:41that was set up by a colleague of Dr. Khan at UCSF,
30:46which I'll have to say
30:47is where he spent most of his career.
30:51And the two in this case conducted
30:55a costing analysis of this facility.
30:58So as you can see,
30:58the summary divides costs
31:00into four different categories.
31:02There's personnel at the top,
31:05antiretroviral medications,
31:06which are the dominant supply
31:08that were used by the program,
31:10lab tests, which is the main service that was provided
31:14by the program,
31:15and then there's this fourth sort of residual category
31:18of less expensive items.
31:20So administration, general supplies,
31:23vehicles, storage, utilities.
31:26You might find pencils in here.
31:29So the standard input categories here
31:31were adapted to conform
31:32to how clinicians think about major components
31:35of antiretroviral treatment.
31:38In this case, the columns show the units
31:40that were tested,
31:41the costs per unit,
31:44the resulting monthly cost for the entire activity
31:47and the entire costs per patient
31:50for each line item.
31:52So in this case, for example,
31:54the full-time doctor costs $1083 per month
31:58and $52 per patient year.
32:02So one interesting takeaway here
32:04is that the authors found
32:06that once the clinic had reached fully operational capacity,
32:10it was able to deliver an ART
32:12at it looks like about $550 per patient per year.
32:16And they also found, which isn't shown here,
32:18that the cost could drop by about a third
32:21with less expensive medications
32:24as well as lower wage scales
32:25and modest increases in patient load.
32:28So just an interesting example
32:31to kind of put some of this into context.
32:34A couple of key costing principles here.
32:37And these are major takeaways
32:38I think from Jim as well.
32:41But in theory you can do some really solid costing work
32:45if you adhere to really these three main principles.
32:48So the most important is that your goal
32:50is to quantify the resources required
32:52for program operation and the associated cost,
32:56and that means developing an inventory of resources
33:00and assigning appropriate costs.
33:02So this task is quite different
33:05than simply accepting a program's budget
33:07or their expenditure report
33:09because that report might include costs
33:11that aren't required for routine program operations,
33:13like special evaluation needs of a funder.
33:17Also, even if the costs are accurate,
33:20you've likely learned nothing
33:21about how the program operates.
33:23So specifically, what types of inputs might be available?
33:27The number two point, you know,
33:29that Jim makes often is that you should be systematic
33:33and thoughtfully thorough.
33:35And by systematic, you know, we mean
33:38that the search for resources
33:40should include all of the usual input categories
33:42like personnel and equipment,
33:45the range of activities, such as marketing
33:47and service delivery.
33:49But, by being thoughtfully thorough, you know,
33:53what Jim refers to is that all of the significant resources
33:57should be included, quantified and costed.
34:00But minor costs,
34:02truly minor things like routine stationary supplies
34:05or routine services, pencils and erasers,
34:10they can often be safely ignored if the data collection
34:13in order to obtain those is too onerous.
34:16So of course that's like a judgment call.
34:19And one suggestion, you know,
34:21that Jim makes is to use a table
34:24and kind of order resources
34:26from the most to the least expensive
34:28so that you can empirically identify,
34:30say a cutoff for each of the resources
34:33that are considered maybe too minor
34:35or that may not play into the overall costs
34:39as much as you might be concerned about.
34:41So, you know, this is really getting into fine, fine detail.
34:44Do you really need to cost that last pencil?
34:48And then finally, you know, as we mentioned before,
34:51adopt the appropriate perspective.
34:53So I've talked about, you know, this at some length,
34:58but it's worth reiterating.
35:04I'll go over, I think I might skip past some
35:06of the typical cost and categories except to say, you know,
35:10these are some examples they may often
35:14go as far as to include patient costs.
35:18They may also vary by program and over time.
35:26The timeframe, you know, which I mentioned earlier,
35:29typically when we think about a costing exercise,
35:32it's about 12 months or it's multiples thereof.
35:36And you know, the reason for this
35:37for annual costing
35:38is that it has the hallmark of, you know,
35:42conforming to our pervasive and programmatic record keeping,
35:45which is really nice,
35:46but it can also help to deal with distortions,
35:50seasonal effects and shocks.
35:51It's smooths as a lot of random variation.
35:54So that can be for like the calendar year
35:56or the fiscal year, typically.
35:59Portraying costs by month,
36:01if it's easier,
36:01might indicate how costs vary over the course of a year.
36:04So that's another consideration.
36:07And you know, again,
36:09if we're talking about fully operational programs,
36:12we wanna make sure that the program is actually matured,
36:16but again, that might ignore some of the startup costs
36:20or other costs that I discussed earlier.
36:23One other little piece of terminology that I want to cover
36:27is gross versus macro, and top down versus bottom up.
36:30So you might have come across this a number of times,
36:32you know, in proposals or mentioned
36:34sort of casually in the literature,
36:37micro-costing focuses on really granular accounting
36:40of inputs as opposed to to macro-costing,
36:45which is, or sorry, gross costing.
36:50I always get them confused.
36:53A gross costing approach that might simply estimate
36:55all of the relevant costs,
36:57typically from program expenditure data,
36:59and then dividing the associated outputs.
37:02So like patient episodes.
37:05In contrast, we've got the bottom up or top down,
37:08which refers to the way in which each resource is allocated
37:12to the total unit cost being estimated.
37:14So top down costing divides
37:17overall program costs or expenditures,
37:20often including those,
37:22which are above the service delivery level
37:27by the number of outputs in order
37:28to calculate the unit cost,
37:31whereas bottom up costing
37:33measures input quantities at the client or activity level.
37:38So gross costing, for example,
37:39is commonly done top down, for example.
37:45And then micro-costing kind of has a bottom up.
37:50I'll skip over.
37:51Well, let's see, Debbie, we're at 43 after.
37:57I wanna mention pricing and valuation very briefly
38:01before moving into some of our other findings
38:05simply because I think there's one example
38:09in the popular press right now
38:12that's really worth consideration,
38:13particularly if you're operating
38:15in low and middle income countries.
38:16And I'll just point to the fourth bullet here, inflation.
38:21So inflation comes up a lot
38:25in the popular press
38:26because we're experiencing rapid inflation
38:28in the US right now,
38:30but I wanna linger on it for a second, simply to say,
38:34so the operation that you use
38:40in order to account for inflation
38:43in different country settings matters a great deal.
38:47So the example here is,
38:48let's say that you've collected
38:51a piece of critical cost data
38:52from a program in Tanzania,
38:54the unit cost of a service
38:55in the year 2000 to say a $100.
38:59So, you know, the typical strategy
39:02is to use a US price inflation tool,
39:08so typically the GDP price deflator
39:12and convert that into US dollars
39:15in order to inflate over time.
39:18There's some recommendations in the literature
39:21that this should be done
39:22in a different order of operations,
39:24and that would be to actually inflate
39:26in local inflation before converting into US dollars.
39:31And the implications for that.
39:32So in the next slide here are pretty substantial.
39:36So, you know, in Tanzania over that same period,
39:39inflation was quite rampant and so it it really changes
39:43the overall dollar amount
39:46that you're looking at
39:47in the last period of the cycle there.
39:50And it's even more volatile
39:52when you include other countries
39:54in sub-Saharan Africa, period.
39:57So, your choice of inflating first before exchanging
40:02into US dollars or exchanging into US dollars
40:05and then using a US inflation process
40:08can have some really substantial implications.
40:12I just wanted to highlight that.
40:13I'm gonna push through some of these final notes here
40:19because I'd like to get to some of our findings.
40:23I do wanna leave you with three key resources at the end,
40:26and those are the consolidated health economics
40:29evaluation reporting standards,
40:30or CHEERS standards.
40:31There's a new update to those standards
40:33that was published in 2022.
40:35And then for those working
40:37in low and middle income country settings,
40:39Vasal and colleagues reference case
40:42for estimating the cost
40:44of global health services and interventions
40:46and the Bill and Melinda Gates Foundation 2014 guidelines
40:49that come through their reference case.
40:52And I'm happy to share those resources after the talk,
40:55but let me get into costing and practice here
40:57because I am running quite low on time.
40:59So the Global Health Cost Consortium,
41:02as Debbie mentioned at the top
41:03of the hour was funded
41:05by the Bill and Melinda Gates Foundation.
41:09I was involved with this project for about four years,
41:12and two of the outputs from the project
41:16were our unit cost study repository,
41:18which I'd invite you to visit.
41:21And that's really a one-stop shop for all
41:24of the published cost data
41:26from HIV and TB interventions
41:28that were conducted
41:29in low and middle income countries settings,
41:30really over the last 20 years or so.
41:33So we went through a pretty long painstaking process
41:36of extracting primary cost data,
41:39that is costs that were not modeled
41:43from the published literature,
41:45standardizing them across
41:48several hundred different variables,
41:51and then making them sort of available to the public
41:56at large in a pretty easy
41:58to use dropdown type of database.
42:01So if you're ever doing any program planning
42:05for HIV or TB interventions,
42:07I'd invite you to visit that site
42:10and see if you can get any utility
42:11from some of the efforts that we put forward.
42:14Another big output was the reference case
42:16that I mentioned before,
42:18guidelines for estimating costs
42:19for Global Health Interventions.
42:22And again, happy to share those after the talk.
42:26Another major output here was we were able
42:29to produce several different publications.
42:33So I'll present a little bit of the findings from that.
42:36So over several years, our team
42:38conducted the systematic search
42:40and screening of peer reviewed and grey literature.
42:43We had a few different goals here.
42:46We wanted to describe the quantity and characteristics
42:48of public published costing data
42:51to identify production patterns over time,
42:54so geographic location,
42:56publication venues like journals
42:58and authorship and to look into key methods
43:02and reporting standards to try to identify gaps
43:05in the costing literature.
43:07So I'll go over some of those findings.
43:09So figure one here shows the results
43:12of that search and screening process.
43:15So we covered articles that were published starting
43:18in January, 2006 through October, 2017.
43:23So it's a couple years old now.
43:26We use six databases,
43:27so PubMed, M Base, Web Science, Cochrane,
43:32and a number of others.
43:34I'm gonna give you high level content here.
43:37So searching also included grey literature,
43:40which you see on the right hand side.
43:42And we identified over 23,000 possible titles,
43:46including nearly 500 known costing studies
43:48that we had started with.
43:50So from these we were able to screen down
43:52to 217 relevant titles
43:54that got included in that unit cost study repository,
43:58that database that I mentioned before.
44:01And of those we published one study among 159
44:11published studies that had taken place
44:12in Sub-Saharan Africa.
44:14So I'm gonna discuss really briefly two different studies
44:18that came out of the effort.
44:19And the first was that 2019 study that I just described.
44:22And the second is a study that's in progress
44:24where we're reviewing findings
44:26from the detailed extraction
44:28of the 243 studies that we started with
44:32in the lines above.
44:34So I might jump a little bit between findings
44:36in these two different venues.
44:37So the first study focused on that evidence
44:41from Sub-Saharan Africa.
44:42This gave us a really nice picture
44:44of where costing data is and isn't collected
44:47on the continent.
44:49The studies we identified took place in 25 countries,
44:52so mostly East Africa and Southern Africa.
44:55And although this sort of geographic variation
44:58is broad, you know,
44:59we found that the quality of evidence
45:01within many of these countries is,
45:03or the quantity rather of evidence within these countries
45:05is quite sparse.
45:07So west Africa doesn't have
45:10a great deal of primary cost data
45:13in the published literature.
45:14A lot of it, you know, is often modeled.
45:19In terms of who is publishing studies
45:21that report on HIV intervention costs.
45:24There's a few notable peer-reviewed journals.
45:26So PLoS One, J AIDS and AIDS are kind of our leaders.
45:33But I think the big takeaway here
45:35is that there's an awful lot
45:36of potential destinations
45:38for published primary cost data
45:42and more so than I think we realized
45:45when we set out in this exercise.
45:48So you can see the large list
45:50of different published studies on the left
45:53or sorry journal destinations
45:55for published studies on the left
45:58and the table in the right.
46:00These are grey literature resources.
46:02So these are the sources of grey lit,
46:05a few key players here, USAID, PEPFAR,
46:09the Futures Group, Health Policy Project
46:12and PANGEA and Bill and Melinda Gates
46:15had their fingers
46:16in a lot of these different published studies.
46:19But these are the destinations.
46:23One thing that I do wanna mention here
46:25is that although we're examining
46:26this subset of primary cost data that was collected
46:29from studies in Sub-Saharan Africa,
46:32there's a little bit of a disconnect here
46:33in terms of where we find that primary cost evidence
46:37is being published and where leaders in the field,
46:40at least nine years ago
46:42when the first CHEERS guidelines paper was published,
46:46that there's, you know,
46:48in that paper there was an explicit effort
46:53to identify leading journals as landing places
46:57for health economic evaluations.
46:59But what we find is that of the 10 that they identified
47:03in that CHEERS statement paper,
47:05only three were we able to find
47:08in our larger systematic search of the HIV literature.
47:12And this is, you know, beyond Sub-Saharan Africa.
47:14This is that paper too that I mentioned.
47:17So this isn't necessarily an indictment of the list
47:20that was generated by the CHEERS authors,
47:23but it does highlight,
47:24I think a large and varied number of journal sources
47:27where these data are published.
47:29And it emphasizes the need to be sure
47:30that these standards, you know,
47:32like the ones laid out by the CHEERS document are being
47:35upheld across different publications.
47:39So, you know, a large growth in publications over time,
47:44a difference in the time to publication.
47:46So peer reviewed studies tended to take 2.8 years on average
47:51to get published,
47:53whereas those in the grey literature 1.4 years.
47:56So this is much more quickly disseminated
48:01into the zeitgeist, you know,
48:04the ability for folks to actually start
48:06to use these data
48:07for their own program planning purposes.
48:13Scale, I'll mention.
48:14This is probably our most substantial finding
48:17and you don't need to spend
48:18a lot of time pouring over this table,
48:19but this was actually quite damning.
48:22If you don't take anything else away
48:24from this presentation,
48:25I think, I hope that that this resonates,
48:29that reporting of scale
48:31is, you know, it's a really critical, important component
48:34of any costing exercise.
48:36But what we found is that in, you know,
48:40the six of the intervention types of all the studies
48:44through which we were able to collect data,
48:46the average number of unit costs
48:50in those published studies,
48:54scale was reported below 50% of the time.
48:58And in some areas as low as 35 or 26% of the time.
49:04So in total, I think the average, you know,
49:06that we found was something
49:08like 44% of unit costs have
49:11explicit measurement of scale associated
49:14with those reports.
49:16So that's a problem,
49:19that's something that needs to improve in the field.
49:22And then I'll also mention perspectives.
49:24So despite I think recommendations and urging from say,
49:30the Gates Foundation's reference case,
49:32which I cited earlier,
49:34which emphasizes that authors should focus
49:36on societal perspectives as much as possible,
49:39we actually find here in terms
49:41of what folks in these articles report
49:46as their given perspective
49:48and when we clean it up and we, you know,
49:51we figure out among those who didn't report
49:53what their perspective was, by the way,
49:5515.6% of of studies
49:58that we uncover don't report a perspective,
50:00which is again, a problem for reporting standards,
50:05only seven or about 2.9% of the studies
50:08that we identified take a societal perspective.
50:12So, you know, most of this is done
50:13at the level of the provider.
50:15So, you know, it gives you a little bit of context,
50:17I think for where there's room for growth.
50:21So in conclusion, on these points, you know,
50:24there's, I wanna say that there's, you know,
50:27a few final recommendations.
50:31We recommend that that future costing
50:32and cost effectiveness studies closely follow
50:35the Gates and the GHCC reference case.
50:40And you know,
50:42that there needs to be more detailed reporting
50:45on a number of different items
50:47that I wasn't able to touch on in this talk.
50:50But, you know, most of all,
50:51I think scale is something that's really missing
50:54in the literature.
50:56So I won't touch on this too much
51:00other than to say there's some future research
51:01that we're working on
51:03to determine the quality of some of these cost data.
51:05So be on the lookout for publication on that in the future.
51:10And now that I've ran through all of our Q and A time,
51:13I do wanna thank you for attending
51:15and I'm happy to take any questions.
51:20<v ->Yeah.</v>
51:23Thank you for the great talk.
51:24So I had a question,
51:25comment about, you know, the societal perspective.
51:28I think in certain scenarios or situation,
51:31this could be a little bit tricky.
51:33For example, if you take vaccine
51:35for some infectious disease in low income countries,
51:39usually the price is heavily negotiated
51:42between the pharmaceutical companies and donors and NGOs.
51:47And if you take the societal perspective,
51:49when you are doing the cost effectiveness analysis,
51:52including, you know, the productivity loss,
51:55the income loss, the vaccine becomes extremely,
52:00you know, cost effective or even cost saving.
52:03So even if they price the vaccine
52:05to $50, $100,
52:07still it becomes cost saving, which is, you know,
52:10it becomes an excuse in hands of, you know,
52:14the pharmaceutical companies to jump up the price of vaccine
52:17because such a cost effective strategy.
52:19So I dunno if you have thought about that.
52:21I usually, I have purposefully avoided
52:24taking societal perspective in those contexts
52:27because I just don't want to put these numbers out,
52:30but I don't know if you have comments, thoughts.
52:34<v ->Yeah, no, I mean that's a great point.</v>
52:37I often don't think far enough along the causal chain
52:40to consider sort of the potential negative externalities
52:43of price gouging, you know, in those contexts.
52:49I guess I'll say, you know,
52:52at least in the context of what we found, you know,
52:55in our search of the literature and you know,
52:57there were a few of those different intervention types
52:59that dealt with pharmaceuticals,
53:02you know, and the costs of those being, you know,
53:04not insignificant drivers of overall program
53:08and unit costs, you know, for HIV care
53:12that most of the studies that we identified,
53:20you know, really do focus in, you know,
53:22don't tend to focus in on the societal level.
53:25So it's not, you know,
53:27perhaps that that type of crisis
53:30hasn't borne out necessarily
53:31as a result of some of this in the literature,
53:34but it's definitely not something that's discussed
53:36as often as I think it should be.
53:37So you raise an excellent point and, you know,
53:39and one that I hope, you know,
53:41gets added to further discussions
53:43in published literature and forums like this.
53:49So I don't have a good, you know,
53:49I don't have a good anecdotal example to share,
53:54but I really, yeah, I like the point.
53:56Yeah. Jeremy.
53:58<v ->Yeah, Drew, thanks so much.</v>
53:59This was really informative.
54:02I actually wonder maybe building
54:03on the previous question,
54:05if you could go back to the slide with the table
54:08from that rural Uganda study,
54:10because my question was about sort of was this a financial
54:15or economic perspective that was being presented
54:18and I was honing in on the ART
54:20because presumably, you know,
54:22through PEPFAR from the programs perspective,
54:25ART should have been without cost
54:29and also without cost to the patient.
54:31So then, so which made me think,
54:33well maybe this is an economic perspective,
54:36but then there are other,
54:37all those other pieces of deliver of getting the medicine
54:39to the patient,
54:40the whole supply chain and whatnot.
54:42And so I was just wondering if you have a sense of like,
54:44was this trying to describe programmatic
54:49or economic perspectives
54:51and, you know, when, if this is economic,
54:54should we truly be thinking about like that whole process?
54:57Like where in the supply chain do you start counting?
55:01<v ->Yeah, you know, I wasn't involved</v>
55:04in this particular study
55:05and so I'm happy to field that question to Jim
55:07'cause he uses this example in his talks.
55:10My sense is that this is programmatic, you know.
55:13It does seem fairly narrow in scope,
55:16but you do, you raise a really good point
55:17I think, you know, that Reza raised as well,
55:19which is, you know,
55:21it depends on who's providing that,
55:24where the supply chain is coming from
55:27and you know, and that obviously
55:28has downstream implications
55:29for, you know, how you deal with the costing procedures
55:35in, you know, these different income country settings,
55:39how you'd inflate and how you'd adjust.
55:42And so supply chains don't often make it into the discussion
55:47in these write-ups, but they need to, right?
55:49Because that's a really critical component.
55:53I don't know if that answers your underlying question.
55:56<v ->Yeah, I think it gets to it for sure.</v>
55:59<v ->Okay.</v>
56:00Yeah.
56:00And you know, like I said,
56:01I'm happy to reach out and and find out
56:03a little this particular example,
56:05but it's question.
56:06<v ->Great, thanks.</v>
56:08<v ->Yeah.</v>
56:09I do realize we're at the top of the hour.
56:11I'm not able to see the chat very well.
56:15So if there's anyone that would like to raise anything
56:17from there that I'm missing,
56:19I'm happy to field those.
56:21<v ->Drew.</v>
56:22I would just say that before David Palteo ran away,
56:26he said that for what it's worth,
56:29his concern is the mirror opposite of Reza's.
56:32When we abandon the societal perspective,
56:35interventions that have broad public benefit
56:38can be greatly undervalued
56:40because they're not germane to the narrow
56:41decision maker and real economic resource use
56:45such as volunteer time and effort
56:47can be ignored because it looks free
56:49to the narrow stakeholder.
56:51So he's, and I would also say
56:54that my question,
56:55and this is one not as an economist
56:58that I've wondered about
56:59for a couple of years,
57:01is that from an equity perspective,
57:04I actually wonder about the household level perspective
57:07as opposed to the individual level perspective.
57:13I've never seen one of these presentations
57:15that looks at the household level
57:18and yet it's at the household level
57:20that cushioning of resources
57:23and equity balances often happen
57:26not at the individual level.
57:28And so I'm just curious from your experience,
57:31have you ever seen work done at the household cost level?
57:37And what do you think
57:38about those as an equity perspective?
57:41<v ->Yeah, I mean I think there's,</v>
57:43I think there's some downstream costs there that, you know,
57:45in theory would get
57:46some subsumed into a societal perspective.
57:50But I struggle to think of specific examples
57:53that kind of cast the umbrella wide enough
57:56or cast a net wide enough
57:57to include other individuals within the household.
58:01So I think that's a really,
58:03that's a really great point.
58:05You know, and it speaks
58:06to the demand side part of this question
58:12in a way that's really, you know,
58:14potentially quite meaningful.
58:16And I would tend to think that as you suspect, you know,
58:23we'd be moving towards even greater downstream benefits
58:28at the household level.
58:29So I am working on one study right now
58:31in Uganda actually
58:32with Jeremy who I think might have jumped off
58:35where we are investigating patient costs
58:40as a component of that intervention
58:43and we do tend to ask in the scope
58:46of collecting patient costs data about opportunity costs
58:52that are germane to the rest of the household.
58:53So we'll often ask questions about household expenditures
58:59and household income generating activities
59:03as well as durable assets of the household
59:07and ways that households can sort of deal
59:09with shocks over time.
59:12'Cause those are all really important questions
59:13I think that play into, you know,
59:16what the experience of a patient is like,
59:18particularly in a lower middle income country setting
59:20where one of these interventions
59:21might have a really meaningful downstream impact
59:24on, you know, on the rest of the household.
59:27So it's something that I spend a lot of time thinking about
59:29'cause I kind of started in that world I think
59:31before coming into the economic evaluation world.
59:34And it's one that I think doesn't currently receive
59:37a great deal of attention, but ought to.
59:40But again, to, you know, to Reza's warning,
59:42I think you have to be careful about, you know,
59:45how much generalizability you pull,
59:49and try to tout from different studies like this
59:51because, right, they have different implications
59:55for different parties
59:56And so, you know,
59:57I'd like to see a standard,
60:00not where one perspective is chosen
01:00:01over another ad nauseum,
01:00:04but where they're both incorporated
01:00:05and where there's a societal
01:00:07and a financial or a programmatic perspective presented
01:00:11within the same paper, you know.
01:00:14And readers of those pieces of evidence
01:00:17can then decide for themselves,
01:00:19like which is the perspective
01:00:20that's the most meaningful?
01:00:21What are the trade-offs, you know, along the way?
01:00:24'Cause it, you know, the value add of that is substantial,
01:00:28to add in a societal perspective piece
01:00:30to a publish economic evaluation.
01:00:33<v ->So I think we do need to wrap up.</v>
01:00:35But thank you Drew.
01:00:37I think that's very helpful
01:00:39and we will have the video available for people.
01:00:45And really appreciate your time
01:00:48in preparing all of this material
01:00:50and sharing it with all of us.
01:00:54<v ->Appreciate the invitation</v>
01:00:56and thanks to those attended today.
01:00:58It's nice to see some faces.
01:01:01<v ->Okay.</v>