New HIT ROI lit review is a methodologic tour de force, but misses the point

JAMIA logoRecently, Jesdeep Bassi and Francis Lau of the University of Victoria (British Columbia) published in the Journal of the American Medical Informatics Association (JAMIA) another in a series of review articles that have been written in recent years to summarize the literature regarding the economic outcomes of investments in health information technology (HIT).  Such articles answer the questions

  • “How much do various HIT technologies cost?”
  • “How much do they save?”
  • “Are they worth the investment?”

They reviewed 5,348 citations found through a mix of automated and manual search methods, and selected a set of 42 “high quality” studies to be summarized.  The studies were quite diverse, including a mix of types of systems evaluated, methods of evaluation, and measures included.  The studies included retrospective data analyses and some analyses based on simulation models.  The studies included 7 papers on primary care electronic health record (EHR) systems, 6 on computer-based physician order entry (CPOE) systems, 5 on medication management systems, 5 on immunization information systems, 4 on institutional information systems, 3 on disease management systems, 2 on clinical documentation systems, and 1 on health information exchange (HIE) networks.

Lau HIT ROI results

Key results:

  • Overall, 70% of the studies showed positive economic results, 24% were inconclusive, and 6% were negative.
  • Of 15 papers on primary care EHR, medication management, and disease management systems, 87% showed net savings.
  • CPOE, immunization, and documentation systems showed mixed results.
  • The single paper on HIE suggested net savings, but the authors expressed doubts about the optimistic assumptions made in that analysis about a national roll-out in only ten years.

My take:

Bassi and Lau have made an important contribution to the field by establishing and documenting a very good literature review methodology – including a useful list of economic measures, a nice taxonomy of types of HIT, and many other tools which they graciously shared online for free in a series of appendices that accompany the article.  They also made a contribution by doing some tedious work to sort through lots of papers and sorting and classifying the HIT economics literature.

But, I think they missed the point.

Like many others, Bassi and Lau have implicitly accepted the mental model that health information technology is, itself, a thing that produces outcomes.  They evaluate it the way one would evaluate a drug or a cancer treatment protocol or a disease management protocol.  Such a conceptualization of HIT as an “intervention” is, unfortunately, aligned with the way many healthcare leaders conceptualize their investment decision as “should I buy this software?”  I admit to contributing to this conceptualization over the years, having published the results of retrospective studies and prospective analytic models of the outcomes resulting from investments in various types of health information technologies.

Process PuckBut, I think it would be far better for health care leaders to first focus on improvement to care processes — little things like how they can consistently track orders to completion to assure none fall through the cracks, bigger things like care transitions protocols to coordinate inpatient and ambulatory care team members to reduce the likelihood that the patient will end up being re-hospitalized shortly after a hospital discharge, and really big things like an overall “care model” that covers processes, organizational structures, incentives and other design features of a clinically integrated network.  Once health care leaders have a particular care process innovation clearly in sight, then they can turn their attention to the health information technology capabilities required to enable and support the target state care process.  If the technology is conceptualized as an enabler of a care process, then the evaluation studies are more naturally conceptualized as evaluations of the outcomes of that process.  The technology investment is just a one of a number of types of investments needed to support the new care process.  The evaluation “camera” zooms out to include the bigger picture, not just the computers.

I know this is stating the obvious.  But, if it is so obvious, why does it seem so rare?

This inappropriate conceptualization of HIT as an intervention is not limited to our field’s approach to economic evaluation studies.  It is also baked into our approach to HIT funding and incentives, such as the “Meaningful Use” incentives for investments in EHR technology, and the incentives created by HIT-related “points” in accreditation evaluations and designations for patient-centered medical home (PCMH), accountable care organizations (ACOs), organized systems of care (OSC), etc.  The designers of such point systems seem conscious of this issue.  The term “meaningful use” was intended to emphasize the process being supported, rather than the technology itself.  But, that intention runs only about one millimeter deep.  As soon as the point system designers put any level of detail on the specifications, as demanded by folks being evaluated, the emphasis on technology becomes instantly clear to all involved.  As a result, the intended focus on enabling care process improvement with technology slides back down to a  requirement to buy and install software.  The people being evaluated and incentivized lament that they are being micromanaged and subject to big burdens.  But they nevertheless expend their energies to score the points by installing the software.

So, my plea to Bassi and Lau, and to future publishers of HIT evaluation studies, is to stop evaluating HIT.  Rather, evaluate care processes, and require that care process evaluations include details on the HIT capabilities (and associated one time and ongoing costs) that were required to support the care processes.

Read More

The debate about what to maximize when selecting candidates for care management programs: Accuracy? ROI? Net Savings? Or Cost-Effectiveness?

When doing analysis, it is really important to clarify up front what it is you are actually trying to figure out.  This sounds so obvious.  But, I am always amazed at how often sophisticated, eager analysts can zip past this important first step.

Health plans, vendors and health care providers are all involved in the design and execution of wellness and care management programs.  Each program can be conceptualized as some intervention process applied to some target population.  A program is going to add more value if the target population is comprised of the people for whom the intervention process can have the greatest beneficial impact.

I have found it useful to conceptualize the process of selecting the final target population as having two parts.  The first part is the process of identification of the population for whom the intervention process is deemed relevant.  For example, a diabetes disease management program is only relevant to patients with diabetes.  An acute-care-to-ambulatory-care transitions program is only relevant to people in an acute care hospital. A smoking cessation program is only relevant to smokers.  The second part is to determine which members of the relevant population are to actually be targeted.  To do this, a program designer must figure out what characteristics of the relevant candidates are associated with having a higher opportunity to benefit from the intervention.  For example, in disease management programs, program designers often use scores from predictive models designed to predict expected cost or probability of disease-related hospitalization.  They are using cost or likelihood of hospitalization as a proxy for the opportunity of the disease management program to be beneficial.  Program designers figure that higher utilization or cost means that there is more to be saved.  This is illustrated in the graph below, where a predictive model is used to sort patients into percentile buckets.  The highest-opportunity 1% have have an expected annual admit rate of almost 4000 admits per 1000 members, while the lowest 1% have less than 200 admits per 1000 members.  A predictive model is doing a better job when more of the area under this curve is shifted to the left.

Although it is common to use expected cost or use as a proxy for opportunity, what a program designer would really like to know is how much good the intervention process is likely to do.  Other variables besides expected cost or use can contribute to a higher opportunity.  For example, in a disease management program, the program might be more worthwhile for a patient that is already motivated to change their self-management behaviors or one that had known gaps in care or treatment non-compliance issues that the intervention process is designed to address.

Applying micro-economics to care management targeting

Once the definition of “opportunity” is determined and operationally defined to create an “opportunity score” for each member of the relevant population, we can figure out which members of the relevant population to actually target for outreach for the program.  Conceptually, we would sort all the people in the relevant population by their opportunity score.  Then, we would start by doing outreach to the person at the top of the list and work our way down the list.  But, the question then becomes how far down the list do we go?  As we go down the list, we are investing program resources to outreach and intervention effort directed at patients for which the program is accomplishing less and less.   Economists call this “diminishing returns.”

As illustrated by the red line in the graph above, there is some fixed cost to operating the program, regardless of the target rate.  For example, there are data processing costs.  Then, if the program does outreach to a greater and greater portion of the relevant population, more and more people say “yes” and the costs for the intervention go up in a more or less linear manner.  As shown by the green line, the savings increase rapidly at first, when the program is targeting the candidates with the greatest opportunity.  But, as the threshold for targeting is shifted to the right, the additional candidates being targeted have lower opportunity, and the green line begins to flatten. The blue line shows the result of subtracting the costs from the savings to get net savings.  It shows that net savings increases for a while and then begins to decrease, as the cost of intervening with additional patients begins to outweigh the savings expected to accrue from those patients.  In this analysis, net savings is highest when 41% of the relevant population of diabetic patients is targeted for the diabetes disease management program.  The black dotted line shows the result of dividing savings by cost to get the return of investment, or ROI.   With very low target rates, too few patients are accumulating savings to overcome the fixed cost.  So the ROI is less than 1.  Then, the ROI hits a peak at a target rate of 18%, and declines thereafter.  This decline is expected, since we are starting with the highest opportunity patients and working down to lower opportunity patients.   Note that in this analysis, increasing the target penetration rate from 18% to 41% leads to a lower ROI, but the net savings increases by 24%.  So, if the goal is to reduce overall cost, that is achieved by maximizing net savings, not by maximizing ROI.

Should we try to maximize accuracy?

In a recent paper published in the journal Population Health Management by Shannon Murphy, Heather Castro and Martha Sylvia from Johns Hopkins HealthCare, the authors describe their sophisticated methodology for targeting for disease management programs using “condition-specific cut-points.”  A close examination of the method reveals that it is fundamentally designed to maximize the “accuracy” of the targeting process in terms of correctly identifying in advance the members of the relevant disease-specific population that will end up being among the 5% of members with the highest actual cost.   In this context, the word accuracy is a technical term used by epidemiologists.  It means the percentage of time that the predictive model, at the selected cut-point, correctly categorized patients.  In this application, the categorization is attempting to correctly sort patients into a group that would end up among the 5% with highest cost vs. a group that would not.  By selecting the cut point based accuracy, the Hopkins methodology is implicitly equating the value of the two types of inaccuracy: false positives, where the patient would be targeted but would not have been in the high cost group, and false negatives, where the patient would not be targeted but would have been in the high cost group. But, there is no reason to think that, in the selection of targets for care management interventions, false negatives and false positive would have the same value. The value of avoiding a false negative includes the health benefits and health care cost savings that would be expected by offering the intervention. The value of avoiding a false positive includes the program cost of the intervention.  There is no reason to think that these values are equivalent.  If it is more important to avoid a false positive, then a lower cut-point is optimal.  If it is more valuable to avoid a false negative, then a higher cut-point is optimal.  Furthermore, the 5% cost threshold used in the Hopkins methodology is completely arbitrary, selected without regard to the marginal costs or benefits of the intervention process at that threshold.  Therefore, I don’t advise adopting the methodology proposed by the Hopkins team.

What about cost-effectiveness?

The concept of maximizing ROI or net savings is based on the idea that the reason a health plan invests in these programs is to save money.  But, the whole purpose of a health plan is to cover expenses for necessary health care services for the beneficiaries.  A health plan does not determine whether to cover hip replacement surgery based on whether it will save money.  They cover hip replacements surgery based on whether it is considered a “standard practice,”  or whether there is adequate evidence proving that the surgery is efficacious.  Ideally, health care services are determined based on whether they are worthwhile — whether the entire collection of health and economic outcomes is deemed to be favorable to available alternatives.  In the case of hip replacement surgery, the health outcomes include pain reduction, physical function improvement, and various possible complications such as surgical mortality, stroke during recovery, etc.  Economic outcomes include the cost of the surgery, and the cost of dealing with complications, rehabilitation and follow-up, and the savings from avoiding whatever health care would have been required to deal with ongoing pain and disability.  When attempting to compare alternatives with diverse outcomes, it is helpful to reduce all health outcomes into a single summary measure, such as the Quality-Adjusted Life Year (QALY).  Then, the incremental net cost is divided by the incremental QALYs to calculate the cost-effectiveness ratio, which is analogous to the business concept of return on investment.  If the cost-effectiveness ratio is sufficiently high, the health service is deemed worth doing.  There is no reason why wellness and care management interventions should not be considered along with other health care services based on cost effectiveness criteria.

The idea that wellness and care management interventions should only be done if they save money is really just a consequence of the approach being primarily initiated by health plans in the last decade.  I suspect that as care management services shift from health plans to health care providers over the next few years, there will be increased pressure to use the same decision criteria as is used for other provider-delivered health care services.

Read More