In the battle between Randomized Clinical Trialists and Observational Studies Advocates, don’t forget the Modelers

This week, I’m attending the annual meeting of the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) in Baltimore.  Yesterday evening, I attended an interesting “smack-down” presentation between advocates for Randomized Clinical Trials (RCTs) vs. advocates for Observational Research.  The session was packed with hundreds of people.  In one corner was Michael Lauer, MD, the director of Cardiovascular Sciences at the National Institutes of Health (NIH).  In the other corner was Marc Berger, MD, the Senior Scientist of Ingenix’s Life Sciences Division.  Both gave examples when their research framework was right and the other got it wrong.  Both acknowledged that RCTs are really essential, and that observational studies will play some role.   It was an interesting session, but I was generally disappointed that neither side recognized the importance of the uninvited model builders.  I’ve been involved in trials, observational research and modeling over the years.  But, in the tongue-in-cheek debate about which methodologists should be kings of the hill, I’m with the modelers.  Let me explain.

How Trialists See the World

The religion of the medical profession includes some strongly-held core beliefs.  One of those beliefs is that the profession should be “evidence based.”  On one level, the premise of Evidence-Based Medicine (EBM) is indisputable.  We should be a profession that makes decisions based on facts, rather than emotions, myths, traditions, self-interest, or whatever else would take the place of facts.  But, the EBM belief system, particularly as preached by the clinical trialist clergy, goes beyond that to imply a decision-making framework that should define the way we determine which treatments to offer or not offer to patients.  The implied EBM decision-making process can be summarized in the following diagram.

When one reads the abstracts of scientific manuscripts describing RCTs published in all the best medical journals, this implied decision-making framework is clearly visible in the conclusions that are drawn and the logical basis supporting those conclusions.

It is important to note the implicit assumptions that underlie this traditional EBM decision-making framework:

  1. There are no important differences between the ideal conditions in a tightly controlled clinical trial vs. real-world use
  2. Health outcomes other than the one selected as the primary endpoint are not important
  3. 95% certainty correctly reflects decision-maker’s values regarding the trade-off between benefit and certainty
  4. Resources are unlimited, so costs need not be considered

Advocates of observational studies often start by questioning one or more of these assumptions.  They point out that the way surgical procedures are carried out or a patient’s adherence to prescribed medication can be different in community settings.  They point out that side effects or complications need to be considered when deciding whether a treatment should be offered.  They question the evidence threshold, particularly for potentially life-saving treatments for patients not expected to survive long enough for the definitive RCT to get published.  And, they point out that our health care system costs far more than other countries without a corresponding improvement in life expectancy or other measured health outcomes, and question whether high tech, new versions of drugs and biotechnology are worth the high cost.

But, how do modelers fit in?

Modelers are those that build computer-based models that use explicitly-stated, quantitative assumptions as the basis for mathematical calculations to estimate outcomes thought to be relevant to decision-making.  Models come in many forms, including decision-analytic models, Markov models, discrete event simulation (DES), and agent-based models (ABM).  The assumptions used as inputs to such models can be supported with data, based on expert opinion, or a mixture of both.

In my experience, both clinical trial and observational study enthusiasts sometimes put down computer-based models as being unrigorous.  They point out that models are based on assumptions, as if that was incriminating.  But, when you take a closer look at the relationships between models, RCTs and observational studies, you notice that modeling logically comes first.  And modeling also logically comes last.  These relationships are illustrated in the following diagram.


Clearly, an RCT is a waste of resources if it can be shown that it is implausible that the proposed treatment would add value.  It is also wrong to do an RCT if it can be shown that it is implausible that the treatment would not add value.  Dr. Lauer explained this point last night in an entertaining way with a reference to a sarcastic paper pointing out that parachutes have never been proven in an RCT to be effective in treating “gravitational challenge.”   But, how does one rigorously assess plausibility other than by offering a model with explicit assumptions that represent the boundaries of what knowledgeable people consider to be plausible?

When the clinical trialist is designing the RCT, they must decide on the sample size.  To do this, they do power calculations.  The basis of power calculations is an assumption of the effect size that would be considered “clinically significant.”  But, if you acknowledge that resources are not unlimited, “clinically significant” is a synonym for “worthwhile,” which is a synonym for “cost-effective.”  But how can one assess “worthwhile-ness” or “cost-effectiveness” in a rigorous way, in advance of an RCT, without constructing a model based on explicitly-stated assumptions?

Once the trial is done, the results must be incorporated into practice guidelines.  Again, if you accept that health outcomes other than the primary endpoint of the trial might be important or that resources are not unlimited, one needs to use a model to interpret the trail results to determine for which subsets of patients the treatment is worth doing.

Then, if observational studies are done to assess the outcomes of the treatment in real-world use, described as “effectiveness studies,” one needs to again use a model to interpret and determine the implications of the data obtained from such studies.

So, if we really want to be logical and rigorous, models must precede RCTs.  And models must also be done after RCTs and observational studies to properly and rigorously determine the implications of the data for practice guidelines.

For next year’s ISPOR annual meeting, I propose to include modelers in the methodology smack-down session.

Read More

Part 2 of Critique of AHA analysis showing higher ACO-related costs: Is being “local” worth 40% more?

The American Hospital Association (AHA) engaged McManis Consulting to develop a model of ACO-related start-up and ongoing costs for Accountable Care Organizations (ACOs). The recently released analysis was based primarily on information gleaned from studying four case examples of provider organizations of different types and sizes that have already established many of the 23 types of “ACO activities” in recent years.  The analysis concludes that ACO-related start-up costs are between $5-12 million, and ongoing costs are between $6-14 million, depending on the size of the organization.

In part 1 of my critique of this model, I commended the work but argued that they had significantly underestimated the cost of process transformation efforts in primary care clinics and the cost of wellness and care management services.  In this part 2, I am exploring the implicit assumptions made by AHA/McManis of the dis-economies of scale of smaller ACOs organized around a single hospital.

The AHA/McManis report provided cost estimates for a mid-size ACO, with 250 primary care physicians (PCP), 500 specialists, and 5 hospitals with a total of 1,200 beds.  Based on an assumption of 2000 patients per PCP, I estimated the size of the population for which such an ACO is taking responsibility at about 500,000 for all payers.  The AHA report also provided ACO cost estimates for a hypothetical single-hospital system, with 200 beds, 80 PCPs, 150 specialists, and, I would estimate, 160K patients.  Based on those estimates, I calculated the implied “premium” for each category of costs, presumably due to the dis-economies from the smaller scale of operations.  These percentages are calculated on a per patient basis, assuming that the average primary care physician’s patient panel size is the same for large and small ACOs.

As shown in the table, the AHS/McManis model estimates that both start-up and ongoing ACO costs are roughly 40% higher for a smaller ACO, which might be the largest possible ACO for many rural health care markets.  The premium percentages vary widely for the different ACO activities.  Surprisingly, the model implicitly assumes that the start-up cost for healthcare information technology (HIT) would be lower on a per-patient basis in a smaller ACO.  However, I assume that these numbers may not be intended by the modelers.  They may have just never looked at their results this way.

Why is this a good way of thinking about the cost estimates?

As health care leaders contemplate new alignments, mergers or acquisitions to create successful ACOs, they need to address the fundamental trade-off of scale.  On the one hand, it has become established industry wisdom that “all health care is local” — a take-off on Tip O’Neil’s 1982 statement “all politics is local.”  According to this view, a successful health care organization — including ACOs — should focus on local needs, local stakeholders and local resources.  But, the obvious down-side of being local is the dis-economies that come from small scale operations.  In smaller organizations, the fixed costs of management, technology and analytic infrastructure are spread over a smaller population of patients.  According to the AHA/McManis model, such dis-economies appear to impose roughly 40% additional cost, which could be conceptualized as the “premium” paid to obtain more local responsiveness. Given that populations of people in different towns have the same organ systems, plagued by mostly the same diseases, it is reasonable to question whether “local-ness” is really worth paying 40% more.

In other industries, creative organizational models have evolved that attempt to achieve both local focus and large scale.  For example, the “franchise” model used in retail, food services and other industries puts product development, brand management, and other functions at the home office, while retaining local ownership and leadership of franchises well suited to attend to the details of service quality and local relationship-building.  When I was at Henry Ford Health System in the 1990s, we talked often about the applicability of the franchise model to health care.  Tom Royer, MD, the leader of the Henry Ford physicians at the time,  jokingly referred to this approach as the “french-fry model.”  In that same time frame, there was a great deal of attention paid in health care organizations to the concept of the “smallest replicable unit” — sweating out the design details of the layout, processes and information systems of a single clinic or nursing unit and attempting to replicate that standardized design to other clinics or nursing units.

I predict that this fundamental trade-off between scale and local-ness will drive a lot of creative thinking over the next few years.  We would be advantaged by learning from other industries that have addressed similar problems and keeping our minds open to rejecting old assumptions about the structure of our health care delivery system.

Read More

Part 1 of Critique of AHA analysis showing higher ACO-related costs: Still underestimating care management cost

The American Hospital Association (AHA) engaged McManis Consulting to estimate the start-up and ongoing costs for Accountable Care Organizations (ACOs). The recently released analysis was based primarily on information gleaned from studying four case examples of provider organizations of different types and sizes that have already established many of the 23 types of “ACO activities” in recent years.

The analysis concludes that ACO-related start-up costs are between $5-12 million, and ongoing costs are between $6-14 million, depending on the size of the organization. These numbers are far higher than the $1.7M start-up figure that has been widely discussed, which was based on the Medicare Physician Group Practice demonstration project.

The AHA wrote a letter to the CMS Director, Don Berwick, to summarize these new calculations and argue that that the draft regulations of the Medicare Shared Savings Program (the ACO program included in the healthcare reform bill) need to include less risk and more gain sharing by providers in order to make the voluntary program worthwhile.

Re-framing the AHA Numbers from a Physician and Population Health Perspective

I applaud the AHA for undertaking quantitative calculations and releasing them for public scrutiny. Whenever I see such numbers, I feel compelled to take out my calculator and start doing some math to see whether the implicit assumptions behind the calculations align with my own “walking around numbers” based on prior experience.  I have previously done such a review of ACO-related model results released by Haywood and Kosel in the New England Journal of Medicine.  And, I previously did a 2-part review of an ACO model released by Milliman, the actuarial consultancy (part 1 and part 2).

When a hospital association thinks about costs, they naturally think of the unit of analysis to be a “hospital system” — hence their reporting of results based on a “1-hospital” and “5-hospital” system. But, when I think of an ACO, I think first of a physician-initiated concept of an ACO, intended to take responsibility of the defined population of patients that have care relationships with those physicians. Therefore, I think of the start-up costs as being most naturally expressed on a per-physician basis. When thinking of ongoing costs — particularly for functions such as network management and clinical programs, I think first of the context of health plans who have traditionally carried out those functions. Health plans most naturally think of such ongoing costs expressed on a “per member per month” (PMPM) basis. To calculate PMPM cost, one has to be able to count “members” (the defined population for which the ACO is taking responsibility). I found it amazing that the AHA/McManis model never mentions the size of the population, revealing a hospital-centered rather than population-focused mental model. To estimate the population size, I assumed that each primary care physician would care for an average all-payer panel size of 2000 patients.

I found the AHA/McManis “activity” categories to be pretty intuitive, but I felt the need to rename them slightly to conform to my own terminology, and to break out the “clinical programs” category to align with the way health plans break out such programs, separating the member-facing wellness & care management activities vs. the provider-facing practice improvement support (including Lean thinking) vs. the provider-delivered hospitalist services.

Based on this lumping, splitting, labeling and math work, I came up with the following revised summary table for the analysis.

The table reveals that start-up costs are an affordable $16K per physician, two-thirds of which are health information technology (HIT) costs. At least some of these HIT costs would be offset by HIT “meaningful use” funding included in the federal stimulus package, and some additional portion would be offset by eventual office efficiencies expected to accrue due to automation. The table shows that ongoing cost amount to $2.35 PMPM, 40% of which is for administrative overhead for leadership and network management.

In my opinion, the most shocking number for start-up cost is the $200 per physician for practice improvement.  In my experience, the work to initially establish the basic set of patient-centered medical home processes and population management processes in physician offices is a huge undertaking, involving taking clinic staff off-line from patient care duties and leveraging practice improvement coaches (including Lean coaches).   $200 does not seem like a reasonable assumption for that work.  The text of the AHA report said that they assumed $10K per “practice” just for NCQA PCMH certification, but the final calculations do not seem to include that amount.  Even if the entire $16K per physician start-up cost was dedicated to starting up such processes, I would conclude it is an under-estimate based on my experience observing this activity in many clinics.

Wellness and Care Management Costs are Substantially Underestimated

The estimated ongoing costs for wellness and care management is only $0.59 PMPM.  This is a very low number, particularly since the AHA/McManis report says “all of the case studies are pursuing disease management strategies aggressively.”  Health plans are far more aggressive, at least in terms of budget.  By comparison, the typical PMPM cost that commercial PPO health plans invest in wellness and care management services would be in the $1-5 range (with $2 being my estimate of most typical), while Medicare Advantage (MA) plans would typically spend $5-15 PMPM on such programs (with $10 being my estimate of most typical).  Therefore, typical MA plans spend 17 times as much as the AHA estimate on wellness and care management.

It is true that many health plans outsource such programs to vendors who typically charge more per hour of service than a clinic is likely to pay for its own internal staff members.  On the other hand, most health-plan sponsored programs are delivered through efficient, high volume telephone call centers.  I am convinced that integrating wellness and care management programs into primary care clinics will make them far more effective.  But, such integration with primary care clinics will also increase the operating cost, particularly for smaller clinics and in scenarios where the providers are delivering such services only to a subset of their patients for which they receive ACO-type reimbursement.  In such a scenario, clinic-based wellness and care management costs must include waste from underutilized care managers and/or travel time for shared care managers.    In any case, I feel the AHA numbers are really low-balling wellness and care management processes.

I noted that the AHA assumptions described in the text of the report included “1 care coordinator per 350 patients with chronic conditions (works out to 1 coordinator for every 1.5 physicians) at $75K per coordinator.”  Based on my calculations, that would add up to between $1.30 and $3.80 PMPM, just for care coordinators, depending on whether you interpreted them to mean all physicians vs. just PCPs.  So, I was unable to get close to the numbers in their final report based on my own validation calculations.

So, does that mean ACOs are not a good investment?

In recent weeks, since the draft regulations were published for the Medicare Shared Savings Program, there has been a roar of voices declaring that the ACO concept is dead on arrival.   In my opinion, and in the opinion of many health care leaders, the surge in ACO pessimism is premature.

The excellent list of 23 ACO-related “activities” outlined in the AHA/McManis report represent a thoughtful long-range “to-do” list that should be considered by provider organizations of all types as being necessary preparations for a future state where they are able to take more responsibility for population health and associated costs.  That general direction seems certain, even if the specific initiatives of various government and private payers and other stakeholders may pose challenges along the way.  Healthcare leaders should not look at the CMS ACO regs and conclude “well, it’s clearly not worth it to invest in population health management capabilities.”

Read More

We don’t need more data. We need better story-telling.

This morning, I read a blog post from someone in the “business intelligence” industry complaining that, after years of innovation, “BI” is not really having the envisioned effects.   The author wondered whether it was because BI, like most other business applications, are not “fun.”  The author suggested that BI applications should consider borrowing from fun social applications with features like “friends” and “like.”

I share the author’s disappointment in how little visible impact we’ve achieved, at least in the health care industry, from years of investment in “business intelligence,” “management information systems,” “data warehouses,” “data mining,” “dashboards,” “report cards” and other systems intended to put more data and reports at our fingertips.

But, I don’t share the view that social networking features are the missing ingredient.  Rather, I think the key to making analytics more engaging and useful is to focus on telling interesting, truthful, actionable stories, backed up by data. People are natural story-tellers and naturally enjoy listening to other people’s stories, particularly if they have an interesting “line” — a beginning that sets the scene, a challenging problem or crisis, and some resolution. The problem with “BI” (and the other data-at-fingertips technologies) is that they are devoid of story line.

When I have had the opportunity to work with young analysts, this is the most important concept that I try to get across.  Fresh out of grad school, the young analysts who have an IT background tend to start with a view that computers can do everything.  They just need to put the power of computers and data in the hands of the “people.”   The young analysts coming out of training in statistics, epidemiology, economics and other research disciplines tend to start with a world view that advanced mathematical methods can do everything.  In developing such young analysts, “job one” is getting them to appreciate the importance of story-telling.  To be effective in changing the world, they have to learn to change minds.  To change minds, they have to keep the attention of their audience — not with outlandish, shocking conclusions or colorful “eye candy” graphics — but with an interesting plot-line that makes their audience eager to see how the story ends.  Once they learn how to tell a story, they can apply that knowledge to the telling of actionable stories, where the ending is a call to action.

The first step in effective story-telling is to develop a good outline. This is a lesson that we all learned from our high school English teachers (or, whatever was your first language!).  Back in the 1990’s, I worked with Dr. Bruce McCarthy at the Henry Ford Health System.  He used to talk about the “logic train” and we would compose our abstract presentations by putting our slides in a line on the floor.  We would joke around by doing a little “chug chug” dance along the line of slides to see if there was a good high level flow to the story.  I repeated this exercise with my 7-year old daughter in the kitchen last month when she was preparing a class speech.  She loved the chug chug dance.  I challenge you to try it out at your next meeting with analysts.

After the outline is clarified, the second step in effective story-telling is to design each slide to clearly and simply communicate its message. This requires focus on the visual design.  In my opinion, a good designer avoids clutter, demanding that every bit of “ink” has a purpose, other than just decoration.  Effective communication takes advantage of the conventions of graphical language unless there is a good reason to violate convention.  For example, if you are telling a story about change over time, people generally expect time to be on a horizontal axis moving from left to right.  They generally assume “up” means “more.”  They expect an arrow to convey sequence or the direction of flow.  They expect thicker lines, bigger fonts, and bolder colors to indicate importance.  Experienced designers of quantitative visuals know that a particular format may work well or not at all depending on the data itself, making it difficult to design an unchanging report that will be consistently effective over time when applied to different data.

This brings me to the most important conclusion.   Despite the huge attention that has been paid to the need for more data and more sophisticated analytic software, the rate-limiting step in our efforts to bring the health care field into the knowledge age is neither data nor technology.  Except for the most mundane monitoring purposes, useful analytics cannot be completely automated.  Effective analytics requires effective analysts.

Read More

We need to cultivate more analysts that can do both rigorous science and practical decision-making

Yesterday afternoon, I had the opportunity to have my first mentoring conversation with Wanyu Liu, a talented young undergraduate at the University of Washington.  Wanyu is studying statistics, economics, and applied computational mathematical sciences.  She is thinking about becoming an actuary.  I always learn the most when I am trying to teach others.  Teaching forces one to clarify thinking.  Mentoring forces one to clarify things that are really important.

So, what did I learn from talking to Wanyu?

I think that there are fundamentally two kinds of analysis.  The first type, scientific analysis, focuses on figuring out how systems work.  They try to discover cause-effect relationships.   To do that, they make observations, notice associations, and use statistical methods to assess the strength of those associations.   If associations are strong enough, and if they are consistent with the researcher’s starting hypothesis about cause-effect relationships, they assert that their data supports the hypothesis.    This work has intrinsic “discovery” value, as well as practical value.  If we know how a system really works, we can figure out how to make it work better.  We can invent new treatments that cure a disease.  Or, we can invent new care processes to make health care more effective or more efficient.  As I explained to Wanyu, epidemiology, biostatistics, health economics and health services research are all fields of study in the tradition of science and social sciences that are focused on scientific analysis.  They are at their worst when the participants are racing to be the first to get p<0.05 and publish a paper.  They are at their best when they are detectives trying to figure out how things work and driving innovation to make things work better.

The second type of analysis is to support decision-making.  All decision-making is fundamentally a choice among available alternatives.  So, decision-analysis basically boils down to using the best information available to estimate all the important outcomes for each of the alternatives, while taking uncertainty into consideration.   Actuaries and financial analysts are primarily focused on supporting decision-making.  They are at their worst when they ignore any outcomes other than financial outcomes, or when they apply their talents to “making the case” for a conclusion not based on data.  They are at their best when they are able to replace subjective intuition with objective, quantitative information to drive important decisions — when they keep us honest and practical.

As I explained to Wanyu, I think the most rewarding careers are those that live in the space between science and decision-making.  The analysts that really change the world are the ones that can:

  • Figure out how the system is really working (science)
  • Collaborate with others to develop ideas for making the system work better (business/clinical)
  • Estimate all the health and economic outcomes for those alternative ideas (decision-analysis)
  • Rigorously evaluate the innovations that are selected for implementation to see whether they really worked (science)

The person who has the rigor of science and the practicality of decision-analysis is the key ingredient in the transformation of our health system.  Or, for that matter, our education system, transportation system, manufacturing system, etc.  Such people are to be cultivated, developed and rewarded for their essential contributions.

Read More

Part 2 of Critique of Milliman’s Whitepaper on Non-Medicare ACOs: Overly optimistic about inpatient use management, alarmingly pessimistic about primary care process transformation

Victoria Boyarsky, Howard Kahn, David Mirkin, and Rob Parke from Milliman recently released a Healthcare Reform Briefing Paper entitled “ACOs beyond Medicare.” 

The paper notes the unfavorable reception by providers to the economics of the draft regs for the Medicare Shared Savings Program, and suggests that emerging ACOs focus instead on commercial (non-Medicare) populations.  In part 1 of this critique, I called into question the authors’ interpretation of their own actuarial model, suggesting that the 28% estimated effectiveness in reducing cost by forming and ACO may be an over-estimate.  In part 2, I am examining their assumptions about the various types of clinical programs expected to be pursued by ACOs.

The authors correctly point out that very few provider organizations currently qualify as a “well-organized multi-specialty group.”  The authors suggest that nascent ACOs should partner with a health plan so the health plan can provide the missing managed care infrastructure, at least on an interim basis until the ACO can build up such infrastructure on their own.  Specifically, the authors consider four types of clinical programs that could be supplied by a health plan to an ACO: inpatient utilization management (UM), outpatient UM, case/disease management, and physician office support.  They define physician office support as including such things as e-visits, e-consults, urgent care, guideline implementation, and offering primary care physician incentives to increase their scope of practice to reduce the need for specialty referrals.  If ACOs partnered with health plans for these programs, according to the Milliman actuarial model, the expected PMPM net savings would be 2.6% — a far cry from the more impressive 28% the model calculated for ACOs formed by well-organized multi-specialty groups.

It is particularly interesting to calculate, based on the Milliman model results, the program-specific effectiveness and return on investment numbers to reveal the authors’ implicit assumptions about impact of the different types of programs.

Program Type

Return of Investment (ROI)

Gross Effectiveness in Reducing the Targeted Category of Cost Gross Effectiveness in Reducing Total PMPM Cost Net Effectiveness in Reducing Total Cost (including program cost)
Inpatient Utilization Mgmt





Outpatient Utilization Mgmt





Case/Disease Mgmt

No return




Physician Office Support





Total 3.1 3.8%



Obviously, the Milliman authors are enthusiastic about inpatient utilization management, assuming it will achieve a 59-to-1 ROI!  The authors are far less optimistic about case management (AKA care coordination), disease management, and physician practice support. These programs represent the core of the primary care practice transformations that are commonly assumed to the be active ingredients of ACO cost savings.  But, the authors are assuming all such innovations, taken together, will lead to a net increase in total cost of 0.4%.

The authors did not offer any evidence or references to support these implicit assumptions.

Based on my own experience, I am far less optimistic about utilization management program effectiveness, particularly over time, as providers tend to figure out “workarounds” to get approval for services they want to be provided to their patients.  The clinical appropriateness criteria that underlie such UM programs tend to be designed to avoid “false positive” rejections, leading to loose criteria that generate a great deal of additional paperwork for both the providers and the managed care staff, with few outright rejections.

Regarding patient-facing clinical programs such as case and disease management, I actually do share the author’s apparent belief that such programs, at least as commonly designed and implemented by health plan staff, are unlikely to achieve substantial net savings. However, I remain far more hopeful that such programs can be substantially more effective if they are integrated into primary care clinic settings. By integrating these programs into primary care, they can benefit from at least some face-to-face interaction and they are tied to the doctor-patient relationship and the medical care plan.  Also, I would be even more optimistic that such programs can reduce cost if they were designed to target the root causes of specific high cost clinical scenarios, rather than being designed to be holistically beneficial.  Although many people in the care management industry tout their programs as being “holistic,”  I prefer programs that are “targeted.”

Finally, I would be more optimistic about the cost-saving effectiveness of such programs if they were improved over time based on valid and rigorous evaluation.  Even after a couple of decades of experience, we really don’t know exactly what types of care management interventions are effective for various clinical scenarios.  The only way to create truly effective programs is to carry out the intervention in a consistent, controlled way, then rigorously evaluate it and then improve the program based on the evaluation results.   Such evidence-driven improvement requires designing rigorous evaluation into the program itself.  Sadly, too many health plans and care management vendors have viewed evaluation as an afterthought, asking evaluators to draw conclusions based on haphazardly-collected data.  Even worse, too many of those same health plans and vendors have viewed effectiveness measurement as a marketing requirement, where they work back from the conclusion that their program is successful to figure the most convincing arguments to support that conclusion.  As an industry, we can and must do better at honest, rigorous evaluation — and design such rigor right into the intervention process itself.

If targeted, rigorously evaluated, provider-based case and disease management programs are more effective than the conventional health plan-delivered programs, then the authors’ advocacy for ACOs to rely on health plans for such programs may not be so desirable.  By assuming negative net savings, the authors seem to be conceding that point, at least with respect to cost reduction aims.

I acknowledge that formal evaluation of the effectiveness of any of these provider-facing or patient-facing programs is very difficult, and the base of valid evidence is virtually non-existent.  I applaud Milliman for taking a stand by releasing quantitative information and allowing others to at least calculate the assumptions they are implicitly making.

One more thing to note.  The savings effects presented in the whitepaper are one-time shifts in cost, not changes to the slope of the trend line.  That means, you must continue to spend the program cost forever to maintain a cost advantage compared to trend.

To sum up, Milliman thinks that health plan-based utilization management can achieve about a 3% one-time downward shift in the cost trend line, and ACO-style clinical process transformation is economically wasteful, at least when executed by health plans.  Food for thought!


Read More

Part 1 of Critique of Milliman’s Whitepaper on Non-Medicare ACOs: Misinterpreting the Milliman “Well Managed” Actuarial Model?

Victoria Boyarsky, Howard Kahn, David Mirkin, and Rob Parke from Milliman recently released a Healthcare Reform Briefing Paper entitled “ACOs beyond Medicare.

In the paper, the authors argue that the Medicare Shared Savings Program (and the associated regulations regarding the formation of ACOs), as defined in the draft regulations released on March 31, 2011, will make financial success elusive in the Medicare population for emerging ACOs.  They argue that providers should consider focusing instead on commercial (non-Medicare) populations.  To “measure the benefits and costs of forming an ACO”  in commercial populations,  the Milliman authors used their actuarial models. Using these models, the authors estimate that the per member per month (PMPM) costs of a typical commercial health plan delivered by “loosely managed, uncoordinated fee-for-service providers” would be $347.42.   According to these models, a “well-managed, well-organized multi-specialty group” would be expected to be able to provide care to the same population for $250.64.  This means that they estimate that forming an ACO will lead to a 28% reduction in cost.

Before I dive into criticism, let me thank Milliman for being willing to prepare a quantitative model and disseminate results freely.  As I noted in my recent critique of another economic model of ACO transformation, I think that such models are extremely helpful to real-world decision-makers because they force people to be explicit about the assumptions they are making, and they provide some quantitative estimates of the outcomes relevant to the comparison of available alternatives so people can make better choices.

But, Milliman’s “Well Managed” Model Cannot Be Used to Estimate Expected Outcomes.

Let me explain.  In the “Non-Medicare ACO” whitepaper, and in many other Milliman documents I’ve seen in recent years, the outputs from Milliman’s “Loosely Managed” vs. “Well Managed” models are described in ways that suggest they were designed to estimate outcomes.  In this whitepaper, the authors describe the “well managed” numbers as using “the utilization rates observed from well-managed multi-specialty groups when treating the same population.”  The authors go on to assert that the “difference between loosely-managed and well-managed claim cost PMPMs illustrates the amount that would be available to share among stakeholders.”  So, they are asserting that their model is producing an estimate of expected outcomes.

But, in my opinion, this is not a valid interpretation.  The Milliman model of “well managed” care is not based on the average performance of real provider organizations that are classified as “well managed.”  Rather, Milliman uses an “extensive amount of data,” augmented with their own experience and judgment, to find the “benchmark” or “target” performance, rather than the average performance.  Moreover, they determine this “benchmark”  separately for each of 60 categories of health care services.  The performance they are reporting as being expected for “well managed” entities is really the performance that would be expected for a theoretical entity that simultaneously achieved benchmark performance in all 60 categories.

Of course, any real organization’s actual performance during any particular time period is partly a result of their excellence and partly a matter of luck (random variation).   The joint probability of any real ACO being lucky enough to be the benchmark performer in each of 60 categories during the same time period is very remote.  The Milliman “well managed” model, therefore, is designed to show a sort of exaggerated hypothetical stretch goal, rather than to estimate the average expected outcome for a real ACO (or health plan) that adopts the set of management interventions that Milliman favors.  When I’ve seen “well managed” numbers presented by Milliman in the past, it was in the context of health plan data.  I was always amazed at how far any particular health plan was from being “well managed” – a performance gap that serves a useful purpose as a call to action for Milliman’s clients who are managers wanting to avoid having their organization labeled as “not well managed.”  But, once I learned what the numbers really meant, I wished that they had been presented in a way that made their true nature clearer.  And, I wished that Milliman published data about the overall performance of actual organizations with different characteristics and using different types of clinical programs.

In this non-Medicare ACO context, the implication of this misinterpretation of the Milliman actuarial model is that the 28% cost reduction associated with well managed ACOs is an over-estimate.


Read More

NEJM report of ACO financial model fails to include risk of delayed start on transformation

On March 23, 2011, Trent Haywood, MD, JD, and Keith Kosel, PhD, MBA published the results in the New England Journal of Medicine (NEJM) web site of a financial model of a hypothetical Accountable Care Organization (ACO).  This model shows that ACOs are likely to lose money on the Medicare Shared Savings Program called for in the Patient Protection and Affordable Care Act during the first three years of implementing the ACO model, based on the up-front investment expected to be required.  The authors conclude that “the high up-front investments make the model a poor fit for most physician group practices.”  They call for modifications to the Medicare Shared Savings Program to make it more generous to participating ACOs.

The model is based on assumptions derived from data from the Physician Group Practice (PGP) Demonstration, carried out by CMS from 2005 to 2010.  In the PGP, the average up-front investment by participants was $1.7M, or $737 per primary care physician (PCP).  The authors calculate that an “unlikely” 20% margin would be required to break even during the 3-year time frame of the Medicare Shared Savings Program scheduled to start in 2012.

Haywood and Kosel are to be commended for taking the time to develop a financial model and publish results.  I think that such models are extremely helpful to real-world decision-makers because they force people to be explicit about the assumptions they are making, and they provide some quantitative estimates of the outcomes relevant to the comparison of available alternatives so people can make better choices.  Unfortunately, in my opinion, the authors misconceptualized the model, creating a risk that people will use the negative results of the model to justify inaction, to their own detriment.

Every decision is a choice among available alternatives.  To create a useful model to support decision-making, an analyst must follow the following four basic steps:

  1. Identify the available alternatives being compared
  2. Identify the outcomes that are relevant to the decision-maker and that are thought to be potentially materially different across the available alternatives
  3. Make quantitative estimates of  the magnitude and uncertainty of all such outcomes for all the available alternatives, and
  4. Apply values (including ethical principles and preferences) to determine which set out outcomes is most desirable or optimal

Although this basic process seems simple and straight-forward, experienced analysts know that each of these steps is devilishly difficult.  In the case of Haywood and Kosel’s financial analysis, in my opinion, they ran into trouble with the first two steps; they failed to identify the available alternatives and misconceptualized the choice or decision that the model is designed to support, and therefore failed to recognize non-Medicare outcomes that differ across the available alternatives.  Of course, an error in any particular step cascades to the remaining steps.

Haywood and Kosel did not explicitly explain the decision their model was intended to support.  But, one could infer from the conclusion that among the intended decisions they were supporting was the decision by physician organizations whether or not to make a $737 per PCP up-front investment and then sign-up for the optional Medicare Shared Savings Program in order to reap a return in the form of increased Medicare revenue.  But, the up-front investment required to create a successful ACO takes the form of fundamental transformation of care processes and the organizational structures, human resources, information systems, and cultural changes required to support them.  Such fundamental transformations affect the entire population served by the nascent ACO, not just Medicare patients.  And, they don’t just affect the providers’ relationship with payers, they also affect the providers’ competitive standing with respect to other providers and their relationship with other stakeholders such as employers, state and federal legislators, accreditation organizations, etc.

The correct conceptualization of the decision facing provider organizations right now is a choice between (1) getting started now with ACO-type transformation or (2) waiting until later to decide if such a transformation is necessary.  Physicians and hospitals that are contemplating the formation of ACOs would be wise to invest in the creation of a model to make explicit estimates of all the relevant financial and non-financial outcomes for the available alternatives.  Such a model will, by necessity, include many assumptions not supported by solid data.  That’s not the fault of a model, nor a reason to justify making decisions based only an intuition (what David Eddy calls “global subjective judgement”).  Rather, prudent health care leaders will invest the time to create and use a model to really understand the sensitivity of the results to various assumptions and the dynamics of the outcomes (how outcomes are likely to play out over time).

My prediction is that, when properly conceptualized as a “start transformation now” vs. “put transformation off until later” decision, such a model is likely to show what personal retirement planning models always show — it pays to get started on things that take a long time to achieve.  If you fall too far behind competitors, you may be unable to catch up later.  On the other hand, if provider organizations opt to get started on transformation, obviously there are many smaller decisions that need to be made, such as which care processes to start on, which particular payer-specific deals to cut, which IT investments to prioritize, etc.

One last point:  Although “pay back period” can sometimes be a useful summary measure of a financial analysis, my advice to to avoid over-simplifying the reporting of model results by reducing it down to a single summary measure.  Model authors would serve decision-makers better by presenting a table with their estimates of all the relevant outcomes for all the alternatives being considered, and possibly showing when those results occur over time.  Then, decision-makers can understand the drivers of their decisions and subsequently summarize the results in various ways that communicate their thinking most effectively using various summary measures such as net present value, return-on-investment, internal rate of return, pay-back period, cost per quality-adjusted life-year, cost-benefit ratio, etc.

Read More

What are agent-based models and how do they relate to ACOs?

Agent-based Modeling (ABM) is a type of computational simulation that involves the creation, inside a computer’s memory, of a collection of objects that are programmed to mimic the behavior of people, companies, countries or anything else that constitutes the “agents” that interact and change over time as part of a complex system. ABM has been used for decades in diverse fields, and has grown more popular as computers have become more powerful and less expensive. In the field of meteorology, ABMs have been designed such that each region of the atmosphere is represented as an “agent” that interacts with adjacent segments of atmosphere to create models that simulate and predict complex weather patterns. ABM has been used to study traffic patterns, where cars and trucks are modeled as “agents” moving across roads.  It has been used in biology to model predator-prey dynamics.  It has been used to study urban sprawl and racial segregation.  It has been used to model the behavior of children in the school yard and the behavior of nations interacting on the world stage. And, in health care, ABM has been used at the CDC and elsewhere to model the transmission and spread of communicable diseases.

ABM is useful when the system of interest is too complex to reduce down to an equation. In my experience, many problems in both health care and health care management fit that description. Think about creating a successful Accountable Care Organization. It involves primary care physicians, specialists, hospitals, patients and health plans interacting, being incentivized in new ways, with changing relationships and changing capabilities regarding information technology, care management and analytics. It seems unlikely that we can reduce this down to a system of equations. But, if we can’t model it with traditional models, should we just fall back on our intuition about what is likely to happen if we pursue different policies and make different investments? How about we use our intuition AND create models. The models allow us to clarify our thinking and explore many different scenarios to see the potential implications of different choices.

According to a Jan 31 article in the Johns Hopkins Gazette, Joshua Epstein is establishing the Johns Hopkins University Center for Advanced Modeling in the Social, Behavioral and Health Sciences (CAM).  Epstein is in the Department of Emergency Medicine, but has cross appointment in other departments including Bloomberg School of Public Health, School of Medicine, Whiting School of Engineering and Krieger School of Arts and Sciences.  They are focusing on disaster medicine, disaster response, public health preparedness, and chronic disease.   And, they are seeking multi-institutional collaboration, starting with the Santa Fe Institute, Pittsburgh National Center for Supercomputing Applications, Virginia Bioinformatics Institute at Virginia Tech, National Center for Computational Engineering at Tennessee and ETH, in Zurich.  Among the earliest models developed by CAM researchers is a planet-level model of 6.5 billion agents to explore the transmission of communicable diseases, including swine flu.

Joshua Epstein giving a lecture on Agent-based Models


In my opinion, ABM is destined to eventually become a standard tool to support knowledge-driven decision-making regarding the transformation of our complex health care system, including the formation of successful ACOs.

Read More

The Smoking Intervention Program, a Provider-based Care Management Process

Smoking cessation is an important public health concern, and has been the subject of a recent Agency for Health Care Policy and Research (AHCPR) guideline, as well as a HEDIS measure.   A point prevalence study conducted with the Henry Ford Health System found a 27.4% prevalence of smoking, and an additional 38.6% former smokers.

The CCE developed a first-generation smoking-dependency clinic which was staffed by trained non-physician counselors and overseen by a physician medical director. The original intervention was a 50-minute initial evaluation and counseling visit, with nicotine replacement therapy prescribed for all patients with a high level of nicotine dependency. This intervention was subsequently updated to reflect the AHCPR recommendation that, unless contraindicated, all smoking cessation patients be prescribed nicotine replacement therapy.

Because relapse is a normal part of smoking cessation, the intervention was explicitly designed to address relapse. This was done through return visits, an optional support group, and follow-up telephone counseling calls throughout the year, as illustrated in the following figure.

The program was designed to be inexpensive and simple to execute within the clinic. This was accomplished by automating the logistics of both the intervention and the collection of outcomes measures. The Flexi-Scan System, an internally developed computer application which helps automate outcome studies and disease-management interventions was used to automate (1) data entry through a scanner, (2) prompting of follow-up calls and mailings, and (3) the generation of medical-record notes and letters to the referring physicians. A database that can be used for outcomes-data analyses is acquired as a part of this process.

As illustrated on the figure below, this first-generation program achieved a twelve-month quit rate of 25%. Such a quit rate is about twice as high as the rate achieved with brief counseling intervention.

To evaluate the cost-effectiveness of this program, a decision analytic model was constructed. This model was constructed using the Markov method.  Key assumptions of the model include the following:

  • One year quit rate for usual care (optimistically assumed to consist of brief physician advice) was 12.5%.
  • Spontaneous quit rate of 1% per year in “out years.”
  • Relapse rate for recent quitters of 10%.
  • Age, Sex distribution based on Smoking Clinic patient demographics
  • Life expectancy of smokers and former smokers by age and sex based on literature (life tables).
  • Cost of clinic intervention – $199
  • Cost of nicotine therapy Smoking Clinic – $101 (Assuming 0.9 Rx/Patient)
  • Usual Care – $33 (Assuming 0.3 Rx/Patient)
  • Future health care costs were not considered
  • Annual discount rate of 5%

The results of this model were presented at the annual meeting of the Society for Medical Decision-Making.  The model results are presented in the form a table called a “balance sheet” (a term coined by David Eddy, MD, PhD).  As shown below, the model estimated that the first-generation smoking-dependency clinic cost about $1,600 for each life year gained.

To help evaluate whether this cost-effectiveness ratio is favorable, a league table was constructed (see below).  The league table shows comparable cost-effectiveness ratios for other health care interventions.  Interpretation of the table suggests that the smoking cessation intervention is highly favorable to these other health care interventions.

League Table

Intervention Cost per Quality-adjusted Life Year Gained
Smoking Cessation Counselling $6,400
Surgery for Left Main Coronary Artery Disease for a 55-year old man $7,000
Flexible Sigmoidoscopy (every 3 years) $25,000
Renal Dialysis (annual cost) $37,000
Screening for HIV (at a prevalence of 5/1,000) $39,000
Pap Smear (every year) $40,000
Surgery for 3-vessel Coronary Artery Disease for a 55 year-old man $95,000

Although this first generation program was effective and cost-effective, it was targeted only at the estimated 16,500 smokers in the HFMG patient population who were highly motivated to quit.

The estimated 66,000 other smokers in the HFMG patient population would be unlikely to pursue an intervention that involved visiting a smoking dependency clinic. Even for the smokers who were highly motivated to quit, the smoking cessation clinic had the capacity to provide counseling to about 500 people each year, or about 3% of these highly motivated smokers.

Second Generation Smoking Intervention Program

In response to this problem, the CCE developed a “second generation” Smoking Intervention Program.” This program uses a three tiered approach which includes (1) a “front-end” process for primary care and specialty clinics to use to identify smokers and provide brief motivational advice, (2) a centralized telephone-based triage process to conduct assessment and make arrangements for appropriate intervention, and (3) a stepped-care treatment tier.

In the “front-end” process, clinic physician and support staff were trained to screen their patients from smoking status and readiness to quit and provide tailored brief advise. Each participating clinic was provided with a program “kit” including screening forms, patient brochures, and posters to assist them in implementing the program. Patients who are interested in further intervention are referred to a centralized triage counselor for further assessment and intervention. These counselors are trained, non-physician care providers. They proactively call each patient referred, conduct an assessment of the patients smoking and quitting history and triage into a stepped-care intervention program.

An important part of this intervention has been providing information to clinicians, including a quarterly report showing the number of patients they have referred to the Smoking Intervention Program, the status of those patients, the type of intervention they are receiving, and the number of patients who report not having smoked in the preceding six months.

The clinician-specific data is presented in comparison to data for the medical group as a whole. These reports have a strong motivational effect on clinicians, as evidenced by a sharp increase in Smoking Intervention Program referrals after each reporting cycle.

As shown above, the second generation program achieved a six month quit rate of about 25%. This rate is virtually identical to the first generation program.  The new program, however, has much larger capacity and lower cost per participant. Patient satisfaction with the Smoking Intervention Program is encouraging, with 85% reporting that they would refer a friend to this program.

Read More
Page 2 of 3«123»