When I’ve had the opportunity to mediate such conflicts, I almost always used the same strategies.

• Step One.  I try to move both parties to some common ground, moving back to some basic statement that seemingly nobody could disagree with.  This generates a tiny bit of agreement momentum.
• Step Two. Apply the momentum generated in step one to getting the parties to agree that, if you took each party’s position to an extreme, the result would be undesirable. The parties are inherently agreeing to re-conceptualize the disagreement from being a choice between polar opposite positions to an optimization problem. The idea is to choose an agreeable value on some spectrum between undesirable extremes.  If the parties make this leap, we are half way to sustainable agreement.

• Step Three.  Get the parties to agree to avoid talking about the answer, and focus on reaching consensus on the factors and assumptions that influence the selection of the optimal answer.  Sometimes, this can be done subjectively, simply listing the factors.  Other times, it is worthwhile to get quantitative, working together on assumptions and calculations to estimate the magnitude and uncertainty of the outcomes at various points along the spectrum of alternative answers.  This quantitative approach has been described as the “explicit method,” and an example of applying it to resolve fierce disagreements about mammography guidelines is described in an earlier post.
• Step Four.  Finally, ask the parties to apply their values to propose and explain an optimum answer, from their point of view.  In this step, the important point is to insist that the parties acknowledge that they are no longer arguing about facts or assumptions, since consensus has already been achieved on those.  If not, then go back to step three. The objective is to untangle and separate factual, logical, scientific debates from the discussion of differences in values.  If those remain tangled up, the parties inevitably resort to talking at each other, rather than engaging in productive dialog.
• Step Five.  Try to achieve a compromise answer.  In my experience, if you’ve really completed steps 1-3, this ends up being fairly easy.
• Step Six.  Work to sustain the compromise.  Celebrating the agreement, praising the participants for the difficult work of compromise, documenting the process and assumptions, and appealing to people to not disown their participation in the process are all part of this ongoing work.   Passive aggressiveness is the standard operating model in many settings, part of the culture of many organizations.  And, it is a very difficult habit to break.

Of course, in the routine work of mediating conflicts, I don’t really explicitly go through these six steps. This conflict resolution approach is in the back of my mind. They are really more like habits than steps.

Sometimes this approach works. Sometimes, it does not.  It can break at any step.

Notice that break downs in most of the steps are basically people issues. People won’t change their conceptualization. They are unwilling to make their assumptions explicit. They are unwilling to acknowledge differences in values. They are unwilling to compromise.

But, sometimes, the process breaks because of the nature of the issue being debated. Sometimes, conceptualizing the debate as an optimization problem between two undesirable extremes fails because there are really not good choices along the spectrum.

For example, when debating the design of a program or policy, I have often encountered a no-win trade-off between keeping it simple vs. addressing each party’s unique circumstances.  If I keep it too simple, people complain that it as a “hammer,” failing to deal with their circumstances.  If I add complexity to deal with all the circumstances, people complain that it is a maze or a contraption.  If I select some middle level of complexity, the complaints are even worse because the pain of complexity kicks in before the value of complexity is achieved.

I’ve seen this no-way-to-win scenario in my own work, in the design of information systems, wellness and care management protocols, practice guidelines and protocols, analytic models, organizational structures, governance processes, contractual terms, and provider incentive programs.  And, I’ve seen this scenario in many public policy debates, such as debates about tax policy, tariffs, banking regulations, immigration, education, and health care reform.  In cases when the extremes are more desirable than the middle ground, the only approach I can think of is to bundle multiple issues together so that one party wins some and the other party wins others, to facilitate compromise.

Two weeks ago, I commented on the results of Nelson’s review of 6 Medicare Care Management demonstration projects over the last decade. At the same time that report was released, Nelson also released a companion report on the 4 “Value-based Payment” CMS demonstrations over the same period. The following is my adaptation of the main results table in this new CBO report.

The most influential of these demonstrations was the Physician Group Practice Demonstration (PGP).  The ten PGP participants included 2 faculty group practices within academic medical centers, 5 non-academic integrated delivery systems, one freestanding group practice, and one network consisting of 60 small practices.  All had experience with care management programs before the demonstration, and all implemented care management programs in the Medicare population for the demonstration.    These care management programs mostly consisted of nurses serving as care managers, focusing primarily on patient education and monitoring for patients with CHF or diabetes or meeting other “high risk” criteria.  Most implemented chronic disease registries for use by the care managers, in addition to using electronic medical records systems that were already in place or in the process of being implemented before the start of the demonstration.

Two of the ten PGP participants received bonuses in the first year, since the Medicare expenditures were more than 2% below the expected expense.  Four participants received bonuses in the second year, five in the third and fourth years, and four in the fifth year.  A formal evaluation of the program conducted after the first two years concluded that the overall effectiveness of the PGP across the ten participants was about 1% gross savings in year two, and even lower in year one.  Net savings, after counting the cost of the bonuses paid to some of the participants, was only 0.1% in year two.

The CBO report points out that even this meager 0.1% net savings might be an over-estimate, because PGP participants changed their diagnostic coding practices, making their populations appear to be sicker, and therefore making the risk-adjusted cost targets artificially high.  Such revenue maximization efforts have been job one in Medicare Advantage plans for years.  The PGP participants succeeded in lifting their risk scores by 8%, which was 3 percentage points higher than the increase in the comparison population.  And, the savings might have been further overestimated because all four of the PGP participants that achieved reward payments in year two already had slower than normal growth in Medicare expenditures before the PGP demonstration began.  All the other PGP participants that did not earn year 2 bonuses had pre-demonstration Medicare growth that was no different than the comparison population.

Despite these discouraging results, the PGP demonstration was nevertheless used as the main evidence base supporting the design of the Medicare Shared Savings Program, calling for the establishment of Accountable Care Organizations (ACOs).  It is also the main evidence base for the associated Pioneer ACO program, for which 32 participating provider organizations have recently been selected.

The Premier Hospital Quality Demonstration focused on 5 disease states and made bonus payments that amounted to only 0.25% of the total Medicare payments for those disease states.  That’s two orders of magnitude less than the size of incentive payments thought to substantially influence performance.  With such a tiny prize, it is not surprising that the quality improvements were assessed to have only a 1-5% incremental impact of process of care quality metrics during a period of time when such process of care metrics were improving nationally.  And, the CBO report concluded that the demonstration had no effect on Medicare expenditures for inpatient hospital care.  In fact, taking into consideration the reward payments, the demonstration led to an increase of costs by 0.3%.

The most successful of the demonstration projects was the Medicare Participating Heart Bypass Center Demonstration, which was used as a model for similar provisions in the health care reform law (PPACA).  In this program, the hospitals negotiated their bundled payments up front, ensuring that Medicare received savings compared to typical fee-for-service cases.  Overall, the program saved Medicare 10%.

So, where did the savings come from?

It’s possible that  they just came from good negotiating by CMS with hospitals that wanted to get out in front of what they saw was an inevitable trend toward bundle payment.  But, let’s assume that the hospitals really had a plan to reduce their costs in proportion to the negotiated decline in their revenue.

In interviews with leaders of the participating hospitals, the important changes in their approaches to patient management that were intended to reduce their costs included:

1. Greater involvement by surgeons in postoperative care
2. Earlier discharge of patients from the intensive care unit
3. Greater standardization of surgical protocols and supplies
4. Substitution of less expensive drugs for more costly ones, and
5. Greater reliance on critical nurse specialists for managing patients’ care in the hospital.

The participating hospitals substantially decreased their length of stay during the demonstration period, although length of stay for bypass surgery was dramatically decreasing nationally during that same time period.

However, probably more important than these process changes was the fact that all the participating hospitals created a physician reimbursement approach that established a fixed per case payment expected to cover all physician payments.  This amount was split in defined percentages among the four types of specialists involved in every bypass case — the thoracic surgeon, cardiologist, anesthesiologist, and radiologist.  Any payments to other specialists was essentially payed from this pool, reducing the payments to the four core specialties. Therefore, the hospitals created a strong incentive for the core specialties to limit referrals to other specialists.  This undoubtedly led to a reduction in utilization of those other specialists and the tests and procedures they generate.  Therefore, some of the savings probably came from reducing revenue to non-core specialists.

My conclusions

1. Don’t bother with incentives unless they are large enough to change behavior.  A 0.1% reward can’t work.  Nor can a 1% reward.  Try 10-20%.
2. Don’t create asymmetrical up-side only incentives.  They are far weaker in terms of motivating change.  And, they create a problem with paying out undeserved rewards for lucky good results.
3. Negotiate the savings up front, rather than just creating a game from which savings might be achieved.

The physician, Mary Newport, MD, is a neonatologist.  She explains that Alzheimer’s is thought to be similar to diabetes in that it involves an impairment in the ability of brain cells to respond to insulin and take in the glucose needed to provide energy.  As a result, brain cells die and eventually brain function is reduced.  She reasoned that the brain cells may avoid death by relying on an alternative fuel, ketones.  She identified coconut oil as a good dietary source of ketones.  So, she introduced coconut oil into her husband’s diet and noted improvement in his brain function.  She documented this improvement with a “clock test,” showing how a hand drawing of the face of a clock done after initiation of coconut oil treatment was more coherent and detailed that a drawing done before the treatment. Excited by the promising results, she wrote a book, started a web-site, and started doing radio and TV interviews to disseminate information about her new treatment.

From the simple explanation, it seems biologically plausible. And, I’m sure that Dr. Newport had nothing but the best intentions, motivated by love for her husband and a desire to help millions of people suffering from Alzheimer’s. And, it is possible that she is absolutely right. Coconut oil may be a simple, inexpensive, non-invasive, effective treatment for the disease.

But, obviously, we would not want to make decisions about treatments from a single data point, where the main outcomes measurement was a subjective assessment about how coherent a hand drawing of a clock was.

It would have been more appropriate for this physician to actually do the work of scientific research before disseminating results.  That would start with writing a study proposal, convincing peers in a study committee for a research granting agency that it was a plausible and promising idea. Then, she would conduct a randomized study, making objective measurements or collecting careful observations by impartial observers.  Then, she would analyze the results to see if there is a statistically significant difference in the outcomes between the treatment group and the control group.  The purpose of the statistical significance test is to assure that there is a low probability that any observed differences are just due to chance.  Finally, she would do the work of writing up a paper and submitting it to a peer reviewed journal to convince expert reviewers that there were no obvious flaws in the methodology.  Only then should she consider further dissemination of the information, such as by writing a book, starting her own web site, and doing TV and radio interviews.

The scientific approach to medical innovation has served us well as a society.  When this physician went straight from one observation to TV interviews, she short-cut the scientific approach.  She may be helping people with Alzheimer’s.  But, she may potentially be distracting Alzheimer’s patients from seeking proven treatments or diverting funding away from competing innovative treatment ideas that have gone through the scientific “front door.”  More generally, she may be harming our society’s commitment to a scientific approach.

The fact that the treatment is a type of food, rather than a patentable drug, admittedly changes the situation.  No drug company wants to fund research on coconut oil.  And, the coconut oil industry is not familiar with clinical research, even if they could benefit from increased demand for treatment of Alzheimer’s.  This is a good argument for why the National Institutes of Health and private research foundations should fund more research related to diet and natural remedies.   It should not be an argument for short-circuiting the scientific approach to health care innovation.

Fortunately, a research team from Oxford is pursuing a randomized clinical trial to test the use of dietary ketones as a treatment for Alzheimer’s.  The Oxford team is testing a specialized ketone ester that is thought to be ten times better than coconut oil in terms of delivering ketones to the interior of brain cells.  Results should be available later this year.  Hopefully, they will show meaningful improvement.

Click here for PDF copy of slides.

All of the 34 care management programs were designed to reduce Medicare costs primarily by maintaining or improving the health of the Medicare beneficiaries, and thereby reducing the need for expensive inpatient hospital stays.  As shown the graph below, different programs showed different effects on the rate of hospital admissions.  On average, the programs showed no effect.
 

Effects of 34 Disease Management and Care Coordination Programs on Hospital Admissions (Percentage Change in Hospital Admissions)

The CBO analyzed whether specific characteristics of programs led to better or worse results. They found that programs where the care management provider’s fees were at risk did not perform better or worse than those with fees not at risk.  However, they did find two things that worked.  They found that programs in which care managers had substantial direct interaction with physicians and those with significant in-person interaction with patients reduced hospital admissions by an average of 7%, while programs that did not have these features had no impact on hospital admissions.

But, after subtracting the cost of the programs themselves, almost none of the programs achieved net savings.

The programs with the most compelling performance included:

• Massachusetts General Hospital and its affiliated physician group reduced hospital admissions between 19-24% among patients selected as “high risk” using a program that was far more tightly integrated with the health care delivery system.  Physicians in the group were involved in the design of the intervention, and care managers were staff members in primary care physicians’ practices.  The patients received the vast majority of their care within the integrated delivery system, so almost all of their health information was available and up-to-date in an electronic medical records system.  Care managers were notified immediately when a patient was admitted to the emergency room or hospital.  They had an opportunity for face-to-face interaction with patients in the clinic.  And, they had access to a pharmacist to address medication issues.
• Two multi-specialty group practices in the Northwest reduced hospital admissions by 12-26% among high risk patients using a program that involved telemonitoring with the “Health Buddy” device that transmitted symptoms and physiologic measurements to a care manager
• Mercy Medical Center in rural Iowa reduced hospital admissions by 17% among patients hospitalized or treated in the ER in the prior year for CHF, COPD, liver disease, stroke, vascular disease, and renal failure using a program that involved care managers, many of which were located in physician offices and/or accompanied patients on their physician visits.

The methods used for these evaluations were far stronger than those used by the self-evaluations typically advertised by vendors of care management services.  In the CBO reports, 30 of the 34 programs were evaluated based on a comparison to a randomly selected comparison group.  The remaining 4 programs were evaluated using a concurrent comparison group selected using the same selection criteria.  In all cases, the programs were evaluated on an “intent to treat” basis, where study subjects were included in the evaluation regardless of whether they participated in the voluntary programs, thereby removing a source of bias that causes mischief in less rigorous evaluations.

To me, the take-away message is that provider-based care management is promising, but health-plan-style telephonic care management has not been successful, even in a senior population, where finding high risk targets is far easier and even when the care management services provider is highly motivated to succeed.

On the one hand, the entertainment industry does face a real problem as people are so easily able to steal their product and widely distribute it to people in the US using social network and other tools based outside of the US.  So, I can see why they are trying to get as much power as they can to shut that down through any of the U.S-based parties that are involved, such as search engines, web site hosts, and credit card clearinghouses.

From their perspective, all that talk about the importance of the internet as a vehicle for freedom and political change is a bit insincere coming from parties that are just trying to make a buck knowingly casting a blind eye to the fact that their customers are deriving a big part of the value of using their services by accessing stolen content.  The entertainment industry is basically just saying that if such parties are not putting in a reasonable amount of effort to block that, they should be considered accomplices to the theft.

But, on balance, I do side with the protesters.  The definition of reasonable amount of effort can very easily become a slippery slope to requiring tiny start-ups to buy unaffordable technology and putting in an unaffordable amount of manual labor into avoiding being accused of being an accomplish to someone else’s crime.  I agree with the protesters that this could easily stifle such start-ups, which would not displease the established big entertainment players.  The entire US-based entertainment industry is just not a very big part of the overall economy, and the internet has become an indispensable part of the entrepreneurship, innovation, political discourse, and social interaction that moves society forward.  The unintended consequences of the proposed SOPA/PIPA legislation, both directly in the US and on the precedent-setting impact on laws in other countries, could be huge and are not worth squeezing a few more dollars out of one small sector of our economy.

Therefore, we should reject the SOPA and PIPA legislation, and continue to pursue more incremental efforts to thwart internet piracy.

More information at http://en.wikipedia.org/wiki/Wikipedia:SOPA_initiative/Learn_more

The Pioneer ACO program was intended to allow provider organizations that had experience and sophisticated population management and care coordination capabilities to get started under a gain-sharing arrangement for Medicare more quickly than the Medicare Shared Savings Program (MSSP).  Compared to the MSSP participants, these Pioneer ACOs will take on greater risk and will be eligible to receive higher gain-sharing payments. They will then have an opportunity to move more rapidly from a gain-sharing to a population-based full-risk capitation payment model in year three, as long as they successfully earned shared savings awards during the first two years.  They will then be allowed to continue through an optional fourth and fifth year. The Pioneer ACOs all commit to negotiating “outcomes-based” reimbursement arrangements with other payers by the end of the second year.

Of the 32 Pioneer ACOs, more than one third are physician organizations, with the remainder being integrated delivery systems or other structures that include both hospital facilities and physicians.

The majority of the Pioneer ACOs are concentrated in 5 states, with California predictably leading the pack with 6 Pioneers, followed by Maine, Michigan, Minnesota and Texas.

The three Michigan-based ACOs include:

• Genesys PHO: a collaboration between Genesys Health System and 160 primary care physicians with 400 participating specialist physicians who deliver health care services in Genesee, Lapeer, Shiawassee, Tuscola and northern Oakland counties.
• Michigan Pioneer ACO: To be managed by the Detroit Medical Center PHO, a partnership of The Detroit Medical Center and its 1100 physicians, who include employed and faculty physicians, but consisting mostly of private practice primary care physicians. The Detroit Medical Center is a large academically integrated system in metropolitan Detroit, owned by Vanguard Health Systems and serving as a teaching and research site for the Wayne State University School of Medicine.
• The University of Michigan Health System, which includes the U-M Faculty Group Practice, part of the U-M Medical School, includes all of the nearly 1,600 U-M faculty physicians who care for patients at the three U-M hospitals and 40 U-M health centers. Although U of M’s ACO was categorized by CMS as an “integrated delivery system,” the Pioneer ACO will also include participation by IHA Health Services Corporation, an Ann Arbor-based group practice that is part of Trinity Healthcare with 175 physicians in 32 practices.

The full list of Pioneer ACOs follows:

CMS released a fact sheet with more details about the Pioneer ACO program.

Four things I’ve noticed about this lifecycle of health care improvement frameworks:

1. They are formulated by conceptual thinkers, but then get taken-over by more tactically-oriented people.  The tactical folks often focus too much on the tools, terminology and associated rituals.  The framework always gets “simplified” to be more suitable for mass consumption.  For example, continuous quality improvement somehow morphed into being primarily about assigning a timekeeper during team meetings and communicating progress on a felt-backed “story board,” rather than finding people with systems-thinking talent and applying that talent to understand sources of variation in complex processes.
2. During the early part of the growth phase, the advocates are always desperate for examples of success, and shower a great deal of attention on early projects that are described using the terminology of the framework and that appear to have succeeded.  The desperation usually leads advocates to lower their standards of evidence during this phase.  This leads to over-promising and unrealistic expectations.  It stimulates lots of superficial imitation by people interested in hopping on the bandwagon.  And, it plants the seeds for the eventual decline, when people determine that their inflated expectations were not met.
3. The decline phase, when the framework is declared to be ineffective, seems to always happen before the framework was ever really implemented in the way envisioned by the original formulators during the incubation phase.
4. All the frameworks are really just restatements of the same underlying concepts, but with different terminology and tools, and different emphasis.  In other words, they all have the same anatomy, but different parts of the anatomy are emphasized.

This last point reminds me of the “humunculus,” also called the “little man.” When I was in medical school in the late 1980s, we used heavy text books that generally did a bad job of teaching the information. One notable exception was clinical neuroanatomy. We used a small, paperback text book playfully entitled “Clinical Neuroanatomy Made Ridiculously Simple” by Stephen Goldberg, MD. It contained a collection of clever drawings designed to explain the structures and functions of the brain and spinal cord. Perhaps the most famous of the drawings was the humunculus.

This drawing was adapted from earlier work by an innovative neurosurgeon named Wilder Penfield, who invented new surgical procedures for patients with epilepsy during the late 1930s.  During those procedures, he used electrodes to stimulate different points on the surface of the brain.  He drew diagrams similar to the drawing above showing that the surface of the brain contained a little man hanging upside down. The diagram shows that a disproportionate portion of the brain surface is dedicated to the sense of touch and muscle movements in certain parts of the body.  Lots of brain surface is dedicated to highly sensitive and nimble areas like the lips, tongue, hands and feet.  Very little brain surface is dedicated to the arms, legs and back.  Many anatomic illustrators have drawn the humunculus as a cartoon character showing how this disproportional emphasis on different parts of the body looks on the little man.

The humunculus is a great teaching tool, making it easy to remember these aspects of clinical neuroanatomy.  But, I think the humunculus is also a useful metaphor for the distorted emphasis that various health care improvement frameworks have placed on various parts of the underlying anatomy of health care improvement.

Over the years, I have assimilated the concepts, terminology and tools from these various improvement frameworks into an approach that attempts to achieve balance, with each aspect of the framework shown without over-emphasis.

This framework puts the patient in the center, surrounded by the health care processes, which are surrounded by improvement processes.  It attempts to balance between focusing on care planning (the clinical decision-making regarding what services are needed) vs. focusing on care-delivery (the teamwork to execute the care plan and provide health care services to the patient).  It balances between measuring outcomes and measuring quality and cost performance.  It balances between implementing best practices through guidelines and protocols vs. improving practices through performance feedback and incentives. By avoiding a distorted over-emphasis on any one part of the anatomy, hopefully it can have greater lasting power than some of the more humunculus-like frameworks that have come and gone.   This framework is described more fully here.

Don Berwick as one of the Notre Dame Four Horsemen on Ward's coffee mug

Apparently, I talk about Dr. Berwick a lot. A few years ago, I received one of my most treasured gifts from colleagues at Blue Cross Blue Shield of Michigan (BCBSM).  It was a coffee mug featuring the famous photograph of the Four Horsemen of Notre Dame, a reference to my undergraduate alma mater.  My colleagues replaced the faces of three of the horsemen with the faces of three of my health care heroes, Drs. Paul Ellwood (the person who coined the terms “health management organization” and “outcomes management”), David Eddy (the clearest thinker on the topics of clinical practice policies and the rational allocation of health care resources), and Don Berwick. The face of the forth horseman they replaced with my own face.  I considered it a great honor to be associated with my heroes, at least on a coffee mug.

My team at BCBSM had heard me repeatedly explain Dr. Berwick’s important contribution to adapting the quality improvement  principles that had been successfully used in manufacturing to the health care field.  Others had been involved in promoting “continuous quality improvement,” “statistical process control,” and “total quality management” in health care. Paul Batalden, Brent James, Eugene Nelson, and Jack Billi come to mind, to name but a few. But, in my opinion, it has always been Berwick that has been the most eloquent and persuasive. He connected the statistical tools emphasized by James with the front line worker involvement emphasized by Batalden. And, he was able to describe how these approaches applied to clinical decision-making as well as care delivery.

At the heart of Dr. Berwick’s contribution was teaching us all to distinguish between the “Theory of Bad Apples” and the “Theory of Continuous Improvement.”

According to the Theory of Bad Apples, errors come from “shoddy work” by people with deficient work performance.  Leaders who uphold this theory focus on inspection to identify such deficient performance, indicated by the undesirable tail in the distribution of provider performance as shown on the left side of the diagram above.  Then, such leaders focus on holding the bad performers “accountable” by applying disciplinary measures intended to motivate improvement in performance and by pursuing other interventions intended to re-mediate the bad performance.  In the health care context, the workers are physicians and the shoddy work is poor quality health care. According to Berwick, the predictable defensive response by the physicians who are targeted for such remedial attention includes three elements: (1) kill the messenger, (2) distort the data and (3) blame somebody else.

Berwick advocates instead for the Theory of Continuous Improvement.  The basic principles of this theory are

T-Shirt of "Berwickians" -- the staff of epidemiologists and biostatisticians at BCBSM

Berwick argued the point made by Dr. Deming:  if we  can reduce fear, people will not try to distort the data.  When learning is guided by accurate information and sound rules of inference, when suppliers of service remain in dialog with those that depend upon them, and “when the hearts and talents of workers are enlisted in the pursuit of better ways, the potential for improvement in quality in nearly boundless.”

I first was influenced by Dr. Berwick back in the 1980′s when he championed these ideas during his tenure at the Harvard Community Health Plan, and subsequently during the 1990′s when he led the National Demonstration Project on Quality Improvement in Health Care and the Institute for Healthcare Improvement.  His face was already on my coffee mug at the time he was nominated to lead CMS.  I was thrilled that someone from our community of people dedicated to clinical process improvement had been recognized and would be serving in a position of such influence.

The Irony of the Political Opposition to Berwick’s Role as CMS Administrator

Dr. Berwick’s candidacy as CMS Administrator faced stiff opposition from Republican leaders who were angry about anything connected to the health care reform law or, for that matter, the Obama administration itself.  The President made the decision to evade this opposition by making a recess appointment of Dr. Berwick.  But, such recess appointments have a limited lifespan.  As the deadline for making a formal, congressionally sanctioned appointment approached at the end of the 2011 legislative session, 42 Republican senators signed a letter reiterating their disapproval of Dr. Berwick as CMS Administrator.   The arguments against Dr. Berwick’s  candidacy, both at the time of his original nomination and again over the last few months, centered around comments that Dr. Berwick has made praising the British health care system.  They concluded from his comments that he was in favor of redistributing wealth to the poor and of rationing, the dreaded “R” word, the thing done by “death panels!”  He was, therefore both a bleeding heart and heartless at the same time.  Dr. Berwick denied these charges, but the opposition was unconvinced and unwilling to back down from a position of persistent opposition to anything connected to “Obamacare.”

The irony is that, of the heroes on my coffee mug, Dr. Berwick is not the one deserving of praise for having insight and bravery concerning the basic tenets of health economics. Instead, it was Dr. David Eddy’s mug that was on my coffee mug because he was brave enough to publish numerous papers in the Journal of the American Medical Association explaining why rationing was the right thing to do (e.g. this one and another one).  Eddy argued that creating evidence-based “practice policies” that rationally allocated health care resources using “explicit methods” was favorable to using implicit methods supported only by “global subjective judgement.”  What a radical thought!

Despite my great admiration for Dr. Berwick, he was the hero that disappointed me as a rationing denier.  In fact, in a 2009 paper published in Health Affairs entitled “What ‘Patient-Centered’ Should Mean: Confessions of an Extremist,” he eloquently argued that we should give any patient whatever they wanted, regardless of the cost and regardless of the evidence of effectiveness.  He discounted the role of the physician as a steward of resources.  I felt the argument was heartfelt and humanistic.  But, I felt it was a cop out.  How strange, then, that the Republican opposition hoisted him on the rationing petard.

Looking Forward to Berwick’s Next Journey

Although it is disappointing to me that Dr. Berwick will no longer be leading CMS, I am eager to see what he chooses to do next.  I’m sure he will continue to make a great contribution to our field.  Without all the administrative and political duties to clog up his day, perhaps we are about to witness a surge in his ongoing contributions to improving health care.

More information: See Health Affairs article and associated Health Affairs Blog Post praising Dr. Berwick.

The AHRQ recently released an excellent pair of white papers offering guidance on the proper evaluation of PCMH initiatives.  The first is a four page overview intended for decisionmakers, entitled “Improving Evaluations of the Medical Home.”   The second is a 56 page document that goes into more detail, entitled “Building the Evidence Based for the Medical Home: What Sample and Sample Size Do Studies Need?”  The whitepapers were prepared by Deborah Peikes, Stacy Dale and Eric Lundquist of Mathematica Policy Research and Janice Genevro and David Meyers from the AHRQ.

The white papers emphasize a number of key points:

• Base evaluation plans on plausible estimates of the effects of PCMH.  Based on a review of the evidence so far, the white paper suggested that a successful program could plausibly hope to reduce cost or hospitalizations, on average, by 15 percent for chronically ill patients and 5 percent for all patients.
• Use a proper concurrent comparison group, rather than doing a “pre-post” analysis. Pre-post analyses, although common, are inconclusive because they can easily by confounded by other factors changing during the same time period, such as economic conditions, health care policy changes, advances in technology, etc.
• Focus on evaluating a large number of practices, rather than a large number of patients per practice.  The authors point out that “a study with 100 practices and 20 patients per practice has much greater power than a study of 20 practices with 100 patients each.”  They warn that small pilot studies with 20 practices or less are unlikely to produce rigorous results without combining the results with many other small studies conducted using the same metrics.  Such pilot studies, which unfortunately are very common, are really only useful for generating hypotheses, not for drawing conclusions.  The authors note that neither positive nor negative results of such small studies should be relied upon.  Small PCMH studies can show no significant impact because they did not have the power to detect such an impact.
• Focus on evaluating health and economic outcomes in subsets of patients such as those with chronic disease.  Satisfaction can be evaluated across the entire population, but if you use data for the entire population to measure hospitalizations, emergency department visits, inpatient days, or health care costs, the lower risk portions of the population contribute noise that obscures the measurement of the effect that is occurring primarily among those most likely to experience such events in the first place.
• Use statistical methods that account for “clustering” at the practice level, rather than treating individual patients as the unit of analysis.  Since the intervention is intended to change processes at the practice level, the individual patients within a practice are not independent of one another.  Clustering must be taken into account not only at the end of the study, when evaluating the data.  It must also be taken into account at the beginning, when determining the number of practices and patients to sample.  For example, if a study includes a total of 20,000 patients, but the patients are clustered within 20 practices, then the effective sample size is only 1,820, assuming patient outcomes are moderately clustered within practices.  When statistical methods treat such patients as independent, they are implicitly treating the sample size in such a situation as 20,000 rather than 1,820.   As a result, evaluators making such an assumption are dramatically over-estimating their power to detect the effect of the PCMH transformation.  If they adjust for clustering at the end, their findings are likely to show a lack of a significant effect, even if the PCMH program really worked.  On the other hand, if they don’t adjust for clustering in the end, there is a great risk of reporting false positive findings.  For example, in a PCMH study with 20 practices and 1,500 patients per practice, where the analysis was done without adjusting for clustering and found a positive result, there is a 60% chance that the positive result is false, based on typical assumptions.

The authors re-ran the models, assuming that the measure of cost and hospitalization would consider only the sub-population of patients with chronic diseases.

The model showed that, based on reasonable assumptions, at least 35 PCMH practices, plus an equivalent number of concurrent comparison practices, would be required to detect the 15% effect that the literature suggests is the plausible effect of PCMH on cost and hospitalizations among patients with chronic diseases.  Even when focusing on the chronic disease sub-population, a pilot evaluation with only 10 PCMH practices and 10 comparison practices could not detect an effect smaller than 30%, an effect size they considered implausible.

I found this modeling exercise to be very informative and very worrisome, given the large number of pilot studies underway that are unlikely to provide conclusive results and the risk that people will try to draw incorrect conclusions when those results become available.  Often, health care leaders find these calculations inconvenient and frustrating, as if the bearers of this mathematical news are being overly rigorous and “academic.”

Note that these concepts and conclusions are applicable not only to evaluations of PCMH, but also of other programs intended to improve processes or capabilities at the level of a physician’s practice, a clinic or a physician organization such as health information technology investments, training staff in Lean methods, or implementing gain-sharing or other performance-based incentives.