Henry Ford observational study that shows 66% in-hospital mortality reduction from hydroxychloroquine seems to be guilty of “heroic propensity score” problem

During the 1990s, I served as the director of the Center for Clinical Effectiveness at the Henry Ford Health System in Detroit.  I was lucky to have an opportunity to hang around with a really talented group of people there, some of which are still there.  Back in the day, I was among the loudest voices advocating for the value of retrospective observational research using databases.  But, at the same time, I was also one of the co-investigators in the largest clinical recruitment site for the largest randomized clinical trial (RCT) done up to that point – the prostate, lung, colorectal and ovarian (PLCO) cancer screening trial.  So I have a foot in both RCT and observational research canoes. Both are important tools that can be used or misused.

Fast forward to the COVID-19 pandemic. Henry Ford established a system-wide COVID-19 Task Force, reviewed the paucity of evidence available, developed consensus on clinical protocols for treatment, and implemented those protocols across six hospitals. Based on its commitment to science and improvement, it also launched randomized clinical trials, including one focused on the efficacy of hydroxychloroquine to prevent SARS-CoV-2 infections in first-responders. It also conducted observational research on the effectiveness of hydroxychloroquine and azithromycin, alone or together, in treating hospitalized COVID-19 patients. Samia Arshad, Marcus Zervos and colleagues took the time to share that observational research in a peer-reviewed journal. That’s how a serious integrated delivery system is supposed to act, and it made me proud to have been associated with them in the past.

As shown in the graph above, the study showed that hydroxychloroquine reduced in-hospital mortality by 66% — or, more carefully stated, the in-hospital mortality “hazard ratio” was 66% when comparing COVID-19 inpatients that received hydroxychloroquine (and not azithromycin) with COVID-19 inpatients that received neither hydroxychloroquine nor azithromycin. Because of strong interest in COVID-19 treatment in general and political controversies surrounding President Trump’s advocacy for hydroxychloroquine in particular, the results were covered in the media and the paper became sufficiently controversial as to prompt the Chief Clinical Officer and the Chief Academic Officer to publish an open letter saying:

Unfortunately, the political climate that has persisted has made any objective discussion about this drug impossible, and we are deeply saddened by this turn of events. Our goal as scientists has solely been to report validated findings and allow the science to speak for itself, regardless of political considerations. To that end, we have made the heartfelt decision to have no further comment about this outside the medical community – staying focused on our core mission in the interest of our patients, our community, and our commitment to clinical and academic integrity.

I admire how the Henry Ford team handled the controversy, and I would defend the value of publishing sincere study findings without regard to the political winds that are blowing.

However, on the question of the value of this particular observational study, I come down on the side of holding out for randomized clinical trials before suggesting changes in treatment protocols, even given the urgent circumstances of a novel disease advancing to pandemic status quickly. Yes, RCTs can be annoyingly slow. But, just as vaccine development has been fast-tracked, so could RCTs of COVID-19 therapeutics. As a life-long advocate and practitioner of observational research, I have a healthy skepticism about all the ways that such research can go astray. Experience has shown that even in ideal circumstances, observational research is tricky business.

A careful read of this particular Henry Ford study shows that this is not ideal circumstances.

In the paper, the authors describe one of the strengths of the study as the fact that the clinical practices studied were based on “regularly updated and standardized institutional clinical treatment guidelines.” Guidelines and protocols are good for reducing unwarranted variation in practice. But they constitute a limitation with regard to the conclusiveness of the study, not a strength.

In any retrospective observational study, the biggest concern is that there may be sources of bias — factors that make the different treatment groups less comparable — particularly factors that cannot be subject to “control” or “adjustment” in the analysis.  The ideal scenario, sometimes described as a “natural experiment” is when you have different sites of care, each of which happens to have a consistent practice pattern that is different from the pattern observed at other sites. Such a scenario resembles a study where the sites were actually randomized to different treatment protocols.  As long as you don’t have big concerns about the comparability of the populations at those different sites, such an analysis can be persuasive, especially if you can “mop up” the remaining small amount of confounding variation using statistical methods such as risk-stratification, risk-adjustment or propensity matching. 

In the case of the Henry Ford study, the investigators had the opposite scenario:  all sites were using the same treatment protocol, the purpose of which is to intentionally give different treatments based on different patient characteristics. Consequently, the patients receiving different treatments are intentionally non-comparable.  They used propensity matching in an attempt to reduce the potential bias, implicitly asserting that if two patients had the same “propensity score” they were in fact comparable patients despite the fact that the treatment protocol and the clinical judgement of the attending physicians determined that they needed different treatments. From the perspective of the investigators, they did the best they could under the circumstances, and I am not suggesting any intent to do hand-waving to distract from the underlying sources of bias. But I do know from experience that statisticians hired by some “wellness” and “disease management” program vendors have intentionally done so over the years. I call this problem the “heroic propensity matching” problem and it is on my “watch out” list along with the “regression to the mean” problem, the “volunteer bias” problem, and my favorite, the “risk factor switcheroo” problem.

My view has always been that statistical methods to deal with biases in observational studies are only appropriate for use in “mopping up” small amounts of bias.  You generally can’t compare humans to mice, nor can you compare pediatric patients to adult patients thinking that propensity matching or risk-adjusting is going to solve the comparability problem.  Consequently, I’m skeptical about results based on a comparison of treatment groups where the patients are intentionally different by virtue of a formal practice protocol, in addition to the usual problematic differences due to clinical judgments — and I’m reluctant to accept propensity matching as an appropriate or adequate remedy.

So, in this scenario, I favor going through the front door and doing real RCTs as quickly as possible.

Read More

Assigning responsibility for Identify and Access Management – Care Analytic News

Care Analytic News Reprint - Thought Leaders Corner - R Ward - 2020-07

Source – Reprint from CareAnalyticNews.com, published by Health Policy Publishing, LLC.

Read More

EMR event log study shows 6 hrs of use per day, but implicitly belittles the clinician’s cognitive effort and the EMR’s support

In a recent paper in the Annals of Family Medicine by Brian Arndt and colleagues at the University of Wisconsin, the authors described the results of an analysis of the user event logs of their Epic EMR.  The authors determined that primary care clinicians used the system nearly 6 hours per day out of an 11.4 hour work day, and that 44% of that time was spent on tasks that the authors categorized as “clerical and administrative.”  It is an interesting paper, but I think it represents a lack of vision and insight on the part of the authors regarding the role that technology can and should play in supporting the cognitive effort of clinicians.
Most specifically, when a clinician was using EMR-based template charting and orders modules, the authors categorized that work as “clerical.”  Such a classification fails to acknowledge that the clinician is (or should be) using such modules to create a coherent, evidence-based plan of care:
  • using condition-specific templates that help remind her of the clinical observations and treatments to consider,
  • viewing reminders and order sets to help her to remember to include important evidence-based services in the plan of care,
  • receiving alerts to help her avoid ordering services that may cause an allergic response, conflict with other medications that the patient is taking, or that are dosed inappropriately considering the body mass of the patient,
  • viewing prompts for a needed referral for care management, or
  • receiving referral guidance to help direct specialty referrals to the specialists that have agreed to integrate care processes with the primary care practice within a clinically integrated network or accountable care organization.

The authors implicitly dismiss the cognitive work being done by the clinician when they are doing “documentation” and “ordering” and the support that the technology is providing to that cognitive work.  They reduce it all down to being wasteful paperwork, and suggest that it be eliminated through voice dictation or assignment of such paperwork to other members of the care team — both of which would preclude the decision support to those important cognitive processes.

I share the authors’ implied frustration with the failure of the current generation of health information technology to live up to the long-standing vision of supporting clinicians’ decision-making and coordination of care across a multi-disciplinary care team.  I acknowledge that today’s EMR-based care planning and coordination functionality feels like clerical work.  I’ve spent time on that problem.  I have two patents on proposed solutions to that problem.  I hope some day to help solve that problem, which I consider to be one of the most important problems in the broad fields of health care improvement, population health management and medical informatics.
I strongly agree with the idea of using user event logs to study how users actually spend their time and how they use application functionality, and I applaud the author’s efforts to validate the event log data with some direct observations.  But, I don’t agree with the authors’ suggestion that others should use their “EHR task categories” because they implicitly reject the vision of real technology-assisted care planning and coordination.  We shouldn’t give up on that vision.  We can and must do better to make it reality.
Read More

If repealing Obamacare is off the table, can we turn attention to improving it through cost-effective clinical protocols?

Paul Krugman recently wrote an article in the New York Times posing the question: if repealing Obamacare is off the table (for now), should people on the “progressive” end of the political spectrum push for a single-payer “Medicare for all” system or just advocate for incremental improvements to the privatized Obamacare model?  He says if we were starting with a blank slate, he would favor the single payer model.  But, he argues, the politics of moving to single payer are too difficult, and the evidence from other countries suggests that a privatized model can achieve comparable outcomes.  Therefore, he argues that progressive politicians should turn their attention to other social policy priorities like subsidized child care and pre-kindergarten education.

I generally agree with Krugman’s proposal to focus on incremental improvements to Obamacare, particularly if the objective is just to maintain and improve access to health insurance.  But that’s not our only objective.  We should also care about the quality and cost of health care.

I’ve long felt that policy to increase access to care should be linked to policy to assure the cost-effectiveness and value of care. Insurance is, by its nature, a collective, cooperative thing. In the long run, the people who are covered under the same insurance risk pool are sharing a finite resource. If they understood that, they would rationally desire for there to be protections against the pooled resource being squandered by other people for low value purposes. In health insurance, such protections primarily take the form of benefit policies. Benefit policies may define which services are not covered because they are considered experimental or cosmetic. They may define limitations based on age, gender, or medical history. They may also set quantity limits on coverage, such as defining the number of physical therapy visits or inpatient psychiatric hospital days covered. They may set lifetime maximum dollar amounts. But, such insurance benefit policies are very blunt instruments.  Insurance companies also protect against low-value uses of health care services using “utilization management” programs, including requiring pre-authorization processes, where doctors are required to submit justifications for proposed services and insurance company employees judge whether the proposed services meet “appropriateness” criteria.  Such utilization management programs create conflict, and insurance companies generally establish criteria that are very loose to minimize the conflict.  As a result, such programs are also very blunt instruments.

Clinical protocols, in contrast, can be more precise instruments, taking into consideration the details of a patient’s clinical situation. Clinical protocols are typically developed by physicians, and are ideally supported by evidence from clinical research studies.  Clinical protocols can take many different forms, and go by different names including “guidelines,” “algorithms,” “care maps,” and “standards of care.”  Whenever we have tried to design clinical protocols, especially for complex and costly care processes such as for low back pain, congestive heart failure, cancer or the care of frail elderly patients, we discovered that different protocols can have very different costs and outcomes.  Thoughtful design, rigorous implementation and continual evaluation and improvement of clinical protocols can lead to large improvements of outcomes and large reductions in cost. But, cost effective protocols do deny some people some treatments that would have helped them a little (just not enough to be “worth” the cost). The whole premise of designing cost-effective protocols depends on the recognition and acceptance of the collective nature of insurance and the finite nature of the resources being shared. Furthermore, it is essential that the people for whom such protocols are applied trust the people and the process of creating and implementing the protocols. In for-profit, commercial insurance companies, there is a fundamental conflict of interest if the owners of the insurance company get to control the design and implementation of the protocols and if they get to keep the money saved from denying services that could have helped people — even a little.

In a single payer system, the entire country (or each state) is treated as a risk pool, and the government plays the role of protocol developer. Some people are OK with that, while others are loathe to assign such authority to governments.  If  we continue to have private insurance companies or accountable care organizations bearing the risk for populations of patients (as in the current Obamacare system), then such organizations can make decisions about clinical protocols.  In either case, we absolutely need to create structures and mechanisms to ensure that the people receiving the care trust the people and process used to create and implement cost-effective protocols.  Some private organizations, such as Group Health Cooperative of Puget Sound (now part of Kaiser Permanente), created some structures and processes designed to build this trust back in the 1990s.

Although most other advanced countries already accept cost-effectiveness and pursue the development and implementation of cost-effective protocols, and although there would be a huge opportunity to reduce cost and improve outcomes by doing so in the U.S., making this policy shift in the U.S. will be very difficult. The U.S. population has been taught to be wary of “rationing” and “death panels” and U.S. doctors have been taught to reject “cookbook medicine.” Nevertheless, moving the health policy discussion in that direction may, in the long term, do some good.   Politicians asking “what’s next” after the apparent end of the quest to repeal Obamacare should consider turning attention to bringing cost-effectiveness to health care through clinical protocols.

Read More

Hospital Value-based Purchasing program 1% incentive is like homeopathic medicine — too diluted to actually work

In the June 15, 2017 issue of the New England Journal of Medicine, Andrew Ryan and colleagues from the University of Michigan published an evaluation of the Medicare Hospital Value-Based Purchasing Program (HVBP).

To summarize, if you offer a 1% incentive, and then dilute it by offering it only for the 40% of hospital patients covered by Medicare, and then dilute it further by spreading it across three domains (clinical process quality, patient experience and mortality), and then dilute each of these by basing them on multiple component metrics, and then dilute it more by choosing metrics that have already been reported for a number of years (and therefore the “low hanging fruit” improvement opportunities may already have been picked), and then further dilute it by offering the incentive mixed in with many other incentives for such things as meaningful use of EMRs…..

Wait for it….
You don’t see impact, even after 4 years.
The thinking behind HVBP is like homeopathy, where the practitioners assert that the more they dilute the homeopathic remedy ingredient, the more powerful the remedy becomes.
Imagine if a company hired a CEO and wanted to incentivize her to achieve growth and profitability. Would they consider a 1% incentive to be meaningful (even without further dilution).  No, the board would choose a number 50 to 75 times higher.
How about an equipment manufacturer choosing an incentive percentage for its sales team?  One percent sound like enough?
I’ve been exasperated for years that our value-based reimbursement designs – for both government and commercial payers – include an incentive that is far too small to motivate the types of changes they are intended to cause.  I fear we are just setting ourselves up for eventually someone saying “well, we tried incentives, and they don’t work.”
Read More

How to use and improve predictive models

Read More

Congressional Budget Office: 27 million will lose coverage and premiums will increase by 50% if repeal ACA without replacement

In my last post, I noted that Congress was eager to repeal Obamacare, but lacked consensus on a replacement.  I noted that Obamacare was designed to cover the sick people and keep premiums affordable through the individual mandate and subsidies.  These two provisions of Obamacare can be reversed through legislative mechanisms that avoid filibusters.  But, removing those provisions, while leaving in place Obamacare’s prohibition of denial for pre-existing conditions creates an unstable insurance market that could lead to an increase in the number of uninsured people, and an increase in the premiums paid by those that continue to be insured.

Today, the Congressional Budget Office published a report offering some non-partisan quantitative estimates of those outcomes:

The clear implication of these projections is that pursuing the repeal of just the portions of Obamacare that can be repealed unilaterally, while leaving in place the popular prohibition of denial of coverage for people with pre-existing conditions would be a disaster.  Hopefully, these projections will help convince Congress to resist the urge to seek quick repeal to appease fervent anti-Obamacare constituents.  Hopefully, Congress will take a breath and take the time to build bi-partisan consensus on a more comprehensive and coherent design for our health care insurance system.

Read More

Are high risk pools really a good replacement for Obamacare architecture?

Congress and the president-elect are enthusiastic about repealing Obamacare, but have not yet achieved any consensus about what to replace it with.  High risk pools figure prominently in various proposals, including Ryan’s “A Better Way” proposal.  But high risk pools are not a new idea.  Thirty five states had them in the years before Obamacare, so we have some experience to draw upon.  In general, they performed poorly, mostly because they were substantially underfunded, leading to high premiums and shameful waiting lists that withheld coverage for the sickest people – those that that needed coverage the most. The following explainer video was prepared last summer by the Kaiser Family Foundation, a health policy think tank.

Sounds Like A Good Idea? High-Risk Pools

High risk pools don’t reduce cost or risk. They just transfer it from private health plan premium payers to taxpayers — mostly state taxpayers. And, if the states fail to fund it properly (as has usually been the case), the wait lists associated with high risk pools creates a particularly cruel mechanism for keeping the most desperate citizens from the lifesaving care they need.

How did Romneycare and Obamacare Avoid the Need for High Risk Pools?

High risk pools consisted of the sickest people in the population. Since sick people incur health care expenses that they can’t afford, the money has to come from somewhere.  It can come from healthier members of the same health plan, from state taxpayers, or from federal taxpayers.  If it is to come from healthier members of the plan, there must enough of such healthy members to share the cost.  The healthy people can’t just wait until they are sick to buy insurance.  If too many healthy people opt out, the premiums for the people in the plan will be too high.  An insurance “death spiral” occurs when high premiums causes some of those healthy members to drop coverage, forcing premiums to go even higher for the remaining healthy members, ultimately leading to the failure of the plan.

So, to achieve coverage for sick people and affordability for all, the health insurance system must be designed to ensure that almost all the healthy people sign up for insurance, and that the healthy people don’t try to wait until they are sick to buy health insurance.   Romneycare and Obamacare attempted to accomplish this through two mechanisms:

  1. A subsidy for premiums for poor people to make them more affordable, and
  2. A “mandate” requiring that everyone have health insurance or pay a penalty.

However, as a political compromise to those that hated the idea of a mandate, the penalties were made quite small, making them only partially effective in getting enough healthy people to join the plan — ultimately causing the premium increases that people point to as evidence that Obamacare is “a disaster.”

Other than high risk pools, what is being proposed as an alternative to Obamacare?

I’ve seen nothing to suggest that there is any new innovation in health care finance that has been proposed as a superior solution.  So, what we’re likely to see is a return to health insurance designs that were used before Romneycare and Obamacare.  These mostly consist of the following:

  1. Reducing taxpayer expense by:
    • Kicking many poor people back out of government funded insurance plans (reversing federal subsidization of Medicaid expansion)
    • Reduce insurance benefits by covering fewer services and shifting more cost to patients for both government funded and private plans (reversing mandatory minimum benefits)
    • Allowing government funded insurance plans to negotiate with drug companies to demand volume discounts (against the wishes of the strong pharma lobby)
    • Increasing competition by reducing state-level insurance regulatory control to allow and facilitate building insurance plans that cut across state lines (against the states’ rights philosophy)
  2. Avoiding the insurance death spiral by:
    • Allowing insurance plans, once again, to reject sick people applying for private health insurance (reinstating pre-existing condition exclusions) and/or
    • High Risk Pools.

However, the president-elect and some members of Congress have claimed to be against some or all of these alternatives.  Except the last one — creating high risk pools. So, I think we’ll be hearing a lot more about that concept over the next few months.  Then, as people learn that high risk pools don’t do any magic and that they have a poor track record, I fear that framers of the “replacement” health insurance system will begin to acknowledge that replacement really means returning back to the other mechanisms listed above.

Read More

Three ways to keep it simple — one of which is bad

“Keep it simple, stupid.”   The “K.I.S.S.” principle.  Generally a good idea, but not always.

Types of Simplification

Consider three types of simplification:

  1. Leaning.  This is about getting rid of waste. When simplifying a design, leaning involves getting rid of unnecessary features.  When simplifying communications, leaning involves getting rid of information that is duplicative, unimportant or just decorative.  Edward Tufte, one of my heroes, is a statistician, artist and graphical designer and zen master of simplicity. He famously rails against “chart junk” and advocates for maximizing the “data – ink ratio.”
  2. Summarizing.  This is about dropping one or more layers of detail.  It is accomplished by grouping smaller details into categories or themes and dropping the details from the communication.  Summarization makes the information “blurry” but still tells the truth.  Summarization satisfies some readers.  To others, it serves as an introduction and and invitation to dive deeper.
  3. Glossing.  This requires making the information conform to a desired level of simplicity, even if it means fibbing. For example, a system may have three components that interact with one another.  Describing the interactions may be tedious to explain.  The interactions may require additional arrows on a diagram.  Glossing it involves escaping this annoying complexity by denying it.   Many companies create diagrams describing the components that make up their product or solution.  As described in Ian Gorton’s book on software architecture, such marketing diagrams are colloquially called “marketecture” diagrams.  Designers of such diagrams often take great liberties with their depiction of the solution, portraying it as being made up of components that conveniently correspond to the sources of value to the prospective customer, even when the actual technology components are organized in an entirely different way.  Glossing it can sometimes be helpful to communicate some deeper truth, almost like a metaphor or parable.  But, often times, glossing involves intentionally obfuscating the truth, making the solution appear to be better or simpler than it really is.

Einstein Simplification

Read More

First take on new CMS Comprehensive Primary Care Plus model

CMS CPC IconThis morning, I read about the recently announced next generation version of the CMS Comprehensive Primary Care model, which will require multi-payer participation and will involve up to 5K practices in 20 regions.

Sounds interesting.  I need to study it in more detail.  But based on my initial assessment:
  • I agree with the idea of pursuing payment and delivery system changes on a multi-payer basis to make it more compelling to providers.
  • I also agree with the idea of prepaying some E&M and then paying reduced FFS for E&M to cover only marginal cost of E&M office visits to make providers incentive-neutral on encounter modes.
  • But I disagree with move away from shared savings and implicit abandonment of the idea of non-governmental primary care-based organized systems of care pursuing care process innovation in favor of CMS taking over responsibility for defining a nationally-standardized multi-payer “care delivery model” and injecting it into individual primary care practices using a CMS-developed  “learning system.”
  • I also disagree with the Track 2 idea of partnering with “CMS-convened” IT vendors and contractual commitment to specific IT capabilities.  That approach basically takes MU, which was a huge distraction from real improvement, to even more obnoxious levels of micro-management.
Overall, I share the Fed’s frustration with the limited impact of previous efforts to transform primary care payment and delivery models, but I think the solution should be to define incentives which are more timely, coherent and consequential, enabled by more meaningful transparency requirements, clearer care relationships and some empowerment of primary care delivery organizations to define their own referral networks.
Read More
Page 1 of 1212345»10...Last »