For almost a year, the weary public has been receiving a confusing mix of information and misinformation and witnessing an inconsistent and sometimes incoherent policy-making process regarding COVID-19 pandemic response. Leaders, journalists, neighbors, and wanna-be-expert social media influencers have all offered opinions about closing schools and businesses, mandatory mask wearing, testing, quarantine directives and other mitigation measures. Now, as a number of vaccines are achieving approval, the debates have expanded to include policies about the prioritization of different sub-populations to receive the initially-limited supply of vaccine.
In a New York Times article published on December 5, 2020, Abby Goodnough and Jan Hoffman reported on the debate about vaccine prioritization, posing a number of interesting questions, which I paraphrase:
- After we finish immunizing front line health care providers and nursing home residents, should the next-in-line be the non-confined elderly and people with serious medical conditions because of high mortality risk?
- Or should our priority be to protect “essential workers” – the 70% of the population that we all depend upon for a normal life and who face a higher infection risk with a longer life expectancy to lose?
- Or should our priority be to protect a more narrowly defined 42% of the population designated as “frontline workers”
- Also, should minorities, the poor and those with a high “social vulnerability index” be prioritized, because the burden of COVID-19 has fallen heaviest on them and they have suffered prior inequities and injustice?
These decisions are being made simultaneously in many venues, including the World Health Organization, at the federal government level (US CDC, US FDA and the Public Health Agency of Canada), and at the level of states, provinces, counties, cities and health care organizations.
Here’s my take
Rationally, we make decisions by considering the difference in outcomes across the available alternatives. We choose the optimal alternative based on the values we implicitly or explicitly assign to both the outcomes and to the process used to make the decision. We assign those values based on principles. The decision making process is always imperfect, subject to missing or inaccurate sources of information, errors in the design and execution of our analyses, cognitive biases, and sometimes influence from selfish or partisan interests.
In this context, what is meant by “values and principles?” In Canada, the federal government has established a formal COVID-19 “ethics framework” that is intended for use by policy-makers and public health professionals making COVID-19 public health decisions. It includes the following list of “values and principles:”
This is a very helpful general starting point, but in my opinion, statements of principles and values are more directly applicable to policy-makers when they are more closely aligned with the health and economic outcomes that are relevant to comparing specific policy alternatives. I offer the following incomplete list of such principles, intentionally sorted to elucidate some important distinctions that need to be made in a coherent policy making process.
Example Principles for COVID-19 Policy-Making
- There is value to avoiding a case of COVID-19.
- There is value to avoiding a severe case of COVID-19, defined as a case requiring hospitalization
- There is value to avoiding a death attributable to COVID
- There is value to avoiding a death resulting from inadequate availability of health care resources due to a case rate that exceeds health care surge capacity
- There is value to saving a life that would have been lost due to both direct and indirect impact of both COVID and pandemic remediation policies
- There is value to prioritizing the welfare of the elderly, because they are at high risk and we want to show respect, thereby giving comfort not only to the elderly but to all that will hopefully become elderly in the future
- There is value to maximizing years of life for the population (and therefore greater value to saving the life of a younger person)
- There is value to avoiding harm in the form of diminished quality of life over time (i.e. “quality-adjusted life years”), such as due to acute and residual effects of COVID
- There is value to prioritizing the welfare of front-line health care workers so as to show appreciation and avoid scarcity of health care services
- There is value to prioritizing the welfare of children because they are innocent and helpless and because they have the greatest life expectancy
- There is value to prioritizing the welfare of minorities and the poor because they are at higher risk for poor COVID outcomes
- There is value to prioritizing the welfare of minorities and the poor to compensate for prior injustices
- There is value to a lower direct economic burden of health care and public health services
- There is value to a lower indirect economic burden from pandemic remediation policies
Examples of Principles Regarding the Policy-Making Process
- There is value to a process that considers the values of many different parties
- There is value to a process that is free from obvious bias and prejudice, particularly against populations that have suffered injustices
- There is value to a process that is democratic
- There is value to a process that is simple
- There is value to a process that is fast
- There is value to a process that is agile, so as to be able to make changes in response to new information and changing needs
- There is value to a process that produces decisions that do not change too often, so as to avoid confusion, inefficient implementation and loss of trust
- There is value to a process that is based on data and valid, accepted scientific methods
- There is value to a process that is supported and controlled by scientific experts, to increase trust that the conclusions are informed and objective
- There is value to a process that is not controlled by public health experts to avoid failure to adequately consider economic outcomes and to avoid being rejected by people that distrust “elites”
- There is value to a process that is supported by trusted political, religious and cultural leaders
- There is value to a process that is transparent and explicit, where policy choices are supported by quantitative estimates of outcomes based on documented assumptions and logic
- There is value to a process that is subjective and opaque, so as to avoid the appearance of being inadequately humanistic
- There is value to a solution that is standardized across all geographic areas, to reduce confusion and increase a sense of fairness
- There is value to a solution that is tailored to meet local needs, to avoid the accusation of “one size fits all” or “Governor taking away our liberty” or “Feds violating the constitution”
As intentionally illustrated by this incomplete list, some of the principles are in conflict with one another. But that does not invalidate the principles, nor the associated values. It merely forces the use of some decision-making process that simultaneously considers multiple values, for multiple outcomes and for multiple decision alternatives.
Should we use QALY’s or Cost-Effectiveness Ratios?
When I started doing clinical policy development back in the early 1990s at Henry Ford Health System, I was greatly influenced by David Eddy, MD, PhD. Dr. Eddy wrote an amazingly useful set of articles in the Journal of the American Medical Association (JAMA) describing his conceptual framework and proposed methods for developing clinical policies (summarized in a convenient book here). Most prominently, Eddy advocated for a process in which models were used to estimate the magnitude of relevant health and economic outcomes across different decision alternatives, based on documented assumptions and logic. He called it the “Explicit Method,” and argued that such transparency is superior to the expert consensus-building processes which were also popular at the time and which Eddy disparaged as “global subjective judgement.” We brought Dr. Eddy to the Fairlane Mansion for a Henry Ford Health System leadership retreat to help convince our colleagues of the value of his approach. It was well received and we proceeded to use it for important clinical policies.
When the explicit method is used to estimate all the relevant outcomes for the decision alternatives, it is useful to decision-makers to simplify through the use of summary measures. During the 1990s, there was growing consensus in the fields of medical decision-making and health economics about the value of summarizing multiple health outcomes using a “quality-adjusted life year” (QALY) metric. In such a metric, saving the life of a younger person is implicitly asserted to be more valuable than saving the life an older person, and a year in the life of a person suffering from pain or disability is asserted to be less valuable than a year in the life of a healthy, able-bodied person (an assertion to which advocates of those with disabilities strongly object). At the same time, there was growing international consensus regarding the value of summarizing health and economic outcomes using a cost-effectiveness ratio, with units of dollars per QALY. In that era, I routinely used QALY and $/QALY metrics in clinical policy analyses, including analyses presented in scientific/professional meetings, published in scientific journals, and most importantly, used to support actual clinical policy-making at the Henry Ford Health System and its HMO, the Health Alliance Plan. They were helpful at the time.
But in the years since then, I’ve soured on the use of the QALY metric and cost-effectiveness ratios, because I’ve found that in the real world, those metrics still generate more heat than light. They make some people angry. They make people think the value judgements are robotic, obscuring what I consider to be an important boundary between the scientific process of developing outcomes estimates for policy alternatives and the non-scientific process of applying shared principles and values to decide which alternative produced the best collection of outcomes. This is particularly true in the U.S., where QALY and $/QALY have been vilified as tools for “rationing” and formally banned for use by some agencies of the US federal government (for reasons described in an earlier post.)
Proposed Six Step Approach
When called upon to facilitate the process of making consequential clinical policies, I propose a six step approach.
- Establish a clear policy-decision-making team, including members with relevant expertise and members representing important stakeholders, with a clear charter by an entity with the authority to do so
- Help the team clarify their principles and values, such as by guiding them to generate lists like the ones above
- Help the team clarify the policy alternatives under consideration
- Help the team clarify the health and economic outcomes that could be expected to be materially different across the policy alternatives. I would encourage them to state those outcomes in a quantitative way that emphasized outcomes experienced by individual people (rather than stopping at ethereal metrics such as “GDP”).
- Provide for them explicit outcomes estimates (including ranges to express uncertainty) for the alternatives based on models that have explicitly stated inputs (data and assumptions) that comprise the “best-available information”
- Help the team select the optimal alternative, applying their principles and values
I’ve previously written a blog post describing an example of my use of this approach to resolve controversy about guidelines for breast cancer screening in the 40-49 age group.
Ideally, when faced with step 6, the decision-making team will realize the difficulty of weighing different outcomes and considering different principles and values. At that point, they are likely to ask for some facilitation. Waiting for them to ask turns out to be an important practice. I would at that point, nudge them away from using QALY and $/QALY summary measures (as described above) and suggest the use of unitless point values, as with the proposed “COVID-19 Harm Index Minimization” approach I proposed in a previous blog post. I’ve successfully used such a unitless point system or “index” with senior policy-making teams in integrated delivery systems, academic medical centers and accountable care organizations in the context of provider performance incentive programs and in strategic planning retreats. I found them to be intuitive, useful and popular among policy-makers, including senior clinical and administrative leaders.
When using such an index, I would insist that the individual outcomes estimates not be buried in an appendix, but rather be prominently included in any final policy documents. The index values should be treated as mere aids to discussion and consensus-building, not as a means to make the selection “mechanical” to avoid responsibility for the decision, nor as a means to avoid transparency and accountability for the assumptions, logic and outcomes estimates that informed the decision.
Advantages and Disadvantages of This Proposed Approach
The main advantage of this approach is that it allows a team of policy-makers to break down a complex process into logical, manageable steps. It creates confidence among the policy-makers and among other stakeholders and the public. By making explicit the assumptions, outcomes estimates, principles and values, the approach establishes accountability and an opportunity to learn from mistakes as new information becomes available, leading to improvement over time.
The main disadvantage is the effort required to develop a model to estimate all the outcomes. In an upcoming blog post, I will describe the role of such models more completely, making the case that a “model in the middle” of a pandemic response system has numerous benefits beyond just supporting policy decisions.
Explicitness Requires Bravery
I’ve alluded above to another disadvantage, from the perspective of the policy-making team and the entity that chartered it. Sometimes policy-makers prefer the safety of ambiguity. If some stakeholders care most about opening the economy, and others feel strongly that it is unethical to weigh dollars against human life, it may feel safer to obfuscate the factors that influenced a policy decision. If the effectiveness of a particular prevention or treatment modality has become politicized, it may be easier to offer only a vague position. To get the benefits of explicitness — including transparency, trustworthiness, accountability, and learning – it is necessary to accept some risk and act with courage.
Not Just COVID
I’ve described this 6-step approach to clinical policy-making in the context of COVID-19 pandemic response policy, with particular focus on vaccine prioritization policy. But this approach is applicable to any consequential health policy, such as controversial practice guidelines, protocols and care maps, medical necessity and appropriateness policies, clinical credentialing policies, health plan medical and benefit policies, and quality and utilization metrics and standards. A repeatable framework, supported by the necessary analytic modelling capabilities, is therefore a core competency for any entity involved in health policy-making, including government agencies (federal, state/province, county, city), large health care provider organizations (health system, CIN, ACO) and health plans. Recognizing the general value of this core competency, it is easier to justify the investment of time and money required to get a good modelling and policy-making process up and running to address what will hopefully be an infrequent need for vaccine prioritization policies. This start-up phase can be shortened substantially by engaging others who have experience and pooling resources and sharing cost with peer entities that have the same need. Of course, we’re happy to help.