Criticism of ProPublica’s Surgical Scorecard fails to consider the possibility of real, useful analytics.

From https://projects.propublica.org/surgeons/

Last week, ProPublica published a scorecard of surgical death and complication rates of more than 17,000 surgeons  for 8 elective procedures using Medicare data.  As with prior releases of health care performance metrics, the response against such “transparency” was swift and bitter.  Among those many responses is a thoughtful blog post entitled “After Transparency: Morbidity Hunter MD joins Cherry Picker MD” by Saurabh Jha, MD in The Health Care Blog.  Definitely worth your time to read.

But, although it is a clever bit of commentary, it implicitly presents a false choice between using data and not using data.

In my opinion, the decision should be conceptualized as:
  • Option 1: Not using data (and relying instead on subjective assessment or chance)
  • Option 2: Using reported metrics, interpreted by people who lack the talent and training to understand the limitations of the metrics and the methods that can address some of those limitations, and
  • Option 3: Using data, interpreted through analysis, conducted by and interpreted with the aid of people with such analytic talent and training.
By talent and training, I don’t mean technology mavens.  Keep your business intelligence professionals, data miners, “big data” experts, and most that claim the fashionable title of “data scientist.” I mean people that have training in epidemiology, biostatistics, health economics and other social science disciplines, and that have sufficient knowledge of health care.  People that can conceptualize theories of cause and effect. People that understand bias and variation.  People that can tell an interesting and actionable story supported by data, rather than just generate a “dashboard” or “score card.”  And, they must be people who have integrity and who are free of conflicts of interest that could prevent them from telling stories that are true.

Before anyone writes off option #3 as idealistic and infeasible, we should at least take the time to think through how we might make it work.

Leave a Reply

Notify me of follow-up comments via e-mail. You can also subscribe without commenting.