“You can’t manage what you can’t measure.” 

While this statement by business sociologist Peter Drucker is certainly true, in some scientific or regulatory instances, it is impossible to measure what you seek to manage. Proxies are measurable and, when strongly correlated with our goal, useful. Proxies can simplify complex problems, allow for performance tracking, and deliver faster feedback. But poorly chosen proxies derail the very goals they’re meant to support.

A Poorly Chosen Proxy

In the opioid wars, Morphine Milligram Equivalents (MMEs) became the proxy of choice. Easily measured, easy to regulate, and track—MMEs appear whenever a prescription is filled. So, it made sense, on the surface, to use MMEs to simplify the goal of reducing opioid prescribing.

Unfortunately, the goal of reducing opioid prescribing is, in fact, a proxy for an even more difficult goal: the reduction of prescription-related substance use disorder. 

MMEs, now a proxy for a proxy—lose their strength of correlation and usefulness, succumbing to the proxy’s pitfalls: misalignment, unintended consequences, and oversimplification. More specifically, the use of MMEs in the war on opioid addiction and overdoses has resulted in unintended harm to individuals afflicted with chronic pain, oversimplifying the clinical nuances of addiction allowing bureaucrats to count pills instead of understanding patients. 

It is a painful demonstration of Goodhart’s Law, 

“When a measure becomes a target, it ceases to be a good measure.”

A new study in JAMA Network Open drives home the disconnect between proxy metrics and meaningful outcomes.

The Law That Capped Judgment

In 2016, New York implemented a limit to the initial prescription of opioids for acute pain to 7 days, stripping any discretion, i.e., clinical judgment, by physicians. The New York law, known as Section 3331, joined 39 other states in state opioid cap laws (SOCLs) aimed at: 

• Reducing the risk of addiction by limiting opioid use
• Reducing unused opioids that could be “diverted” for misuse

The research involving Medicare beneficiaries looked at prescribing behavior before and after implementing New York’s SOCL Section 3331 for individuals undergoing total joint replacement (TJR). As they write, 

“Adequate post-TJR pain control is a key marker of successful surgery, and inadequate control is associated with impaired recovery, resulting in delayed or unmet physical therapy milestones, increased health services use, and reduced quality of life.”

California, which had the highest number of Medicare TJRs and no SOCL, served as a control. The primary endpoint was the total MMEs filled after discharge. They considered the first 7 days after surgery and 30- and 90-day intervals consistent with the law and “key clinical practice milestones.” [1]

The patient cohort included 85,000 Medicare beneficiaries undergoing 93,000 total joint replacements, roughly a third being treated before New York’s SOCL implementation. The mean age was 73; 60% were women, and 89% were White.

The Data Speaks

• As the graph demonstrates, both states had similar significant declines in opioid prescribing throughout the 90-day post-operative period. The decrease in NY was 47%, and in California, 43%.
• In that initial 7-day period, the number of prescriptions filled and the quantity of opioids prescribed were reduced more so in New York than California.
• While the likelihood of patients filling at least one opioid prescription within 7 days after surgery dropped in California (−7.76 %) and New York (−5.27 %), paradoxically, Section 3331 resulted in a relative increase in opioid fills compared to California and more fills during the subsequent 31 to 90-day recovery period.

“Overall, our findings suggest that Section 3331 may have achieved its intended objective of reducing opioid prescribing for acute pain in the short-term 7-day post-TJR period.” [emphasis added]

However, that’s where the success story ends. The unstated but actual goal, reducing prolonged opioid exposure, addiction risk, and leftover meds for diversion—remained unmeasured and possibly unmet. Exposure was longer in New York when those later refills are counted. And we have no data on addiction, diversion, or patient satisfaction.

What’s Missing from the Metrics

Most critically, the focus on the proxy, MMEs prescribed, left “meaningful changes in the pain needs of patients, morbidity, or mortality for future research.” The increase in refills after the initial 7 days is ambiguous. Do they suggest the persistence of pain requiring longer treatment or reflect misuse or diversion? Refills tell us nothing about “risky” behavior. 

We don’t know—because the measure we chose can’t tell us.

Goodhart’s Law In The Real World

Goodhart’s Law manifests itself in many ways, as this study demonstrates. MMEs are too simple a measure of the complexity of real-world prescribing that is a confluence of practitioner beliefs, institutional protocols, and policy environment. California’s trends in prescribing behavior were nearly identical to New York’s, without the benefit or limitations of SOCL. MMEs fail because they are correlates, not causes. MMEs are a rough measure of opioid exposure, at best, a risk factor among many for substance use disorder. 

They are neither a measure of pain management nor relief. MMEs also allow for gaming the system. Prescribers might under-treat pain out of fear—or overprescribe to avoid complaints—neither of which addresses patient well-being. Proxies distort behavior and, as Goodhart suggests, undermine intended outcomes. 

Recognizing these pitfalls, one might hope for mitigation strategies. That could include:

• Thoughtful consideration of downstream unintended consequences,
• Choosing metrics more closely aligned with desired outcomes
• Using balanced targets, e.g., MME, pain scores, and diagnosis of substance use disorders, to expose the real trade-offs and discourage gaming. 

Unfortunately, legislation is drawn in blacks and whites, and rarely are clinicians invited to the table. Researchers are equally culpable for using available data rather than more costly, in time and money,  bespoke data. These researchers write that “much work remains to restrict opioid prescribing in the later post-TJR period, suggesting there is an opportunity to refine Section 3331 to reduce perioperative prescribing.” This refines the proxy and does little for the clinical outcomes of safe pain management and lowered addiction risk. 

Proxy metrics like MMEs were never designed to bear the weight of legislation, yet we keep piling on. In chasing numbers, we lose sight of patients—of pain that goes untreated, of addiction risks that go unmeasured, of policies that solve for optics instead of outcomes. 

Real patients suffer when we treat metrics as medicine.

If we want better results, we need better questions, metrics, and a better understanding of what and who we’re actually trying to heal.

[1] Secondary endpoints included MMEs per prescription and day, number of refills and the likelihood of at least one opioid fill, the likelihood of an opioid prescription longer than 7 days, and total opioids prescribed.