A Sufficient Statistics Approach to Ex Ante Health Policy Evaluation

John A. Graves

Models projecting the impact of reforms to health insurance programs and markets play an important role in shaping U.S. health policy. In 2017, for example, Congressional attempts to repeal and replace the 2010 Affordable Care Act (ACA) were hampered by public outcry after the Congressional Budget Office (CBO) projected that upwards of 23 million people would become uninsured. The twists and turns of earlier debates over the ACA–and before it, the Clinton health plan–also were shaped by modelers’ assessments of how reform would impact insurance coverage, premiums, health care spending, and government costs.

Microsimulation models used by the CBO and by others to produce these estimates draw on economic theory and on a large and growing literature evaluating past state and federal reform efforts. Yet while models derive inputs from this shared evidence base, the evidence is uncertain and not in uniform agreement. Models also differ in their structure, underlying data sources and assumptions. It should come as no surprise, then, that models often produce widely varying projections of the same reform proposal.

This current state of affairs has subjected microsimulation models to criticism over their “black box” like qualities and their tendency to produce estimates with a limited accompanying sense of uncertainty or sensitivity to alternative parameter values and assumptions. Moreover, modelers have understandably but unfortunately shied away from producing comparative assessments of overall welfare impact. Existing models typically produce an array of intermediary point estimates on welfare-relevant outcomes (e.g., changes in coverage, premiums, spending and government costs) and leave it to policymakers to weigh those factors when comparing policy choices.

This approach to health policy modeling has a number of important shortcomings. First, despite modelers’ attempts to caveat the high degree of uncertainty in their estimates, modeled projections are often afforded a false sense of precision in high-stakes policy debates. This results in decisions being made in spite of a high degree of uncertainty surrounding the budgetary, health and coverage impact of proposed reforms. Second, the “black box”-like opacity of microsimulation models makes it difficult for researchers to know whether and how their work can inform modeling efforts. Finally, the development, execution, and maintenance costs of microsimulation models are considerable. Combined, these factors contribute to high barriers to understanding and a muddled sense of how the health economic and policy research enterprise could be further refined to improve policy decision making.

This study outlines an approach to ex ante policy evaluation that addresses many of the above shortcomings. First, I outline a generalized discrete time modeling framework for assessing the cost, coverage and welfare impact of health reform policies. This framework has roots in health economic modeling methods used worldwide for health technology assessment, and in the “sufficient statistics” approach to welfare evaluation developed in public finance. I demonstrate that this modeling framework can encompass many existing approaches to health policy microsimulation, but also facilitates simple yet powerful counterfactual policy aassessments based on reduced form estimates. That is, the framework provides researchers with a simple tool to investigate the coverage and cost impacts of reform alternatives without the need for a detailed individual-level microsimulation model. As a proof of concept, I demonstrate how differences-in-differences evidence on the impact of Medicaid expansion on coverage, combined with regression-discontinuity estimates on willingness to pay for subsidized health insurance (Finkelstein, Hendren and Shepard 2019) can be harnessed to model the impact of further expansion of coverage via public programs versus via increased subsidies for private coverage.

Second, within this framework I tie together diverse approaches to assessing uncertainty and the welfare impacts of policy. Specifically, I draw linkages between standard wefare impact measures used in health technology assessment (e.g., net health benefit and net monetary benefit) and the marginal value of public funds (MVPFs), a summary measure of the costs and benefits of public policies (Hendren 2017). This linkage allows for a systematic approach to understanding parameter and modeling uncertainty based on probalistic sensitivity anlayses (PSAs) and value of information (VOI) methods.

Intuitively, VOI methods quantify the opportunity cost of decision making under uncertainty. At a given policy efficiency or willingness-to-pay threshold (e.g., a MVPF value of 0.8, above which a policy might be implemented but below which it may not), uncertaninty may or may not affect optimal decision making. If policy decisions based on comparative assessments of MVPF are insensitive to varying assumptions or to estimation uncertainty in model parameters, then the value of information on these parameters is low – i.e., it is not worth pursuing additional research to reduce uncertainty. If policy decisions are sensitive to this uncertainty, however, then these methods provide a guidepost for priortizing and refining future research. The value of information on individual parameters in a model can be estimated, and these