Welfare Analysis Meets Decision Theory

Part One: Net Welfare Benefit and the Value of Information

Part one in a three-part series on how the tools of welfare analysis and decision science can be used to inform the direction, scope, and design of research and decision-making.

Introduction

Suppose a policymaker is considering strategies to reduce poverty. For the sake of simplicity, let’s assume the policymaker is weighing the tradeoffs of creating or expanding an in-kind benefit transfer, such as a job training program.

Implementing the program has a measurable impact on welfare-relevant outcomes (e.g., education, lifetime earnings, health, etc.). However, the program also imposes costs on society. These costs reflect both the budgetary cost (i.e., the total sum of program expenditures), as well as other changes in behavior, expenditures, or tax revenues that either bolster or drag on social welfare.

Before we proceed, it is useful to formalize the above and define summary measures of policy benefits \(W(\mathbf{\pi},\alpha)\) and costs \(C(\mathbf{\pi},\alpha)\). The vector \(\mathbf{\pi}\) captures welfare-relevant parameters such as the average willingness-to-pay for the policy, program costs, and any fiscal externalities (FEs) that occur as a consequence of the policy.¹ We define and measure welfare benefits and costs for each of \(D\) total policy strategies \(\boldsymbol{\alpha} = (\alpha_1, \cdots , \alpha_D)\).

¹ For example, these fiscal externalities might include changes to government tax revenues stemming from changes in participation in the labor force and/or other social programs.

In a series of influential papers, Hendren and colleagues define the Marginal Value of Public Funds (MVPF) as the ratio of benefits to costs (Finkelstein and Hendren 2020; Hendren and Sprung-Keyser 2022, 2022; Hendren 2016):

\[ MVPF(\mathbf{\pi},\alpha) = \frac{W(\mathbf{\pi},\alpha)}{C(\mathbf{\pi},\alpha)} \tag{1}\]

The MVPF measures the marginal value of an additional dollar spent on a policy. That is, the MVPF quantifies how the welfare benefits accrued by implementing a policy compare to the costs of adopting it.

While we will leave the theoretical details to the aforementioned citations, we can summarize the MVPF of an in-kind benefit transfer as:

\[ MVPF^{\text{inkind}}(\mathbf{\pi},\alpha) = \frac{W(\mathbf{\pi},\alpha)}{1+FE(\mathbf{\pi},\alpha)} \tag{2}\] where \(FE(\mathbf{\pi},\alpha)\) is the fiscal externality associated with the policy, and \(W(\mathbf{\pi},\alpha)\) is the willingness to pay among infra-marginal recipients of the program.

In this example, we are comparing a single policy strategy (in-kind benefit program) to an alternative strategy of doing nothing. However, in principle we could think of alternative (competing) strategies to achieve the same objective—each with its own welfare cost and benefit estimates—that might be under consideration.

The State of Current Knowledge

Suppose that previous research has quantified the welfare benefits and costs—such that the hypothetical policymaker has at her disposal some information that can guide her decision on which policy (if any) to pursue.

The state of current information is summarized in Table 1.

Table 1: Summary of Current Knowledge

	MVPF [95% interval]	Benefits [95% Interval]	Costs [95% Interval]
inkind	0.89 [0.73,1.08]	1.21 [0.99,1.46]	1.35 [0.31,0.39]

MVPF as a Decision Tool

Table 1 highlights that the policymaker’s decision is fraught with uncertainty. Based on current information, the estimated MVPF of the in-kind benefit is 0.894. However, the 95% interval shows that current knowledge is consistent with the policy more than paying for itself (i.e., an MVPF above 1), and with the policy resulting in some redistribution (i.e., an MVPF below 1).

In short, based on current evidence, it is hard to say for sure whether the policy should be adopted.

Our policymaker’s decision problem actually goes one step further. The MVPF summarizes the private benefits that accrue from the in-kind benefit transfer; it may be that social preferences are such that, from a societal perspective, a policy with an MVPF below 1.0 is not worth pursuing.

We can formalize this idea by defining an additional parameter \(\lambda\) summarizing societal willigness-to-pay (WTP). This threshold value could simply be based on an MVPF of one (i.e., the benefit must be at least as great as the cost). Or, if society values some redistributive consequence of the policy, \(\lambda\) could be set based on a value less than one.²

² For example, Finkelstein, Hendren, and Shepard (2019) make comparative assessments of health insurance subsidization policies by specifying a social welfare function over Constant Relative Risk Aversion (CRRA) utility and a defined coefficient of risk aversion (\(\sigma = 3\)). This results in \(\lambda = 0.2\). But researchers do not necessarily have to specify the structure of the social welfare function to define a decision-making benchmark. A value tied to an existing policy with strong social support could also suffice. For instance, Finkelstein, Hendren, and Shepard (2017) also consider a benchmark (\(\lambda = 0.88\)) based on the MVPF of the Earned Income Tax Credit (EITC)—a popular means-tested cash transfer program. Finally, Hendren (2020) argues for the use of efficient welfare weights that project the welfare costs and benefits of any specific policy into the MVPF of a tax transfer between the affected populations.

³ For the sake of this example, welfare benefits are drawn from a lognormal distribution with mean \(\log(1.2)\) and standard deviation \(0.1\). Welfare costs are based on Equation 2 and a fiscal externality that is normally distributed with mean \(0.35\) and standard deviation \(0.2\).

Figure 1 visualizes this uncertainty for 1,000 draws from the uncertainty distributions of \(W^{\text{inkind}}(\mathbf{\pi},\alpha)\) and \(C^{\text{inkind}}(\mathbf{\pi},\alpha)\) in Table 1.³ In addition, for three hypothetical values of \(\lambda\), each point is shaded based on whether the implied MVPF is above or below \(\lambda\).

The dotted lines show a ray from the origin with slope \(1/\lambda\); values that fall under and to the right of the line are those where the MVPF\(\geq \lambda\) and those that fall up and to the left of the line are those where MVPF\(<\lambda\). Finally, the green diamonds plot the average welfare benefit and cost values as summarized in Table 1.

(a) lambda = 1.0

(b) lambda = 0.90

(c) lambda = 0.5

Figure 1: Welfare Cost and Benefits Overlaid With Decision Rules Based on \(\lambda\)

Figure 1 (a) highlights that when \(\lambda=1\), the MVPF of the in-kind benefit strategy rarely exceeds the societal willingness-to-pay threshold. In other words, there are few points in the joint uncertainty distribution where the value falls below and to the right of the dotted line. Thus, there is some—but not much—uncertainty in the decision to not implement the policy under this decision criterion.

By comparison, Figure 1 (c) shows that when \(\lambda=0.5\), there is no decision uncertainty; our policymaker would conclude that the the in-kind benefit should be adopted—and this decision is consistent across all values in the uncertainty ranges.

The Opportunity Cost of Imperfect Information

The above discussion highlights that depending on social preferences, policy decisions based on current information may carry an opportunity cost of making the wrong decision. This is most evident in Figure 1 (b), where based on the average welfare cost estimates in Table 1 (plotted as a green dot in Figure 1 (b)), the policymaker might elect to pursue an in-kind benefit program. In other words, the implied average MPVF basd on these values is 0.894. If \(\lambda = 0.9\), this value is below the WTP threshold—so the policy would appear to pass a societal cost-benefit test.

However, Figure 1 (b) also makes clear the available information is also consistent with the “true” MVPF falling above \(\lambda\). That is, roughly half of the points are shaded gold (i.e., the policy does not pass our defined societal cost-benefit test) and half are shaded black (i.e., MVPF\(<\lambda\)). If the true MVPF is below \(\lambda\), then society would incur a net welfare loss from implementing the policy.

Contrast this with the scenario in Figure 1 (c), which is based on \(\lambda = 0.5\). In that scenario, there is uncertainty in the underlying welfare estimates but no uncertainty in policy adoption decisions. The same decision (to implement the in-kind benefit program) occurs across the entire uncertainty range, so there is no opportunity cost to making the “wrong” decision.

Net Welfare Benefit

Let’s now formalize the above observations by defining the net welfare benefit as the benefits of a given policy minus \(\lambda\) multiplied by the costs:

\[ NWB(\boldsymbol{\pi},\alpha,\lambda) = W(\boldsymbol{\pi},\alpha) - \lambda \cdot C(\boldsymbol{\pi},\alpha) \tag{3}\]

Intuitively, policies where \(NWB \geq 0\) indicate situations where the MVPF is equal to or greater than \(\lambda\). Put simply, when the NWB is positive, the policy passes the societal cost-benefit test; when it is negative, it does not.

With the NWB defined, it is useful to think about uncertainty along two distinct but related dimensions:

1. Variation in the NWB that derives from sampling/structural/modeling uncertainty in estimates of \(W^{\text{inkind}}(\mathbf{\pi},\alpha)\) and \(C^{\text{inkind}}(\mathbf{\pi},\alpha)\).

2. Variation in optimal decisions, which occurs when variation from (1) causes frequent sign changes in the NWB.

In other words, depending on the value of \(\lambda\), variation in welfare outcomes (costs and benefits) does not necessarily imply variation in optimal decisions; it may be \(\lambda\) is sufficiently low (or high) that policy decisions do not vary across the uncertainty distribution.⁴

⁴ This is the scenario depicted in Figure 1 (c).

From Net Welfare Benefit to Value of Information

The idea that decisions based on current information may carry a welfare-valued opportunity cost underlies decision theoretic concepts of the value of information (VOI).

Before we define various VOI concepts and measures, let’s lay out the practical questions a VOI-based approach to welfare analysis based on the MVPF can answer:

1. Which strategies are cost-effective given social preferences and based on current evidence? To answer this question, we can assess whether current knowledge–cast against a summary measure of social preferences such as \(\lambda\)—is sufficient to confidently make policy decisions. Because we may not want to make a stance on the particular value of \(\lambda\), we can vary these assessments over a plausible range of values and estimate the expected welfare loss of making the wrong decision at a given value of \(\lambda\).

2. Should we invest more resources to reduce uncertainty in our decisions? If so, it will be important to isolate which sources of uncertainty we should focus on. Furthermore, we want to think about the most efficient way to sample the population to improve our knowledge on these dimensions.

The Welfare Loss from Imperfect Information

To answer the first question, we need a summary measure of the expected welfare loss from making the wrong decision.

Define the optimal strategy as

\[ \alpha^* =\arg\max E_{\boldsymbol{\pi}} \big [ NWB(\boldsymbol{\alpha},\boldsymbol{\pi},\lambda) \big ] \] so that \(NWB(\alpha^*,\boldsymbol{\pi},\lambda)\) is the net welfare benefit evaluated at the optimal strategy, i.e.,

\[ NWB(\alpha^*,\boldsymbol{\pi},\lambda) = \max_{\boldsymbol{\alpha}} E_{\boldsymbol{\pi}} \big [ NWB(\boldsymbol{\alpha},\boldsymbol{\pi},\lambda) \big ] \]

Next, define a welfare loss function

\[ L_{\alpha} = \max_{\boldsymbol{\alpha}} NWB(\boldsymbol{\alpha},\boldsymbol{\pi},\lambda) - NWB(\boldsymbol{\alpha},\boldsymbol{\pi},\lambda) \tag{4}\]

The loss function evaluated at \(\alpha^*\) is

\[ L_{\alpha^*} = \max_{\boldsymbol{\alpha}} NWB(\boldsymbol{\alpha},\boldsymbol{\pi},\lambda) - NWB(\alpha^*,\boldsymbol{\pi},\lambda) \tag{5}\]

And its expected value is given by

\[ E_{\boldsymbol{\pi}} \big [ L_{\alpha^*} \big ] = E_{\boldsymbol{\pi}} \big [ \max_{\boldsymbol{\alpha}} NWB(\boldsymbol{\alpha},\boldsymbol{\pi},\lambda) \big ] - NWB(\alpha^*,\boldsymbol{\pi},\lambda) \tag{6}\]

The expected welfare loss in Equation 6 is also known as the expected value of perfect information (Jackson et al. 2022; Wilson 2015):

\[ E_{\boldsymbol{\pi}} \big [ L_{\alpha^*} \big ] = EVPI(\boldsymbol{\alpha},\boldsymbol{\pi},\lambda) \]

Estimating the Expected Welfare Loss

The code below calcualtes the EVPI for the MVPF of the in-kind benefit transfer above, for a particular value of \(\lambda\):

set.seed(23)
n <- 1e5   # Number of uncertainty distribution draws
w <- rlnorm(n, log(1.2),0.1)   # welfare benefit is lognormal(ln(1.2),0.1)
fe <- rnorm(n, 0.35,.02)  # fiscal externality is norm(0.35,0.02)
c <- 1+fe  # cost is denominator of MVPF

lambda <- 0.9  # set a societal WTP value
nwb <- w - lambda * c  # calculate the net welfare benefit
evpi <-mean(pmax(0,nwb))-max(0,mean(nwb))  # expected value of perfect information
evpi

[1] 0.04441101

The estimated EVPI (0.044) is non-zero, indicating that there is overall value in reducing uncertainty in the policymaker’s decision to implement the in-kind benefit transfer. However, note that the EVPI varies over different values of \(\lambda\). If \(\lambda\) were instead 0.5, we would make the same decision (to implement the policy) across the entire uncertainty range—so there is no value in obtaining new information on uncertain inputs into the MVPF estimate. This can be seen in the code below:

lambda <- 0.5 # set a societal WTP value
nwb_alt <- w - lambda * c  # calculate the net welfare benefit
evpi_alt <-mean(pmax(0,nwb_alt))-max(0,mean(nwb_alt))  # expected value of perfect information
evpi_alt

[1] 0

Figure 2 is based on repeating the above exercise across a variety of \(\lambda\) values (x-axis) and plotting the resulting EVPI estimate for each (y-axis):

Figure 2: Expected Value of Reducing Decision Uncertainty Through Future Research, by Societal Willingness-to-Pay Value (lambda)

Summary and Next Steps

Figure 2 tells us that based on existing evidence, we have an answer to our first question: if the societal willigness-to-pay value (\(\lambda\)) is in the neighborhood of 0.75-1.15, then there is high information value in future research to reduce uncertainty in \(W(\mathbf{\pi},\alpha)\) and \(C(\mathbf{\pi},\alpha)\).

With this information in mind—and so long as pursuing more information has value—we can move on to the second question and start to ask (a) which specific sources of uncertainty (i.e., individual parameters or sets of parmeters in \(W(\mathbf{\pi},\alpha)\) and \(C(\mathbf{\pi},\alpha)\)) drive the overall EVPI estimate? And how can we prospectively design a clinical trial or randomized evaluation in such a way as to most efficiently reduce information uncertainty to a point where a better decision can be made? Answering these questions is the focus of part two in this series.

References

Finkelstein, Amy, and Nathaniel Hendren. 2020. “Welfare Analysis Meets Causal Inference.” Journal of Economic Perspectives 34 (4): 146–67.

Finkelstein, Amy, Nathaniel Hendren, and Mark Shepard. 2017. “Subsidizing Health Insurance for Low-Income Adults: Evidence from Massachusetts.” National Bureau of Economic Research Working Paper Series, no. w23668 (August).

———. 2019. “Subsidizing Health Insurance for Low-Income Adults: Evidence from Massachusetts.” American Economic Review 109 (4): 1530–67.

Hendren, Nathaniel. 2016. “The Policy Elasticity.” Tax Policy and the Economy 30 (1): 51–89.

———. 2020. “Measuring Economic Efficiency Using Inverse-Optimum Weights.” Journal of Public Economics 187 (July): 104198. https://doi.org/10.1016/j.jpubeco.2020.104198.

Hendren, Nathaniel, and Ben Sprung-Keyser. 2022. “The Case for Using the MVPF in Empirical Welfare Analysis.” National Bureau of Economic Research.

Jackson, Christopher H., Gianluca Baio, Anna Heath, Mark Strong, Nicky J. Welton, and Edward C. F. Wilson. 2022. “Value of Information Analysis in Models to Inform Health Policy.” Annual Review of Statistics and Its Application 9 (March): 95–118. https://doi.org/10.1146/annurev-statistics-040120-010730.

Wilson, Edward C. F. 2015. “A Practical Guide to Value of Information Analysis.” PharmacoEconomics 33 (2): 105–21. https://doi.org/10.1007/s40273-014-0219-x.