Deterministic, Threshold + Scenario Analyses & PSAs

Learning Objectives and Outline

Learning Objectives

Explain the purpose of deterministic sensitivity analysis and provide examples of one-way versus two-way analyses.
Detail the advantages/disadvantages of deterministic sensitivity analysis.
Discuss common sources of uncertainty in decision models.
Understand & characterize PSAs

Outline

One-way sensitivity analysis.
Two-way sensitivity analysis.
Limitations and extensions.
Threshold analysis
Scenario analysis
Probabilistic Sensitivity Analysis (PSAs)

Sensitivity

ICER for Prevention strategy is just above WTP threshold of $50,000/QALY.
How sensitive are these results to changes in specific model inputs?

One-way sensitivity analysis

Usually the starting point for sensitivity analyses
Sequentially testing one variable at a time (i.e., Age, BMI, QALY, other clinically important parameters), while holding everything else constant
Determining how this variation impacts the results
One-way sensitivity analyses are often presented in a tornado diagram
- Used to visually rank the different variables in order of their overall influence on the magnitude of the model outputs

Examples from publications

Rotavirus study

Other examples

Interactive Amua Session

Primary Results: Progressive Disease

Strategy	ICER
Status Quo	-
Treatment	49,513
Prevention	139,630

Treatment is cost-effective at WTP=$50,000/QALY—but barely.
How sensitive is this result to the input parameter values used?

Two-way sensitivity analysis

A way to map the interaction effects between two parameters in a decision analysis model
Varies 2 parameters at a time
Explores the robustness of results in more depth

Examples from publications

HIV prevention

Markov model examining strategies for HIV prevention among serodiscordant couples seeking conception (woman does not have HIV and male has HIV)
We know that if the male partner is consistently on medication for HIV (i.e., resulting in virologic suppression), then the risk of transmission is small regardless of the woman taking PrEP (pre-exposure prophylaxis)
And we also know that PrEP has traditionally been really costly

HIV prevention

Financial incentives for acute stroke care

Under pay for performance policies in the US, physicians or hospitals are paid more for meeting evidence-based quality targets
Study objective: Illustrate how pay-for-performance incentives can be quantitatively bounded using cost-effectiveness modeling, through the application of reimbursement to hospitals for faster time-to-tPA for acute ischemic stroke

Financial incentives for acute stroke care

When administered quickly after stroke onset (within three hours, as approved by the FDA), tPA helps to restore blood flow to brain regions affected by a stroke, thereby limiting the risk of damage and functional impairment

Financial incentives for acute stroke care

Limitations of deterministic sensitivity analyses

Caution: Limitations!

Limited by the subjectivity of the choice of parameters to analyze
That’s why we also run PSAs!, i.e., varying ALL input parameters at the same time, using priors to play a distribution around each value

Scenario analysis

Motivation

Suppose we wanted to model the impact of interventions under different assumptions – e.g., instead of assuming everyone starts the medication in pregnancy; let’s assume they were on it pre-pregnancy, which has different implications for “medication start up risks” and costs (starting the medication in pregnancy requires a lengthy hospital visit, whereas if someone is already stable on the medication pre-pregnancy, they don’t need this hospital stay).
It may be more efficient to define different scenarios rather than add additional strategies to the model structure itself.

Scenario analysis

Focuses more on model assumptions rather than parameter uncertainty

Could include separate analysis on:

Subgroups/sub-populations, including different age cohorts & risk levels

Different perspectives (societal; modified societal; etc)

Scenario analysis

> Hypothetical scenarios (“optimistic” and “conservative” scenarios; for > example, if we have little evidence of long-term survival associated > with medication X, we might have an optimistic versus conservative > scenario)

Time horizons

Examples

Threshold analysis

Answers the question: What the input parameter needs be to meet the country thresholds of:
- $50,000/QALY gained
- $100,000/QALY gained
- $150,000/QALY gained
- $200,000/QALY gained

Examples

Probabilistic Sensitivity Analysis (PSAs)

Roles of PSA’s in decision science

Quantify the degree of decision uncertainty in our model.
Is it worth pursuing additional research to reduce uncertainty?

Different Types of Uncertainty

First-order: Stochastic uncertainty from simulating individual patients.

Each patient will have a different “experience” in the model, which will create variation in model outputs (e.g., total costs, QALYS) both within the model and across different model runs.
Not relevant for Markov cohort models because those models are deterministic—they capture the average experience of a population, and do not simulate individual patient trajectories within that population.
Relevant source of uncertainty for discrete event simulation and microsimulation models.
Can often be minimized via modeling choices (i.e., simulate a lot of patients!)

Different Types of Uncertainty

First-order: Stochastic uncertainty from simulating individual patients.

Second-order: Uncertainty in the “true” value of underlying parameters.

Model parameters are often estimated with uncertainty (e.g., 95% confidence interval)
You may have assumed or calibrated parameters not rooted in a published research study; there is uncertainty involved in these processes, too!

Different Types of Uncertainty

First-order: Stochastic uncertainty from simulating individual patients.
Second-order: Uncertainty in the “true” value of underlying parameters.

Model structure uncertainty: Different choices on how to construct the structure of your model will result in different outcome estimates.

Different choices for cycle correction (e.g., half-cycle, Simpson’s 1/3, etc.)
Different choices for how to construct transition probability matrices (e.g., rate-to-probability conversion formulas vs. embedding via Matrix exponentiation)

Heterogeneity vs. Uncertainty

Source: Briggs et al., “Model Parameter Estimation and Uncertainty: A Report of the ISPOR-SMDM Modeling Good Research Practices Task Force-6”

When Does Uncertainty Matter?

When Does Uncertainty Matter?

When Does Uncertainty Matter?

When Does Uncertainty Matter?

When Does Uncertainty Matter?

In this example, model outputs are sensitive to uncertainty, but decisions are not.

When Does Uncertainty Matter?

In this example, model outputs are sensitive to uncertainty, but decisions are not.

When Does Uncertainty Matter?

Both model outputs and decisions are sensitive to uncertainty.

When Does Uncertainty Matter?

Both model outputs and decisions are sensitive to uncertainty.

Markov Cohort Models

DES and Microsimulation

Probabilistic Sensitivity Analysis

2. How Do We Conduct Probabilistic Sensitivity Analyses?

Idea

Run the model many times, each time randomly drawing a given parameter value from its uncertainty distribution.
Collect the parameter values and model outputs (e.g., total costs and QALYs) in a probabilistic sensitivity analysis (PSA) dataset.
Analyze the PSA results to construct uncertainty estimates for ICERs, NMB/NHB, etc.
PSA results can also be used for value of information that quantify decision uncertainty and the value of future research to reduce uncertainty.

How Do We Draw PSA Values?

Central limit tells us that distribution for many estimated parameters is normal (when independent random variables are added together, their properly normalized sum tends toward a normal distribution, even if the original variables are not normally distributed)
However, we often do not rely on a single parameter estimate, but rather on a range of estimates from the literature.
In any PSA, we want to specify parameter uncertainty in such a way as to capture the overall level of uncertainty in model parameters.

How Do We Draw PSA Values?

Parameter Type	Distribution
Probability	beta
Rate	gamma
Utility weight	beta
Right skew (e.g., cost)	gamma, lognormal
Relative risks or hazard ratios	lognormal
Odds Ratio	logistic

For example,

Why do probabilities follow a beta distribution?
- The beta distribution is bounded between 0 and 1, exactly matching the range of probability values.
- Beta(α+successes, β+failures); 20 successes/100 trials = Beta(20, 80)

How Do I Draw Values

https://yuhanxuan.shinyapps.io/shiny4dist/

How Do I Draw Values

97.5% - 2.5% = 95%

So 95% of the probability mass falls between 0.6 and 0.8

How many do I draw?

Until results (ICER/NHB/etc.) stabilize – minimal change with additional iterations.
Start with 1,000 – 5,000 – 10,000

Constructing a PSA Sample

For a given iteration j

Draw separate PSA values from the uncertainty distributions in your model.
Run the model and calculate model outputs (e.g., total costs and QALYs for each strategy).
Record the PSA parameter values and the outcome results in a table.
Repeat 1-3 many times.

Common PSA Distributions in Amua

Parameter Type	Distribution	Amua
Probability	beta	Beta(shape1,shape2,~)
Rate	gamma	Gamma(shape, scale, ~)
Utility weight	beta	Beta(shape1,shape2,~)
Right skew (e.g., cost)	gamma, lognormal	LogNorm(shape,scale,~)
Relative risks or hazard ratios	lognormal	LogNorm(shape,scale,~)
Odds Ratio	logistic	Logistic(location, scale)

Exmample: Uncertainty in Utility Weight

Base case value: 0.95
Sample from Beta(95,5,~)
Alternatively, sample from Beta(950,50,~)
First sample = 100 (reflecting greater uncertainty); Second sample = 1,000 observations

Exmample: Uncertainty in Utility Weight

3. How Do We Summarize PSA Results?

How Do We Summarize PSA Results?

Plot costs and QALYs of each iteration to show degree of variation in estimates.
Figure plots values at each iteration, the average across 1,000 iterations (large points) and ellipses that capture ~95% of points.

Cost Effectiveness Acceptability Curves

CEACs summarize the degree of uncertainty as captured by our PSA.
CEAC represents the (Bayesian) probability of each option being cost-effective at different levels of the cost-effectiveness threshold \lambda.

Source

Cost Effectiveness Acceptability Curves

Constructing the CEAC

Define a WTP value.

\lambda = 50000

Constructing the CEAC

Use the PSA sample to calculate the Net Monetary Benefit (NMB) and/or the Net Health Benefit (NHB) of each strategy.

Net Monetary Benefit

TOTQALY * \lambda - TOTCOST

Net Health Benefit

TOTQALY - \frac{TOTCOST}{\lambda}

PSA Sample

PSA_ID	totcost_trtA	totcost_trtB	totcost_trtC	totcost_trtD	totcost_trtE	totqaly_trtA	totqaly_trtB	totqaly_trtC	totqaly_trtD	totqaly_trtE
1	19619.47	25588.07	37065.38	23997.77	40797.18	18.5372	18.6151	18.7262	18.6529	18.6529
2	11776.93	17379.42	36873.02	19453.59	45567.65	17.0105	17.1083	17.1966	17.1353	17.1353
3	13292.49	19268.95	33789.89	19886.42	41933.20	17.2514	17.3591	17.5108	17.4620	17.4620
4	14652.37	19102.10	25153.63	19245.88	33977.42	16.1276	16.1314	16.3777	16.2537	16.2537
5	13286.67	15912.61	26997.83	18687.76	39152.52	15.9383	16.0365	16.1411	16.0760	16.0760
6	14958.85	17505.56	32929.46	20602.67	49327.30	16.0158	16.0886	16.2868	16.1828	16.1828

Net Monetary Benefit

Note: Values shown are for a single value of lambda (50,000/QALY)
PSA_ID	NMB_A	NMB_B	NMB_C	NMB_D	NMB_E
1	907239	905168	899245	908648	891849
2	838749	838035	822957	837314	811200
3	849279	848687	841748	853212	831165
4	791727	787466	793733	793440	778708
5	783630	785914	780057	785112	764647
6	785829	786922	781411	788536	759811

Identify the Optimal Strategy

For each iteration, determine which strategy maximizes NMB/NHB.

This is the optimal strategy for a given \lambda value.

Identify the Optimal Strategy

PSA_ID	NMB_A	NMB_B	NMB_C	NMB_D	NMB_E
1	907239	905168	899245	908648	891849
2	838749	838035	822957	837314	811200
3	849279	848687	841748	853212	831165
4	791727	787466	793733	793440	778708
5	783630	785914	780057	785112	764647
6	785829	786922	781411	788536	759811

Identify the Optimal Strategy

PSA_ID	MAX_IS_A	MAX_IS_B	MAX_IS_C	MAX_IS_D
1	0	0	0	1
2	1	0	0	0
3	0	0	0	1
4	0	0	1	0
5	0	1	0	0
6	0	0	0	1

How Often is the Stratgy the Optimal?

The average of this binary indicator across all PSA model runs is the fraction of the time each strategy is optimal for a given value of \lambda.

How Often is the Strategy the Optimal?

lambda	MAX_IS_A	MAX_IS_B	MAX_IS_C	MAX_IS_D	MAX_IS_E
50000	0.1667	0.1667	0.1667	0.5	0

How Often is the Strategy the Optimal?

Now repeat this exercise across a range of values for \lambda.

How Often is the Strategy the Optimal?

lambda	MAX_IS_A	MAX_IS_B	MAX_IS_C	MAX_IS_D
20000	1.0000	0.0000	0.0000	0.0000
40000	0.1667	0.1667	0.0000	0.6667
50000	0.1667	0.1667	0.1667	0.5000
60000	0.0000	0.3333	0.1667	0.5000
80000	0.0000	0.0000	0.1667	0.8333
100000	0.0000	0.0000	0.1667	0.8333
120000	0.0000	0.0000	0.3333	0.6667
140000	0.0000	0.0000	0.5000	0.5000
160000	0.0000	0.0000	0.5000	0.5000
180000	0.0000	0.0000	0.6667	0.3333
200000	0.0000	0.0000	0.6667	0.3333

How Often is the Strategy the Optimal?

We can now plot these data:
- x-axis: \lambda.
- y-axis: Fraction/percent of the time each strategy is optimal.
This is the Cost-Effectiveness Acceptability Curve