Fourteen DID estimators walk into a bar \(\tau\)

HPOL8539

John Graves

September 21, 2022

Policy Evaluation Methods Until 2018

Policy Evaluation: 2018-2021

The rest of us:

Policy Evaluation: 2021

Econometricians:

Policy Evaluation: 2021

The rest of us:

After Today

Today’s Plan

General principles for DID designs with staggered adoption.
Data wrangling for DID designs with staggered adoption.
learnr exercise for eight DID estimators.

We’re going to ignore standard TWFE and TWFE Event Study estimators, given all the problems.
We’ll also focus on dynamic treatment effect estimates, though some estimators also yield static estimates.
We’ll also skip over extended POLS and extended Mundlak (random effects), though implementing them is straightforward (see last week’s exercises).

We’ll come back to “conditional DID” in a future class (i.e., the design variables will not include covariates)

General principles for DID designs with staggered adoption.

Data selection and setup is key.

Important

If all groups are eventually treated, you’ll need to “trim” the data of all observations on or after the last treated cohort date.

General principles for DID designs with staggered adoption.

Data selection and setup is key.

Important

Define a treatment cohort variable with values set to the first year of treatment for the unit. Never treated units should be assigned a value of 0 for this cohort variable.

General principles for DID designs with staggered adoption.

Data selection and setup is key.

Important

When constructing a relative-time variable (i.e., time - cohort_tx_time), never treated units should get a value of -Inf.

General principles for DID designs with staggered adoption.

Once you do all the above, you are teed up to fit any of the NextGen DID estimators.

Wooldridge (2021): Extended TWFE, POLS and Mundlak (Random Effects) regression.
Callaway and Sant’Anna (2021) : Group-time ATTs
Borusyak, Jaravel, and Spiess (2021) : Imputation-based DID
Gardner (2021) : Two-stage DID
De Chaisemartin and d’Haultfoeuille (2020) : Generalized “fix” for two-way fixed effects.
Athey et al. (2021): Matrix completion methods
Cengiz et al. (2019): Stacked regression

NextGen DID Estimators

I won’t go through details today, but you should familiarize yourself with whatever method you choose.

NextGen DID Estimators

Standard practice used to be just (naively) fitting a TWFE-based regression.
- This is fine in the simple 2x2 DID case.
- It breaks down in the presense of staggered treatment adoption and heterogeneous treatment effects.
Wooldridge (2021) demonstrates that a carefully specified regression can give us a line of sight through the danger zones.

NextGen DID Estimators

In general, the other new estimators are explicit about making “clean” comparisons and avoiding “forbidden” comparisons.
General Idea #1: estimate “clean” treatment effects for various treatment cohorts, and then aggregate them together into relative time (or calendar time).
General Idea #2: Imputation of missing potential outcomes.

NextGen DID Estimators

Benefit of new estimators: in principle, you’re being transparent about your comparisons and the “weights” you use to aggregate across groups.
Double-edged sword: many of the estimators have R & Stata commands, but they can be a bit of a “black box.”
Roth et al. (2022) provides a very nice, approchable summary of the recent literature. Link to paper.

Data Setup

Data Prep Step 0: Raw Data

Let’s start out with the simplest possible data structure.

Units (e.g., patients, states, etc.) unit
Time year
Outcome y_it
Time-varying (binary) treatment indicator d_it
A unit-specific (time invariant) covariate x_i
A unit- and time-specific covariate x_it

We actually won’t use x_i or x_it today, so we can ignore them for now.

Data Prep Step 0: Raw Data

Let’s take a look at the first few rows of our data:

unit	year	y_it	x_i	x_it
1	2005	0.627	1.553	-0.484
1	2006	-0.157	1.553	0.472
1	2007	-0.261	1.553	-0.406
1	2008	-0.319	1.553	-0.735
1	2009	-1.670	1.553	-2.017
1	2010	-2.920	1.553	-2.331

Data Prep Step 1a: Identify Treatment Cohorts

Our next objective is to identify the various treatment cohorts in our data.
To do this, we will ask “when is the first year this unit is treated”?
Based on the answer to this question, we will be able to identify all of the cohorts based on the first year they are treated.

Data Prep Step 1a: Identify Treatment Cohorts

Identify first year each unit was treated

df_ <- 
  df %>% 
  #########
  # Step 1a
  #########
  arrange(unit,year) %>% 
  # What year is the unit first treated in?
  mutate(first_treated = as.integer(lag(d_it==0) & d_it==1)) %>% 
  group_by(unit) %>% 
  # Is the unit never treated (i.e., d_it is never 1)
  mutate(never_treated = as.integer(sum(d_it)==0))  %>% 
  ungroup() %>% 
  mutate(first_treated = year * first_treated) %>% 
  group_by(unit) %>% 
  # Treatment cohort 
  mutate(treatment_cohort = max(first_treated)) %>% 
  ungroup()

Data Prep Step 1a: Identify Treatment Cohorts

Identify never-treated units (if any)

df_ <- 
  df %>% 
  #########
  # Step 1a
  #########
  arrange(unit,year) %>% 
  # What year is the unit first treated in?
  mutate(first_treated = as.integer(lag(d_it==0) & d_it==1)) %>% 
  group_by(unit) %>% 
  # Is the unit never treated (i.e., d_it is never 1)
  mutate(never_treated = as.integer(sum(d_it)==0))  %>% 
  ungroup() %>% 
  mutate(first_treated = year * first_treated) %>% 
  group_by(unit) %>% 
  # Treatment cohort 
  mutate(treatment_cohort = max(first_treated)) %>% 
  ungroup()

Data Prep Step 1a: Identify Treatment Cohorts

Treatment cohort names are the year of first treatment.
Treatment cohort name is “0” for never treated units.

df_ <- 
  df %>% 
  #########
  # Step 1a
  #########
  arrange(unit,year) %>% 
  # What year is the unit first treated in?
  mutate(first_treated = as.integer(lag(d_it==0) & d_it==1)) %>% 
  group_by(unit) %>% 
  # Is the unit never treated (i.e., d_it is never 1)
  mutate(never_treated = as.integer(sum(d_it)==0))  %>% 
  ungroup() %>% 
  mutate(first_treated = year * first_treated) %>% 
  group_by(unit) %>% 
  # Treatment cohort 
  mutate(treatment_cohort = max(first_treated)) %>% 
  ungroup()

Data Prep Step 1b: Trim Observations

Important

If all groups are eventually treated, you’ll need to “trim” the data of all observations on or after the last treated cohort date.

There is no natural comparator for the last-treated group, since all other groups are already treated.
This step alone can take care of a significant amount of the bias from “forbidden comparisons” in TWFE DID.

Data Prep Step 2: Define Relative Time and Calendar Time Indicators

Set relative time to -Inf for never treated units.

df_ <- 
  df %>% 
  #########
  # Step 1a
  #########
  ... 
  ##########
  # Step 2
  ##########
  # Relative time variable
  mutate(rel_time = year - treatment_cohort) %>% 
  mutate(rel_time = ifelse(never_treated==1, -Inf, rel_time))  %>% 
  # Create dummy indicators for treatment cohorts, years, and relative time. 
  dummy_cols(c("treatment_cohort","year","rel_time"))  %>%  
  # Rename to remove the minus sign
  rename_at(vars(contains("rel_time_")),function(x) gsub("-","lag",x)) %>% 
  # Never treated units should have values of 0 for all relative-time indicators. 
  mutate_at(vars(contains("rel_time_")),function(x) ifelse(is.na(x),0,x))

Data Prep Step 2: Define Relative Time and Calendar Time Indicators

Can easily create dummy variable indicators using FastDummies package.
Due to negative values, the - sign can complicate variable names, so we just replace with lag.

df_ <- 
  df %>% 
  #########
  # Step 1a
  #########
  ... 
  ##########
  # Step 2
  ##########
  # Relative time variable
  mutate(rel_time = year - treatment_cohort) %>% 
  mutate(rel_time = ifelse(never_treated==1, -Inf, rel_time))  %>% 
  # Create dummy indicators for treatment cohorts, years, and relative time. 
  dummy_cols(c("treatment_cohort","year","rel_time"))  %>%  
  # Rename to remove the minus sign
  rename_at(vars(contains("rel_time_")),function(x) gsub("-","lag",x)) %>% 
  # Never treated units should have values of 0 for all relative-time indicators. 
  mutate_at(vars(contains("rel_time_")),function(x) ifelse(is.na(x),0,x))

Data Prep Step 2: Define Relative Time and Calendar Time Indicators

Never treated units should have values of 0 for all relative-time indicators.

df_ <- 
  df %>% 
  #########
  # Step 1a
  #########
  ... 
  ##########
  # Step 2
  ##########
  # Relative time variable
  mutate(rel_time = year - treatment_cohort) %>% 
  mutate(rel_time = ifelse(never_treated==1, -Inf, rel_time))  %>% 
  # Create dummy indicators for treatment cohorts, years, and relative time. 
  dummy_cols(c("treatment_cohort","year","rel_time"))  %>%  
  # Rename to remove the minus sign
  rename_at(vars(contains("rel_time_")),function(x) gsub("-","lag",x)) %>% 
  # Never treated units should have values of 0 for all relative-time indicators. 
  mutate_at(vars(contains("rel_time_")),function(x) ifelse(is.na(x),0,x))

Final Data

Let’s take a look at a few rows of our final data:

unit	year	y_it	d_it	x_i	x_it	first_treated	treatment_cohort	rel_time	treatment_cohort_2013	year_2010	year_2011	year_2012	year_2013	year_2014	year_2015	rel_time_lag1	rel_time_lag2	rel_time_lag3	rel_time_0	rel_time_1	rel_time_2
1	2010	-2.920	0	1.553	-2.331	0	2013	-3	1	1	0	0	0	0	0	0	0	1	0	0	0
1	2011	-2.979	0	1.553	-1.304	0	2013	-2	1	0	1	0	0	0	0	0	1	0	0	0	0
1	2012	-0.744	0	1.553	-0.905	0	2013	-1	1	0	0	1	0	0	0	1	0	0	0	0	0
1	2013	0.194	1	1.553	-1.677	2013	2013	0	1	0	0	0	1	0	0	0	0	0	1	0	0
1	2014	-0.641	1	1.553	-2.088	0	2013	1	1	0	0	0	0	1	0	0	0	0	0	1	0
1	2015	-1.243	1	1.553	-0.602	0	2013	2	1	0	0	0	0	0	1	0	0	0	0	0	1

Learnr Exercise

We’ll now turn to the learnr exercise to fit the various DID estimators.
Link to exercise

References

Athey, Susan, Mohsen Bayati, Nikolay Doudchenko, Guido Imbens, and Khashayar Khosravi. 2021. “Matrix Completion Methods for Causal Panel Data Models.” Journal of the American Statistical Association 116 (536): 1716–30.

Borusyak, Kirill, Xavier Jaravel, and Jann Spiess. 2021. “Revisiting Event Study Designs: Robust and Efficient Estimation.” arXiv Preprint arXiv:2108.12419. https://arxiv.org/abs/2108.12419.

Callaway, Brantly, and Pedro HC Sant’Anna. 2021. “Difference-in-Differences with Multiple Time Periods.” Journal of Econometrics 225 (2): 200–230.

Cengiz, Doruk, Arindrajit Dube, Attila Lindner, and Ben Zipperer. 2019. “The Effect of Minimum Wages on Low-Wage Jobs.” The Quarterly Journal of Economics 134 (3): 1405–54.

De Chaisemartin, Clément, and Xavier d’Haultfoeuille. 2020. “Two-Way Fixed Effects Estimators with Heterogeneous Treatment Effects.” American Economic Review 110 (9): 2964–96.

Gardner, John. 2021. “Two-Stage Difference-in-Differences,” April.

Roth, Jonathan, Pedro HC Sant’Anna, Alyssa Bilinski, and John Poe. 2022. “What’s Trending in Difference-in-Differences? A Synthesis of the Recent Econometrics Literature.” arXiv Preprint arXiv:2201.01194. https://arxiv.org/abs/2201.01194.

Wooldridge, Jeffrey. 2021. “Two-Way Fixed Effects, the Two-Way Mundlak Regression, and Difference-in-Difference Estimators,” September.