Birds of a feather flock together: Comparing controlled pre-post designs

Abstract

Objective: To formalize comparative interrupted time series (CITS) using the potential outcomes framework; compare two version of CITS-a standard linear version and one that adds postperiod group-by-time parameters-to two versions of difference-in-differences (DID)-a standard version with time fixed effects and one that adds group-specific pretrends; and reanalyze three previously published papers using these models.

Data sources: Outcome data for reanalyses come from two counties' jail booking and release data, Medicaid prescription drug rebate data from the Centers for Medicare and Medicaid Services (CMS), and acute hepatitis C incidence from the Centers for Disease Control and Prevention.

Study design: DID and CITS were compared using potential outcomes, and reanalyses were conducted using the four described pre-post study designs.

Data collection/extraction methods: Data from county jails were provided by sheriffs. Data from CMS are publicly available. Data for the third reanalysis were provided by the authors of the original study.

Principal findings: Though written differently and preferred by different research communities, the general version of CITS and DID with group-specific pretrends are the same: they yield the same counterfactuals and identify the same treatment effects. In a reanalysis with evidence of divergent preperiod trends, failing to account for this in standard DID led to an 84% smaller effect estimate than the more flexible models. In a second reanalysis with evidence of nonlinear outcome trends, failing to account for this in linear CITS led to a 28% smaller effect estimate than the more flexible models.

Conclusion: We recommend detailing a causal model for treatment selection and outcome generation and the required counterfactuals before choosing an analytical approach. The more flexible versions of DID and CITS can accommodate features often found in real data, namely, nonlinearities and divergent preperiod outcome trends.

Keywords: econometrics; evaluation design and research; observational data/quasi-experiments; program evaluation.