We conduct a series of robustness checks to explore the consistency of the ELE parameter estimates. To the extent that these estimates display consistency, it strengthens the evidence provided by the original model specification and, thereby, the conclusions that can be drawn from the analysis. These robustness checks include reestimating the main model with the following variants:
- Alternative specifications of the control variables to determine the source of the ELE effect:
- To start, we remove the policy variables, unemployment rate, and child population from the main model specification (that is, this model includes only state and quarter fixed effects). This simple unadjusted difference-in-difference model removes all time-varying covariates and approximates the average ELE treatment effect from the descriptive data, relative to the chosen set of comparison states (alternative 1).
- We then add the policy variables to the simple model (all at once and each individually) to determine if their inclusion alters the magnitude and significance of the ELE variable (alternative 2).
- We also add the unemployment rate and child population variables to the simple model to determine if their inclusion alters the magnitude and significance of the coefficient on the ELE variable (alternative 3).
- We replace all of the administrative simplification dummy variables with a policy index, ranging from 0 to 5 in the Medicaid model and 0 to 10 in the Medicaid/CHIP model (alternative 4).
- Alternative specifications with respect to how the comparison group is defined, excluding non-ELE states in a systematic manner to determine if specific control states drive the main results. These tests are important because the non-ELE states control for what the baseline trend in Medicaid/CHIP enrollment would have been in the absence of ELE.
- We include all 41 non-ELE states as the comparison group in the Medicaid/CHIP and Medicaid models (alternative 5).
- We use the same methodology from the main model to select comparison states, but exclude non-ELE states in which the time trend interaction term is statistically significant at the 10 percent level (alternative 6) and at the 1 percent level (alternative 7). In the Medicaid/CHIP model, there are 22 comparison states in alternative 6 and 35 in alternative 7, compared with 25 in the main model. In the Medicaid-only model, there are 30 and 36 comparison states in alternatives 6 and 7, respectively, compared with 33 in the main model.
- We use a similar but more restrictive method to select comparison states (alternative 8). Instead of interacting the ELE indicator with the time trend, we interact each quarter dummy with the ELE variable and exclude non-ELE states in which we reject the null hypothesis that the joint interaction terms are zero at the 5 percent level. This method increases the likelihood of choosing comparison states that have the same quarter-to-quarter pattern in enrollment before 2009 and excludes more comparisons states relative to the main model scenario. Under this alternative, there are 15 comparison states in the Medicaid/CHIP model and 19 comparison states in the Medicaid model.
- We exclude non-ELE states that are statistical outliers and might not serve as ideal comparison states. For this exercise, we remove 8 non-ELE states from the Medicaid/CHIP model and 9 non-ELE states from the Medicaid-only model that had observations with studentized residuals greater than 2.5 and less than –2.5 in the main model specification (alternative 9).
- Similarly, we reestimate the simple unadjusted Medicaid/CHIP and Medicaid-only difference-in-difference models, including one non-ELE state at time to determine which comparison states have the strongest influence on the ELE coefficient magnitude. We then rank the states based on the estimated ELE coefficient when they are included in the model and reestimate the main model, excluding the comparison states that resulted in the 5 highest and the 5 lowest ELE effects, respectively (alternative 10). We also estimate a variant that excludes comparison states with the 10 highest and 10 lowest ELE effects (alternative 11).
We also estimate several other alternative models to support the robustness of the ELE variable, but the results are not included here.54 For instance, we include a control for whether the state expanded coverage to children who have lawfully resided in the United States for fewer than five years under the new CHIPRA option.55 We also add controls for the receipt of Cycle I (awarded September 2009) or Cycle II (awarded August 2011) CHIPRA outreach grants.
54 These results are discussed in the final report to ASPE (Blavin et al 2012) and are available upon request.These results are discussed in the final report to ASPE (Blavin et al 2012) and are available upon request.
55 We also created different ELE policy variables—“ELE through SNAP” and “ELE through tax returns”—to explore whether there appeared to be a differential effect based on the type of ELE program implemented, but the limited experience with ELE to date constrains our ability to make such an assessment.