CHIPRA Mandated Evaluation of Express Lane Eligibility: First Year Findings. Appendix C: Sensitivity Analysis Results for the Seds Analysis

12/01/2012

As mentioned in the main text, the results from alternatives 1 and 3 show that the inclusion or exclusion of policy variables in the main model—which uses the designated sets of comparison states described in the methods section—has a small impact on the magnitude of the ELE variable, but has a more substantial effect on the level of statistical significance. In addition, removing the unemployment rate and child population (alternative 2) from the main model has a negligible effect on the ELE results. 

Appendix Table C.2 further explores the policy variables by analyzing the effect of going from alternative 3 (the simple unadjusted difference-in-difference model with state and quarter fixed effects and demographic controls), to adding each of the policy covariates one at a time, and the effect of subtracting one policy at a time from the full model. The ELE coefficient in the Medicaid model remains statistically significant at the 10 or 5 percent level in each model where a policy variable is subtracted one at a time. Removing the simulated parent eligibility variable lowers the magnitude associated with the ELE coefficient by about 1.4 percentage points, from 5.6 to 4.2, but does not alter the statistical significance. Removing joint application and administrative verification of income has a small negative effect on the ELE variable’s statistical significance (p-value increases from around .03 to just over .05), but does not have much of an effect on the magnitude. The next column under the Medicaid enrollment model shows that adding the parent eligibility variable, presumptive eligibility, and administrative verification of income one at a time adds some statistical significance on the ELE variable in the basic unadjusted model (p-values range from .07 to .09). In contrast, the p-values associated with the ELE variable in the models that add the other variables in this column are just over .1. 

In contrast, the first two columns of Table C.2 show that the level of statistical significance in the Medicaid/CHIP model is more sensitive to the exclusion or inclusion of specific policy controls. The ELE effect becomes statistically insignificant, with p-values just above .1, when the following variables are removed one at a time from the fully adjusted model: parent eligibility, presumptive eligibility (Medicaid), no in-person interview (Medicaid), and administrative verification of income (CHIP). The ELE effect in the unadjusted model remains statistically insignificant when you add each policy variable one at a time. However, the direction and magnitude are relatively stable across all these models.

These results provide reassurance that the magnitude of the ELE effect is not driven by other observed policy or economic changes correlated with ELE implementation or with policy choices among non-ELE states. Our results show that the ELE variable remains positive and similar in magnitude even when we do not control for the other policy changes and when the policy variables are removed from the main model one at a time. We also find that the coefficient on the ELE variable remains statistically significant and relatively constant in magnitude after controlling for recent (2009 onward) child immigrant expansions and level of outreach grants states received under CHIPRA (results not shown).

The results from alternatives 1 through 4 also provide confidence that the main results are not driven by measurement error or multicollinearity. Although we are confident in the accuracy of the ELE implementation dates, there is some uncertainty about the accuracy of the dates for the other policy variables, especially for changes that occurred earlier in the period of analysis. As an additional test, we find that the ELE effect remains positive and statistically significant even when we lag all the time-varying covariates by one or two fiscal quarters (results not shown). The finding that the ELE effect remains when the policy variable dummies are aggregated into a single index (alternative 4) suggests that multicollinearity is not distorting the overall ELE effect. The main regression models suffer from high levels of multicollinearity, as measured by the variance inflation factor (VIF). However, we find that the VIF is in a normal range when we remove the time-varying covariates from the main model, while the ELE coefficient changes very little. 

The results from alternative models 6 through 11 further validate the ELE effect found in the main Medicaid model by showing that the ELE effect persists even after using various groupings of non-ELE comparison states, while casting some doubt on the robustness of the Medicaid/CHIP model. In alternative 6, we increase the significance threshold to 10 percent for rejecting the null of no difference in pre-ELE enrolment trend differences, thereby dropping three additional states in both models. In alternative 7, we decrease the significance threshold to 1 percent and include 10 additional comparison states in the Medicaid/CHIP model and 3 additional states in the Medicaid models. Using the alternative procedure based on quarter-quarter trend differences (alternative 8), we only include 15 comparison states in the Medicaid/CHIP model and 19 comparison states in the Medicaid model. Under these three alternatives, the ELE effect remains relatively unchanged in the Medicaid model, whereas the ELE coefficient is less than one percentage point smaller and no longer statistically significant in the Medicaid/CHIP model (the p-value ranges from .12 for alternatives 6 and 7 to .37 for alternative 8).   

The composition of the comparison groups differs across the Medicaid/CHIP and Medicaid models because more non-ELE states have significantly different CHIP enrollment trends compared to the ELE state average before 2009. However, when we use the same comparison states in the Medicaid/CHIP model that were selected in the Medicaid model, we find that the ELE effect is positive (in the neighborhood of 4.5 percent) and statistically significant across all specifications of the comparison group exclusion tests.  

In alternative 9, we remove outlier states that might not serve as the most appropriate comparison for ELE states and find that the ELE effect is even stronger relative to the main model results. For alternatives 10 and 11, we re-estimate the simple Medicaid/CHIP and Medicaid-only models, including one non-ELE state at time to determine which control states have the strongest influence on the ELE coefficient. We then rank the states based on the estimated ELE coefficient when they are included in the model. For the Medicaid/CHIP and Medicaid-only models under alternative 10, we remove the top and bottom five states based on this ranking. We find that removing these 10 states resulted in a stronger ELE effect in the Medicaid-only model and a slightly lower, but still statistically significant, effect in the total Medicaid/CHIP model. We also find that the ELE effect remains statistically significant and comparable in magnitude in the Medicaid-only model even after we remove the top and bottom 10 states in the distribution (alternative 11).


Table C.1. Trends in Medicaid and CHIP Enrollment Among ELE States 2007-2011 Quarterly SEDS Enrollment

  Alabama Georgia Iowa Louisiana Maryland New Jersey Oregon South Carolina
Fiscal Quarter Medicaid CHIP Medicaid CHIP Medicaid CHIP Medicaid CHIP Medicaid CHIP Medicaid CHIP Medicaid CHIP Medicaid CHIP
2007
Q1 415,365 73,279 795,998 300,361 193,627 20,147 684,290 N/A 411,498 13,154 502,388 84,486 210,385 39,306 435,924 N/A
Q2 445,496 74,028 778,842 305,543 193,458 23,818 654,435 N/A 405,412 14,032 505,305 85,349 206,272 42,729 432,026 N/A
Q3 410,315 74,636 766,860 293,950 195,454 22,929 653,464 1,007 405,019 N/A 506,695 86,368 207,011 45,819 425,657 N/A
Q4 406,007 77,017 746,212 296,789 197,677 23,979 664,038 1,681 421,591 N/A 518,131 84,536 206,914 47,071 421,542 N/A
2008
Q1 408,910 77,336 768,798 287,188 198,910 23,830 668,018 1,831 428,742 N/A 511,346 76,221 201,705 48,209 426,195 N/A
Q2 410,629 78,786 811,349 262,657 199,748 23,670 675,222 1,961 440,742 N/A 513,078 76,348 207,752 49,813 432,470 N/A
Q3 377,328 78,825 789,035 249,180 202,815 24,157 679,026 2,206 434,206 N/A 520,369 77,837 213,527 51,876 434,878 2,048
Q4 400,104 79,909 830,734 238,469 207,137 23,490 687,150 3,571 437,971 N/A 526,339 78,940 216,740 52,509 448,738 5,779
2009
Q1 416,215 79,017 847,744 229,499 214,778 22,939 690,591 4,196 447,788 N/A 531,616 79,263 221,583 55,044 451,701 9,286
Q2 432,325 78,358 867,210 225,703 220,602 22,544 697,224 4,494 454,580 N/A 540,700 79,643 237,484 58,954 458,260 12,078
Q3 440,129 76,959 886,505 225,921 227,470 23,552 703,050 4,883 461,224 N/A 576,744 83,874 256,255 56,086 464,802 14,944
Q4 450,346 77,320 904,278 220,884 253,877 25,516 714,674 5,504 475,584 N/A 587,883 87,812 252,962 50,900 480,170 16,196
2010
Q1 460,127 83,270 930,800 223,020 256,707 27,392 726,581 5,567 481,651 N/A 603,131 90,680 277,529 57,981 486,792 16,946
Q2 470,262 84,659 925,626 225,482 261,969 29,594 735,413 5,513 487,604 N/A 612,515 76,337 290,688 61,217 491,284 16,832
Q3 480,396 85,918 944,438 222,570 266,722 32,614 742,666 5,756 497,440 N/A 621,941 78,001 297,234 65,869 497,079 17,401
Q4 492,001 81,880 951,748 217,224 270,934 34,318 749,170 5,966 508,743 N/A 630,845 96,154 288,775 64,634 504,903 17,862
2011
Q1 499,069 80,945 998,573 217,940 272,312 36,615 748,284 5,933 515,244 N/A 639,755 98,300 312,517 75,283 524,395 N/A
Q2 505,911 82,846 989,334 215,607 274,665 38,780 743,877 6,017 522,863 N/A 645,531 99,533 320,783 78,493 527,402 N/A
Q3 507,888 86,354 909,930 218,471 276,872 39,909 741,076 6,210 531,628 N/A 653,144 101,191 326,705 80,950 519,413 N/A
Q4 521,664 88,589 1,004,598 217,157 281,189 40,607 746,196 6,336 537,051 N/A 661,540 101,055 332,096 84,023 529,382 N/A

Source: CMS Statistical Enrollment Data System (SEDS) as of March 30, 2012, verified and provided by CMS.

Notes: (1) Medicaid enrollment only includes children who were ever enrolled in Title XIX or Title XXI Medicaid during the fiscal quarter. CHIP enrollment only includes children who were ever enrolled in a separate CHIP during the fiscal quarter. (2) Values in bold were imputed by the Urban Institute using methods described in the paper. (3) N/A indicates that the state did not have a separate CHIP during the quarter.

CHIP = Children's Health Insurance Program; CMS = Centers for Medicare & Medicaid Services; ELE = Express Lane Eligibility; SEDS = Statistical Enrollment Data System; Q = quarter.


Appendix Table C.2. Estimated ELE Effects for Alternative Models That Add or Remove Each Policy Variable 2007-2011 Quarterly SEDS Data

  Total Medicaid/CHIP Enrollment Medicaid Enrollment
  Fully Adjusted Main Model, Subtracting One Policy at a Time Unadjusted Basic Model, Adding One Policy at a Time Fully Adjusted Main Model, Subtracting One Policy at a Time Unadjusted Basic Model, Adding One Policy at a Time
Main Regression Model 0.0420* (0.024) 0.0420* (0.024) 0.0562** (0.026) 0.0562** (0.026)
Subtracted or Added Policy Variable
Separate CHIP 0.0408* (0.024) 0.0370 (0.029) 0.0575** (0.025) 0.0375 (0.026)
Simulated Eligibility Threshold for Children 0.0434* (0.022) 0.0255 (0.024) 0.0567** (0.026) 0.0389 (0.025)
Simulated Eligibility Threshold for Parents 0.0296 (0.018) 0.0488 (0.035) 0.0419** (0.021) 0.0539* (0.032)
Joint Application 0.0388* (0.022) 0.0345 (0.028) 0.0519* (0.026) 0.0404 (0.025)
Presumptive Eligibility-Medicaid 0.0416 (0.025) 0.0355 (0.023) 0.0557** (0.027) 0.0412* (0.022)
Admin. Verification of Income-Medicaid 0.0419* (0.024) 0.0360 (0.028) 0.0512* (0.026) 0.0433* (0.025)
No In-Person Interviews-Medicaid 0.0408 (0.024) 0.0366 (0.027) 0.0541** (0.026) 0.0409 (0.025)
Continuous Eligibility-Medicaid 0.0452* (0.025) 0.0301 (0.024) 0.0607** (0.028) 0.0378 (0.023)
Presumptive Eligibility-CHIP 0.0418* (0.024) 0.0342 (0.025) N/A N/A
Admin. Verification of Income-CHIP 0.0428 (0.027) 0.0368 (0.028) N/A N/A
No In-Person Interviews-CHIP 0.0405* (0.023) 0.0358 (0.029) N/A N/A
No Asset Test-CHIP 0.0457* (0.027) 0.0305 (0.025) N/A N/A
Continuous Eligibility-CHIP 0.0424* (0.024) 0.0315 (0.026) N/A N/A
R-squared 0.99 0.99 0.99 0.99
Sample Size 660 660 820 820

Source: CMS Statistical Enrollment Data System (SEDS) as of March 30, 2012, verified and provided by CMS.

Notes: (1) Robust standard errors clustered at the state level are in parentheses. (2) *p<.1, **p<.05, ***p<.01. (3) All models include state and quarter fixed effects and demographic controls (coefficients not shown). (4) Total enrollment includes children who were ever enrolled in Medicaid or CHIP during the fiscal quarter. Medicaid enrollment only includes children who were ever enrolled in Title XIX or Title XXI Medicaid during the fiscal quarter.

CHIP = Children's Health Insurance Program; CMS = Centers for Medicare & Medicaid Services; ELE = Express Lane Eligibility; SEDS = Statistical Enrollment Data System

View full report

Preview
Download

"rpt.pdf" (pdf, 1.1Mb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®