Approaches to Evaluating Welfare Reform: Lessons from Five State Demonstrations. 3. Analysis and Recommendations


The presence of large entry effects induced by welfare reform can call into question the validity of impact estimates for applicant cases, even if the evaluation features an experimental design. Only two of the five evaluations we reviewed included substantial efforts to study entry effects by analyzing application and termination behavior over time. If longitudinal data on applications are not available, time-series analyses are not feasible. If data are available, there may not be sufficient resources for the analysis in an evaluation largely focused on experimental impact estimates. Even if adequate longitudinal data and analytic resources are available for a particular state, the results of the estimation of entry effects may be sensitive to statistical assumptions employed by evaluators.

Remarkably little research exists on entry effects. Therefore, we recommend additional research on entry effects, which may be separate from random-assignment evaluations of state welfare reform initiatives, since the data collection and analytic needs for each type of study differ. Evaluations of entry effects could look at monthly welfare applications and terminations across several states, using standardized statistical methodologies and data sources such as historical caseload records from the states, the federal Integrated Quality Control System, or the Survey of Income and Program Participation. A major goal of studies using data from several states should be to identify the sorts of policy changes that are most likely to be associated with large entry effects over time. Another goal of such research should be to identify ways to combine nonexperimental entry effect estimates with experimental impact estimates to assess the overall consequences of a state's welfare reform program for applicant cases.