Approaches to Evaluating Welfare Reform: Lessons from Five State Demonstrations. 3. Analysis and Recommendations

10/01/1996

Properly implemented, random assignment ensures that the baseline characteristics of experimental and control cases are, on average, the same. A major advantage of this equivalence at baseline is that subsequent differences between experimental and control groups can be attributed entirely to exposure to welfare reform policies. When the sample used to analyze impacts from welfare reform is reduced in size, either because of incomplete or incorrect data or because of the analysis of subgroups defined by program-related events since random assignment, the strict comparability of the experimental and control samples may be lost.

The problem of incomplete or incorrect data can be reduced through state efforts to ensure the quality of administrative records and through evaluator efforts to increase survey response rates. We also recommend that evaluators use all observations for which valid outcome data and basic baseline characteristics are present, rather than restricting the sample because nonessential baseline information is missing. If desired, evaluators may impute missing values of nonessential baseline characteristics. By not deleting observations needlessly, both large sample sizes and the representativeness of the overall research sample are preserved. This makes the resulting impact estimates more applicable to the entire population of cases from which the research sample was drawn.

In defining outcomes for inclusion in the impact study, evaluators of state welfare reform programs should adopt analytic strategies that take maximum advantage of the strengths of an experimental design. In particular, we recommend that evaluators define outcomes in ways that enable values to be assigned for all or nearly all recipient and applicant cases in the research sample. When there is interest in a particular outcome for a subgroup defined by events since random assignment (such as recidivism rates for cases that have left welfare), we recommend that alternative outcomes be considered for analysis (such as the number of welfare spells or the percentage of months spent on welfare since random assignment).