Approaches to Evaluating Welfare Reform: Lessons from Five State Demonstrations. D. ESTIMATING IMPACTS FOR SUBGROUPS DEFINED BY EVENTS SINCE RANDOM ASSIGNMENT


No matter how carefully an evaluator assembles administrative or survey data for an impact analysis, outcome or background data will be missing or invalid for at least a small fraction of the sample. Frequently, however, outcome or background data are missing or invalid for a large portion of the research sample (perhaps 20 percent or more). One reason for this situation is that the data were not collected completely by administrators or survey workers, although the corresponding information is (in theory) available for all relevant cases. Another reason for missing or invalid data is that the outcome itself is defined by program-related events subsequent to random assignment (such as participation in welfare or in job training), so no outcome data can possibly be collected for some cases. Both situations present problems for the analysis of impacts from the welfare reform package.