Approaches to Evaluating Welfare Reform: Lessons from Five State Demonstrations. Notes


(1)Appendix C provides a glossary of evaluation terms used in this report.

(2)The specification of a statistical model refers to (1) the number, type, and measurement of control variables, (2) the measurement of policy variables (for example, participation in the reform program can be measured as a dichotomous yes/no variable or as a continuous months-of-exposure variable), (3) the measurement of outcome variables (for example, current employment can be measured as a dichotomous yes/no variable or as a continuous hours-per-week variable), (4) functional form (that is, whether the relationship between independent and dependent variables is linear or some nonlinear function), and (5) the assumed distribution of the error term. Misspecification of a model along these or other dimensions can result in it generating biased estimates of the impacts of a program reform.

(3)With an experimental design it still may be useful to use a multivariate model to increase the precision of impact estimates.

(4)Nonexperimental methods for estimating the impacts of specific reforms may be employed in an experimental context just as in a nonexperimental context. However, as discussed in Chapter VI, such approaches rarely work well.

(5)An experiment would have to randomly assign all members of the population who could conceivably be at risk of entering the program. Alternately, as discussed in Chapter VI, an experiment could randomly assign a large number of sites, and compare entry rates in experimental and control sites.

(6)Although there is random assignment of sites in this type of design, we do not consider this a truly experimental design, because the sample of sites is typically much too small to rule out the confounding of site-specific factors with the effects of the program.