Approaches to Evaluating Welfare Reform: Lessons from Five State Demonstrations. a. Is a Survey Necessary?


If administrative data cover all of the major outcomes of interest in the evaluation, conducting a survey to get at additional outcomes may not be worthwhile. If resources permit, surveys may be used to study the major outcomes in more depth or to obtain data on secondary outcomes. Such surveys, however, may be too expensive for small states or small evaluations to pursue. To be useful to the impact evaluation, enough resources should be available to obtain high response rates (see Section C).

On the other hand, some interventions primarily target outcomes for which there are no readily available administrative data. Examples include "family cap" provisions (under which no additional benefits are awarded when another child is born to someone on assistance), provisions designed to increase school attendance and immunization rates, and changes in the AFDC-UP program that are intended to promote marriage and family stability. Even in these instances, a survey may not be the only option. There are often less readily available sources of administrative data (such as birth records, school records data, or Medicaid records) that may be more cost-effective or may provide data of better quality than survey data. These alternative data sources have limitations as well, but should be considered carefully. The plan for a survey to be done by the evaluator several years after implementation may take the pressure off state agency staff to obtain other administrative data and, thus, have a counterproductive effect. If alternative sources of administrative data are not planned for early on, important opportunities may be lost, since some of these sources (such as school records) require signed consent forms, and such signatures are most effectively obtained at program intake.