Approaches to Evaluating Welfare Reform: Lessons from Five State Demonstrations. A. OBJECTIVES AND METHODS OF THIS STUDY


In recent years, the federal government has required most evaluations of welfare reform demonstrations to be based on an experimental design.(3) In welfare reform demonstrations, an experimental design means that some welfare cases are randomly assigned to the demonstration program (these are referred to as experimental group cases) and others are randomly assigned to the pre-reform program (these are referred to as control group cases). A comparison of individuals randomly assigned to the experimental and control groups provides an estimate of the impact of the reforms that controls for factors external to the demonstration, such as changes in the economy. In addition to requiring a common design for the demonstration evaluations, the federal government also specified a core set of outcome measures that must be included in the evaluations, including the employment and earnings of welfare recipients and their rates of marriage and separation.

Because the welfare reform demonstration evaluations generally share a common research design and core set of outcome measures, they repeatedly confront the same research issues.(4) These issues must be appropriately resolved if the evaluations are to produce reliable research results that are useful for assessing and making welfare policy. Given the similarity in many of the programs and in the overall research designs, the diversity of approaches that the evaluators take to specific research issues is somewhat surprising. Some evaluators do not address a particular issue, either because they fail to recognize it as a potential problem or because they recognize it but judge that doing nothing is the most appropriate course of action. Among those who actively respond to an issue, a variety of technical approaches may be used.

DHHS, as it observed the welfare reform demonstration evaluators struggling with a common set of research issues, recognized the need to systematically identify the issues and assess the approaches to them. This led to discussions between DHHS and MPR during fall 1995 that resulted in two conclusions that provided structure for the study of waiver evaluations:

  1. To effectively review the designs for the welfare reform demonstration evaluations and monitor their progress, the federal government sought an assessment of the most important research issues common to the evaluations and the appropriateness of various approaches to them. This same information would also benefit the states and evaluators in designing and carrying out the evaluations.
  2. Without the federal waiver process, more of the initiative for designing, implementing, and evaluating welfare demonstration programs would shift to the states. Many of the research issues surrounding the waiver demonstration evaluations would be present in those future evaluations. Thus, a compilation of research issues and approaches arising in the context of those evaluations would almost certainly be beneficial to future state welfare reform evaluations occurring outside the federal waiver process.

These observations implied two broad objectives for this study. One objective was to identify for the federal government, the states, and the evaluators the principal research issues surrounding the design and execution of the waiver demonstrations and evaluations and to assess the appropriateness of various technical approaches to those issues. The study's other objective was to document those issues and approaches in a sufficiently general way that the information would be useful in designing and conducting welfare demonstrations and evaluations outside of the federal waiver context.

To achieve these objectives, DHHS and MPR agreed that the study would be based on the actual experiences and decisions of states and evaluators conducting waiver demonstrations and evaluations. They also agreed that its focus would be on the estimation of impacts of the demonstrations, rather than the cost neutrality calculations or other federally mandated components of the evaluations. MPR would obtain this information primarily by reviewing documents prepared by the federal government, the states, and the evaluators pertaining to selected waiver demonstrations and evaluations. These documents included the following:

  • The state's waiver request, including its evaluation design
  • The terms and conditions under which DHHS and USDA granted a state waivers to implement its welfare reform demonstration
  • The evaluator's plan for the demonstration evaluation
  • Quarterly and annual reports on the progress of the evaluation
  • Interim evaluation reports, when available (no final reports were available)

When this study was being conducted in 1996, most of the waiver evaluations had not been under way long enough to have produced interim reports; this limited what could be learned by reviewing project documents. As a supplemental source of information on the evaluations, MPR staff members spoke by telephone with the state officials in charge of the selected evaluations, the directors of the evaluation contracts and (in some states) other researchers working on the evaluations.(5) These calls were made to fill in gaps in information obtained from documents and to obtain the latest information about the status, methods, and findings of the evaluations.

An advisory panel of experts on welfare evaluations was a final source of information for this project.(6) The advisory panel reviewed MPR memos identifying the key issues surrounding the evaluations, reviewed MPR summaries of state demonstrations and evaluations, and met with DHHS and MPR staff to comment on these items and to provide additional information about the evaluations and guidance regarding the course of the study.