Approaches to Evaluating Welfare Reform: Lessons from Five State Demonstrations. 2. State Approaches


In the four state evaluations reviewed that have experimental designs, three are relying primarily on baseline data from administrative records:

  • In California, UC DATA has built a longitudinal file with up to five years of preimplementation data on all cases in the welfare reform research sample, on the basis of data recorded in the state's Medicaid data system. Variables include participation in Medicaid, AFDC, and other programs related to Medicaid eligibility. They also have assembled over five years of historical UI records data on employment and earnings for individuals in research sample cases. Constructing these longitudinal files required a major investment but has led to a research infrastructure that now is supporting a wide range of research projects. Another database records demographic characteristics of individuals from the time the case enters the research sample (generally two months after application for new applicants), using data extracted from the county-level AFDC data systems.
  • In Colorado, baseline data from administrative records reflected case characteristics in the month of random assignment, except that there was a plan to try to obtain UI records data for a period before random assignment.
  • In the Michigan TSMF evaluation, information was available from the state's AFDC data system, by person, on basic demographics and on AFDC/SFA participation in the 24 months prior to random assignment. By case, information was available on active welfare status, welfare participation before random assignment, number of children, number of adults, presence of earnings, and so forth, by month. However, little information was available on cases denied for both AFDC and SFA; in the end, these cases were dropped from the sample. The evaluator argued that this exclusion is not a concern, because the intervention largely affects whether a family is approved for AFDC versus SFA, but not whether it is denied for both.

Special forms or surveys were not used to collect baseline information in Colorado and Michigan. California supplemented the administrative data with a telephone survey, and Minnesota relied completely on an intake form:

  • In California, the plan was to conduct the first telephone survey within a few months after random assignment, but the start of the survey was substantially delayed, limiting its usefulness as a baseline survey. The delay was caused in part by the time it took to obtain sample frame information from county-level automated data systems. Delays in instrument development were also a factor, as many stakeholders were involved in reviewing and adding to the instrument. In practice, the "baseline" or Wave I survey took place about a year after random assignment began for ongoing cases and has continued to lag random assignment substantially for applicant cases. Because of this, the survey results are being used only as descriptive background information on the survey sample, not to provide independent variables for the impact analysis.
  • Minnesota used a special baseline data collection form, administered to all ongoing and applicant cases just before random assignment. The individual applying for assistance or subject to redetermination would meet with an intake worker to fill out the Background Information Form. The form took about 10 minutes to fill out, and the response rate was 99 percent. For those already or previously on assistance, some data on their public assistance history were entered by intake staff from the automated system. In addition, the client was given a self-administered Private Opinion Survey on issues such as barriers to work, and attitudes toward work and welfare. The response rate for the Private Opinion Survey was 83 percent.