Approaches to Evaluating Welfare Reform: Lessons from Five State Demonstrations. 2. State Approaches


On the basis of the state evaluations we have reviewed, the surveys planned or in progress in current waiver demonstrations are broad in scope, generally include samples of from 1,000 to 2,000 cases, and are scheduled to occur either at regular intervals or once relatively late in the demonstration period. Information collected tends to include background information and two types of outcomes: (1) economic outcomes not obtainable from administrative data, such as hours of work, wages, participation in non- JOBS education and training, and costs of work such as child care and transportation; and (2) noneconomic outcomes, such as family structure, fertility, health status, health behaviors, and food security. The large number of outcomes pursued in some instances has led to lengthy and expensive surveys. The sample sizes appear modest (particularly for assessing impacts on outcomes such as family structure) because such impacts are expected to be small and therefore more difficult to detect.

The surveys planned or conducted in the five states are described here:

  • In Wisconsin, MAXIMUS planned to survey all research sample cases after they leave assistance and obtain a job, to obtain further information about their employment and other outcomes not captured in administrative data. Then, MAXIMUS would resurvey these individuals annually. Surveys of those who leave assistance were to be used to obtain more detailed information on sample members' jobs than is available in the wage records data, such as information on health insurance, job type, hours and wages, job satisfaction, advancement potential, job stability, and (for those not working) barriers to work and reasons not working. In addition, the survey was intended to collect data on changes in family composition, although collecting data from vital records was also planned. (This survey was delayed because of difficulties in obtaining a sample frame from administrative data, and had not began as of spring 1996.)
  • In California, the Wave I "baseline" English/Spanish survey included about 2,200 ongoing cases and has covered about 250 approved applicant cases so far; a Wave II (follow-up) survey is ongoing. Another followup is scheduled for 18 months after the start of Wave II, if funding is available. The surveys are lengthy and comprehensive, although some background items included in Wave I were omitted in Wave II. An additional survey is being conducted that oversamples speakers of four less common languages (Cambodian, Vietnamese, Laotian, and Armenian); it includes the same items as in the surveys of the English/Spanish population plus items concerning English studies and refugee and immigration status and experiences.
  • In Colorado, the evaluators planned three waves of followup--at 9, 24, and 36 months after random assignment--with small samples; targets at 9 months were 1,500 cases; at 24 months, 1,275 cases; and at 36 months, 956 cases, equally divided between experimental and control cases. A wide range of outcomes was to be covered in the surveys; some would be covered in only one or two of the follow-up interviews, to reduce the interview length.
  • In the Michigan TSMF evaluation, Abt plans one survey, 48 months after random assignment, of a random subsample of approximately 1,200 cases (600 experimental/600 control), including both ongoing and applicant cases. The focus of the survey is to be on family stability and on employment-related activities (including employment not measured in UI records), as well as on participation in training, education, and community service.
  • In Minnesota, MDRC plans two rounds of follow-up surveys (to be conducted by RTI). The first followup was designed to include only cases in urban counties, 12 months after random assignment; the sample was to consist of 2,250 cases each in groups E1 and C1, 1,350 in group E2, and 150 in group C2. The second followup is to occur 36 months after random assignment and to be of a sample selected from all seven counties. The first followup focused on employment-training participation, understanding of the program, food use, and job characteristics. The second followup is to cover employment-training participation, total family income, family and child well-being, food use, and attitudes on work and welfare.

All of these surveys have a broad focus, and many include collecting more detailed data on outcomes already available to some extent in administrative data.