Moving People from Welfare to Work. Lessons from the National Evaluation of Welfare-to-Work Strategies.. Research Design


NEWWS used a random assignment research design to estimate the effects of the studied programs. Welfare recipients were randomly assigned to one of two or three research groups, depending on the site. One of the groups was always a control group. In all the sites, control group members were eligible for welfare as usual. In addition, they were eligible for child care assistance similar to that offered to program group members, provided that they were participating in nonprogram activities in which they had enrolled on their own.

In the three sites that operated both an LFA program and an HCD program, three-way random assignment was performed. Welfare recipients in these sites were randomly assigned to one of three groups: the LFA group, whose members received LFA program services; the HCD group, whose members received HCD program services; or the control group, whose members could not receive services through a welfare-to-work program. A three-way design was also used in Columbus, except that in Columbus the program groups differed only with respect to the case management they received (integrated or traditional).

In Detroit, Oklahoma City, and Portland, two-way random assignment was used to test the effectiveness of existing programs rather than of programs designed for research purposes. In these sites, welfare recipients were randomly assigned to a group that enrolled in the program or to a control group that was not eligible for any welfare-to-work program services.

The study design allowed for many revealing comparisons. The key ones examined the programs' economic effects on adults and spillover effects on families (that is, indirect effects on noneconomic outcomes and effects on children). To determine the net effect of each program, the outcomes for each program group were compared with those for the control group in the same site. In the three-way sites, it was also possible to estimate the relative effects of alternative program approaches by comparing the outcomes for the two program groups directly. What makes the design in the three-way sites particularly robust is that, by making comparisons between programs operated in the same site, it holds constant contextual features (such as population characteristics and local economies) that might vary from site to site and affect the programs' results.

The random assignment research design used in all the sites is what makes NEWWS such a rigorous investigation of the effectiveness of various welfare-to-work approaches. Because people were assigned to groups at random within each site, one can be sure that there were no systematic differences between people in the program and control groups when they entered the study. Therefore, any subsequent differences in outcomes between groups in the same site -- whether between two program groups or between a program group and the control group -- can be confidently attributed to a particular type of program. These differences, called impacts, can relate to any type of outcome -- for instance, rates of participation in education activities, reading test scores, employment rates, earnings levels, number of months on welfare, or assessments of children's well-being (to name but a few of the outcomes examined in NEWWS). Throughout this document, statements concerning whether the NEWWS programs increased or decreased some outcome (such as earnings) refer to their impacts, that is, to differences between how program and control group members fared during the five-year follow-up period -- not to changes in any given research group's behavior over time. (For a discussion of the advantages of using impacts rather than outcomes to assess program effectiveness, see Box 1.) Unless otherwise noted, all the impacts discussed are statistically significant, meaning that they are unlikely to be due to chance.