Impact on Young Children and Their Families 2-Years After Enrollment: Methods: How Did We Study Impacts on Children? . Design of the National Evaluation of Welfare-to-Work Strategies


As noted in Chapter 1, the Family Support Act of 1988 recommended an experimental evaluation of the impacts of the JOBS Program, that is, an evaluation involving random assignment of families to control and experimental groups. In such a design, it can be assumed (and documented) that the families in the different research groups did not differ in terms of background characteristics prior to their assignment to the different research groups. After random assignment, apart from the experiences associated with the research group they are assigned to, families in each site are all exposed to the same broad context, for example, in terms of the job market and local economy. Given that the families did not differ initially, because the assignment to research groups is done randomly rather than according to the backgrounds of the families, and because the families reside in the same broad economic and social context within each study site, significant differences between research and control groups detected upon following up the families can be assumed to reflect the differences in their experiences associated with assignment to differing research groups.

The National Evaluation of Welfare-to-Work Strategies, currently being conducted by the Manpower Demonstration Research Corporation (MDRC), follows such an experimental research design. The purpose of the NEWWS is to assess the impacts of various types of welfare-to-work strategies under the auspice of the JOBS Program on adult human capital and economic outcomes, including program effects on:

  • employment
  • employment stability
  • earnings
  • total family income (including earnings and benefits)
  • receipt of welfare and other government benefits;
  • government expenditures
  • maternal educational attainment, and
  • maternal reading and math literacy skills.

Findings regarding adult human capital and economic impacts in three of the seven sites in the evaluation have been reported on previously (Hamilton, Brock, Farrell, Friedlander, and Harknett, 1997), and findings on economic impacts in the full set of eleven programs in the seven sites are being released in parallel with the present report (Freedman, Friedlander, Hamilton, Rock, Mitchell, Nudelman, Schweder, and Storto, L., 2000). In addition to the study of program impacts, the NEWWS also includes components examining the implementation of the JOBS program in differing sites (Hamilton and Brock, 1994; Hamilton et al., 1997) and a cost-benefit analysis (see Hamilton et al., 1997).

In the three NEWWS sites in which the Child Outcomes Study is embedded, there are two experimental groups. Each of the experimental groups involves a different program approach within JOBS: the labor force attachment (LFA) approach emphasizes a quick transition into the labor force through job search activities, while the human capital development (HCD) approach emphasizes enhancing welfare recipients' skills through education and training, as a means to obtaining employment at higher wages and with better prospects of advancement. As noted in Chapter 1, these approaches represent differing views on how best to foster economic self-sufficiency in welfare recipients (Hamilton et al., 1997). The labor force attachment approach assumes that participating in the workplace is the best way to learn work behaviors and skills. The human capital development approach assumes that building "human capital," (skills related to employment), is an important step to take prior to employment, that will help assure higher earnings and greater job stability. These three study sites thus use a "planned variation" research design, in which the outcomes of contrasting program approaches can be compared within each of the sites. A more detailed description of the program approaches can be found in the publications reporting on the NEWWS thus far (Freedman et al., 2000; Hamilton and Brock, 1994; Freedman and Friedlander, 1995; Hamilton et al.,1997), and is also provided in Chapter 3 for the particular sites included in the NEWWS Child Outcomes Study.

The NEWWS is being carried out in seven sites across the country. In three of the seven sites (Atlanta, Georgia; Grand Rapids, Michigan; and Riverside, California), families in the evaluation were randomly assigned to one of three research groups: the two program groups (the human capital development group or the labor force attachment group), or to a control group. In a fourth site (Columbus, Ohio), two different forms of case management were contrasted with a control group: an integration of case management for income maintenance and JOBS participation, and case management that focused on these separately. In three further sites (Detroit, Michigan; Oklahoma City and surrounding counties in Oklahoma; and Portland, Oregon), families were randomly assigned to only one of two groups: the site's pre-existing welfare-to-work program or the control group.

Once the income maintenance case worker had reached a decision that a welfare recipient or applicant was not exempt from legislatively mandated JOBS participation, the recipient then received notification to report to a JOBS program orientation. In the Child Outcomes Study sites (Atlanta, Grand Rapids, and Riverside), random assignment to a JOBS program occurred at the orientation. At random assignment, recipients were given a presentation about the evaluation, an assessment of their basic reading and math skills was administered, and they were asked to provide background information. Those who met the criteria for inclusion in the evaluation (noted below) were then randomly assigned to a research group within the evaluation. As we will note below, there was a further step in the random assignment process in the Riverside site than in the other two research sites.

Families were considered eligible for inclusion in the NEWWS when they met the following criteria:

  • They had applied for or were receiving Aid to Families with Dependent Children (AFDC) at the time of enrolling in the evaluation.
  • They were not exempt from participation in the JOBS program, meaning that the recipient was not ill or incapacitated, caring for a household member who was ill or incapacitated, pregnant past the first trimester, or living in an area where program services were unavailable, and did not have a child younger than age three (or age one at state option, an option taken by three states with sites in the NEWWS, but affecting only one of the study sites in the Child Outcomes Study: Grand Rapids).

It is important to note a variation on the random assignment process that occurred only at the Riverside site, and was necessitated by program regulations at the state level. At this site, regulations required that a distinction be made between those deemed "in need of basic education" and those deemed "not in need of basic education." Individuals were considered in need of basic education when they met any one of the following conditions: they (1) did not have a high school diploma or General Educational Development (GED) degree, (2) had a low score (214 or below) on either the reading or math component of the assessment (the GAIN Appraisal test), or (3) required remediation in English. In a further step in the random assignment process in this site only, individuals in need of basic education, and those not in need of basic education were then assigned to different random assignment processes. Those who were considered to be in need of basic education were randomly assigned to any one of the three research groups. However, those considered not in need of basic education could be randomly assigned only to the labor force attachment or control groups (see Hamilton et al., 1997).

As a result, when contrasts of research groups are carried out in the Riverside site, those in the human capital development group are compared to control group members who are likewise considered in need of basic education, whereas members of the labor force attachment group (who could be in need or not in need) are compared to all control group members. The use of a subset of the control group in the Riverside site for comparisons with the human capital development group is apparent in the program impact tables in Chapters 6, 7, and 9. The fact that all families in the human capital development group in Riverside were in need of basic education, whereas this was not the case for the human capital development groups in the other two sites, should be kept in mind when looking across the three sites at the findings for the human capital development program.

For mothers assigned to either the labor force attachment group or the human capital development group, participation in the activities of the JOBS Program was mandatory. That is, the mother was required to participate in JOBS program activities, or she faced the possibility of sanctioning (reduction in welfare benefits). Mothers in the control group, while eligible for Aid to Families with Dependent Children benefits, were not required to participate in any JOBS activities. Control group members were, however, free to seek out education and training programs in their communities at their own volition, and were guaranteed child care while participating in such approved activities, as required by the Family Support Act provisions.

It is important to note that the experimental evaluation of the JOBS program does not focus on the effects of participating in the JOBS program per se. Rather, the experimental evaluation assesses the impact of assignment to a JOBS experimental group, and thus exposure to program messages, services, and mandate to participate. The evaluation carefully documents how many mothers in each experimental group participated in JOBS program activities, and also considers the implications of participation for the major outcomes. However, when each of the experimental groups is contrasted with the control group in the analyses of program impacts, the experimental groups include all those who were assigned to those groups, whether or not they actually participated in the program. That is, the experimental groups include individuals who did not participate in any JOBS activities despite the mandate to participate, and might have been sanctioned as a result, as well as those who did participate.

For all of the sample members in the NEWWS, background information and attitudinal data are available from information collected just prior to random assignment (standard client characteristics and Private Opinion Survey). In addition, immediately prior to random assignment, recipients were given an assessment of reading and math skills. Because these data were collected prior to random assignment, they provide us with "baseline data" about the families, or data unaffected by assignment to a research group. In addition, administrative data are available for each of the families in the sample from county and state Aid to Families with Dependent Children records, and from state unemployment insurance records.

While baseline and administrative data are available for all sample members in the NEWWS, a subset of the full evaluation sample is also participating in two follow-up survey waves: one completed approximately two years after random assignment and another five years after random assignment. The sample for the client survey is a stratified random sample of the full evaluation sample; that is, those participating in the client survey were randomly selected, with certain subgroups systematically oversampled to permit analyses of specific subgroups. Only respondents who spoke English or Spanish, and thus could be interviewed in one of these languages, were included in the client survey sample. Analyses of the client survey data are weighted to permit generalization to the full population of individuals eligible for the NEWWS at each site. The two and five-year follow-up surveys provide maternal response measures on such issues as participation in educational and training activities and perceptions of such activities, educational attainment, employment, earnings, receipt of benefits, and use of child care while the mother is employed.