1 This synthesis is one of many papers and reports that have been issued as part of the NEWWS Evaluation, conducted by the Manpower Demonstration Research Corporation (MDRC) under contract to the U.S. Department of Health and Human Services (HHS), with support from the U.S. Department of Education. Child Trends, as a subcontractor, is conducting the analysis of outcomes for preschool-age children (the Child Outcomes Study). This document synthesizes results presented in two 2000 reports, both published by the U.S. Department of Health and Human Services (Administration for Children and Families and Office of the Assistant Secretary for Planning and Evaluation) and the U.S. Department of Education: Impacts on Young Children and Their Families Two Years After Enrollment: Findings from the Child Outcomes Study, prepared by Sharon M. McGroder, Martha J. Zaslow, Kristin A. Moore, and Suzanne M. LeMenestrel (Child Trends); and Evaluating Alternative Welfare-to-Work Approaches: Two-Year Impacts for Eleven Programs, prepared by Stephen Freedman, Daniel Friedlander, Gayle Hamilton, JoAnn Rock, Marisa Mitchell, Jodi Nudelman, Amanda Schweder, and Laura Storto (MDRC). This synthesis is very much indebted to the authors of these reports, who conducted the many analyses included here. In addition, certain concepts and language in this document reflect the joint efforts of the two research teams. Note, however, that because of differences in the data sources used for economic impacts and differences in the definition of the sample of children examined, which are explained in detail later in this document, some statements regarding economic impacts and young child impacts may differ somewhat between this synthesis and the above-cited report on impacts on young children.
2 Sample members were randomly assigned to research groups over approximately a two-year period in each site, starting in June 1991 in Riverside and ending in December 1994 in Portland. Thus, the results presented here cover the period between June 1991 (the first sample members' entry into the study) and December 1996 (the last month of the two-year follow-up for the last sample members who entered the study in Portland).
3 See Zaslow et al., 1998, and McGroder et al., 2000, for further discussion of these hypotheses.
4 See, for example, Wilson, Ellwood, and Brooks-Gunn, 1995; Zaslow et al., 1995; Zaslow et al., 1998; and McGroder et al., 2000. The conceptual model used here was developed by Child Trends early in the NEWWS Evaluation, and has undergone refinements over subsequent years.
5 See Haveman and Wolfe, 1995.
6 The primary sources for the pathways in this model are correlational studies of the relationships between income, employment, child care, and child outcomes; previous work by MDRC and others on interventions and their effects on income and employment; and the underlying theories about how welfare-to-work or employment policies might affect children. The model depicts the effects of parent actions on children, but not the effects of children on parents. This is likely an overly linear representation of the relationships between parents and children.
7 The nontargeted outcomes in box D include those that theoretically could affect child outcomes. Measurement of some of these nontargeted outcomes is difficult, if not impossible; as a result, not all of them were examined in the NEWWS Evaluation.
8 A subsample of these interviewees also took adult reading and math achievement tests.
9 These additional data were actually collected for 3,194 surveyed individuals, but 176 of the families were deemed inappropriate, for various reasons, for the in-depth Child Outcomes Study, resulting in a sample size of 3,018.
10 Additional data already collected include in-depth information on the implementation of the programs the extent of individuals' participation in various program activities, the quality of the activities, and staff practices and opinions as well as on the costs of different aspects of welfare-to-work programs.
11 Note that parents with a severely ill or disabled child were generally not mandated to participate in welfare-to-work programs in the early to mid 1990s. While the proportion of families exempted from the participation requirement for this reason during this time period was very small, such families would not have been included in the samples examined in the NEWWS Evaluation.
12 The Oklahoma City sample included only welfare applicants (and not recipients), who were thus more likely to have more prior work experience and less prior welfare receipt.
13 Tests of statistical significance were not conducted on the differences in sample characteristics discussed in this paragraph.
14 In this section, as well as the following section on targeted outcomes, the benchmarks for characterizing the magnitude of program impacts are based on ranges of impact findings from previous experimental evaluations of welfare-to-work programs. The specific thresholds vary by outcome. Impacts on participation of 20 percentage points or more are considered "large," impacts of 10 to 20 points are "moderate," and impacts below 10 points are "small." For measures of high school diploma or GED attainment, welfare expenditures, and employment, impacts of 10 percentage points or more are considered "large," impacts of 5 to 10 points are "moderate," and impacts below 5 points are "small." Impacts on earnings or income of more than $900 per year (or $1,800 over two years) are considered "large," impacts of $300 to $900 per year (or double these amounts over two years) are "moderate," and impacts of less than $300 per year are "small."
15 Length of stay in the programs, and thus participation in program activities, was shorter than in voluntary programs, but not shorter than in typical mandatory welfare-to-work programs. This reflects the fact that people cycle off welfare frequently. In the studied programs, most people did not remain on welfare continuously, and thus most were not subject to the program participation requirements, for the full two years of follow-up. In the seven programs for which detailed participation statistics have been published so far, the length of time during the two-year follow-up period that program group members spent enrolled in the programs ranged, on average, from 11 months in the Riverside LFA and HCD programs to 17 months in the Atlanta HCD program. The average number of months of participation in program activities for only those who ever participated at all ranged from 3 months in the Riverside LFA program to 9 months in the Atlanta HCD program. (See Hamilton et al., 1997, and Scrivener et al., 1998.)
16 This section focuses on economic impacts as measured by the administrative records databases described in Section II. McGroder et al., 2000, Chapter 9, in presenting economic impacts for the adults in the Child Outcomes Study, primarily used client survey data. Each of these data sources has its advantages and disadvantages. Measures of income using administrative records data, for example, cover the entire two-year follow-up period, but include income only from cash welfare, Food Stamps, and Unemployment Insurance-reported earnings. Measures of income using client survey data include many more potential sources of income, but cover only one month (the last month of the two-year follow-up period). As a result of these differences, some summary statements about economic impacts in McGroder et al., 2000, differ somewhat from those in this document.
17 Impacts could differ for the samples for several reasons. First, the client survey sample includes individuals from all seven evaluation sites, while the COS sample was drawn from only three of the sites. Second, even within the three sites in which the COS is nested, there are demographic differences between sample members with preschool-age children and those with older or even younger children, as discussed in Section IV. Finally, in general, different data sources and response rates can produce different impacts for the client survey sample compared with the full sample. See Appendix E in Freedman et al., 2000, for a discussion of the reliability and generalizability of results based on the client survey and Appendix F for a comparison of impacts estimated from survey and unemployment insurance data.
18 As noted earlier, the two-year follow-up period for many of the programs predated significant expansions in the Medicaid program and the establishment of the Children's Health Insurance Program (CHIP).
19 Both access to and allowable payments for child care were the same for control and program group members in the NEWWS evaluation. If a control group member enrolled on her own in a community education or training activity or become employed, she would have been entitled to the same type of child care assistance that a program group member would have received. If a state would only pay for licensed child care, for example, then only this type of care would be paid for either control or program group members. Program group members, however, probably heard messages about the importance of child care and the advantages of particular types of child care more frequently than did control group members, owing to program group members' increased exposure to caseworkers and other program-related staff.
20 For more information on the methods of making child care payments, reimbursement rates, and child care allowances to assist welfare recipients who had earned income in the Atlanta, Grand Rapids, Riverside, and Portland sites, see Hamilton et al., 1997, and Scrivener et al., 1998.
21 For example, parents with no school-age children would have to respond "no" when asked if any of their children had been suspended from school. As mentioned earlier, approximately 49 percent of the parents in the client survey sample had all school-age children.
22 In presenting results for young children, this section discusses impacts on the "focal" child in the COS families in the three sites in which this study was embedded, not impacts on the focal child's siblings, most of whom were older. Here, impacts on these siblings are included with those of all school-age children or with those of children of all ages across the seven evaluation sites (depending on the child outcome measure). McGroder et al., 2000, presents child impacts for the focal child as well as the focal child's siblings. Summary statements of child impacts in that report often combine all children in the COS families and thus may differ somewhat from the "young child" summary statements in this document.
23 This does not necessarily mean that the child was placed in the foster care system; an unruly teenager, for example, could have been sent to live temporarily with a relative.
24 It is not clear what might have led to this child impact, the only one found in Oklahoma City. For families with all school-age children in the Oklahoma City sample, there were no impacts on two-year employment, earnings, or income. There was, however, a decrease in reported health care coverage, a decrease in the proportion of parents married and living with their spouse, and an increase in the proportion of parents who were single and living only with their children. (See Tables 2 and 3.) There are no clear connections between these findings and the increase in children's receiving help for behavioral or emotional problems.
25 The results of two quite conservative tests the Tippet test and the Fisher test, developed in the literature on research synthesis (Cooper and Hedges, 1994) indicate that at least one of these two impacts is, in fact, statistically significant in view of the large number of programs examined.
26 At a 42-month follow-up point, New Chance had increased the proportion of mothers living without any of their children as well as without at least one of their children. In addition, a higher proportion of children were in foster care. (See Quint et al., 1997, pp. 138-144.) New Chance researchers could not identify a clear reason for this finding, but hypothesize that increased exposure to program staff, resulting in a higher likelihood of identifying child welfare issues among program group mothers than control group mothers, might have played a role. In addition, program group mothers were more likely to have moved out of their parents' home, a situation that might have placed them more at risk for child neglect if there were no older family members around to help watch the children. Finally, the program did increase maternal depression, and this may have increased childrearing problems.
27 It is possible that the provision of "on-site" child care centers in these programs, where mothers could drop off their children, played a role in this finding. In both Grand Rapids and Columbus, some of the providers of job club and education activities (e.g., the public school system in Grand Rapids) operated such child care centers in the same building as the activities took place, staffed by well-trained child care workers, that mothers could use while they were participating in job club or attending education classes. Program administrators in Grand Rapids remember at least a few cases where child care workers from these centers, knowing that most of the mothers of the children under their care were welfare recipients, alerted the welfare department to possible cases of child neglect or abuse. Although these centers would have been open to individuals in the Grand Rapids LFA as well as HCD programs, and to those in the Columbus Integrated as well as Traditional programs, length of stay in the programs (and on welfare) was longer in the Grand Rapids HCD program than in the LFA program, and longer in the Columbus Traditional program than in the Integrated program, thus increasing the length of time that children could have been in these child care centers and thereby the chances that a case of neglect or abuse would be identified and brought to the attention of the welfare department. The impacts on child removal from the home, however, are larger in the Grand Rapids HCD and Columbus Traditional samples that consist of families with all school-age children (who would have had less use of these child care centers), compared with the samples that consist of families with children of all ages, suggesting that while this particular hypothesis may explain part of these impacts, it is not a complete explanation.
28 Note that statistical tests were not applied to the within-site differences in child impacts for the employment-focused programs as compared with the education-focused programs in this analysis. In addition, this analysis compared, for most of the subgroups in the Riverside site, individuals in the education-focused program all of whom lacked a high school diploma or GED, had low literacy levels, or had limited English skills with individuals in the employment-focused program, who may or may not have had these credentials or skills.
29 See Freedman et al., 2000, p. 193.
30 U.S. Department of Education, 1997.
31 U.S. Department of Education, 1997.
32 As noted earlier, this does not necessarily mean that the child was placed in the foster care system.
33 Vital and Health Statistics: Health of Our Nation's Children, 1994.