Impact on Young Children and Their Families 2-Years After Enrollment: Methods: How Did We Study Impacts on Children? . Design of the NEWWS Child Outcomes Study


The JOBS program departed from earlier welfare-to-work programs in that it mandated the participation of parents with children as young as age three (or younger at state option). Previous welfare-to-work programs were often voluntary, and had focused their attention on mothers with school-age children. In the context of the JOBS Program, preschool-age children were expected to be particularly likely to experience changes in their daily routines and child care situations. The NEWWS Child Outcomes Study, being carried out by Child Trends under subcontract to the Manpower Demonstration Research Corporation, was launched as a special substudy within the larger NEWWS, in order to study whether and how the development of preschool-age children was affected over time when their mothers were assigned to a JOBS program.

As noted in Chapter 1, there are reasonable bases for hypothesizing quite divergent program impacts for children (ranging from negative impacts, to neutral impacts, to positive impacts, or to impacts only for specified subgroups). Thus, the NEWWS Child Outcomes Study does not begin with a specific hypothesis about the direction of effects on children but rather seeks to document the full range of impacts both in the aggregate and for specified subgroups. At the same time, a priority is placed in our examination of the impacts for children on assessing whether the JOBS Program had unfavorable impacts on children (the "harm hypothesis"). For policy makers, two important bases for assessing the impacts of the JOBS Program are whether it had positive effects on family economic self-sufficiency and, at a minimum, did not harm children.

In the present study, we report all program impacts on children that are statistically significant. These program impacts are reliable: they are very unlikely to have occurred just on the basis of chance. As such, these program impacts warrant continued monitoring. In the Child Outcomes Study, we will want especially to monitor whether the kinds of measures on which statistically significant impacts were found at the two-year point continue to show differences at the final follow-up (five years after the families enrolled in the evaluation), and if such differences grow in magnitude.

We also report on whether a statistically significant result meets a further criterion: that of "policy relevance." At the start of the study and as the study proceeded, researchers and policy makers met to grapple with the question of the point at which child impact findings should be taken into account in considerations about policy. A decision was made that statistically significant findings that were of a particular magnitude should be considered relevant to policy discussions: specifically, statistically significant child impact findings of a third of a standard deviation.

This threshold sets aside impact findings that are so small that, while they are reliable statistically and warrant continued monitoring over time, may at this point in time have limited importance in terms of children's development. At the same time, the threshold for policy relevance does not require that an impact be large in magnitude (1) in order to meet the criterion. By setting the threshold in this way, we can be reasonably confident that we are being inclusive in identifying instances of possible harm (as well as of possible beneficial effects on children), without focusing on effects that are so small as to be of limited importance for children's development.

In presenting results, we go beyond consideration of significant and policy relevant effects to discuss the patterning of findings. We also identify those impacts for which effect sizes substantially exceeded the threshold for policy relevance, in that effect sizes were .50 or larger. The strongest evidence on which to base conclusions about impacts on children is a consistent patterning of impact results, particularly when impacts meet or exceed the criterion for policy relevance. A patterning of results, for example, might show consistently favorable impacts for families in a particular site, or a particular program approach. A patterning of results might also pertain to a type of child outcome, with findings in one aspect of development (such as health) consistently affected favorably (or unfavorably) across programs.

The NEWWS Child Outcomes Study is being carried out in three of the seven sites of the full evaluation. These sites -- Atlanta (Fulton County), Georgia; Grand Rapids (Kent County), Michigan; and Riverside (Riverside County), California -- were chosen on the grounds that they each conducted at least one round of random assignment at the JOBS office, involve a contrast of all three research groups (labor force attachment, human capital development and control groups), and permit an examination of the JOBS Program as implemented in differing regions of the country (with differing populations and differing economic, as well as social, contexts). Chapter 3 includes a discussion of the site characteristics and a description of how the JOBS Program was implemented in each of the three sites of the NEWWS Child Outcomes Study.

The NEWWS Child Outcomes Study is "embedded" within the larger NEWWS; that is, each of the families in the Child Outcomes Study completed the procedures of the full evaluation, including the component of the full evaluation that involved collection of survey data. Thus, we have baseline data, administrative data, and the two-year follow-up survey data from the full evaluation for these families, and we will eventually have five-year follow-up surveys as well.

For the families participating in the NEWWS Child Outcomes Study, the two and five-year follow-up surveys are more extensive than for other families in the survey sample, including extra sections focusing on the development of the child and on aspects of family life and child care that may be important to child outcomes. In addition, families in the NEWWS Child Outcomes Study are asked at the time of the five-year follow-up for their permission to contact the focal child's primary teacher, in order to ask that the teacher complete a mailed questionnaire concerning the child's academic progress and behavior in school (the "Children's School Progress Survey").

In order to be eligible for inclusion in the NEWWS Child Outcomes Study, families participating in the NEWWS in the Atlanta, Grand Rapids, and Riverside evaluation sites had to meet these additional criteria:

  • Each family had to have a child between about 3 and 5 years at the time of enrollment. In each family a child of between about 3 and 5 served as the focal child, or child focused upon in the evaluation. If the family had more than one child in this age range, one was selected randomly to be the focal child.
  • Single fathers were excluded.

In all, 5,905 families were identified as eligible for inclusion in the NEWWS Child Outcomes Study. Of these, 3,670 families were selected to be interviewed for the two-year follow-up. Overall, a total of 3,194 (or 87 percent) of selected families completed the two-year follow-up survey, with response rates ranging from 80 percent (in Riverside) to 91 percent (in Atlanta and Grand Rapids).

Four further criteria were established in order for families to be included in the analyses of the two-year follow-up data for the Child Outcomes Study:

  • The focal child had to be the biological or adoptive child of the mother participating in the evaluation.(2)
  • In order to examine developmental outcomes for children within a particular age range, we excluded families from the sample in which the focal child was beyond 99 months (i.e., eight years, three months) of age at the time of the two-year follow-up.(3)
  • Families were not included in the two-year follow-up study if the interviewer would have had to travel 100 miles or more in order to complete the two-year follow-up interview in the family's home. Instead, telephone (rather than in-home) interviews were conducted with families who had moved 100 mile or more away, when the families could be located for such interviews. However, these telephone interviews included only the "core" sections of the interview and did not include the sections specific to the Child Outcomes Study. Because these additional sections (including the assessment of the child's cognitive development) are essential to the examination of child impacts, the Child Outcomes Study analysis sample was restricted to those with in-home interviews including the Child Outcomes Study modules.(4)
  • Families were not included in the two-year follow-up Child Outcomes Study data analyses if the interview data indicated that the mother and child had not seen each other for the last three months or more. The mother was the source of some of the child outcome measures and also reported on activities that she engaged in with the child (for example, reading with the child). A separation of three months or more would limit the basis on which the mother could make her assessments of child behavior and report on her involvement in activities with the child.(5)

A total of 176 (or 5.5 percent of) respondents to the two-year follow-up survey were dropped from the Child Outcomes Study analysis sample. Thus, the sample for the present analyses of the NEWWS Child Outcomes Study includes a total of 3,018 families. Of these families, 1,422 are from the Atlanta site of the evaluation, 646 are from the Grand Rapids site, and 950 are from the Riverside site. Chapter 4 describes the characteristics of these families at the time they entered the NEWWS.(6)

For the present report, which focuses on child outcomes at the time of the two-year follow-up, we will rely upon data from the following sources:

  • Baseline data: the information on standard client characteristics (collected by welfare staff during routine intake interviews with clients prior to random assignment, including information on such issues as AFDC history, educational background, marital status); the Private Opinion Survey (a client-completed survey of opinions, attitudes, and psychological well-being); and direct assessments of maternal reading and math literacy.
  • Administrative data: automated county and state administrative records provide data on earnings, employment, and welfare receipt.
  • Two-year follow-up survey: Participants in the NEWWS Child Outcomes Survey received both the "core" interview given to all those receiving the two-year follow-up client survey in the full evaluation sample, and a 20-minute interview specific to the Child Outcomes Study.

Core component. The core interview provides us with maternal report measures of participation in education and training programs, educational attainment, employment, earnings, benefits, and child care use while the mother was employed. Some measures in the core interview also concern the well-being of all of the children in the family. Respondents who were determined at baseline to be in need of education, and who were in the human capital development group or the control group, completed math and literacy tests as part of the two-year follow-up as well.

Child Outcomes Study component. The component of the interview specific to the Child Outcomes Study is the source of maternal report measures of the focal child's health and social development, as well as of a direct assessment of the focal child's cognitive school readiness. The specific child outcome measures are described below. This component of the survey also provides maternal report and interviewer rating measures of the home environment and of the mother-child relationship, and maternal report measures of the focal child's child care participation, the mother's psychological well-being, household composition, and the receipt of child support and the child's contact with the father.

While the present report focuses on outcomes two years after random assignment in all three of the sites included in the NEWWS Child Outcomes Study, we note that two special studies have been conducted that involve a subset of the Child Outcomes Study sample specifically in the Atlanta site. In order to provide a descriptive portrayal of families with young children close to the start of the evaluation, 790 families in the Atlanta site participated in an additional wave of data collection called the Descriptive Survey, on average three months after baseline. A descriptive account of these families and of the children's development shortly after the start of the evaluation is presented in a report entitled How Well Are They Faring? AFDC Families with Preschool-Aged Children in Atlanta at the Outset of the JOBS Evaluation (Moore, Zaslow, Coiro, Miller, and Magenheim, 1995).

A second special study, the JOBS Observational Study, is also being conducted in the Atlanta site among a subsample of families from the Descriptive Study. This study is supported by a consortium of private and public funders including the Foundation for Child Development, the William T. Grant Foundation, the George Gund Foundation, an anonymous funder, and (for pretest work only) the U.S. Department of Health and Human Services. The study seeks to provide detailed and sensitive measures of mother-child interaction at two points in time: soon after baseline (4-6 months), and a period of years after baseline, when longer-term effects of the program can be assumed to have occurred (4 ½ years after baseline). The goals of this study are to ask whether the JOBS program affects mother-child interaction during the early months of the program, and at a point in time when longer-term adaptations to the program have been made; to examine the role of parenting behavior in shaping any impacts of JOBS on children's development; and to assess the contributions of different approaches to measuring parenting behavior (observational; interview-based) in the context of an evaluation study.