Impacts on Young Children and Their Families Two Years After Enrollment: Summary Report. Overview

02/06/2004

As we seek to understand the effects on families of the 1996 welfare reform law, we can build on the foundation of a rigorous evaluation study focusing on the effects on families of welfare-to-work programs implemented under the previous welfare law, the Family Support Act of 1988. The National Evaluation of Welfare-to-Work Strategies (the NEWWS, funded by the U.S. Department of Health and Human Services and the U.S. Department of Education) is evaluating the impact of a set of welfare-to-work programs operated under "JOBS" (the Job Opportunities and Basic Skills Training Program). A pioneering feature of this national evaluation is that it simultaneously considers program impacts on adult economic outcomes and on the development and well-being of the children in the families.

This summary report presents a summary of findings from one of a set of three complementary reports from the National Evaluation of Welfare-to-Work Strategies' two-year follow-up (with results from a further follow-up, completed five years after families enrolled, to be presented in the future). We focus in the present summary report on the findings related to impacts on children, reporting results from a special component of the evaluation, the Child Outcomes Study (see McGroder, Zaslow, Moore and LeMenestrel, 2000, for a detailed presentation of findings). This component of the evaluation focuses in depth on children's development and well-being for a sample of families with young (preschool-age) children at the start of the evaluation, drawn from three of the evaluation's seven research sites. A second report in this series focuses primarily on economic impacts in all seven of the evaluation's research sites, with a more limited examination of impacts on children of all ages (Freedman, Friedlander, Hamilton, Rock, Mitchell, Nudelman, Schweder, and Storto, 2000). A third report draws together findings on children from the in-depth look at young children in the Child Outcomes Study, and brief markers of well-being collected regarding children of all ages in families in the full evaluation sample (Hamilton, with Freedman and McGroder, 2000).

The Child Outcomes Study of the National Evaluation of Welfare-to-Work Strategies examines the impacts on both the parental and child generations of two distinct approaches to welfare reform implemented as part of the federal JOBS Program: a labor force attachment approach (emphasizing a rapid transition to employment), and a human capital development approach (emphasizing a longer-term strategy of education and training in order to obtain a better job). These strategies are precursors of the welfare reform programs now being implemented under the 1996 welfare law, the Personal Responsibility and Work Opportunity Reconciliation Act (PRWORA). The labor force attachment approach under JOBS is especially germane because of its emphasis on moving clients quickly into employment, the clear priority of new policies. However, the human capital development approach may provide an informative model for states as caseloads drop, and those families remaining on welfare face more barriers to employment (such as low literacy or limited education).

Although welfare policies were initiated many years ago with the aim of protecting children in poor families, most of the evaluation research concerning these policies has focused on adult economic outcomes. This is perhaps not surprising, given that the most clearly targeted outcomes of these programs have been economic. The Family Support Act explicitly stated as its goal the reduction of long-term welfare dependency. Further, this law did not call for services aimed directly at enhancing the development of children (such as early childhood educational intervention, or developmental screening); rather authorized services focused on increasing adult employment.

Nevertheless, a mother's assignment to a welfare-to-work program has the potential to affect the development of children, for example, by affecting the material resources available within the family, and by affecting children's experiences of care both within and outside of the home. The Child Outcomes Study of the National Evaluation of Welfare-to-Work Strategies examines whether children can be affected by their mothers' assignment to a welfare-to-work program, how their development and well-being are affected (favorably or unfavorably), if at all, and in what ways any impacts on children come about.

The Child Outcomes Study uses a rigorous experimental design. Two years after mothers were randomly assigned to one of the two JOBS welfare-to-work strategies or to a control group, outcomes for children (at that point between about 5 and 7 years of age) were examined. The children's cognitive development and academic achievement were measured through a combination of direct assessment (an assessment of the children's cognitive school readiness) and maternal report (for example, mothers' reports of academic problems). The children's behavioral and emotional adjustment were measured through maternal report (for example, using measures of the child's behavior problems and positive social behaviors). The children's health and safety were also measured through maternal report (for example, using an interview measure indicating whether the child has had an accident, injury or poisoning requiring emergency medical attention; an interview measure widely used in national surveys in which the mother indicates whether she sees the child's overall health as excellent, very good, good, fair, or poor). Validation work indicates that the global health rating reflects primarily physical health problems (Krause and Jay, 1994).In general, all measures selected for use in the Child Outcomes Study have demonstrated sufficient validity and reliability (Bracken, 1984; Polit, 1996; Zill, 1985; Peterson and ZILL, 1986)

In addition to examining mean scores on measures of cognitive school readiness, problem and positive behavior, and overall health, we also examined program impacts on the proportion of children with extreme scores on these measures in the interest of ascertaining whether JOBS welfare-to-work programs changed the distribution of children's outcomes  for example, reducing the proportion at the "unfavorable" end and/or increasing the proportion at the "favorable" end  which is possible even if the programs had no impacts on mean scores. Thus, in some cases a single response to a survey question can give rise to two or more impacts (e.g., one relating to the mean and one relating to the distribution).

The Child Outcomes Study was conducted in three sites: Atlanta, Georgia; Grand Rapids, Michigan; and Riverside, California. Families enrolled in the evaluation between September 1991 and January 1994. Data collection for the two-year follow-up was completed in January 1996, prior to the implementation of the new welfare law. Findings from this study must be seen in light of the fact that mothers were exempt from participation in JOBS welfare-to-work activities if they were needed at home to care for an ill or incapacitated family member, including a child. As a result, children with a health condition requiring such care were not included in the evaluation. The 1996 welfare law no longer provides an explicit exemption for a mother with an ill or incapacitated child.

In this experimental evaluation, mothers randomly assigned to a control group in each of the study sites were eligible for all welfare benefits. However, they did not receive the special messages and case management of a JOBS program, they were not mandated to participate in JOBS program activities, and they did not have access to the particular work preparation activities through JOBS. Control group members were eligible for child care assistance, similar to that offered to program group members, if they were participating in work preparation activities in which they had enrolled on their own.

The summary report that follows this overview and the full report of the two-year follow-up of the Child Outcomes Study provide extensive information on the impacts of the six JOBS programs examined here (i.e., the human capital development and labor force attachment approaches in each of the three study sites) on economic outcomes in these families with young children, as well as on such further measures of family functioning as marital status, father involvement as reported by mothers, maternal psychological well-being and parenting. In this overview, we focus specifically on the findings regarding impacts on children.

Overall, the six JOBS programs examined had relatively few statistically significant impacts on children on average, that is, across all families assigned to a given program. Nevertheless, there were more impacts than one would expect due to chance. In addition, findings differed according to the aspect of the children's development examined, with impacts in the area of cognitive development favorable, in the area of health unfavorable, and in the area of behavior mixed (including both favorable and unfavorable impacts).

Child impact findings at the aggregate level tended to differ according to site. Whereas impacts on child outcomes were relatively few, they tended to be favorable in Atlanta, unfavorable in Riverside, and mixed in Grand Rapids. Where impacts were found, they were generally small. Only a single impact(1) was of sufficient magnitude to be called "policy relevant." Policy relevant findings in the present study are those that meet the criterion, set at the start of the study, of having an effect size of at least a third of a standard deviation.(2) The strongest evidence on which to base conclusions about impacts on children is a consistent patterning of impact results, particularly when impacts meet or exceed the criterion for policy relevance.

In addition to examining impacts for the sample as a whole in each site, analyses also addressed the question of whether findings differed for children from families with different background characteristics. There was an initial concern that children in families at higher risk (in terms of such background characteristics as the mother's education, her work history, number and spacing of children, and maternal psychological well-being) might show a pattern of unfavorable program impacts. Mothers in higher-risk families might be less able to fulfill the requirement to participate in a welfare-to-work program, or might experience substantial stress in doing so. Children in higher-risk families might be more vulnerable to changes in family routines and their own care situations. Yet on the other hand, JOBS programs might provide exactly the kind of support the higher-risk families need in order make progress toward economic self-sufficiency. The combination of messages, enhanced case management, program requirements, and services might be particularly effective for such families. If this is indeed the case, then children in higher-risk families might be particularly likely to show favorable impacts of one or both JOBS program approaches.

As for the findings for children across all families, statistically significant impacts for subgroups of families were relatively few, though exceeding what might have been expected on the basis of chance. Findings for children in higher-risk families, when they emerged, tended to be favorable, though small, when the mother had been assigned to a human capital development program or to Atlanta's labor force attachment program(3). By contrast, program impacts, some of which reached the criterion for policy relevance, tended to be unfavorable for children from higher-risk families in the other two labor force attachment programs.

Contrary to the initial hypothesis, it was among lower-risk families that there were indications of some concentrations of unfavorable impacts for children, and these did not appear to vary by program approach. Rather, a pattern of unfavorable impacts occurred for children in three particular programs: Grand Rapids' labor force attachment program, and both of Riverside's programs. In addition, a number of the program impacts for children in lower-risk families met the criterion for policy relevance, and nearly all of these were unfavorable impacts. Further, many (one-third) of the unfavorable and policy relevant impacts found for children in lower-risk families substantially exceeded the threshold for policy relevance, in that effect sizes were .50 or larger (which is considered "moderate" to "large" in magnitude; Cohen, 1988).

Further analyses were conducted to begin to explicate the processes underlying program impacts on children. While the analyses of program impacts on children summarized above are experimental in nature, the analyses used regarding the processes underlying program impacts on children, mediational analyses, are non-experimental. Whereas experimental analyses provide strong evidence regarding the existence of program impacts on children, they cannot address the question of how these impacts came about. Because mediational analyses are non-experimental, they do not allow firm causal inferences to be made regarding the pathways through which these impacts come about. Nevertheless, if one hopes to gain insight into the processes through which a given program had impacts, then non-experimental statistical analyses are necessary.

Mediational analyses examined five selected program impacts on children in greater depth. The particular child impact findings focused upon in this way were selected in keeping with the overall patterning of findings of favorable impacts for children in the area of cognitive development, unfavorable health impacts, and mixed behavioral impacts. These analyses rely on information about processes and child outcomes measured contemporaneously and used a relatively modest statistical approach in modeling pathways; as such, results should be considered preliminary. Additional waves of data from the five-year follow-up, as well as statistical methods that more effectively partition effects among multiple mediators, more completely control for selection effects, and allow alternative models to be tested explicitly (e.g., structural equation modeling), are needed to provide more definitive answers regarding the multiple pathways through which program impacts on children came about.

These analyses suggest that the mechanisms through which children can be affected by a given welfare-to-work program include both outcomes that were directly targeted (e.g., employment), as well as outcomes not directly targeted by the programs (e.g., parenting, maternal psychological well-being). In particular, though, the findings highlight the role played by mechanisms more proximal to the child, especially maternal psychological well-being and parenting. As yet, the roles of child care, health insurance, and income in mediating the five child impacts selected for pathways analyses have not been identified, although it should not be concluded that these factors do not in general affect outcomes for children. Findings also indicate that welfare-to-work programs sometimes have effects on aspects of family life that are important to children that go in opposing directions (favorable and unfavorable). For example, Atlanta's labor force attachment program increased mothers' sense of time stress (which was associated with less favorable behavioral outcomes in children), but simultaneously led to improvements in parenting (which was associated with more favorable behavioral outcomes in children). Impacts on children reflect the net effect of such influences. Opposing influences of this kind on key aspects of family life may help explain the small number and size of significant impacts on child outcomes. Understanding the nature and direction of multiple influences on family life is central to strengthening pathways that yield favorable impacts on children and weakening the pathways that have detrimental implications for children.

In sum, the results indicate that the welfare-to-work programs implemented as part of the JOBS Program did have significant impacts on children's developmental outcomes, but these impacts were not widespread and were generally small. When impacts did occur, they were favorable in the area of the children's cognitive development and academic achievement, unfavorable in the area of the children's health and safety, and mixed in the area of behavioral and emotional adjustment. Looking at subgroups of families, there was a pattern of favorable impacts for children from higher-risk families assigned to human capital development programs or to the Atlanta labor force attachment program. Yet, at the same time, there was a concentration of unfavorable, and policy relevant, impacts for children from lower-risk families in three of the programs studied.

Given these findings, we must consider it a possibility that the welfare-to-work programs implemented under the 1996 law can have impacts  both favorable and unfavorable  on children's development. It is indeed possible that the intensified obligations and opportunities of the new policy are resulting in stronger or more pervasive impacts on children than were found for the JOBS Program. Furthermore, the variations in impacts suggest the value of research that would examine other policy strategies, for example, child care, health care, and other work supports, to determine how to promote favorable, and avoid unfavorable, child impacts. Thus, it is important to continue to measure the influences of welfare-to-work programs and other policy strategies on both the parent and child generations.

References

Bracken, B. A. (1984). Bracken Basic Concept Scale: Examiner's Manual. The Psychological Corporation, Harcourt Brace Jovanovich, Inc.

Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd edition). Hillsdale, NJ: Erlbaum and Associates.

Freedman, S., Friedlander, D., Hamilton, G., Rock, J., Mitchell, M., Nudelman, J., Schweder, A., and Storto, L. (2000). Evaluating alternative welfare-to-work approaches: Two-year impacts for eleven programs. Washington, D.C.: U.S. Department of Health and Human Services, Administration for Children and Families and Office of the Assistant Secretary for Planning and Evaluation, and U.S. Department of Education.

Hamilton, G., with Freedman, S., and McGroder, S. (2000). Do mandatory welfare-to-work programs affect the well-being of children? A synthesis of child research conducted as part of the National Evaluation of Welfare-to-Work Strategies. Washington, D.C.: U.S. Department of Health and Human Services, Administration for Children and Families and Office of the Assistant Secretary for Planning and Evaluation, and U.S. Department of Education.

Krause, N. M. and Jay, G.M. (1994). What do global self-rated health items measure? Medical Care, 32 (9), 930-942.

McGroder, S.M., Zaslow, M.J., Moore, K.A., and LeMenestrel, S.M. (2000). The National Evaluation of Welfare-to-Work Strategies: Impacts on young children and their families two years after enrollment: Findings from the Child Outcomes Study. Washington, D.C.: U.S. Department of Health and Human Services, Administration for Children and Families and Office of the Assistant Secretary for Planning and Evaluation, and U.S. Department of Education.

Peterson, J., and Zill, N. (1986). Marital disruption and behavior problems in children. Journal of Marriage and Family, 48, 295-307.

Polit, D. F. (1996). Self administered teacher questionnaire in the New Chance 42-month survey. New York: Manpower Demonstration Research Corporation.

Zill, N. (1985). Behavior Problem Index. Washington, D.C.: Child Trends, Inc.