Do Mandatory Welfare-to-Work Programs Affect the Well-Being of Children?

Prepared for:
U.S. Department of Health and Human Services
Administration for Children and Families
Office of the Assistant Secretary for Planning and Evaluation

U.S. Department of Education
Office of the Under Secretary
Office of Vocational and Adult Education

Prepared by:
Gayle Hamilton
with
Stephen Freedman
Manpower Demonstration Research Corporation
and
Sharon M. McGroder
Child Trends

 

This document was prepared as part of the National Evaluation of Welfare-to-Work Strategies (NEWWS).  The Manpower Demonstration Research Corporation (MDRC) is conducting the NEWWS Evaluation under a contract with the U.S. Department of Health and Human Services (HHS), funded by HHS under a competitive award, Contract No. HHS-100-89-0030.  Child Trends, as a subcontractor, is conducting the analyses of outcomes for young children (the Child Outcomes Study).  HHS is also receiving funding for the evaluation from the U.S. Department of Education.  The study of one of the sites in the evaluation, Riverside County (California), is also conducted under a contract from the California Department of Social Services (CDSS).  CDSS, in turn, is receiving funding from the California State Job Training Coordinating Council, the California Department of Education, HHS, and the Ford Foundation. The findings and conclusions presented herein do not necessarily represent the official positions or policies of the funders.

"

Acknowledgments

This synthesis is based on the work of two teams of researchers, one at MDRC and one at Child Trends, and this document owes much to them. The productive collaboration among these research teams and their affiliate organizations was instrumental in bringing to fruition this study of the effects of mandatory welfare-to-work programs on children.

Gratitude is also due to the people in the NEWWS Evaluation research samples. These people shared detailed information about themselves and their children, thoughtfully completed batteries of tests and indices, and, in many cases, opened their homes to enable researchers to obtain particularly sensitive information and directly assess children's well-being.

Finally, the ongoing cooperation of welfare department staff and administrators in the evaluation's seven sites has been indispensable. Their willingness to subject their programs to rigorous evaluation requiring an elaborate research design and detailed data has been of crucial importance throughout the NEWWS Evaluation. Welfare policymakers will reap the benefits of their commitment to discovering the true effects of various welfare program approaches for many years to come.

Introduction

Since its inception the primary goal of the Aid to Families with Dependent Children (AFDC) program, as well as successor programs funded under Temporary Assistance for Needy Families (TANF), has been to provide government support for poor children. Over the years, this public assistance has become more and more predicated on custodial parents' involvement in work or mandatory welfare-to-work program activities, as policymakers have sought to balance the goal of fostering poor children's well-being with that of encouraging adults' self-sufficiency. Currently, there are strong incentives for states to run mandatory, work-focused welfare-to-work programs: States face financial penalties if they fail to meet TANF-defined participation standards, which require large proportions of welfare recipients to be working or in work-related activities, and states must require recipients to work after two years of assistance. In addition, federal funds now may not be used to support most families on welfare for longer than five years, and a number of states and localities have shorter welfare time limits.

This document examines the effects of welfare-to-work programs on the children of the adults (almost all single mothers) mandated to participate in such programs. Synthesizing the results from two recently completed reports from a large-scale evaluation  the National Evaluation of Welfare-to-Work Strategies (NEWWS Evaluation)  the two-year effects of 11 welfare-to-work programs that operated in seven sites in the early to mid 1990s are summarized.⁽¹⁾ The sites included in the evaluation are Atlanta, Georgia; Columbus, Ohio; Detroit and Grand Rapids, Michigan; Oklahoma City, Oklahoma; Portland, Oregon; and Riverside, California. While the programs operated under the federal Job Opportunities and Basic Skills Training (JOBS) program that preceded TANF, and thus did not invoke a time limit on eligibility for welfare, they shared TANF's primary goal of moving welfare recipients into paid work and off assistance, and they reflect a range of approaches, implementation features, and environments: Some were strongly employment-focused while others emphasized basic education; they varied in how broadly the program participation mandate was applied to the welfare caseload and how strictly it was enforced, in the amount of child care support provided for program participation or employment, and in methods of case management; and the programs served different welfare populations and operated in a variety of labor markets. Although the NEWWS evaluation was designed to address the effects on children of requiring parents to participate in welfare-to-work programs, there are many other policies  for example, child care and health insurance policies that can affect children, and those policies can be examined only indirectly in this evaluation.

To determine program effects on children, the NEWWS Evaluation uses a very strong research design: a random assignment experiment. In each evaluation site, adults who were required to participate in the program were assigned, by chance, either to a program group that had access to employment and training services and whose members were required to participate in the program or risk a reduction in their monthly welfare grant or to a control group that received no services through the program but could seek out such services from the community⁽²⁾. (Control group members were eligible for child care assistance, similar to that offered to program group members, if they were participating in nonprogram activities in which they had enrolled on their own.) Notably, in four of the sites, there were two program groups (plus a control group). In three of the sites, one program group was employment-focused while the other program group was education-focused; in the fourth site, the two program groups varied in their case management staffing structure. This random assignment design assures that, within each site, there were no systematic differences between the background characteristics of families in the program and control groups when they entered the study. Thus, any subsequent differences in outcomes between the groups  for adults, children, or families as a whole  can be attributed with confidence to the effects of the programs. These differences between outcomes are called impacts, and all those reported are statistically significant and hold for the whole sample unless otherwise noted.

Overview of Findings

No aspects of the 11 welfare-to-work programs studied as part of the NEWWS Evaluation were targeted directly to children. Theoretically, however, the programs could affect children through their effects on the parents required to participate in them, as described in the conceptual model presented below. Program effects on parents' education levels, welfare status, employment, earnings, and income could potentially result in changes in family routines, parents' self-esteem or stress, or parents' supervision of children or use of child care arrangements. Thus, as a first step, it is necessary to examine how the programs were implemented and their impacts on adults.

Impacts on Adults

The 11 programs differed in the messages that they sent to welfare recipients about how best to obtain and retain employment: Some stressed getting a job quickly and others stressed initial investments in basic education or training. Most of the programs imposed a mandatory participation requirement on all recipients, and several used financial sanctions (that is, welfare grant reductions) extensively to enforce this mandate. All programs substantially increased participation in activities designed to promote employment, beyond what would have happened in their absence.

Most of the programs had impacts on their targeted outcomes: Over the two-year follow-up period, four programs that had an education focus increased the probability that welfare recipients would obtain a high school diploma or General Educational Development (GED) certificate. All programs decreased some aspect of welfare dependency, and 8 of the 11 increased two-year employment levels. While most programs increased individuals' reliance on earnings, as opposed to welfare, net income for these individuals was largely unchanged.

Impacts on nontargeted outcomes were found as well: Some programs led to a reduction in health insurance coverage, and most programs increased the use of paid child care. Across the programs, there were few effects on fertility or family structure, housing status, or mothers' psychological functioning, stress, or parenting.

Impacts on Children

Impacts on children were based on data collected in three child development areas: behavioral and emotional adjustment; cognitive functioning and academic achievement; and health and safety. In-depth data are available for preschool-age children in three of the sites (as part of a special Child Outcomes Study); more limited data are available for children of all ages in all seven sites. An examination of findings from both sets of data suggests the following regarding all children:

• Measured effects on children were infrequent. In addition, most effects could be considered small in magnitude.
• Both favorable and unfavorable child impacts were found. Notably, however, they were consistently favorable in the cognitive development area, consistently unfavorable in the health area, and both favorable and unfavorable in the behavioral and emotional adjustment area.
• Child impacts were not systematically different for mothers subject to employment-focused programs than for those subject to education-focused programs: They were not clustered in one of the two types of program, and neither type had consistently favorable effects while the other type of program had consistently unfavorable effects.

An examination of impacts on subgroups of young children (as part of the Child Outcomes Study) indicates the following:

• As was true for all studied children, few child impacts were found for subgroups of young children who, as of study entry, were at either high risk or low risk for poor development. (This analysis was conducted for preschool-age children in three of the sites in which employment-focused and education-focused programs were operated simultaneously.)
• The few impacts on children at higher risk for poor development were small, and in two of the three sites tended to be favorable for education-focused programs and unfavorable for employment-focused programs.
• The few impacts on children at lower risk for poor development were larger, tended to be unfavorable, did not tend to vary by program approach, and were clustered in three programs.

Further research and longer follow-up are needed to clearly determine the mechanisms through which some of the programs affected children. Nonexperimental methods  which lack the rigor of the experimental methods that produced the findings reported above but are needed to examine the processes through which programs might affect children  were used in an attempt to explain the few found child impacts. The results are thus suggestive, but not definitive. They suggest that, for families with all school-age children, programs that place little emphasis on helping welfare recipients obtain good child care or that result in decreases in family income may tend to have unfavorable impacts on children. (There is also some indication that increases in employment may be connected with unfavorable child effects, but this finding held true for one source of data on employment and not for the other.) Most likely, these program characteristics or effects interact with each other in particular (as yet unknown) ways to affect children. Other examined program features or effects  whether programs were employment- or education-focused, the extent to which a mandatory participation requirement was enforced, increases in parents' high school diploma or GED receipt, or decreases in health insurance coverage  do not appear, by themselves, to relate to impacts on children. Analyses of selected impacts on the younger children in the study also suggest that programs might affect children to the extent that they affect mothers' employment and/or affect children's home environment (for example, mothers' psychological well-being and parenting). These analyses of preschool-age children did not find that increases in the use of child care, decreases in health insurance coverage, or changes in family income played roles in explaining the selected child impacts examined.

Because the 11 programs operated under JOBS (Job Opportunities and Basic Skills Training program), which preceded TANF, they did not invoke the TANF time limit on eligibility for welfare, try to meet its participation goals, impose full-family financial sanctions, or put in place the generous financial work incentives of many current programs. They also did not have available to them the recent and substantial increases in federal funding for child care or expanded eligibility for health insurance through Medicaid and the State Children's Health Insurance Program. It is plausible that some of the current generation of programs will produce larger effects on adults than those reported here; as a result, it is possible that they may have larger effects on children. The new policies also may result in stronger and more divergent impacts on children with varying initial levels of being at risk for poor development⁽³⁾.

The remainder of this synthesis expands on the above findings. Section II presents a conceptual model of how mandatory welfare-to-work programs might affect children. Section III describes aspects of child well-being examined in the NEWWS Evaluation. Section IV discusses the characteristics of the adults and children in the evaluation samples. Sections V, VI, and VII summarize program implementation and program effects on targeted and nontargeted outcomes, highlighting any situations where effects were different for mothers in a special Child Outcomes Study sample focusing on young children, as compared to mothers with children of all ages. Section VIII  the heart of the document  presents child impacts.

How Mandatory Welfare-to-Work Programs Might Affect Children

At the outset of the NEWWS Evaluation, it was an open question as to whether, and how, welfare-to-work programs designed for adults would affect children. For 20 years prior to 1988, women receiving welfare who had children under age 6 generally were not required to participate in such programs. With the passage of the Family Support Act in 1988, women with children as young as age 3 (or as young as age 1, at state option) were newly designated as mandatory participants. Thus, in the early 1990s there was much interest in how welfare-to-work programs might affect children, particularly preschool-age children, who were seen as particularly vulnerable to changes in their family situation. Expectations regarding possible effects on children varied. On the one hand, given that all aspects of the programs were aimed at changing adult behavior, one might expect few effects on children, particularly if effects on adults were not dramatic. On the other hand, if effects on adults for example, increased employment or participation in program activities such as job search or education, or increases or decreases in family income were large or pervasive enough, one might expect effects on children, either positive or negative.

The most prevalent current theories about how mandatory welfare-to-work programs might affect children hypothesize that program effects on adults' employment, earnings, and income may, in turn, affect the resources available to children's development, either positively or negatively⁽⁴⁾. The resources available to children shape the daily experiences that contribute to their health, safety, and development. These resources can be material (for example, housing) or social (for example, interactions between mothers and children)⁽⁵⁾. As a positive example, welfare-to-work programs that raise income might allow families to afford better and safer housing. Additionally, employment may improve mothers' self-esteem, enhancing their ability to be a role model for their children. As a negative example, the reduction of working mothers' time at home may result in decreased overall supervision of their children. Additionally, the requirement to participate in a welfare-to-work program or the experience of holding a new job may result in increased stress for mothers, affecting parenting practices.

Figure 1 depicts the theoretical model described above⁽⁶⁾. The pivotal box in this model, labeled "Targeted Outcomes" (box C), represents the adult outcomes targeted by welfare and employment policies and programs (box A) that can be affected through implementation of the policies and programs (box B). Changes in the targeted outcomes, which can affect the resources available to families as well as family socialization patterns, can produce effects on nontargeted outcomes (box D). These nontargeted outcomes represent other avenues through which child outcomes (box E) might be affected⁽⁷⁾.

 

As evident in the model, it is important to establish that impacts on the targeted outcomes or nontargeted outcomes exist. If impacts on children are found, but no impacts are apparent for the targeted or nontargeted outcomes, it will be unclear what led to the child impacts. (While the rigorous NEWWS research design can provide solid evidence about the existence of impacts on children, it does not allow firm causal inferences to be made about the processes through which mandatory welfare-to-work programs may affect children's well-being.) Associations found among program features, adult impacts, environmental effects, and child impacts, however, can give clues about possible pathways of effects. These can then be investigated through further research and result in modifications to the conceptual model.

Aspects of Child Well-Being Examined in the NEWWS Evaluation

In order to allow sufficient time for the full effects of the NEWWS Evaluation programs to manifest themselves, sample members in the evaluation are being followed for five years from the time they entered the study. Comprehensive data on economic outcomes, including information on quarterly unemployment insurance-reported earnings and monthly welfare and Food Stamp payments, are being collected over the five-year follow-up period for 44,569 single parents (the full sample) who have been randomly assigned to research groups across the seven evaluation sites. (See box A in Figure 2.) At this point in the evaluation, economic outcome data covering the first two years of follow-up have been analyzed and results have been made public.

In addition, a wealth of information was collected through interviews with a subsample of 9,675 individuals, randomly chosen from all research groups in all seven sites, two years after they entered the study. Interviewees in this client survey sample were asked about their experiences in the program, educational attainment, family composition, housing status, employment and wage progression, and total family income, as well as about their children's behavior, school progress, and health and safety⁽⁸⁾. (See box B in Figure 2.)

For a subset of the surveyed individuals (3,018 surveyed sample members in three of the seven evaluation sites who had a child aged 3 to 5 at study entry), additional, more detailed data were collected at the time of the interview for the young child⁽⁹⁾. (If a family had more than one child aged 3 to 5, one was randomly selected to be studied in depth.) (See box C in Figure 2.)

 

These "focal" children, who would have been between about ages 5 and 7 at the two-year interview, were given a direct assessment of cognitive development, and the mothers completed multiple and more detailed measures regarding the children's behavioral and emotional adjustment and physical health and safety. Mothers in the Child Outcomes Study (COS) sample were also asked additional questions concerning nontargeted aspects of the welfare-to-work programs: for example, mothers' psychological well-being, child care arrangements, fathers' involvement with children and the payment of child support, and the family's home and neighborhood environment.

Data relating to all children of the 9,675 mothers interviewed in seven sites and to the young focal children of the 3,018 mothers interviewed in three sites were collected for three child developmental domains or outcome areas: behavioral and emotional adjustment; cognitive functioning and academic achievement; and health and safety. Appendix A details the measures used at the two-year follow-up point to assess changes in each of these areas. Data collection on children (and adults) is ongoing in the NEWWS Evaluation⁽¹⁰⁾, and Box 1 describes the future analyses planned.

Characteristics of Adults and Children in the Samples

In most aspects, the samples in the seven NEWWS Evaluation sites are diverse. Across all sites, almost all adult sample members were female single parents and, on average, 30 years old with two children at the time of study entry. The samples in Grand Rapids, Detroit, and Oklahoma City included teen parents, who represented, at most, 10 percent of each site's full sample. In Grand Rapids, Detroit, Oklahoma City, and Portland, mothers had children as young as age 1 at study entry; in these four sites, about two-fifths of the sample members entered the program when their youngest child was under age 3. The remainder of the samples in these four sites, as well as the full samples in the other three sites, were about evenly split between parents whose youngest child, as of study entry, was aged 3 to 5 and those whose youngest child was aged 6 or over. Depending on the site, between one-third and one-half of sample members gave birth to their first child when they were teenagers.⁽¹¹⁾

 

The ethnic make-up of the samples varied across the sites, reflecting the ethnic composition of the localities from which the samples were drawn. In Atlanta and Detroit, almost all sample members were African-American. About half of the sample members in Grand Rapids, Riverside, Columbus, and Oklahoma City, and two-thirds of those in Portland, were white. Only Riverside had a substantial portion (one-third) of Hispanics.

Slightly more than one-half of sample members had a high school diploma or General Educational Development (GED) certificate when they entered the program, and in all seven sites at least some study enrollees had some college or post-secondary schooling. On average, however, sample members had completed just 11 years of school as of study entry. None of the programs served populations who, as a whole, had much work history; fewer than half the individuals in all sites but Oklahoma City had worked at some point during the year prior to study entry. In all sites except Oklahoma City, between a quarter and a half of sample members had received welfare cumulatively for at least five years.⁽¹²⁾ Furthermore, up to a quarter of sample members in any site met a definition of "most disadvantaged"; that is, they did not have a high school diploma or GED, lacked any work history in the year prior to enrolling in the program, and already had received welfare cumulatively for two years or more before entering the study.

In contrast to the full samples of parents described above, adults in the Child Outcomes Study (COS) sample (a subset of parents in the Atlanta, Grand Rapids, and Riverside samples) consisted solely of mothers with preschool-age children. At study entry, the mothers' average age was 27 in the Grand Rapids sample and 29 in the Atlanta and Riverside samples, below the average age of parents in the full sample.⁽¹³⁾ Nevertheless, COS mothers had, on average, a slightly higher number of children than parents in the full sample. In addition, these mothers were less likely than those in the full sample to have ever been married and more likely to have a high school diploma or GED. Finally, in two of the three sites (Atlanta and Riverside), COS mothers were less likely than those in the full sample to have ever worked full time for six months or more for one employer. Some of these sample differences for example, the younger average age of COS mothers and their relative lack of work experience  are typical of mothers with young children. Other sample differences  for example, COS mothers being more likely to have a high school diploma or GED  reflect the fact that women on welfare with all older children are often those who have more barriers to finding a job and leaving welfare, while women with young children, particularly those not married, are perhaps more likely to have just recently started receiving welfare, and the most advantaged of this group will leave welfare within a few years.

The children of adult sample members across the seven NEWWS Evaluation sites ranged in age from 1 to 17. Approximately 49 percent of all families had all school-age children, 23 percent had no school-age children, and 28 percent had both school-age and preschool-age children. As noted above, about two-fifths of the families in four sites included a child as young as age 1 or 2. At the other end of the age spectrum, approximately one-third of the families in all seven sites included at least one high school-age child, that is, a child between ages 13 and 17. Other characteristic data for children are available only in Atlanta and only for a small number of young children who were between ages 3 and 5 at study entry. (See Box 2.)

Program Implementation

As shown in Figure 1, features of program implementation lead directly to the targeted outcomes of welfare and employment policies and programs and can also influence nontargeted outcomes. This section discusses the program services, mandates, and activities shown in box B of the figure. Results for the full sample are highlighted. Where evident, distinctions are drawn between program implementation as it was experienced by the seven-site client survey sample (for which impacts on primarily school-age children are available) and by the COS sample in three of the evaluation sites (for which impacts on preschool-age children are available).

• The evaluation sites implemented very different welfare-to-work programs; in fact, in four of the sites, two types of programs were operated within each site, to allow the study to rigorously compare the effects of specific program approaches.

 

For analysis purposes, distinctions are drawn in the evaluation between employment-focused programs and basic education-focused programs, as well as between programs with high and low levels of enforcement of the participation mandate. Taking into account these two dimensions of program characteristics, as well as the types of program activities to which welfare recipients were initially assigned, four categories of welfare-to-work program approaches emerge, shown in Table 1.

The distinction between employment- and education-focused approaches is central to the NEWWS Evaluation. To promote ongoing work and self-sufficiency among welfare recipients, states have traditionally implemented one or the other of these two approaches. The employment-focused approach emphasizes placing people in jobs quickly, even at low wages, reflecting a view that the workplace is where welfare recipients can best build their work habits and skills. The education-focused approach, which emphasizes education and training as a precursor to employment, is based on the belief that the required skill levels for many jobs are rising and that an investment in the "human capital" of welfare recipients will allow them to obtain better and more secure jobs. The two approaches convey different messages to welfare recipients about the best route to self-sufficiency, and they emphasize different program components. One aim of the NEWWS Evaluation is to determine the relative effects of the two approaches on both adults and their children.

Four of the sites in the evaluation  Atlanta, Grand Rapids, Riverside, and Columbus  operated two different programs simultaneously, to enable rigorous side-by-side tests of the comparative effectiveness of various approaches. Each of the first three of these sites implemented a "labor force attachment" (LFA) program as well as a "human capital development" (HCD) program, versions of employment-focused and education-focused programs that magnified the differences between the two types of approaches. The fourth site, Columbus, implemented a program using a "traditional" case management model, in which welfare eligibility and employment program functions were performed by separate sets of staff, as well as a program using an "integrated" case management model, in which these two functions were performed by the same staff. In the remaining three sites in the evaluation  Oklahoma City, Detroit, and Portland  the sites' established programs were studied. In all, 11 programs were examined in the seven NEWWS Evaluation sites.

• Employment-focused programs differed significantly from education-focused programs in the message that was sent to welfare recipients about how to obtain employment and in the sequence and emphasis of required program activities. These differences were evident in all three studied samples: the full sample, the client survey sample, and the Child Outcomes Study (COS) sample.

The four employment-focused programs (see Table 1) provided job search assistance to a large segment of their caseload and encouraged enrollees to find work as quickly as possible. The three LFA programs, however, differed from the Portland program in two important ways. First, the LFA programs routinely assigned individuals to job search as their first activity, whereas Portland offered GED preparation classes to people who were thought to have a good chance of attaining a GED certificate relatively quickly. Second, Portland case managers, more often than those in the LFA programs, encouraged enrollees to hold out for a job that paid well above the minimum wage and offered the best chance for long-lasting and stable employment. In contrast, LFA case managers, especially in Riverside, stressed the value of taking any job, even a low-paying one, and trying to advance.

In the education-focused programs, a large percentage of program enrollees were initially assigned to some type of skill-building activity. Their first assignments depended, in part, on their educational levels on entering the program. Those with low reading or math skills were assigned to adult basic skills classes; those with higher skills but lacking a high school diploma or GED were assigned to GED preparation classes; and non-English speakers could be assigned to English as a Second Language (ESL) classes. Finally, those with a high school diploma or GED could be assigned to vocational training or employment-oriented skills courses at local community colleges. All in all, however, assignments to GED preparation or basic education courses predominated, vocational training program assignments were less common, and enrollment in college was minimal. Riverside's HCD program was unique among this group in that it did not serve high school graduates and GED holders who, at program entry, scored above minimum levels in reading and math tests.

• The 11 programs varied widely in the degree to which a participation mandate was enforced and in their use of financial sanctions (welfare grant reductions), but the six programs in which COS sample members participated enforced the mandate and had moderate to high sanction rates.

As specified in the research design, no control group members were subject to a participation requirement and, as a result, none of them experienced any sanctions. In contrast, a wide cross section of program group members were enrolled in most programs, and participation was monitored closely. Failure to participate could result in a sanction, that is, a reduction in a family's total welfare grant. Sanction rates were high in four programs (Grand Rapids LFA and HCD and Columbus Integrated and Traditional), where at least 26 percent of sample members were sanctioned at some point during the two-year follow-up period, and low in two programs (Detroit and Oklahoma City), where less than 5 percent of sample members were ever sanctioned. Sanction rates for the remaining programs fell between these two extremes and were considered to be moderate to somewhat high.

• All 11 programs increased participation levels in activities designed to promote employment during the two-year follow-up period.

Many control group members took part in employment-related activities, such as basic education, skills training, post-secondary education, or formal job search, on their own initiative at some point during the two-year follow-up period. All programs, however, were able to increase participation levels in such activities above those achieved by the control groups. Of the nine programs with at least a moderate enforcement of the participation mandate, all but one (Grand Rapids LFA) produced large impacts on participation, ranging from 21 percentage points (Grand Rapids HCD and Columbus Traditional) to 40 percentage points (Riverside HCD).⁽¹⁴⁾ As expected, all of the employment-focused programs produced large increases in participation in job search activities, but two also produced small increases in participation in education and one produced a small increase in training. Most of the education-focused programs raised participation levels in education or training. These programs also, to a lesser extent, increased participation in job search.⁽¹⁵⁾

Impacts on Targeted Outcomes

Figure 1 suggests that numerous features of mandatory welfare-to-work programs are hypothesized to directly impact the adults subject to such programs and that these targeted effects on adults can potentially, through effects on nontargeted outcomes, influence child well-being. This section briefly summarizes effects on targeted outcomes for the 11 programs in the seven evaluation sites. The effects of the programs are measured by comparing outcomes for program groups with those of control groups; the resulting differences, or impacts, can be confidently viewed as the effects of the programs.⁽¹⁶⁾ As in the previous section, while most shown results are for the full sample, contrasts between impacts for the client survey sample and for the COS sample, where they exist, are highlighted.⁽¹⁷⁾

• Four of the 11 programs increased the probability that sample members would obtain a high school diploma or GED, including two of the three education-focused programs to which COS sample mothers were subject.

While most education-focused programs increased participation in basic education among those who entered the study without a high-school diploma or GED, only three of these programs (Grand Rapids and Riverside HCD and Columbus Traditional) increased GED certificate attainment. For all sample members, not just those who entered the study without these education credentials, impacts on GED receipt in the three programs ranged from 3 to 8 percentage points. (Table 2 shows these impacts for the several different samples for which child impacts are discussed later in this document.) Portland's employment-focused program, which used a varied first activity approach, achieved similar gains in GED receipt. (The other three employment-focused programs had no effect on GED attainment.)

• All 11 programs reduced some aspect of welfare dependency to some degree, but among COS sample mothers welfare reductions were not as universal.

For the full sample, seven programs decreased cumulative welfare expenditures by 10 percent or more relative to the control groups, a historically large effect. (See Appendix B.) The Portland and Grand Rapids LFA programs produced unusually large decreases of 17 and 19 percent, respectively. The Riverside LFA welfare expenditure impact was also large (14 percent). Among COS sample mothers, four of the six programs to which they were subject reduced welfare dependency. For this sample, the Grand Rapids and Riverside LFA programs decreased cumulative welfare expenditures by at least 15 percent, a large impact; the Atlanta LFA and Riverside HCD programs decreased expenditures by a smaller percentage. Finally, while welfare expenditure reductions were found for the full samples in the Atlanta and Grand Rapids HCD programs, they were not evident for the COS samples in these same programs.

 

Eight of the 11 programs increased two-year employment levels. Among COS sample members, five of the six programs to which they were subject increased employment, and impacts on employment rates were generally substantially higher for this sample than for the client survey samples in these five programs.

A majority of control group members worked for pay at some point during the two-year follow-up period. For the client survey sample, three of the employment-focused programs (all except Atlanta LFA) and four of the seven education-focused programs (Atlanta, Grand Rapids, and Riverside HCD; and Detroit) increased two-year employment levels. (See Table 2.) Among COS sample members, all programs except the Atlanta LFA program increased two-year employment levels. Employment impacts ranged from 6 percentage points in the Grand Rapids HCD program to 25 percentage points in the Riverside LFA program. These employment impacts were substantially larger than those for the client survey sample in all but one of the five programs. The biggest difference occurred in the Riverside LFA program, where the impact for the COS sample was nearly 7 percentage points higher than the impact for the client survey sample.

• Nine of the 11 programs produced two-year earnings gains, although many of these impacts were not statistically significant for the client survey sample. Impacts on total two-year earnings were found for the COS sample in three of the six programs: Riverside LFA and HCD and Grand Rapids LFA.

For the client survey sample, earnings gains for three of the programs (Riverside LFA and HCD and Portland) were moderate and statistically significant, while earnings gains for the other programs were smaller and not statistically significant. (All but two of the 11 programs produced two-year earnings gains for the much larger full samples.) (See Table 2.) For COS sample mothers, impacts on total two-year earnings were found in the Riverside LFA and HCD and Grand Rapids LFA programs. The largest earnings impact occurred in the Riverside LFA program where, on average, mothers in the COS program group earned an impressive 71 percent more than their control group counterparts.

• While most programs increased sample members' reliance on earnings, as opposed to welfare, their net incomes were largely unchanged. As a result, the programs lifted few additional families above the poverty line. This was the case for the client survey sample as well as the COS sample.

Across all 11 programs in the second year of follow-up, control group members in the client survey sample averaged between $6,055 (Oklahoma City) and $8,596 (Columbus) in combined income from earnings, welfare, and Food Stamps. Few programs substantially altered these combined income levels; in general, reductions in welfare, Food Stamps, and other benefits matched or exceeded earnings gains. (See Table 2.) Including estimates of the earned income tax credit (EITC) as income (not shown in Table 2) produced little change in this finding. Interestingly, one program (Riverside LFA) increased the proportion of COS sample members with incomes at or above the poverty level by 6 percentage points, a result not found for the Riverside LFA full sample. For COS sample members, there were no program effects on child support awards or payment.

• For the client survey sample, two programs had the effect of pushing a proportion of families deeper into poverty. For families in the COS sample in these two programs, this result was not statistically significant.

The Grand Rapids and Riverside LFA programs increased, by 6 and 7 percentage points, respectively, the proportion of sample members living deeply in poverty, that is, below 50 percent of the poverty line. (See Table 2.) While increases on this measure are apparent for the COS samples in these two programs, they are not statistically significant.

Impacts on Nontargeted Outcomes

As shown in Figure 1, mandatory welfare-to-work programs can affect nontargeted outcomes (box D) through effects on the targeted outcomes (box C). For example, an increase in employment, if the increase is in jobs that do not provide health benefits, could affect the proportion of people with health care coverage. Program implementation features (box B) can also affect nontargeted outcomes (box D). For example, interactions between welfare recipients and case managers that stress the importance of quality child care, or networks among welfare recipients that develop through job search clubs or other program activities and end up providing child care provider "tips," can, in turn, change the child care environments for welfare recipients' children. This section briefly summarizes program effects on the nontargeted outcomes for which data are available in the NEWWS Evaluation. Again, the effects on nontargeted outcomes are measured by comparing outcomes for the program and control groups, and differences in impacts for the client survey and COS samples are highlighted.

• Some programs led to a reduction in health insurance coverage for both children and parents.

In total, four programs  Riverside LFA, Portland, Columbus Integrated, and Oklahoma City  decreased health care coverage levels (as reported by parents) as of the end of the follow-up period.⁽¹⁸⁾ (See Table 3, which shows these effects for several different samples for which child impacts are discussed later in this document.) Portland program impacts were not statistically significant, but were just above the 10 percent level used as the standard throughout the NEWWS Evaluation analyses. For the client survey sample, the other seven programs had no impacts on health care coverage rates for children or parents. COS sample members in the Grand Rapids and Riverside LFA programs also reported a decrease in coverage, for focal children as well as any other children. (See Box 3 for a discussion of the dynamics that may have led to health insurance coverage reductions.)

 

• The programs differed in their messages and practices concerning child care.

All 11 programs offered child care assistance to welfare recipients who needed this service while they were participating in program activities or while they were employed.⁽¹⁹⁾ In the Atlanta LFA and HCD, Oklahoma City, Portland, and Detroit programs child care assistance was emphasized either by site staff or by the welfare department's organizational structure. In both Atlanta programs case managers actively promoted the availability of child care reimbursement as a benefit of program participation. In Oklahoma City state-wide emphasis on access to child care made assistance to clients readily available while they were in the program and after they left welfare for work. Oklahoma City had no set caps on the amount of child care assistance that clients could receive. Atlanta and Oklahoma City, unlike the other programs, reimbursed only for care given by licensed providers. In Portland case managers did not push specific types or locations of providers, but they did emphasize the necessity for individuals to make arrangements and assisted those who were unable to make arrangements on their own. In Detroit case managers placed a priority on making child care payments, and made referrals to licensed providers in the area on request, but the choice of provider (including choosing licensed or unlicensed care) was left to the parent.

Both the Grand Rapids and Columbus programs would reimburse expenses from child care in licensed as well as unlicensed care, but expected parents to make their own arrangements. Referrals to licensed providers in the area could be made at the parents' request.

Child care providers were not difficult to come by in any site except Riverside, where case managers encouraged the use of low-cost, more informal arrangements, both to contain program costs and because case managers believed that parents would be more able to afford such arrangements after program or other government supports expired.

The authorization of transitional child care benefits for those who left welfare for work was easiest in the Detroit, Portland, Columbus, and Oklahoma City programs. Few individuals who went to work in the other six programs received these benefits.⁽²⁰⁾

• Most programs increased the use of paid child care during employment.

Between 15 and 32 percent of control group members (depending on the site) in all 11 programs used paid child care while employed at some point during the two-year follow-up period. Nine programs  all four of the employment-focused programs and five of the seven education-focused programs produced impacts on paid child care, with the impacts in two programs (Portland and Detroit) above 10 percentage points in magnitude. (See Table 3.) Relatively few control group members (less than 15 percent in any site) used transitional child care benefits. Five programs increased the use of such benefits, but these impacts were sizable only in the Atlanta LFA and Portland programs, where the increases in the receipt of these benefits were 7 and 11 percentage points, respectively. Among COS sample members, only the two Riverside programs increased child care during employment for the focal child (measured during the month prior to the client survey interview), and this care tended to be informal and to occur during nonstandard hours (evenings or weekends). Thus, in no site were COS focal children more likely than their control group counterparts to be in formal child care during their mother's employment. (See Box 4 for a discussion of how child care use and employment might have interacted.)

• The programs had little, if any, effect on fertility or family structure or living arrangements. Any effects found were small and not clustered among certain programs. This was true for the client survey sample as well as for the COS sample.

Across all 11 programs, only Columbus Traditional had any effect on fertility; this program resulted in a 3 percentage point decrease in the proportion of sample members who had a baby since study entry. (See Table 3.) Only two programs had impacts on marital status: in Portland a 5 percentage point increase in the proportion of sample members living as an unmarried couple (not shown in Table 3), and in Riverside LFA a 3 percentage point reduction in the proportion of sample members married and living with a spouse. Similarly, only two programs had impacts, small in size, on family household composition. Impacts on housing status were not extensive either, although five programs did produce impacts (not shown in Table 3). The largest of these was a 5 percentage point increase in the proportion of program group members in the Columbus Traditional program who lived with family or friends and paid rent. For COS sample members, none of the programs had an impact on fertility or marital status, and only Riverside's LFA program had an impact on families' living arrangements, with fewer fathers living with their biological focal child.

 

• The few effects on mothers' psychological functioning and stress or on parenting can be considered small. These particular nontargeted outcomes were examined only for the COS sample.

Measures specific to the COS component of the two-year client survey were used to assess mothers' psychological well-being and parenting. Psychological well-being was assessed in three areas: time stress, locus of control or self-efficacy, and depression. Increases in mothers' feelings of time stress were found in both Atlanta programs and in the Riverside HCD program. There were no impacts on mothers' feelings of control over their own lives. Only the Grand Rapids LFA program had an impact  an unfavorable one  on mothers' depressive symptoms. Parenting was assessed on such dimensions as maternal warmth, maternal aggravation, and maternal cognitive stimulation. Impacts were found on two of the parenting measures for the Atlanta HCD program (both favorable), on four of the measures for the Atlanta LFA program (all favorable), and on one measure for the Grand Rapids LFA program (unfavorable). No impacts on parenting were found for the other programs. (The impacts discussed in this paragraph are not shown in a table.)

Child Impacts

This section summarizes impacts for children, using data from the Child Outcomes Study (COS) sample in three sites and from the client survey sample in all seven evaluation sites. COS data provide a rich, in-depth look at a subset of young children; child data from the client survey sample, though more limited, cover more sites and programs, provide information about a large number of children who were age 6 or over at study entry, and are available for families in four sites who had children as young as age 1 at baseline. (See Appendix C for a discussion of how the children in the control groups in this study and national samples of children compared developmentally at the two-year follow-up point.) Given, however, that many of the client survey questions applied only to school-age children, most analyses reported below narrow the client survey sample to only those who had all school-age children at study entry. ⁽²¹⁾(Results for client survey sample members with children of all ages are shown in Appendix D.) Child impacts, as was the case with the previously discussed impacts, are measured by comparing outcomes for children of program group members with outcomes for children of control group members. Child impacts are presented for three child outcome areas: behavioral and emotional adjustment, cognitive functioning and academic achievement, and health and safety.⁽²²⁾

Spillover Effects of Welfare-to-Work Programs on Children's Behavioral and Emotional Adjustment

• For the young children in the COS sample, program impacts on behavioral and emotional adjustment were infrequent and both favorable and unfavorable.

For focal children in the COS sample, indicators of behavioral and emotional problems included the Behavior Problems Index (BPI) and the Positive Behavior Scale/Social Competence Subscale (PBS/SCS). For each of the six COS programs, 11 outcome measures were developed from these two instruments. Two of the six programs had any impacts on focal children in this outcome area, producing a total of five impacts. (See Table 4.) Two of the five impacts were in the Grand Rapids LFA program; they were unfavorable and they related to the BPI. The three remaining impacts were in the Atlanta LFA program, and they were both favorable and unfavorable and related to the BPI and the PBS/SCS.

• For school-age children across all the evaluation sites, 8 of the 11 programs produced at least one impact on behavioral and emotional adjustment. These effects were both favorable and unfavorable.

In the client survey sample, children's behavioral and emotional adjustment was measured by asking parents whether their children (1) had been suspended from school, (2) were receiving or requiring help for behavioral or emotional problems, or (3) were in a special class or school for such problems. Among the subgroup of families with all school-age children, for whom these measures would be directly applicable, between 22 and 35 percent (depending on the site) of the control group parents reported that at least one of their children had been suspended from school since study entry; between 16 and 45 percent (depending on the site) reported that at least one child was currently receiving or requiring help for behavioral or emotional problems; and between 6 and 17 percent (depending on the site) reported that they had a child (or children) attending a special class or school for behavioral or emotional problems. (See Table 5.) Eight of the 11 programs produced at least one impact on children in this outcome area. (See Table 6.) Three programs decreased the incidence of at least one behavioral problem, and five programs increased the frequency of at least one. Only two programs, however, had an impact on more than one of the three behavioral adjustment measures.

Effects of Welfare-to-Work Programs on Children's Cognitive Functioning and Academic Achievement

• For the young children in the COS sample, favorable program impacts on cognitive development were found, although they can be considered small in magnitude. Impacts were concentrated in the Atlanta LFA program.

For focal children in the COS sample, the Bracken Basic Concept Scale/School Readiness Composite (BBCS/SRC) was used to measure cognitive development. For each of the six COS programs, three outcome measures were developed from the BBCS/SRC scores (reflecting the average score and the distribution of scores), and one outcome measure consisted of an index of two survey questions asked of mothers about academic problems. Three of the six programs had impacts in this outcome area, and they were all in a desirable direction. (See Table 4.) The Atlanta LFA program increased the average BBCS/SRC score, increased the proportion of focal children scoring in the top quartile, and decreased the proportion scoring in the bottom quartile. The Atlanta and Grand Rapids HCD programs slightly increased the proportion of children scoring above the 75th percentile, but the mean score was unaffected.

• For school-age children across all evaluation sites, few program impacts on academic achievement were found.

Children's school progress was measured by asking parents whether their children (1) had repeated a grade or (2) were attending a class for learning problems. Between 8 and 23 percent (depending on the site) of control group families with all school-age children reported that at least one of their children had repeated a grade in school during the two-year follow-up period; between 14 and 33 percent reported that any of their children were currently attending a special class for learning problems. (See Table 5.) For program group families with all school-age children, two programs — Riverside LFA and Columbus Integrated — had any impacts in this area; again, these were in a desirable direction. (See Table 6.)

 

 

 

Effects of Welfare-to-Work Programs on Children's Health and Safety

• While most mothers in the COS sample rated their children as being in very good or excellent health, in one site the welfare-to-work programs resulted in a decrease in this rating.

In the area of health and safety, mothers were asked to rate their focal child's overall health by answering the question: "Would you say that your child's health in general is excellent, very good, good, fair, or poor?" For each of the six COS programs, two outcome measures were developed from the responses. In addition, a third outcome measure concerned whether the focal child had had a serious accident, injury, or poisoning. Between 77 percent (Atlanta) and 82 percent (Riverside) of control group mothers in the COS sample reported that their focal child was in very good or excellent health; between 13 percent (Atlanta) and 24 percent (Grand Rapids) reported that their focal child had had a serious accident, injury, or poisoning since study entry.

Two of the six programs — Riverside LFA and HCD — had impacts in this area. In both of these programs, the found impacts, which were based on the mother's health rating, were unfavorable: Mothers' average rating of the general health of their focal child decreased slightly, and a smaller proportion of children were reported by mothers to be in very good or excellent health. (See Table 4.) Specifically, as a result of the Riverside LFA and HCD programs, mean health ratings (on a scale of 1 to 5 points) decreased by .23 and .20 points, respectively; the proportion of focal children rated as in very good or excellent health decreased by about 12 and 10 percentage points and the proportion rated as in fair or poor health increased by 5 and 4 percentage points, respectively.

It should be noted that the primary measure here was one of global health and not specific health problems. In addition, the health assessments were made by mothers, and not by impartial doctors or through a review of health records. While it is entirely possible that the Riverside programs truly changed children's health status, it is also possible that these findings reflect changes in mothers' perceptions of their children's health. As discussed earlier, the Riverside LFA and HCD programs produced large increases in the likelihood that COS mothers would be employed at some point during the two-year follow-up period; their two-year employment rates, relative to control groups', increased by 65 percent in the Riverside LFA program and by 45 percent in the Riverside HCD program. The next largest increase in two-year employment rates was a 15 percent increase, achieved in the Grand Rapids LFA COS sample. It is possible that mothers in the Riverside LFA and HCD COS samples, given their much greater likelihood of employment, perceived their focal children as being in poorer health than mothers in the control groups. For these mothers, even relatively minor focal child health problems (e.g., ear infections) could have caused disruptions in their daily lives, because they would have needed to either stay home from work to care for the child or perhaps quickly make alternative, non-group child care arrangements so they could go to their jobs. Among control group mothers, who were much less likely to be employed, these same relatively minor health problems might not have been as disruptive and thus memorable.

• For all children across the evaluation sites, impacts on the likelihood of events suggesting other child health or safety issues were not common.

Children's health and safety was measured by asking parents if any of their children (1) had been removed from their care or (2) had had a serious accident, injury, or poisoning. Among all control group parents in the seven sites, a surprisingly high proportion — up to 8 percent in a site — reported that a child had been removed from their care during the two-year follow-up period because they could not care for or handle the child.⁽²³⁾ (See the upper panel of Table D.1.) Between 18 and 37 percent of all control group members reported that during the previous two years at least one of their children had had an accident, injury, or poisoning requiring a visit to a hospital emergency room or clinic. Among control group parents with all school-age children, these statistics were similar. (See Table 5.) When all families are considered, one of the 11 studied programs (Columbus Traditional) had an impact on children's being removed from their mother's care (an increase in the incidence of this event) and no programs affected the likelihood of children having an accident, injury, or poisoning that required immediate medical attention. (See the lower panel of Table D.1.) When only families with all school-age children are considered, two programs — Columbus Traditional and Grand Rapids HCD — increased the incidence of children being removed from their home and no programs had an impact on the latter outcome. (See Table 6.)

Clustering of Impacts in Child Outcome Area

• For the young children in the COS sample, impacts (favorable or unfavorable) were not clustered (that is, concentrated) in a particular child outcome area.

Two programs had at least one impact on behavioral and emotional adjustment measures; three programs had at least one impact on cognitive functioning and academic achievement measures; and two programs had any impacts on health and safety measures. The 14 impacts found were about evenly split among the three child outcome areas. (See Table 7 for a summary of the COS focal child impacts.) Notably, however, the behavioral and emotional adjustment impacts were both favorable and unfavorable; all of the cognitive functioning and academic achievement impacts were favorable; and the health and safety impacts were unfavorable. There is some evidence that suggests that the diverging directions of the impacts on behavioral outcomes for these young children may reflect the fact that some of the programs affected underlying processes, such as parenting, in different ways.

• For school-age children across all evaluation sites, impacts tended to be clustered in the behavioral adjustment area; relatively few impacts were found in the areas of school progress or health and safety.

For this group of families, eight programs had at least one impact on children's behavioral and emotional adjustment; only two programs had any impacts on either academic progress or health and safety. As was the case for the young children in the COS sample, the behavioral and emotional impacts were both favorable and unfavorable; the few impacts on academic progress were favorable; and the few impacts on health and safety (both concerning removal of a child from a mother's care) were unfavorable.

Clustering of Impacts by Program

• For the young children in the COS sample, impacts were somewhat clustered (that is, concentrated) in the Atlanta LFA program.

For focal children of mothers subject to this program, there was a decrease in the proportion of children scoring at the low end on the Behavior Problems Index (BPI) — the one unfavorable impact of the Atlanta LFA program. All other impacts for this program were favorable: There was a decrease in the average frequency of BPI externalizing behavior or emotional problems and an increase in the proportion of children scoring at the high end on the Positive Behavior Scale. In addition, there was an increase in the average Bracken School Readiness Composite test score, a decrease in the proportion of children scoring at the low end Table 7 of the test, and an increase in the proportion scoring at the high end. (See Table 4 and Table 7.) In contrast, other programs had two child impacts at most: In both Riverside programs, mothers' average rating of the general health of their focal child decreased slightly, and a smaller proportion of children were reported by their mother to be in very good or excellent health.

 

• For school-age children across all evaluation sites, impacts were not clustered in particular programs.

As noted earlier, seven child outcomes were measured for children in this age group. At most, impacts were found on three of the outcomes in any given site: The Riverside LFA program increased the proportion of parents reporting that a child had been suspended from school during the previous two years or that a child was attending a special class for behavioral or emotional problems and decreased the proportion of parents reporting that a child had repeated a grade in the past two years. (See Table 6.) Thus, impacts for this program were both unfavorable and favorable.

Size of Impacts

Few evaluations of welfare-to-work programs that have examined effects on children used a random assignment research design. As a result, in contrast to the situation for adult impacts, few benchmarks for characterizing the magnitude of child impacts exist. Nevertheless, this section attempts to assess the size of the found child impacts.

• For the young children in the COS sample taken as a group, almost all child impacts can be considered small in magnitude.

One of the child impacts for this sample — the decrease in the proportion of mothers in the Riverside LFA program rating their young child as in excellent or very good health — had an effect size of one-third of a standard deviation. All other young child impacts were of a smaller magnitude, although, as will be discussed below, a few of the child impacts for lower-risk subgroups were larger. For example, the increase in the average Bracken School Readiness Composite test score for children of mothers in the Atlanta LFA program represents an increase of .14 of a standard deviation and indicates that focal children of program group mothers knew, on average, almost two more school readiness concepts than focal children of control group mothers. (A total of 61 concepts are assessed in this test.)

• For school-age children across all evaluation sites, most of the child impacts can be viewed as small, but some are clearly larger and some are of concern. In two sites, for example, there was an increase in the proportion of parents reporting that a child had been removed from their care.

Most of the child impacts for this sample can be judged as small. Some of the impacts, however, are not so small. For example, the Oklahoma City program increased by 17 percentage points the proportion of parents reporting that at least one of their children was currently receiving or requiring help for behavioral or emotional problems (35 percent of program group parents reported this situation compared with 18 percent of control group parents).⁽²⁴⁾ Some of the impacts, though smaller in absolute size, are of concern owing to their nature. As one example, four programs increased the proportion of parents reporting that at least one of their children was currently attending a special class for behavioral or emotional problems (although an additional two programs had impacts in the opposite direction on this measure). As another example, the Columbus Traditional and the Grand Rapids HCD programs increased the proportion of parents reporting that a child had been removed from their care during the two-year follow-up period because they could not care for or handle the child. In the Columbus Traditional program, this increase was 6 percentage points (8 percent of program group parents compared with 2 percent of control group parents — a four-fold increase) and the Grand Rapids HCD program produced a 4 percentage point increase here (9 percent of program group parents compared with 5 percent of control group parents).⁽²⁵⁾

It is unclear why there was an increase in the proportion of parents in two sites reporting that a child had been removed from their care. A similar result was found in a random assignment evaluation of the New Chance program, a voluntary demonstration project for young women who had children as teenagers and were high school dropouts.⁽²⁶⁾ The hypotheses that have been suggested for the New Chance finding, however — increased exposure to program staff, an increase in mothers moving out of their parental homes, and increased maternal depression — largely do not "hold true" for the Columbus Traditional and Grand Rapids HCD programs. While program group members in these two programs would have had more contact with case managers than their control group counterparts, increasing the chances that child abuse problems might have been identified, they would not have had any increase in exposure to program staff relative to their counterparts in the Columbus Integrated and Grand Rapids LFA programs. Impacts on child removal from the home were not found in these latter two programs.⁽²⁷⁾ In addition, few impacts on family living arrangements, which might lead to child neglect or abuse, were found in the two programs. These impacts were the following: For families with children of all ages in the Columbus Traditional program, there was a 5 percentage point increase, noted earlier, in the proportion living with family or friends and paying rent. For the same group of families in the Grand Rapids HCD program, there was a 3 percentage point decrease in the proportion of families whose household included relatives and a 3 percentage point increase in the proportion of families whose household did not include the parent's children. Finally, symptoms of maternal depression were examined in the Grand Rapids programs (not in the Columbus programs), and only for mothers in the COS sample. For that group of mothers in the Grand Rapids HCD program, however, no effect on mothers' depressive symptoms was found (although an increase in such symptoms was found for COS mothers in the Grand Rapids LFA program).

Balance Between Favorable and Unfavorable Impacts

• Overall, for both the young children in the COS sample and the school-age children across all evaluation sites, program impacts on child development were as likely to be favorable as unfavorable.

There were equal numbers of favorable and unfavorable impacts on the focal child. (See Table 7.) As previously noted, however, favorable impacts were concentrated in the area of cognitive development and the few health impacts found were unfavorable. Similarly, for school-age children in all 11 programs, there was an almost equal mix of favorable and unfavorable impacts. (See Table 6.)

Variations in Impacts by Subgroup

• Impacts were examined separately for the young children in the COS sample who, as of study entry, were at high risk or at low risk for poor development. Few impacts were found within these subgroups. Among these, impacts on children at higher risk were small, and in two of the three sites tended to be favorable for education-focused programs and unfavorable for employment-focused programs; impacts on children at lower risk were larger, tended to be unfavorable, did not tend to vary by program approach, and were clustered in three programs. This type of subgroup analysis was not conducted for school-age children across all sites.

In subgroup analyses examining child impacts for COS sample members, four subgroup divisions were defined, based on family baseline characteristics, that past research has suggested contain a higher-than-average proportion of children at risk for poor development. These overlapping subgroups included families with three children or more or at least two children born less than two years apart (the sibling configuration risk subgroup); families in which the mother, at baseline, did not have a high school diploma or GED or had low scores on reading or math tests (the educational risk subgroup); families in which the mother, at baseline, had received at least five years of welfare, reported at least four barriers to employment, or had never worked full time for six months or more for the same employer (the work risk subgroup); and families in which the mother, at baseline, reported symptoms of depression and a lack of control over her own life (the maternal psychological well-being risk subgroup). Within the COS control group, families who met the criteria of any of these four subgroups had children who, as of the two-year follow-up point, were generally not developing as well as children in families who did not meet these criteria.

Prior research, however, has suggested that the accumulation of risk may be more important than any particular risk factor for children's development. While a child may be able to overcome a single risk factor, the accumulation of risk may "tip the scales" against a child, and result in unfavorable child outcomes. For the COS subgroup analysis, families who met the criteria of no subgroup or only one subgroup were considered to be in the lower cumulative risk subgroup; families who met the criteria of two, three, or all four of the subgroups were considered to be in the higher cumulative risk subgroup. Thus, all families were in one of these two subgroups.

Relatively few young child impacts were found for each of the four defined overlapping subgroups and for the cumulative risk subgroups. The impacts on focal children at higher risk for poor development were small, but in two of the three sites tended to be favorable for education-focused programs and unfavorable for employment-focused programs. The impacts on focal children at lower risk for poor development were larger, tended to be unfavorable, and did not tend to vary by program approach.⁽²⁸⁾ The unfavorable impacts for focal children at lower risk were clustered in the Grand Rapids LFA program and in both of the Riverside programs. As noted earlier, this type of subgroup analysis was not conducted for school-age children across all evaluation sites.

Impact Differences Between Employment- and Education-Focused Programs

• For the young children in the COS sample, as well as for families with all school-age children in the 11 programs, child impacts were not systematically different for mothers subject to employment-focused programs than for those subject to education-focused programs.

The strong experimental research design implemented in the Atlanta, Grand Rapids, and Riverside sites, comparing the effectiveness of LFA and HCD programs, permits a clear assessment of whether program approach — employment- or education-focused — explains any child impacts found. As noted earlier, among the young children in the COS sample, 6 of the 14 child impacts found were within the Atlanta LFA (employment-focused) program. The Atlanta HCD (education-focused) program produced only one child impact. (See Table 4.) However, a pattern of more focal child impacts in LFA programs than in HCD programs did not occur in Grand Rapids and Riverside, the two other sites in which the COS was nested. In fact, the child impacts found for the two Riverside programs related to an identical child health measure and were unfavorable for both the LFA and HCD programs in Riverside. Thus, for the COS sample as a whole, child impacts were not consistently favorable or unfavorable in LFA or HCD programs either. As discussed earlier, however, the situation was somewhat different when subgroups within the COS sample were examined.

Among families with all school-age children in the 11 NEWWS Evaluation programs, employment- and education-focused programs also did not produce systematically different child impacts. (See Table 6.) In the three evaluation sites (Atlanta, Grand Rapids, and Riverside) in which the research design permitted direct comparisons of these two program approaches, there were no clear differences in child impacts. Overall, for the client survey sample in the 11 programs, child impacts were not clustered in one of the two types of program, and neither type of program had consistently favorable effects while the other had consistently unfavorable effects.

Possible Explanations for the Few Child Impacts That Were Found

Further research is needed to clearly determine the mechanisms through which some of the programs affected children. As noted above, the strong experimental research design implemented in three of the evaluation sites allows for a rigorous examination of whether program approach affects child impacts. To examine whether other program features explain child impacts, however, nonexperimental approaches, which do not have the rigor of the experimental results discussed so far in the document, are required. At this point in the NEWWS Evaluation, two approaches have been used. Possible patterns in child impacts were ascertained by taking advantage of the large number of programs studied in the school-age child analysis and simply assessing whether child impacts clustered according to the size of the 11 programs' impacts on targeted and nontargeted outcomes. In addition, statistical mediational analyses were carried out for selected focal child impacts in the COS sample. Results from both approaches, which provide only suggestive explanations and do not indicate causality, are briefly summarized below.

• Did programs with different sanctioning practices produce systematically different child impacts? For families with all school-age children in the 11 programs, measured child impacts were not necessarily different for mothers subject to programs with moderate to high enforcement of the participation mandate than for mothers in programs with low enforcement.

There is no obvious relationship between the frequency with which programs imposed sanctions and the observed patterns of child impacts for the client survey sample. (See Table 8, which arrays the same child impacts shown in Table 6 according to the magnitude of each program's impacts on various program implementation features and targeted outcomes. The child impacts are simply rearranged in each panel of the table.) For families with all school-age children, the two Columbus programs and the Atlanta HCD program had the highest sanction rates. Few child impacts were found for these three programs, with two favorable impacts for the Columbus Integrated program; an unfavorable child impact (an increase in the incidence of a child being removed from the mother's care) for the Columbus Traditional program; and no statistically significant child impacts for the Atlanta HCD program. Among the four programs in which sanctioning rates were the lowest for families with all school-aged children, child impacts were both favorable and unfavorable.

• Did programs that increased the likelihood of parents' obtaining a high school diploma or GED tend to have favorable child impacts? For families with all school-age children in the 11 programs, measured child effects were not any more favorable in programs that did, in fact, increase the proportion of parents with such educational degrees than in those that did not have this result.

Impacts on parents' receipt of a high school diploma or GED do not appear to be associated with favorable child impacts. (See Table 8.) The largest impact on receipt of these credentials was found in the Riverside HCD program, a program that did not show any favorable impacts on the child outcome measures available for the client survey sample. The other program that produced smaller impacts on high school diploma or GED receipt also failed to result in any favorable child impacts.

• Did programs that decreased health insurance coverage tend to have unfavorable child impacts? For families with young children in the COS sample as well as families with children of all ages in the 11 programs, there was not a pattern of unfavorable child impacts in the programs that decreased health care coverage.

For families in the COS sample, only two of the six programs — the Grand Rapids and Riverside LFA programs — decreased health care coverage as of the end of the two-year follow-up period. The decrease in mothers' rating of their focal child's health found for both the Riverside LFA and HCD programs does not appear to be connected to this finding: Both Riverside programs resulted in unfavorable child health rating impacts (similar in size in the two programs), while only the Riverside LFA program produced a decrease in family health care coverage. In addition, no other focal child impacts were found for Riverside's LFA (or HCD) program. For families in the COS sample, the Grand Rapids LFA program resulted in an unfavorable impact on focal children's externalizing behavior problems, but no child health rating impacts or other child impacts were found.

 

For families with children of all ages in the 11 programs, four programs — Riverside LFA, Portland, Columbus Integrated, and Oklahoma City — decreased health care coverage levels. Only two of these programs had any child impacts for this group of families: The Riverside LFA program had one favorable and two unfavorable child impacts; the Columbus Integrated program had one favorable child impact. (These results are not shown in Table 8.)

• Did programs that increased employment produce systematically different child impacts? For families with all school-age children in the 11 programs, there is some indication that increases in employment in the first two years of follow-up may be associated with unfavorable child impacts, but this finding held true for one source of data on employment and not for the other.

For families with all school-age children, the large employment impacts in the Riverside LFA program corresponded to unfavorable increases in school suspension rates and in attendance at a special class for behavioral or emotional problems, but also to a favorable decrease in grade repetition. (See Table 8, which presents two-year employment impacts based on Unemployment Insurance earnings records, as is the case throughout this document.) Employment impacts in the Riverside and Grand Rapids HCD programs also corresponded to unfavorable increases in attending a special class for behavioral or emotional problems and, in the case of the Grand Rapids program, to an unfavorable increase in the incidence of a child being removed from his or her mother's care. Programs that, for families with all school-age children, did not have impacts on employment in follow-up years one or two had a mixture of favorable and unfavorable child impacts.

When two-year employment impacts for families with all school-age children are based on parental reports of employment obtained through the client survey, the association between increases in employment and child impacts is no longer apparent. Using this data source, large or moderate employment impacts were found for the Riverside LFA, Riverside HCD, Portland, Columbus Integrated, and Detroit programs.⁽²⁹⁾ The Riverside LFA program resulted in an unfavorable increase in school suspension rates and a favorable decrease in grade repetition; both the Riverside LFA and HCD programs produced an unfavorable increase in attendance at a special class for behavioral or emotional problems. Child impacts for the Columbus Integrated and Portland programs, however, were all favorable: in the Columbus Integrated program a decrease in attending a special class for behavioral or emotional problems and a decrease in attending a special class for learning problems, and in the Portland program a decrease in behavioral or emotional problems. The Detroit program had no child impacts.

In sum, a clear connection between increases in employment and unfavorable child impacts is not evident.

• Did programs that increased or reduced combined income produce systematically different child impacts? For families with all school-age children in the 11 programs, a relationship may exist between impacts on combined income and child impacts. In particular, there is some evidence that decreases in combined income may be related to unfavorable child impacts in this older-child sample.

Among families with all school-age children in the client survey sample, one program — Portland — increased combined income in year 2 of the follow-up period; this program also increased the proportion of families with incomes at or above the poverty level. Portland had one statistically significant child impact, a favorable decrease in the proportion of families reporting a child with behavioral or emotional problems. Two programs — Riverside LFA and Grand Rapids LFA — decreased income; these two programs, as well as the Atlanta HCD and Riverside HCD programs, also had the effect of pushing a proportion of families more deeply into poverty in follow-up year 2, that is, below 50 percent of the poverty line. In the Riverside LFA program, three child impacts were found, two unfavorable and one favorable. The Grand Rapids LFA and the Riverside HCD programs each had one child impact, which was unfavorable. The Atlanta HCD program had no child impacts. The remaining programs had no effect on combined income or poverty status for this sample, and had a mixture of favorable and unfavorable child impacts.

• Did programs with different patterns of child care use and different child care assistance practices have systematically different child impacts? For families with children of all ages in the 11 programs, the data suggest that child care policies may be related to child impacts for some programs.

Patterns of child care use did not differ widely across the programs. Most programs produced an increase in the use of paid child care (relative to control groups), but varied to the extent that the child care increase was a function of increases in employment. The programs did differ, however, in their practices or policies concerning child care assistance. Given the impacts observed in most programs on the use of paid child care during employment, more information on the type of child care assistance offered by each program can illuminate the nature of the child care increases. For example, the interpretation of child care impacts similar in size for two programs might be different if one program paid only for licensed care while the other program emphasized low-cost, informal care.

In Atlanta, Oklahoma City, Portland, and Detroit, as noted earlier, child care assistance was a high priority for program staff. In addition, the Atlanta and Oklahoma City programs would reimburse sample members only for licensed child care. For families with children of all ages, these five programs' child impacts were generally favorable, although few were statistically significant. (These results are not shown in Table 8.) The Atlanta LFA program decreased the proportion of families with a child attending a special class for behavioral or emotional problems and the Atlanta HCD program decreased the proportion of families with a child who had recently repeated a grade in school. The Oklahoma City, Portland, and Detroit programs did not have any statistically significant child impacts for this sample. Staff in the Grand Rapids and Columbus programs largely expected parents to make their own child care arrangements. For families with children of all ages, the Grand Rapids programs did not have any child impacts, and the Columbus programs each had one impact: a favorable one in the Columbus Integrated program (a decrease in attendance at a special class for learning problems) and an unfavorable one in the Columbus Traditional program (an increase in the incidence of a child having been removed from the mother's care). In Riverside, low-cost, informal child care was encouraged. In this site, two unfavorable child impacts were found for the LFA program: an increase in the proportion of families with a child who had been recently suspended from school and an increase in attendance at a special class for learning problems.

• What relationships do statistical analyses suggest? Mediational statistical analyses of selected impacts on young children in the COS sample suggest that welfare-to-work programs can affect children to the extent that they affect mothers' employment and/or affect children's home environment (for example, mothers' psychological well-being and parenting).

Five of the 14 impacts on focal children in the COS sample (which reflect the general pattern of favorable cognitive, unfavorable health, and mixed behavioral impacts) were examined in more detail through an analysis that attempts to identify factors that appear to statistically explain the relationships between outcomes. According to this nonexperimental analysis, for example, the Atlanta LFA program's favorable impact on focal children's average school readiness score appears to be related to the program's favorable impacts on mothers' employment and parenting skills. As another example, the Riverside programs' unfavorable impacts on focal children's maternal health rating statistically appear to be related to mothers' increase in work hours and lower likelihood of receiving welfare at the end of the two-year follow-up period in the Riverside LFA program and to mothers' increased feelings of time stress in the Riverside HCD program. The Riverside LFA program's decrease in health insurance coverage for adults and children was not found to be linked to the unfavorable child health rating impacts; similarly, the Riverside HCD program's increase in the use of child care was not statistically related to these child impacts either.

These mediational analyses also suggested that child impacts may reflect a combination of both favorable and unfavorable program impacts on targeted and nontargeted outcomes. The Atlanta LFA program's favorable impact on focal children's reported externalizing behavior problems, for example, appears to be related to the program's favorable impact on parenting skills, despite two of the program's unfavorable impacts: an increase in mothers' time stress and an increase in the proportion of mothers who felt that they were "pushed" by the welfare office to find a job or go to school.

• In summary, what might explain the few found child impacts?

As discussed above, the nonexperimental approaches that have been used so far in the NEWWS Evaluation to attempt to explain the few found child impacts lack by necessity the rigor of the experimental analyses presented in the rest of the document. The two nonexperimental approaches rest on many assumptions which may or may not be true. In addition, the two approaches invoked different assumptions, were applied to different sets of families, and sought to explain impacts on different child outcome measures. As a result, they also will not necessarily yield the same explanations for the found child impacts.

Nevertheless, the results for the families with all school-aged children in 11 programs suggest that programs that place little emphasis on helping welfare recipients obtain good child care or that result in decreases in family income may tend to have unfavorable impacts on children. (There is also some indication that increases in employment may be connected with unfavorable child effects, but this finding held true for administrative records data on employment and not for client survey data on employment.) More likely, these program characteristics or effects interact with each other in particular (currently unknown) ways to affect children. Other examined program features or effects — whether programs were employment- or education-focused, the extent to which a mandatory participation requirement was enforced, increases in parents' high school diploma or GED receipt, and changes in health insurance coverage — do not appear, by themselves, to relate to impacts on children.

The results for the families with preschool-age children in six programs suggest that programs might affect children to the extent that they affect mothers' employment and/or affect children's home environment (for example, mothers' psychological well-being and parenting). This analysis did not find that increases in the use of child care, decreases in health insurance coverage, or changes in family income played a role in explaining the selected child impacts examined.

All of these findings suggest avenues for future research, and the longer-term impact data that will soon be available for both adults and children will provide a rich and more powerful data set with which to pursue these possible explanations of child impacts.

Conclusion

The NEWWS Evaluation is one of the first random assignment evaluations of mandatory welfare-to-work programs to examine program effects on children. The analyses presented in this synthesis indicate that mandatory welfare-to-work programs targeted to adults, with no services provided directly to children, can have spillover effects on the well-being of children. An examination of two years of follow-up found that the 11 programs studied in the evaluation did not have widespread, large, or consistent effects on the children of the parents (primarily mothers) required to participate in the mandatory programs. But favorable and unfavorable child impacts were found in some of the programs. Further research is needed to determine the mechanisms through which some of the programs affected children. It is important that the parents and the children in the NEWWS Evaluation samples are being followed for a total of five years. Forthcoming analyses of five-year data will indicate whether the impacts on children observed in the first two years of follow-up persist, are magnified, or decline by the end of five years. In addition, new child impacts may emerge over time. As policymakers continue to seek to both encourage adult self-sufficiency and foster poor children's well-being, these and future findings from the NEWWS Evaluation warrant a close watch.

Appendix A: Two-Year Child Outcome Measures

Child outcomes in the NEWWS Evaluation two-year follow-up are based on the two-year client survey and the Child Outcomes Study (COS) survey component, which was added to the client survey given to mothers with children aged 3 to 5 at baseline in three sites. Both components of the two-year follow-up included measures of children's development in three broad areas: behavioral and emotional adjustment, cognitive functioning and academic achievement, and health and safety. Some measures from each of these areas are available for all children of all parents in the two-year client survey sample in the seven evaluation sites; additional, more detailed measures in each area are available for randomly selected preschool-age children of the mothers in the COS sample (these children are identified as the "focal" children).

Measures of Behavioral and Emotional Adjustment

All Children

• Behavioral and emotional problems questions. Parents were asked if any of their children were currently getting help for any emotional, mental, or behavioral problem and if any of them were going to a special class or school or getting special help for behavioral or emotional problems. Parents were also asked if, at any point since study entry, they had felt, or someone had suggested, that any of their children needed help for any emotional, mental, or behavioral problem.
• Suspended or expelled from school question. Parents were asked if, since study entry, any of their children had ever been suspended, excluded, or expelled from school.

Focal Child

• Behavior Problems Index (BPI). Parents were asked to indicate whether series of statements are not true, sometimes true, or often true of the focal child. Statements describe behavior such as: the child is high strung, tense, or nervous; the child cheats or tells lies; the child has trouble getting along with other children. Total BPI scores are examined, as well as subscale scores for externalizing behavior problems (such as arguing, bullying, breaking things, lying, and cheating) and internalizing behavior problems (such as feeling unhappy, sad, depressed, unloved, or worthless).
• Positive Behavior Scale/Social Competence Subscale (PBS/SCS). The PBS assesses positive social behaviors, such as self-esteem, self-control, obedience, and persistence. Parents were asked to indicate whether behaviors are not true, somewhat true, or often true of the focal child. Examples of the behavioral descriptions in the subscale are: the child is helpful and cooperative; the child shows concern for other people's feelings; the child is admired and well liked by other children.

Measures of Cognitive Functioning and Academic Achievement

All Children

• Academic problems questions. Parents were asked if, since study entry, any of their children had repeated any grade for any reason and if any of them were currently going to a special class or school or getting special help in school for learning problems.

Focal Child

• The Bracken Basic Concept Scale/School Readiness Composite (BBCS/ SRC). The Bracken scale is a well-established and widely used measure of children's cognitive school readiness. The 61 items used in the COS consist of 5 subtests that assess children's knowledge of colors, letters, numbers and counting, comparisons, and shapes. (The full scale consists of 11 subtests.)

Measures of Health and Safety

All Children

• Accident or injury questions. Parents were asked if, since study entry, any of their children had had an accident, injury, or poisoning requiring a visit to a hospital emergency room or clinic, and if, during the same time frame, any of their children had been removed from their care because they could not care for or handle them.

Focal Child

• Child health rating. Parents rated the focal child's overall health in response to the following question: "Would you say that your child's health in general is excellent, very good, good, fair, or poor?"

Appendix B: (Table) Impacts on Average Total Welfare Payments Received in Years 1 and 2

Appendix C: A Comparison of National Samples of Children and Control Group Children Two Years After Study Entry

Throughout this document, comparisons are made between children of parents in control groups and children of parents who were subject to welfare-to-work programs. In contrast, the following compares the developmental status of children in national samples with children whose parents were in control groups.

Behavioral and Emotional Adjustment

The focal children in the Child Outcomes Study (COS) sample control groups, who were 5 to 7 years old at the two-year follow-up point, were reported by mothers to have more frequent behavior problems than children of the same age in a national sample. Behavior Problems Index (BPI) scores indicate that control group focal children in the COS sample tended, on average, to have more frequent total, externalizing, and internalizing behavior problems at the two-year follow-up point than children in the National Longitudinal Survey of YouthChild Supplement (NLSYCS) sample.

School suspension rates, one measure of children's behavioral and emotional adjustment, were higher for children in control group families in the client survey sample with all school-age children at study entry than for those in a national sample of eighth graders. About one-fifth of all client survey sample members in the control groups reported that at least one of their children had been suspended from school at some point during the two-year follow-up period. (See the upper panel of Table D.1.) Note, however, that many of the client survey questions applied only to children of school age. Narrowing the client survey sample to only those who had all school-age children at study entry, about one-quarter of the control group parents reported that at least one of their children had been suspended from school since study entry. (See Table 5.) In comparison, 11 percent of 1988 eighth graders nationally had ever been suspended from school.⁽³⁰⁾

Cognitive Functioning and Academic Achievement

The focal children in the COS sample control groups were significantly less cognitively ready for school than children of the same age in a national sample. Using age-standardized scores on the Bracken Basic Concept Scale/School Readiness Composite (BBCS/ SRC), average scores for COS control group children in Atlanta and Grand Rapids corresponded to the 19th percentile of the national NLSYCS sample; the average scores for Riverside COS control group children corresponded to the 21st percentile. Thus, they displayed less cognitive readiness for school than children in a national sample.

Grade repetition rates were generally higher for children in control group families in the client survey sample with all school-age children at study entry than for those in a national sample. The national average for repeating a grade is approximately 10 percent for children aged 5 to 18.⁽³¹⁾ Overall, 7 to 19 percent (depending on the site) of all client survey sample members in the control group reported that at least one of their children had repeated a grade in school during the two-year follow-up period. (See the upper panel of Table D.1.) The range for this same measure was 8 to 23 percent for those with all school-age children at baseline. (See Table 5.) Notably, in four of the seven evaluation sites (Atlanta, Columbus, Detroit, and Oklahoma City), these figures approached or exceeded 20 percent  double the national average.

Health and Safety

Focal children in the COS sample control groups were as likely to be rated by their mothers as being in very good or excellent health as children in a national sample. Between 77 percent (Atlanta) and 82 percent (Riverside) of control group mothers in the COS sample reported that their focal child was in very good or excellent health. In comparison, in 1993, mothers of 79 percent of children aged 5 to 7 in the National Health Interview Survey sample reported that their children were in very good or excellent health. As noted earlier, however, parents with a severely ill or disabled child were generally not mandated to participate in welfare-to-work programs in the early to mid 1990s. While the proportion of families exempted from the participation requirement for this reason was very small during this time period, such families would not have been included in the COS sample, whereas a national sample of children would include some severely and chronically ill children.

Similarly, children of control group members in the client survey sample did not appear to be at high risk on health and safety measures. A surprisingly high proportion of all control group members  up to 8 percent in a site  reported that a child had been removed from their care during the two-year follow-up period because they could not care for or handle the child.⁽³²⁾ (See the upper panel of Appendix D.) Between 18 and 37 percent of all control group members reported that during the previous two years at least one of their children had had an accident, injury, or poisoning requiring a visit to a hospital emergency room or clinic. In comparison, nationally, 12 percent of children under the age of 18 in 1988 had had an accident, injury, or poisoning in the previous year. It is not known, however, how many of these children required medical treatment for these incidents.⁽³³⁾

 

Appendix D: (Table) For All Families in the Client Survey Sample: Control Group Child Outcomes and Impacts on Child Outcomes

 

References

Cooper, Harris, and Larry V. Hedges, editors. 1994. The Handbook of Research Synthesis. New York: Russell Sage Foundation.

Duncan, Greg, Jeanne Brooks-Gunn, and Pamela Kato Klebanov. 1994. "Economic Deprivation and Early Childhood Development." Child Development 65(2): 296-318.

Freedman, Stephen, Daniel Freedlander, Gayle Hamilton, JoAnn Rock, Marisa Mitchell, Jodi Nudelman, Amanda Schweder, and Laura Storto. 2000. Evaluating Alternative Welfare-to-Work Approaches: Two-Year Impacts for Eleven Programs. Washington, DC: U.S. Department of Health and Human Services, Administration for Children and Families and Office of the Assistant Secretary for Planning and Evaluation; and U.S. Department of Education, Office of the Under Secretary and Office of Vocational and Adult Education.

Hamilton, Gayle. 1995. The JOBS Evaluation: Monthly Participation Rates in Three Sites and Factors Affecting Participation Levels in Welfare-to-Work Programs. Washington, DC: U.S. Department of Health and Human Services, Office of the Assistant Secretary for Planning and Evaluation.

Hamilton, Gayle, and Thomas Brock. 1994. The JOBS Evaluation: Early Lessons from Seven Sites. Washington, DC: U.S. Department of Health and Human Services, Administration for Children and Families and Office of the Assistant Secretary for Planning and Evaluation; and U.S. Department of Education, Office of the Under Secretary and Office of Vocational and Adult Education.

Hamilton, Gayle, Thomas Brock, Mary Farrell, Daniel Friedlander, and Kristen Harknett. 1997. Evaluating Two Welfare-to-Work Program Approaches: Two-Year Findings on the Labor Force Attachment and Human Capital Development Programs in Three Sites. Washington, DC: U.S. Department of Health and Human Services, Administration for Children and Families and Office of the Assistant Secretary for Planning and Evaluation; and U.S. Department of Education, Office of the Under Secretary and Office of Vocational and Adult Education.

Haveman, Robert, and Barbara Wolfe. December 1995. "The Determinants of Children's Attainments: A Review of Methods and Findings." Journal of Economic Literature 33: 1829-1878.

McGroder, Sharon M., Martha J. Zaslow, Kristin A. Moore, and Suzanne M. LeMenestrel. 2000. Impacts on Young Children and Their Families Two Years After Enrollment: Findings from the Child Outcomes Study. Washington, DC: U.S. Department of Health and Human Services, Administration for Children and Families and Office of the Assistant Secretary for Planning and Evaluation; and U.S. Department of Education, Office of the Under Secretary and Office of Vocational and Adult Education.

McLoyd, Vonnie C., Toby Epstein Jayaratne, Rosario Ceballo, and Julio Borquez. 1994. "Unemployment and Work Interruption Among African-American Single Mothers: Effects on Parenting and Adolescent Socioemotional Functioning." Child Development 65(2): 562-589.

Moore, Kristin A., Martha Zaslow, Mary Jo Coiro, Suzanne Miller, and Ellen Magenheim. 1995. The JOBS Evaluation: How Well Are They Faring? AFDC Families with Preschool-Aged Children in Atlanta at the Outset of the JOBS Evaluation. Washington, DC: U.S. Department of Health and Human Services, Administration for Children and Families and Office of the Assistant Secretary for Planning and Evaluation; and U.S. Department of Education, Office of the Under Secretary and Office of Vocational and Adult Education.

Pavetti, LaDonna A. 1992. "The Dynamics of Welfare and Work: Exploring the Process by Which Young Women Work Their Way Off Welfare." Prepared for the APPAM Annual Research Conference, October 29, Malcolm Wiener Center for Social Policy, John F. Kennedy School of Government, Harvard University.

Quint, Janet C., Johannes M. Bos, and Denise F. Polit. 1997. New Chance: Final Report on a Comprehensive Program for Young Mothers in Poverty and Their Children. New York: Manpower Demonstration Research Corporation.

Scrivener, Susan, Gayle Hamilton, Mary Farrell, Stephen Freedman, Daniel Friedlander, Marisa Mitchell, Jodi Nudelman, and Christine Schwartz. 1998. Implementation, Participation Patterns, Costs, and Two-Year Impacts of the Portland (Oregon) Welfare-to-Work Program. Washington, DC: U.S. Department of Health and Human Services, Administration for Children and Families and Office of the Assistant Secretary for Planning and Evaluation; and U.S. Department of Education, Office of the Under Secretary and Office of Vocational and Adult Education.

U.S. Department of Education, National Center for Education Statistics. 1997. The Condition of Education, 1997, NCES 97-388, by Thomas M. Smith, Beth Aronstamm Young, Yupin Bae, Susan P. Choy, and Nabeel Alsalam. Washington, DC.

Vital and Health Statistics: Health of Our Nation's Children. 1994. Series 10: Data from the National Health Interview Survey (NHIS), No. 191, Figure 1.

Wilson, Julie Boatright, and David T. Ellwood. 1993. Welfare to Work Through the Eyes of Children: The Impacts on Children of Parental Movement from AFDC to Employment. Cambridge, MA: Malcolm Weiner Center for Social Policy, John F. Kennedy School of Government, Harvard University.

Wilson, Julie Boatright, David T. Ellwood, and Jeanne Brooks-Gunn. 1995. "Welfare-to-Work Through the Eyes of Children." In P. Chase-Lansdale and J. Brooks-Gunn (eds.), Escape from Poverty: What Makes a Difference. New York: Cambridge University Press.

Zaslow, Martha, Kristin Moore, Donna Ruane Morrison, and Mary Jo Coiro. 1995. "The Family Support Act and Children: Potential Pathways of Influence." Children and Youth Services Review 17(1/2): 231-249.

Zaslow, Martha, Kathryn Tout, Sheila Smith, and Kristin Moore. 1998. "Implications of the 1996 Welfare Legislation for Children: A Research Prospective." Social Policy Report of the Society for Research in Child Development XII(3).

Notes

¹  This synthesis is one of many papers and reports that have been issued as part of the NEWWS Evaluation, conducted by the Manpower Demonstration Research Corporation (MDRC) under contract to the U.S. Department of Health and Human Services (HHS), with support from the U.S. Department of Education. Child Trends, as a subcontractor, is conducting the analysis of outcomes for preschool-age children (the Child Outcomes Study). This document synthesizes results presented in two 2000 reports, both published by the U.S. Department of Health and Human Services (Administration for Children and Families and Office of the Assistant Secretary for Planning and Evaluation) and the U.S. Department of Education: Impacts on Young Children and Their Families Two Years After Enrollment: Findings from the Child Outcomes Study, prepared by Sharon M. McGroder, Martha J. Zaslow, Kristin A. Moore, and Suzanne M. LeMenestrel (Child Trends); and Evaluating Alternative Welfare-to-Work Approaches: Two-Year Impacts for Eleven Programs, prepared by Stephen Freedman, Daniel Friedlander, Gayle Hamilton, JoAnn Rock, Marisa Mitchell, Jodi Nudelman, Amanda Schweder, and Laura Storto (MDRC). This synthesis is very much indebted to the authors of these reports, who conducted the many analyses included here. In addition, certain concepts and language in this document reflect the joint efforts of the two research teams. Note, however, that because of differences in the data sources used for economic impacts and differences in the definition of the sample of children examined, which are explained in detail later in this document, some statements regarding economic impacts and young child impacts may differ somewhat between this synthesis and the above-cited report on impacts on young children.

²  Sample members were randomly assigned to research groups over approximately a two-year period in each site, starting in June 1991 in Riverside and ending in December 1994 in Portland. Thus, the results presented here cover the period between June 1991 (the first sample members' entry into the study) and December 1996 (the last month of the two-year follow-up for the last sample members who entered the study in Portland).

³  See Zaslow et al., 1998, and McGroder et al., 2000, for further discussion of these hypotheses.

⁴  See, for example, Wilson, Ellwood, and Brooks-Gunn, 1995; Zaslow et al., 1995; Zaslow et al., 1998; and McGroder et al., 2000. The conceptual model used here was developed by Child Trends early in the NEWWS Evaluation, and has undergone refinements over subsequent years.

⁵  See Haveman and Wolfe, 1995.

⁶  The primary sources for the pathways in this model are correlational studies of the relationships between income, employment, child care, and child outcomes; previous work by MDRC and others on interventions and their effects on income and employment; and the underlying theories about how welfare-to-work or employment policies might affect children. The model depicts the effects of parent actions on children, but not the effects of children on parents. This is likely an overly linear representation of the relationships between parents and children.

⁷  The nontargeted outcomes in box D include those that theoretically could affect child outcomes. Measurement of some of these nontargeted outcomes is difficult, if not impossible; as a result, not all of them were examined in the NEWWS Evaluation.

⁸  A subsample of these interviewees also took adult reading and math achievement tests.

⁹  These additional data were actually collected for 3,194 surveyed individuals, but 176 of the families were deemed inappropriate, for various reasons, for the in-depth Child Outcomes Study, resulting in a sample size of 3,018.

¹⁰  Additional data already collected include in-depth information on the implementation of the programs  the extent of individuals' participation in various program activities, the quality of the activities, and staff practices and opinions  as well as on the costs of different aspects of welfare-to-work programs.

¹¹  Note that parents with a severely ill or disabled child were generally not mandated to participate in welfare-to-work programs in the early to mid 1990s. While the proportion of families exempted from the participation requirement for this reason during this time period was very small, such families would not have been included in the samples examined in the NEWWS Evaluation.

¹²  The Oklahoma City sample included only welfare applicants (and not recipients), who were thus more likely to have more prior work experience and less prior welfare receipt.

¹³  Tests of statistical significance were not conducted on the differences in sample characteristics discussed in this paragraph.

¹⁴  In this section, as well as the following section on targeted outcomes, the benchmarks for characterizing the magnitude of program impacts are based on ranges of impact findings from previous experimental evaluations of welfare-to-work programs. The specific thresholds vary by outcome. Impacts on participation of 20 percentage points or more are considered "large," impacts of 10 to 20 points are "moderate," and impacts below 10 points are "small." For measures of high school diploma or GED attainment, welfare expenditures, and employment, impacts of 10 percentage points or more are considered "large," impacts of 5 to 10 points are "moderate," and impacts below 5 points are "small." Impacts on earnings or income of more than $900 per year (or $1,800 over two years) are considered "large," impacts of $300 to $900 per year (or double these amounts over two years) are "moderate," and impacts of less than $300 per year are "small."

¹⁵  Length of stay in the programs, and thus participation in program activities, was shorter than in voluntary programs, but not shorter than in typical mandatory welfare-to-work programs. This reflects the fact that people cycle off welfare frequently. In the studied programs, most people did not remain on welfare continuously, and thus most were not subject to the program participation requirements, for the full two years of follow-up. In the seven programs for which detailed participation statistics have been published so far, the length of time during the two-year follow-up period that program group members spent enrolled in the programs ranged, on average, from 11 months in the Riverside LFA and HCD programs to 17 months in the Atlanta HCD program. The average number of months of participation in program activities for only those who ever participated at all ranged from 3 months in the Riverside LFA program to 9 months in the Atlanta HCD program. (See Hamilton et al., 1997, and Scrivener et al., 1998.)

¹⁶  This section focuses on economic impacts as measured by the administrative records databases described in Section II. McGroder et al., 2000, Chapter 9, in presenting economic impacts for the adults in the Child Outcomes Study, primarily used client survey data. Each of these data sources has its advantages and disadvantages. Measures of income using administrative records data, for example, cover the entire two-year follow-up period, but include income only from cash welfare, Food Stamps, and Unemployment Insurance-reported earnings. Measures of income using client survey data include many more potential sources of income, but cover only one month (the last month of the two-year follow-up period). As a result of these differences, some summary statements about economic impacts in McGroder et al., 2000, differ somewhat from those in this document.

¹⁷  Impacts could differ for the samples for several reasons. First, the client survey sample includes individuals from all seven evaluation sites, while the COS sample was drawn from only three of the sites. Second, even within the three sites in which the COS is nested, there are demographic differences between sample members with preschool-age children and those with older or even younger children, as discussed in Section IV. Finally, in general, different data sources and response rates can produce different impacts for the client survey sample compared with the full sample. See Appendix E in Freedman et al., 2000, for a discussion of the reliability and generalizability of results based on the client survey and Appendix F for a comparison of impacts estimated from survey and unemployment insurance data.

¹⁸  As noted earlier, the two-year follow-up period for many of the programs predated significant expansions in the Medicaid program and the establishment of the Children's Health Insurance Program (CHIP).

¹⁹  Both access to and allowable payments for child care were the same for control and program group members in the NEWWS evaluation. If a control group member enrolled on her own in a community education or training activity or become employed, she would have been entitled to the same type of child care assistance that a program group member would have received. If a state would only pay for licensed child care, for example, then only this type of care would be paid for either control or program group members. Program group members, however, probably heard messages about the importance of child care and the advantages of particular types of child care more frequently than did control group members, owing to program group members' increased exposure to caseworkers and other program-related staff.

²⁰  For more information on the methods of making child care payments, reimbursement rates, and child care allowances to assist welfare recipients who had earned income in the Atlanta, Grand Rapids, Riverside, and Portland sites, see Hamilton et al., 1997, and Scrivener et al., 1998.

²¹  For example, parents with no school-age children would have to respond "no" when asked if any of their children had been suspended from school. As mentioned earlier, approximately 49 percent of the parents in the client survey sample had all school-age children.

²²  In presenting results for young children, this section discusses impacts on the "focal" child in the COS families in the three sites in which this study was embedded, not impacts on the focal child's siblings, most of whom were older. Here, impacts on these siblings are included with those of all school-age children or with those of children of all ages across the seven evaluation sites (depending on the child outcome measure). McGroder et al., 2000, presents child impacts for the focal child as well as the focal child's siblings. Summary statements of child impacts in that report often combine all children in the COS families and thus may differ somewhat from the "young child" summary statements in this document.

²³  This does not necessarily mean that the child was placed in the foster care system; an unruly teenager, for example, could have been sent to live temporarily with a relative.

²⁴  It is not clear what might have led to this child impact, the only one found in Oklahoma City. For families with all school-age children in the Oklahoma City sample, there were no impacts on two-year employment, earnings, or income. There was, however, a decrease in reported health care coverage, a decrease in the proportion of parents married and living with their spouse, and an increase in the proportion of parents who were single and living only with their children. (See Tables 2 and 3.) There are no clear connections between these findings and the increase in children's receiving help for behavioral or emotional problems.

²⁵  The results of two quite conservative tests  the Tippet test and the Fisher test, developed in the literature on research synthesis (Cooper and Hedges, 1994)  indicate that at least one of these two impacts is, in fact, statistically significant in view of the large number of programs examined.

²⁶  At a 42-month follow-up point, New Chance had increased the proportion of mothers living without any of their children as well as without at least one of their children. In addition, a higher proportion of children were in foster care. (See Quint et al., 1997, pp. 138-144.) New Chance researchers could not identify a clear reason for this finding, but hypothesize that increased exposure to program staff, resulting in a higher likelihood of identifying child welfare issues among program group mothers than control group mothers, might have played a role. In addition, program group mothers were more likely to have moved out of their parents' home, a situation that might have placed them more at risk for child neglect if there were no older family members around to help watch the children. Finally, the program did increase maternal depression, and this may have increased childrearing problems.

²⁷  It is possible that the provision of "on-site" child care centers in these programs, where mothers could drop off their children, played a role in this finding. In both Grand Rapids and Columbus, some of the providers of job club and education activities (e.g., the public school system in Grand Rapids) operated such child care centers in the same building as the activities took place, staffed by well-trained child care workers, that mothers could use while they were participating in job club or attending education classes. Program administrators in Grand Rapids remember at least a few cases where child care workers from these centers, knowing that most of the mothers of the children under their care were welfare recipients, alerted the welfare department to possible cases of child neglect or abuse. Although these centers would have been open to individuals in the Grand Rapids LFA as well as HCD programs, and to those in the Columbus Integrated as well as Traditional programs, length of stay in the programs (and on welfare) was longer in the Grand Rapids HCD program than in the LFA program, and longer in the Columbus Traditional program than in the Integrated program, thus increasing the length of time that children could have been in these child care centers and thereby the chances that a case of neglect or abuse would be identified and brought to the attention of the welfare department. The impacts on child removal from the home, however, are larger in the Grand Rapids HCD and Columbus Traditional samples that consist of families with all school-age children (who would have had less use of these child care centers), compared with the samples that consist of families with children of all ages, suggesting that while this particular hypothesis may explain part of these impacts, it is not a complete explanation.

²⁸  Note that statistical tests were not applied to the within-site differences in child impacts for the employment-focused programs as compared with the education-focused programs in this analysis. In addition, this analysis compared, for most of the subgroups in the Riverside site, individuals in the education-focused program  all of whom lacked a high school diploma or GED, had low literacy levels, or had limited English skills  with individuals in the employment-focused program, who may or may not have had these credentials or skills.

²⁹  See Freedman et al., 2000, p. 193.

³⁰  U.S. Department of Education, 1997.

³¹  U.S. Department of Education, 1997.

³²  As noted earlier, this does not necessarily mean that the child was placed in the foster care system.

³³  Vital and Health Statistics: Health of Our Nation's Children, 1994.