U.S. Department of Education
Office of the Deputy Secretary
Planning and Evaluation Service
Office of Vocational and Adult Education
Prepared by:
GayleВ Hamilton, StephenВ Freedman, LisaВ Gennetian, CharlesВ Michalopoulos, JohannaВ Walter, DianaВ Adams-Ciardullo, and AnnaВ Gassman-Pines
ManpowerВ DemonstrationВ ResearchВ Corporation
SharonВ McGroder, MarthaВ Zaslow, JenniferВ Brooks, and SurjeetВ Ahluwalia
Child Trends
with
Electra Small and Bryan Ricchetti
ManpowerВ DemonstrationВ ResearchВ Corporation
"
Acknowledgements & Dedication
This report rests on the commitment, cooperation, and hard work of hundreds of people in dozens of agencies during the 12 years that the NEWWS Evaluation was conducted. Critical to the evaluation was the support and assistance of state and local welfare department and other agency administrators and staff in the study's localities: the states of California, Georgia, Michigan, Ohio, Oklahoma, and Oregon and the counties of Riverside (in California); Fulton (in Georgia); Kent and Wayne (in Michigan); Franklin (in Ohio); Oklahoma, Cleveland, and Pottawatomie (in Oklahoma); and Multnomah and Washington (in Oregon). The willingness of these staff to allow their programs to be studied using an elaborate research design, to share insights into how their programs were implemented, and to allow and facilitate detailed data collection was of crucial importance. The following key staff are owed particular thanks:
- in California Bruce Wagstaff, Paul Nakashima, Paul Warren, Debra Gamble-Hojjatie, Dana Herron, Chuck Morga, Janet Secco, and Karen Sutton; and in Riverside County, Dennis Boyle, Marilyn Kuhlman, Ron Quinn, John Rodgers, Shirley Smith, Barbara Black, Herman Copsy, Terry Welborn, Pat Virzi, John Harvey, and Susan Ogden;
- in Georgia Michael Thurmond, Sylvia Elam, Linda Bryant, Veronica Carpenter-Thomas, Doug Greenwell, Jeffrey Blankenship, Ed Nelson, Marti Colglazier, Blanie Scroggins, and Susan Williamson; and in Fulton County, Ralph Mitchell, Shirley Tate, Doretha Watkins, Gwen Bailey, Judy Byerly, Freda Carroll, Dallas Chambers, Mary Parker, Linda Turner, and Nancy Chesna;
- in Michigan Gerald Miller, Marva Hammons, Dan Cleary, Gary Howitt, Leo Greco, Steve Miller, Dick Hall, F. Robert Edwards, Dick Branch, Diane Clark, Nancy Duncan, Charles Overbey, Nancy Colbert, Vicki Enright, John Weimer, William Walker, and Marlene Hagans; in Kent County, Everett Vermeer, John Cole, Jim Poelstra, Andy Zylstra, Ken VanLoo, John Rosendahl, Char Kramer, and Marilyn Pennebaker; and in Wayne County, Samuel Chambers, Barbara Borden, Johnnie Fox, Eda Fields, Princess Nunley, Richard Stylski, Barbara Allen, and Kathleen Cook;
- in Ohio Michael Haas, Joel Rabb, Richard Deppe, Michael Koss, Scott Kozlowski, Nancy Mead, and Brenda Newsome; and in Franklin County, John Hahn, Leila Hardaway, Annette Mizelle, Toni Smith, and Georgianna Hayes;
- in Oklahoma Robert Case, Raymond Haddock, Susan Hall, Woody Hogue, Sondra Jacob, Ann Kent, Stuart Kettner, Curtis Rachels, Debbie Toon, Paul Walker, Jacque Lippel, and Sedelia Koper; and in Oklahoma, Cleveland, and Pottawatomie Counties, Judith Atkinson, Farilyn Ballard, Bill Bynum, Neil Freeman, Carolyn Gault, Herbert Jones, Margaret Jones, Beverly Morris, Jim Struby, David Reeves, and Margaret Thompson;
- in Oregon Kevin Concannon, Gary Weeks, Sandie Hoback, Stephen Minnich, Jerry Burns, Debbie White, Susan Blanche-Kappler, Elizabeth Lopez, Ron Taylor, Margaret Armantrout, Edward Buckner, Ward Kent, Larry Morris, Bob Putman, Bill Barrong, and Rich Grace; in AFS District Two, Maureen Casterline, Jean Stryker, Judith Brown, David Flock, Frank Gembinski, Erma Hepburn, Jean Pullen, and Marge Reinhart; and in Multnomah and Washington Counties, Pat Adair, Rod Brown, June Cook, Cathy Craner, Oren Cyphers, Hilda Davis, Bob Earnest, Carol Eckel, Dorothy Fuller, Angel Grogen, Veda Latin, Bruce Lowry, Linda Montgomery, Ann Pickar, Kei Quitevis, Will Reinhart, Pam Ruddell, C. L. Thames, Roger Zwemke, Jodi Davich, Mardica Hicks, Maureen Judge-Morris, Nan Poppe, and Julie Wyckoff-Byers.
Gratitude is also due the members of the NEWWS Evaluation research samples. These people shared detailed information about themselves and their children, thoughtfully completed batteries of tests and indices, and, in many cases, opened their homes to enable researchers to obtain particularly sensitive information and directly assess their children's well-being. As policymakers continue to seek new and better ways to increase employment among adult welfare recipients, lift families out of poverty, and foster poor children's well-being, the information pertaining to the study's sample members and their families will provide much guidance for many years to come.
| This report is dedicated to the memory of Daniel Friedlander (1947-1999), an adroit and insightful researcher of social programs and an enthusiastic and dedicated colleague, mentor, and friend. Daniel was instrumental in developing the research design for the NEWWS Evaluation, formulating its key research questions, and developing the tools for analysis. Daniels unswerving commitment to rigorous experimental research and to clear, precise writing is an enduring inspiration to those who carry on his work. |
Funding
The Manpower Demonstration Research Corporation (MDRC) is conducting the National Evaluation of Welfare-to-Work Strategies under a contract with the U.S. Department of Health and Human Services (HHS), funded by HHS under a competitive award, Contract No. HHS-100-89-0030. Child Trends, as a subcontractor, is conducting the analyses of outcomes for young children (the Child Outcomes Study). HHS is also receiving funding for the evaluation from the U.S. Department of Education. The study of one of the sites in the evaluation, Riverside County (California), is also conducted under a contract from the California Department of Social Services (CDSS). CDSS, in turn, is receiving funding from the California State Job Training Coordinating Council, the California Department of Education, HHS, and the Ford Foundation. Additional funding to support the Child Outcomes Study portion of the evaluation is provided by the following foundations: the Foundation for Child Development, the William T. Grant Foundation, and an anonymous funder.
The findings and conclusions presented herein do not necessarily represent the official positions or policies of the funders.
To obtain other publications from the NEWWS Evaluation, and for information on how to access the NEWWS Evaluation restricted and public use data files, go to http://aspe.hhs.gov/hsp/NEWWS/index.htm.
Introduction
For the past 30 years, federal and state policymakers have been looking for new and better ways to increase the employment of welfare recipients. Beginning in the late 1960s, in response to dissatisfaction with the Aid to Families with Dependent Children (AFDC) program, one of the nation's principal safety nets for poor families, Congress began to reshape it, creating a program to encourage welfare recipients to find jobs. In 1988 the Family Support Act (FSA) established a system of mutual obligation within the AFDC benefit entitlement structure: Government was to provide education, employment, and support services to AFDC recipients, who were, in turn, required to participate in the Job Opportunities and Basic Skills Training (JOBS) program. The most recent federal reform effort, the 1996 Personal Responsibility and Work Opportunity Reconciliation Act (PRWORA), replaced AFDC with a flexible, state-directed block grant program, Temporary Assistance for Needy Families (TANF); limited most families to five years of lifetime federal TANF assistance (with some states setting shorter welfare time limits); and created financial incentives for states to run mandatory, work-focused welfare-to-work programs. PRWORA gives states more flexibility than they had before in designing their programs (which has encouraged, for example, some states to implement generous financial work incentives or tough financial sanctions for noncooperation) and more responsibility for moving the nation's poor into the labor market.
Mandatory, work-focused welfare-to-work programs are thus not a new idea. The challenge for welfare policymakers and state and local program administrators, however, is to determine how to design and implement such programs so as to best achieve the goal of fostering adults' long-term economic self-sufficiency without unintentionally jeopardizing the well-being of their children. This report provides guidance on the topic by analyzing the long-term effectiveness of 11 programs in seven sites that aimed to move substantial numbers of people from welfare to work. These programs were begun under FSA but operated under TANF in the later years of the evaluation. (The results in this report pertain to the time period between 1991 and 1999.) Overall, the programs shared TANF's primary goal of moving people from welfare to work. Further, they reflect a range of approaches, implementation features, and environments: Some were strongly employment-focused while others emphasized basic education; they varied in how broadly the program participation mandate was applied to the welfare caseload and how strictly it was enforced, in the amount of child care support provided for program participation or employment, and in methods of case management; and the programs served different welfare populations and operated in a variety of labor markets. Finally, all 11 programs were studied using a strong random assignment design, resulting in reliable information about their relative effectiveness.
This report analyzes the comparative long-term effectiveness of two very different types of programs: employment-focused programs and education-focused programs. Results from the 11 programs studied provide evidence on the relative effectiveness of these two approaches in two ways: First, an innovative research design implemented in three sites is used to directly compare versions of these two approaches within each site, resulting in unusually reliable information about their relative effectiveness. Second, cross-site comparisons of all 11 programs studied are used to determine if more examples of the two approaches, operating in different labor markets and for different populations, uphold or refine the findings from the direct comparisons in three sites. In addition, the cross-site comparisons are used to ascertain if any versions of these approaches emerge as particularly successful. Results are presented for two key subgroups of welfare recipients for whom different program approaches might be expected to work differently: those with a high school diploma or GED and those without these credentials. Covering a five-year follow-up period and using a wealth of data pertaining to single parents (mostly mothers) as well as their children, the report thus addresses the following critical question: What works best and for whom?
The five-year results presented in this report were produced as part of the National Evaluation of Welfare-to-Work Strategies (NEWWS). This evaluation is being conducted by the Manpower Demonstration Research Corporation (MDRC) under contract to the U.S. Department of Health and Human Services (HHS), with support from the U.S. Department of Education. Child Trends, as a subcontractor, is conducting the Child Outcomes Study (COS), the part of the evaluation that examines effects on young children. The NEWWS Evaluation includes programs in seven sites across the country: Atlanta, Georgia (Fulton County); Grand Rapids, Michigan (Kent County); Riverside, California (Riverside County); Columbus, Ohio (Franklin County); Detroit, Michigan (Wayne County); Oklahoma City, Oklahoma (Oklahoma, Cleveland, and Pottawatomie counties); and Portland, Oregon (Multnomah and Washington counties).(1) In these seven locations, more than 55,000 people were randomly assigned to research groups as part of the study.
This chapter begins by presenting a framework for understanding the five-year results and then describes the research questions and design of the overall NEWWS Evaluation. Next, the environments in which the 11 programs studied were operated are discussed, along with the most critical program features. This is followed by an examination of the welfare-to-work program treatments experienced by individuals in the various evaluation research groups over time, given the long follow-up examined in the report. Finally, two-year NEWWS Evaluation results are recapped, and the contents of Chapters 2-13 are briefly described.
A Framework for Understanding Program Results
The FSA gave program administrators a great deal of flexibility in designing the 11 programs studied in the National Evaluation of Welfare-to-Work Strategies. (Box 1.1 provides a comparison of the key features of FSA and TANF.) That flexibility, combined with local economic, political, and funding environments, resulted in 11 programs that vary on several dimensions. This report focuses primarily on one of those dimensions: the self-sufficiency approach used.
FSA/AFDC (1988-1996)
PRWORA/TANF (1996-2002)
|
While the overarching goal of programs in the past 30 years to foster the self-sufficiency of welfare recipients through increased employment and decreased welfare receipt has not changed, there has been disagreement on how best to move individuals from welfare to work. One strategy emphasizes quick employment, reflecting the belief that individuals can best build their employability and improve their skills, eventually achieving self-sufficiency, through actual work, even if their initial jobs are minimum wage and without fringe benefits. The other strategy emphasizes initial investments in short-term education and, in some cases, training, reflecting the view that these investments will eventually enable individuals to obtain higher-wage, longer-lasting jobs with health insurance coverage. Most programs have blended the two strategies and emphasized elements of both. Past research has shown that a program's location on the continuum between these two strategies and the mix of services it provides to enrollees can have an effect on the patterns and magnitude of program impacts measured in the short and long term.(2)
The programs in this report have been categorized by their approach: either employment-focused or education-focused. Three sites in the evaluation (Atlanta, Grand Rapids, and Riverside) simultaneously implemented a Labor Force Attachment (LFA) program and a Human Capital Development (HCD) program, versions of employment-focused and education-focused programs that magnified the differences between the two types of approaches. The six programs in these three sites provide the best test of the relative effectiveness of the two approaches.(3) Another site, Columbus, was also asked to implement two different programs in a head-to-head test. One program used an "integrated case management" staffing structure, in which one worker assumes responsibility for both eligibility and employment and training for her clients. The other program used a "traditional case management" staffing structure, in which separate workers handle the eligibility and employment and training duties. These programs, called the Columbus Integrated and Traditional programs, both used an education-focused approach.(4) Program administrators in the other three sites chose which self-sufficiency approach to implement based on their own goals. Of the 11 programs studied, four programs (Atlanta LFA, Grand Rapids LFA, Riverside LFA, and Portland) were employment-focused; the remaining seven were education-focused.
In the three LFA versions of the employment-focused program, almost all enrollees were first assigned to job search. In Portland, the other employment-focused program, many, but not all, individuals were assigned to job search as a first activity. Some individuals, usually those who were determined to have more barriers to work than other members of the caseload, were first assigned to education or training activities. In the three HCD education-focused programs, as well as in the four other education-focused programs, almost all individuals were first assigned to either education or occupational skills training activities.
This report also focuses on another program implementation dimension: mandatoriness. Past research suggests that the degree to which a program enforces a participation mandate for the welfare caseload is a determinant of whether a program can have an effect.(5) High or low enforcement of the mandate is a product of three factors: how wide a cross section of the welfare caseload is enrolled in a program; how closely a program monitors individuals' participation; and how swiftly and consistently a program imposes financial sanctions (that is, reductions in monthly welfare grants) on those who do not participate. Nine of the programs were high enforcement programs; Detroit and Oklahoma City were not, mostly because of limited program and staff resources.
Table 1.1 categorizes the 11 programs according to their self-sufficiency approach and level of enforcement of the participation mandate. Section IV of this chapter discusses in greater detail these dimensions of the programs, as well as others that may have affected program impacts. It is important to keep in mind, however, that these program categorizations and descriptions apply to these particular programs as they would have been experienced by evaluation sample members in their first three years of follow-up. In the years corresponding to sample members' fourth and fifth years of follow-up, some of the programs changed their approach (in response to TANF). Section V of this chapter describes these changes.
| Employment-focused approach | Education-focused approach | ||
|---|---|---|---|
Job search first | Varied first activity | Education or training first | |
High enforcement | High enforcement | High enforcement | Low enforcement |
| Atlanta LFA Grand Rapids LFA Riverside LFA | Portland | Atlanta HCD Grand Rapids HCD Riverside HCD Columbus Integrated Columbus Traditional | Detroit Oklahoma City |
NOTES: LFA = Labor Force Attachment program. HCD = Human Capital Development program. | |||
Research Questions and Design
Within the categorization scheme described above, the report analyzes program effects for single-parent welfare recipients and their children, focusing on results for the five years after individuals entered the programs. Data included in this report are from administrative records (unemployment insurance, state and county welfare payments, and Food Stamp payments) and from client surveys and parent and child assessments conducted over the five years after individuals entered the study. The report specifically addresses the following questions:
- Which welfare-to-work program approaches were most successful in helping welfare recipients to receive the program services or attain the skills or credentials that could enhance their chances of finding employment?
- Which approaches were most successful in helping welfare recipients to find paid work? Did any approaches help individuals get "good" jobs, that is, full-time jobs with health benefits?
- Which approaches were most successful in helping individuals to leave welfare and to remain off welfare? What were effects on Food Stamps as well?
- Which approaches were most successful in increasing welfare recipients' income and helping them move out of poverty?
- Which approaches were most successful in achieving self-sufficiency for those who lacked basic education credentials, were at high risk for long stays on welfare, or were at most disadvantage in finding jobs on their own?
- Did any approaches affect the health care coverage of parents or their dependent children?
- Did any approaches affect the household and personal circumstances of welfare recipients ¾ for example, their marital status, family composition, or fertility, or the likelihood that they would experience abuse by an intimate partner?
- Did any approaches affect the likelihood of using child care, the type of child care used, or the types of activities in which children participated on a typical day?
- Did any approaches favorably or unfavorably affect the well-being of children of various ages as a result of the services provided to or the mandates imposed on parents?
- In particular, did any approaches favorably or unfavorably affect the well-being of children who were preschool-age when their parents entered the study children often viewed as most vulnerable to the effects of parents being out of the home?
- Finally, which approaches were more expensive or less expensive? From the perspectives of government and the welfare recipients themselves, for which approaches did benefits outweigh the costs, and by how much?
The NEWWS Evaluation uses an unusually rigorous research design, a random assignment experiment, to estimate program effects. In each site individuals who were required to participate in the program were assigned, by chance, to either a program group, which had access to employment and training services and whose members were required to participate in the program or risk a reduction in their monthly welfare grant, or a control group, whose members were not subject to a participation mandate and received no services through the program, but could seek out such services from the community.(6) This random assignment design ensures that there are no systematic differences between the background characteristics of people in the program and control groups within each site when they enter the study. Thus, any subsequent differences in outcomes between the groups can be attributed with confidence to the programs. These differences, called impacts, are the primary focus of this report.
Although this design assures that the impact estimates of each program are extremely reliable, there are some limitations. Local conditions, including labor markets, prevailing wages, welfare grant levels, political environments, program funding levels, and staff administration, can all have an effect on the magnitude of impact estimates. For this reason, comparisons of impacts across the 11 employment- and education-focused programs in this report are only suggestive of the relative effectiveness of either approach in the short term.(7) The most definitive judgments on the relative effectiveness of the two approaches come from the results in the three sites in this evaluation that tested versions of the two approaches side by side.
Finally, it should be kept in mind that in some sites it was not possible to bar all control group members from receiving welfare-to-work program services for the entire five-year follow-up period examined in this report. While no control group members were exposed to the services provided and mandates imposed by sites' welfare-to-work programs for the first three years of the follow-up period, their status differed by site in the fourth and fifth years, as detailed in Section VI of this chapter. As a result, in several sites impacts measured for years 4 and 5 of follow-up may underestimate the effects that would have been found had the control embargo been in effect for a full five years in all sites. Given this uncertainly, some of the analyses throughout the report separate impacts for years 1 to 3 from those for years 4 and 5.
Program Environments
When planning this evaluation, HHS and MDRC sought to include sites that would demonstrate operation in a diverse range of conditions, though they would not represent all welfare-to-work programs in the country. As shown in Table 1.2, sites varied along several dimensions, such as geographic location, labor market, and welfare grant level.(8) More striking, however, are the changes in the economy, and the concomitant changes in welfare policy and caseloads, that all sites experienced from 1991, when the first group of individuals was randomly assigned as part of the NEWWS Evaluation, to 1999, the end of the five-year follow-up period for the last group of individuals randomly assigned. (Random assignment took place over a roughly two-year time period in each site, falling somewhere between mid 1991 and the end of 1994, depending on the site.) Part of the richness of the evaluation results thus stems from the examination of program effects over a period of time that reflected unprecedented economic growth and a sea change in welfare policies.(9)
Characteristic | Atlanta | Grand Rapids | Riverside | Columbus | Detroit | Oklahoma City | Portland |
|---|---|---|---|---|---|---|---|
| Population, 1990 | 648,779 | 500,631 | 1,170,413 | 961,437 | 2,111,687 | 832,624 | 895,441 |
| Population, 1997 | 722,540 | 539,425 | 1,447,791 | 1,017,274 | 2,127,087 | 889,411 | 1,015,954 |
| Population growth, 1990-1997 (%) | 11.4 | 7.7 | 23.7 | 5.8 | 0.7 | 6.8 | 13.5 |
| Employment growth, 1991-1996 (%) | 14.8 | 15.9 | 11.9 | 7.7 | 5.4 | 9.1 | 15.1 |
| 1991-1999 | 25.1 | 28.9 | 28.2 | 11.8 | 12.8 | 16.5 | 17.8 |
| Unemployment rate (%) | |||||||
| 1991 | 5.3 | 7.8 | 9.8 | 3.8 | 10.5 | 6.0 a | 5.4 b |
| 1992 | 7.4 | 7.5 | 11.6 | 4.6 | 10.5 | 5.5 a | 7.3 b |
| 1993 | 6.4 | 5.3 | 11.9 | 4.5 | 8.3 | 5.5 a | 6.6 b |
| 1994 | 5.8 | 4.2 | 10.5 | 3.7 | 6.7 | 5.0 a | 4.9 b |
| 1995 | 5.4 | 3.8 | 9.6 | 2.9 | 6.0 | 4.0 a | 4.1 b |
| 1996 | 4.9 | 4.0 | 8.2 | 2.9 | 5.5 | 3.6 a | 5.2 b |
| 1997 | 4.6 | 3.2 | 7.5 | 2.7 | 4.7 | 3.5 a | 5.0 b |
| 1998 | 4.1 | 2.9 | 6.6 | 2.5 | 4.3 | 3.9 a | 4.9 b |
| 1999 | 3.8 | 3.1 | 5.5 | 2.5 | 4.2 | 2.7 a | 5.2 b |
| AFDC/TANF caseload c | |||||||
| 1991 | 18,507 | 7,660 | 23,325 | 23,192 | 87,992 | 12,305 | 11,234 |
| 1992 | 21,801 | 7,389 | 25,581 | 24,135 | 88,584 | 13,392 | 11,817 |
| 1993 | 23,113 | 7,508 | 27,775 | 24,739 | 89,083 | 14,259 | 11,961 |
| 1994 | 23,121 | 7,137 | 32,044 | 24,807 | 88,337 | 14,257 | 11,981 |
| 1995 | 22,043 | 7,052 | 24,650 | 23,240 | 88,614 | 13,959 | 11,231 |
| 1996 | 19,620 | 5,836 | 25,076 | 19,474 | 74,051 | 12,488 | 10,097 |
| 1997 | 15,754 | 5,362 | 23,519 | 17,363 | 70,052 | 10,239 | 6,721 |
| 1998 | 12,007 | 4,551 | 19,278 | 13,341 | 59,060 | 8,273 | 5,306 |
| 1999 | 8,907 | 3,303 | 15,091 | 11,082 | 43,278 | 7,280 | 5,229 |
| Welfare grant level for a family of 3 ($) | |||||||
| 1993 | 280 | 474 | 624 | 341 | 459 | 324 | 460 |
| 1998 | 280 | 459 | 611 | 362 | 459 | 292 | 503 |
| Food Stamp benefit level for a family of 3 ($) d | |||||||
| 1993 | 292 | 252 | 202 | 292 | 252 | 292 | 287 |
| 1998 | 329 | 314 | 321 | 329 | 314 | 329 | 310 |
| Income disregard policies | Standard; fill-the-gap | Standard e | Extended; fill-the-gap | Standard | Standard e | Standard | Standard |
Maximum that a family of 3 could earn and receive AFDC, January 1993 ($) | |||||||
| In months 1-4 of employment | 756 | 831 | 1,175 | 632 | 809 | 606 | 810 |
| In months 5-12 of employment | 544 | 594 | 823 | 461 | 579 | 444 | 580 |
| After 12 months of employment | 514 | 564 | 793 | 431 | 549 | 414 | 550 |
Maximum that a family of 3 could earn and receive TANF, December 1998 ($) | |||||||
| In months 1-4 of employment | 756 | 774 | 1,447 | 974 | 774 | 704 | 616 |
| In months 5-12 of employment | 544 | 774 | 1,447 | 974 | 774 | 704 | 616 |
| After 12 months of employment | 514 | 774 | 1,447 | 974 | 774 | 704 | 616 |
| SOURCES: U.S. House of Representatives, 2000; Gaquin and Littman, 1999; Hall and Gaquin, 1997; Hamilton and Brock, 1994; Hamilton et al., 1997; Scrivener et al., 1998; State Policy Documentation Project; U.S. Department of Labor, Bureau of Labor Statistics; Center on Social Welfare Policy and Law, 1994; Center for Law and Social Policy, 1995; site contacts. NOTES: Data are for counties: Atlanta (Fulton County), Georgia; Grand Rapids (Kent County), Michigan; Riverside (Riverside County), California; Columbus (Franklin County), Ohio; Detroit (Wayne County), Michigan; Oklahoma City (Oklahoma, Cleveland, and Pottowatomie Counties), Oklahoma; Portland (Multnomah and Washington Counties), Oregon. a Data are for Oklahoma County. The unemployment rates for Cleveland County are: 1991, 4.4%; 1992, 3.5%; 1993, 3.5%; 1994, 3.5%; 1995, 2.9%; 1996, 2.6%; 1997, 2.6%; 1998, 3.2%; 1999, 2.1%. The unemployment rates for Pottowatomie County are: 1991, 7.6%; 1992, 5.9%; 1993, 5.8%; 1994, 5.7%; 1995, 4.5%; 1996, 4.8%; 1997, 4.8%; 1998, 5.4%; 1999, 3.7%. b Data are for Multnomah County. The unemployment rates for Washington County are: 1991, 4.5%; 1992, 6.1%; 1993, 5.3%; 1994, 3.7%; 1995, 3.2%; 1996, 3.9%; 1997, 3.8%; 1998, 3.9%; 1999, 4.1%. c Annual average monthly caseloads, as reported by the state or county. In Atlanta and Portland averages are for calendar years; in all other sites averages are for state fiscal years. d Assumes the receipt of the maximum AFDC/TANF payment. e Although Michigan implemented nonstandard earned income disregards during the evaluation period through the To Strengthen Michigan Families initiative, all sample members in the NEWWS Evaluation were excluded from them. | |||||||
To be included in the NEWWS Evaluation, sites needed large enough welfare caseloads to meet the sample size requirements of the research design. Accordingly, all seven sites include urban areas. Detroit, with a population that was slightly over 2 million in both 1990 and 1997, is the largest urban area studied in the evaluation. Riverside, with a population of over 1 million in 1990, experienced the most growth during this time period, adding almost 24 percent to its population by 1997. Population growth in the other sites during this seven-year period ranged from 6 to 14 percent.(10)
As population grew, so did labor markets. In three sites employment expanded significantly between 1991 and 1999: the employed labor force grew by 29 percent in Grand Rapids, 28 percent in Riverside, and 25 percent in Atlanta. The other four sites experienced 12 to 18 percent gains.
Rising employment, particularly in localities with rising population, does not necessarily indicate declining unemployment rates. Unemployment rates in all seven sites, however, decreased over this period. Following national trends, unemployment rates peaked in 1992 and, in general, were lowest in 1999. Early in the evaluation period unemployment rates in Detroit and Riverside topped 10 percent. Although rates in both localities steadily declined, Riverside rates remained at 8 percent in 1996, significantly higher than the national average. By 1999, unemployment rates in all evaluation sites were below 6 percent. Throughout the evaluation period the Columbus labor market was notably robust; its unemployment rate never exceeded 5 percent, even during the high point of the national recession, and dropped to less than 3 percent in 1999.
Because individuals in the program and control groups within each site were subject to the same labor market, the quality of the economy by itself should not necessarily affect impact estimates; program and control groups shared the same advantages of a tight labor market or disadvantages of a slack one. However, different economic environments can influence the type of people receiving welfare and thus required to participate in welfare-to-work programs. For example, in a good labor market individuals with more serious barriers to work are likely to be left on the welfare rolls.
The size of welfare (AFDC or TANF) caseloads varied with the size of site populations, ranging in 1991, the beginning of the evaluation, from about 7,500 in Grand Rapids to almost 90,000 in Detroit. In general, welfare caseloads grew in the early part of the evaluation period, peaked in 1993 or 1994, and declined to their 1991 levels or below by 1996, shrinking further to 1999. In almost all evaluation sites, welfare caseloads in 1999 were no more than half the size they were in 1991; decreases in Riverside and Oklahoma City were somewhat smaller, with caseloads reduced by 35 and 41 percent, respectively.
There was considerable variation in welfare grant levels among the sites. In 1993 monthly maximum cash payments for a family of three ranged from $280 in Atlanta to $624 in Riverside. Welfare grant levels in 1998 were similar to 1993 levels, with slight reductions in the maximum amount payable in three sites, a slight increase in two sites, and identical levels in two sites. Food Stamp payments, for which means standards are federally set, varied less across the sites, from $202 in Riverside to $292 in Atlanta, Columbus, and Oklahoma City in 1993, with slightly higher benefit levels in all sites in 1998.(11) To some extent, low welfare grants are offset by higher Food Stamp payments, but this does not change the overall rankings of sites on benefit levels.
All states were required to disregard (that is, not count) some earned income when calculating a family's welfare grant and, over time, more generous earnings disregards were put into place in some of the evaluation sites. At the beginning of the NEWWS Evaluation, five sites applied standard earnings disregard rules. Under these, for the first four months of employment $120 of earnings and an additional one-third of the remainder were disregarded. This $120 disregard included both a $30 flat disregard and a $90 disregard for work expenses. In months 5-12 of employment, the additional one-third disregard was eliminated, leaving the total disregard at $120. After the first year of employment only the $90 work expenses disregard was allowed. In addition, individuals were allowed to disregard child care expenses up to $175 per child aged 2 or over and $200 per child under age 2.(12) Atlanta and Riverside applied nonstandard disregard rules that permitted employed recipients to keep more of their welfare check. Throughout the evaluation period Atlanta employed "fill-the-gap" budgeting. Under fill-the-gap, working welfare recipients can earn up to the state-determined "standard of need" before losing all welfare benefits. For example, in 1993 the standard of need for a family of three was $424 (per month). A parent with two children could earn up to $756 in each of the first four months of employment and still remain on AFDC, $544 in months 5-12, and $514 per month thereafter.
Throughout the course of the NEWWS Evaluation five-year follow-up period, states implemented further earnings disregard policies. In California, for example, the state received a waiver at the end of 1993 to eliminate the time limit on the standard earnings disregard applied to the calculation of welfare benefits and also instituted a version of fill-the-gap. By 1998, relative to 1993, all but one of the evaluation sites had implemented policies that allowed welfare recipients to keep more of their earnings, affecting the likelihood that a sample member could work while remaining on welfare. Increases were greatest in Riverside where, in 1998, a family of three could earn as much as $1,447 and still receive TANF. In Columbus and Oklahoma City, welfare recipients in 1998 similarly could earn almost double what they could in 1993 and remain eligible for welfare.
Differences in welfare grants, earnings disregard standards, and the use of fill-the-gap budgeting may explain some variation in program impacts, even though these grant levels and policies applied to both program and control group members in each site. Impacts on welfare payments in low-grant states are likely to be somewhat lower than those in high-grant states, other things being equal, because there are fewer welfare dollars to reduce. In addition, in low-grant states even low-paying jobs may be more attractive than welfare, providing a greater incentive to work. At the same time, in states that have higher grant levels, or generous earnings disregards, it may be easier for individuals to combine work and welfare in a way that will increase total household income and raise the family standard of living, particularly after factoring in the Earned Income Tax Credit (EITC).(13)
Program Features
This report presents the long-term impacts, or effects, that the 11 NEWWS programs had on outcomes such as employment, earnings, welfare receipt, and child and family well-being. This section provides a context for interpreting the results in the chapters that follow by showing the range of programs on key implementation dimensions. As will become apparent, there is no typical "package" of welfare-to-work program features. As examples, the most work-focused programs are not necessarily the toughest, and those that use integrated case management do not necessarily monitor their enrollees' progress more effectively than others. Given this information, it is important to interpret each program's impacts as a result of its entire "bundle" of services and features. It is likely that a combination of features rather than one specific feature is associated with successful outcomes and specific impacts.
Nonetheless, this section focuses on the two implementation dimensions used to categorize the 11 NEWWS programs: the self-sufficiency approach used and the level of participation mandate enforcement. These particular dimensions are discussed at length here because they clearly demonstrate the division between the programs and provide a general framework for thinking about program results. In addition, this section explores other implementation features that provide an important context for interpreting the impact results in later chapters.
It should be kept in mind that sample members were subject to the welfare-to-work programs being studied only as long as they were receiving welfare. Welfare recipients frequently cycle on and off the welfare rolls. In the first two years of follow-up in these programs, for example, sample members received welfare and were required to participate in the programs for an average of 9 to 17 months, depending on the site. On average, sample members were either participating in a program activity, employed while subject to the program, or sanctioned for program nonparticipation between 22 and 68 percent of these months, depending on the site.(14) (Those not in one of these statuses in any given month may have been waiting for case managers to refer them to an activity or sanction them, waiting for support services to be arranged or for activities to begin, temporarily deferred from participation for reasons such as illness or caring for an ill relative, or temporarily "lost track" of by program staff.) While such figures on program "dosage" might seem surprising, it is important realize that the goal of welfare-to-work programs is to enable individuals to leave welfare and/or get a job. As a result, one would hope that sample members had not been participating in program activities during every month in the follow-up period, since this would mean that they had never left welfare and/or found employment during the period.
As noted earlier, the implementation features discussed here primarily relate to how sample members would have experienced the programs in the first three years of follow-up in the evaluation. Section V describes changes in program focus in the last years of the five-year follow-up period analyzed in the report. Table 1.3 summarizes for all programs the implementation features discussed in this section.
| Program | Degree of Emphasis on Self-Sufficiency Approach | Degree of Enforcement of the Participation Mandate | Type of Child Care Support Provided for Participation | Extent of Partnership Between Eligibility and Self-Sufficiency Staff | ||||
|---|---|---|---|---|---|---|---|---|
| Employment | Education | Enrollment | Monitoring | Sanctioning | Message | Availability | ||
| Atlanta LFA | High | Moderate | Broad- delayed | Moderate | High | Encouraged use; licensed care only | No shortage | Limited |
| Atlanta HCD | Low | High | Broad- delayed | Moderate | High | Encouraged use; licensed care only | No shortage | Limited |
Grand Rapids LFA | High | Low | Broad | High | Very high | Suggested use; choice of provider | No shortage | Limited |
Grand Rapids HCD | Low | High | Broad | High | Very high | Suggested use;choice of provider | No shortage | Limited |
Riverside LFA | High | Low | Broad | High | High | Encouraged low-cost, informal care | Occasional shortage | Limited |
Riverside HCD | Moderate | High | Broad | High | High | Encouraged low-cost, informal care | Occasional shortage | Limited |
Columbus Integrated | Low | High | Broad | Moderate | Very high | Suggested use; choice of provider | No shortage | Strong |
Columbus Traditional | Low | High | Broad | Low | Very high | Suggested use; choice of provider | No shortage | Limited |
Detroit | Low | High | Selective | Low | Low | Organizational emphasis on providing assistance; choice of licensed or approved provider | No shortage | Very limited |
Oklahoma City | Low | High | Selective | Low | Low | Organizational emphasis on providing assistance; licensed care only | No shortage | Limited |
Portland | High | Moderate | Moderately selective | High | Moderate | Emphasis on necessity of arrangements; choice of provider | No shortage | Strong |
Self-Sufficiency Approaches
As discussed earlier, welfare-to-work program strategies usually emphasize either quick employment or an initial investment in education or training. The 11 programs in the NEWWS Evaluation blended elements of both strategies to varying degrees.
The kinds of messages that case managers send about education and work, the emphasis that they place on different program activities, and the activities in which program enrollees actually participate help to determine whether a client is more likely to get a job shortly after she enters the program or after she has tried to build her skills. The following program descriptions incorporate both the directions that case managers gave and the activities in which enrollees were most likely to participate.(15) Box 1.2 gives a brief description of the services offered by the programs in this evaluation.
Four of the programs are categorized as employment-focused and seven as education-focused. In the descriptions below, programs within each of the two categories are listed in rough rank order, from those that are most purely education- or employment-focused to those that blend the two approaches.
In general, the welfare-to-work programs studied in this evaluation made available to their enrollees the following services and classes:
*For a more detailed description of service components in the 11 programs, see Hamilton et al., 1997; Scrivener et al., 1998; Farrell, 2000; Storto et al., 2000; and Scrivener and Walter, 2001. |
1. Education-Focused Programs
The Oklahoma City program encouraged long-term education and training activities instead of active job search almost universally. Case managers communicated to clients the importance of education, even in job clubs, as a way to increase skills for later entry into the labor market.
The Atlanta HCD, Grand Rapids HCD, and Columbus Integrated and Traditional programs emphasized increasing skills through formal education and training before entry into the labor market. Because of the generally low educational attainment of participants in these programs, basic education was a common first activity, though Grand Rapids also encouraged participation in vocational training programs. Clients in these programs were given considerable latitude in choosing the kind of education activity they wanted to pursue.
The Detroit program underwent a substantial shift in focus over the study period. Initially, the program emphasized long-term education and training assignments before clients engaged in work search. About midway through the study period clients were referred to a program that required job search first.
The Riverside HCD program, which enrolled only individuals without a high school diploma or GED, generally assigned clients to basic education as a first activity. Short stays in these classes and active job search once a literacy benchmark was reached were stressed by case managers throughout clients' participation. Job developers assisted HCD clients in job club.
2. Employment-Focused Programs
Case managers and program staff in the Riverside, Grand Rapids, and Atlanta LFA programs emphasized that employment was the goal of program participation and that job search should be the first activity for participants. Clients were given very little choice in their first program assignment. In Riverside participants were encouraged to take even part-time and low-paying jobs as a first step to self-sufficiency and were assisted by full-time job developers who searched for job leads and followed up on job placements.(16) While Grand Rapids staff stressed to clients the importance of finding work, they believed that it might be justifiable for clients to turn down temporary or part-time jobs. Those who wished to enroll in education programs were encouraged to do so in addition to, not instead of, working. Atlanta case managers indicated the availability of education and training services as a second step after initial job search. Many Atlanta enrollees did, in fact, participate in education or training if they completed job search without finding a job.
While Portland program staff emphasized that employment was the goal of program participation, not all enrollees were assigned to job search first. For individuals who first enrolled in education or training activities, usually those who were thought by case managers to be the more disadvantaged members of the caseload, program staff communicated that improving employability was the goal of their assignment. Portland also employed full-time job developers to work with participants once they began actively looking for a job though, unlike other developers in work-focused programs in this evaluation, they encouraged participants to seek "good" jobs, that is, higher-paying jobs with benefits.
Degree of Participation Mandate Enforcement
In addition to the messages about work and education that case managers send to clients and the relative mix of services that a program provides, the degree to which a program enforces a participation mandate has also been shown to affect program impacts. The three elements of enforcement include the broadness with which a program enrolls from its caseload, how well it monitors participants' progress, and how strictly the participation requirements are enforced. In other words, a high or low ranking indicates the likelihood that a client would be told to participate, the likelihood that her case manager would know if she had not been participating, and how swiftly or surely she would be sanctioned for not participating. Nine of the 11 NEWWS programs were "high enforcement" programs; that is, they were rated the equivalent of "high" on at least two of the three elements. Two programs, Detroit and Oklahoma City, were rated the equivalent of "low" on all three elements of enforcement. (See Table 1.3.)
The rest of this section describes how each program was rated with respect to the enforcement of the participation mandate. Within each element of enforcement sites are listed in a rough rank order, from high to low. A number of factors can contribute to a program's overall ranking on an element; a site may be high on one but low on others, but these factors have not been formally weighted.
1. Broadness of Enrollment
How likely was it that an individual would have been required to participate in the welfare-to-work programs?(17)
Single parents with children aged 3 or over were required to participate in all programs studied in this evaluation, with some programs requiring participation of women with children as young as age 1. Individuals who had health barriers or were taking care of a household member who was ill or incapacitated, were pregnant, or were already working 30 hours per week could be exempted from this mandate.(18)
At a number of points, administrator and case manager discretion, combined with funding and resource constraints, could affect a welfare applicant's or recipient's chances of enrolling in a welfare-to-work program. First, five of the programs required women with children as young as age 1 to enroll. Since over 40 percent of the welfare cases nationwide in 1994 included a child under age 3,(19) expanding the mandate to this group significantly increased the proportion of the caseload that could be served by the program. Second, case managers might not tell all of those who meet the demographic criteria to enroll. Third, individuals might not show up for the program orientation because they do not wish to participate or they become exempt or leave welfare in the period between referral and orientation date, especially if the period is long.(20) Finally, even recipients who attend an orientation could be deferred from future activities at case managers' discretion.(21)
The Grand Rapids LFA and HCD, Columbus Integrated and Traditional, and Riverside LFA and HCD programs enrolled broadly, including virtually their entire mandatory caseload.(22) Both of the Grand Rapids programs included parents with children as young as age 1 in their participation mandate.
The Atlanta LFA and HCD programs aimed to enroll their entire mandatory caseloads; however, budget limitations created a waiting list, sometimes as long as six months, before those who had been referred to the program could actually enroll. During a waiting period welfare recipients with the fewest barriers to work leave the rolls on their own; thus, the clients who actually enroll may be slightly more disadvantaged than they would be if there were no waiting list.(23) Indeed, the Atlanta sample includes more long-term recipients than samples in most other sites. Because the Atlanta programs did refer virtually all members of their mandatory population to the program, and enrolled all those who were left after the delay, their enrollment is termed "broad-delayed" in Table 1.3.
The Portland program extended its mandate to parents of very young children (as young as age 1), but selectively enrolled from its mandatory population. Some individuals determined "hard-to-serve," that is, less employable, either would not be referred for enrollment in the program or, after attending a program orientation, would not be assigned to further activities. For these reasons Portland can be considered moderately selective.
The Detroit and Oklahoma City programs also extended their mandate to women with very young children, but were more selective than other programs. Like Atlanta, Detroit had a waiting list for "slots" in the program. Guided by the principle that the program would rather spend scarce resources on those who wished to participate than on cajoling those who might never participate, staff tended to give priority to "mandatory" clients who volunteered for the program. In addition, case managers spent a large proportion of their time authorizing child care and support service payments, leaving little time to focus on individuals who were not eager to enroll. Oklahoma City referred all those eligible to its program; however, since the program was also limited by resources and rising caseloads, much of the responsibility for enrolling in program activities fell on the client. Case managers assisted clients in finding appropriate services, but the self-directed enrollment allowed resistant individuals to avoid the mandate. As discussed in Chapter 2, this evaluation examines only the experiences of individuals applying for welfare ("applicants") in the Oklahoma City program; the treatment of those already receiving welfare ("recipients") may have been different from the situation described here.
2. Closeness of Participation Monitoring
How often or how quickly would an enrollee be contacted by her case manager if she was not participating?
Once clients begin participating, they may drop out of activities or attend irregularly because they have a new job, have new problems with child care or transportation, or no longer want to participate. Close monitoring can help case managers maintain and increase participation among their caseload, facilitate the authorization of transitional benefits for individuals who leave welfare for work, or speed case closures for individuals who become ineligible. In order to monitor participation closely, case managers must learn about attendance problems from activity providers, determine the reasons for them, contact clients about their options or the consequences of nonparticipation, and then inform the income maintenance branch of a case's outcome. How closely an individual will be monitored depends on the level of information that case managers get from the activity instructors and providers and on the time that case managers have to devote to this task.
The Riverside LFA and HCD, Grand Rapids LFA and HCD, and Portland programs all intensively monitored their participants' progress. Overall, more case managers in these sites indicated receiving a lot more information about attendance from providers than those in most other sites. In addition, case managers reported that it took them between one and two weeks to both hear about attendance problems from providers and contact clients about their attendance, the shortest in the range of time among the programs.
The Atlanta LFA and HCD and Columbus Integrated programs engaged in moderate monitoring of their clients. Information sharing between providers and case managers was not as regular in these programs as in the intensive-monitoring programs, and it took between two and a half and three and a half weeks to get information from providers. These programs did, however, contact clients in less than two weeks once they learned of attendance problems.
The Oklahoma City, Detroit, and Columbus Traditional programs engaged in less intensive monitoring of their clients than the other programs. Regular protocols for obtaining attendance information from providers were not in place for at least two of the programs. It took a little longer, on average, for all three programs to get information from providers than it did for the moderate-monitoring programs. Moreover, it took between two and three weeks for case managers to contact clients about their attendance problems, on average one week longer than for the moderate-monitoring programs.
3. Level of Mandatoriness
How much would an individual be encouraged, or coerced through financial sanctions, to participate in a program activity if she did not want to?
The great majority of welfare recipients who are required to participate in welfare-to-work programs believe, prior to hearing details about the program, that they will have trouble participating, citing barriers such as a lack of child care or transportation or having a health or emotional problem.(24) All the programs in this evaluation provided monetary assistance to help participants (both program and control group members) with child care and transportation, but they also relied on case managers to work with clients to remove participation barriers or to coerce participation through the imposition of a financial sanction.(25) Most of the programs were strongly committed to enforcing the participation mandate for their welfare caseload, though the degree to which clients were more likely to be cajoled or coerced differed. Individuals in Detroit and Oklahoma City were not as likely to be cajoled into participating if they did not want to, though this was largely the consequence of limited program funding and staffing.
The Grand Rapids LFA and HCD and Columbus Integrated and Traditional programs were very highly committed to the enforcement of clients' participation obligation. Case managers sent strong messages about the consequences of nonparticipation and, in instances of noncompliance, imposed financial sanctions swiftly on a large percentage of their caseloads.
While other programs informed clients of the necessity of program participation, they gave them more chances to comply than Grand Rapids or Columbus. Atlanta LFA and HCD program case managers were somewhat less comfortable with enforcing participation requirements through financial sanctions, though they did so on a regular basis. More clients were sanctioned in the Atlanta HCD program than in its LFA program, though the messages that case managers sent about requirements were not different.
Riverside LFA and HCD program staff tended to view sanctions as one tool to get clients to attend activities and initially emphasized to clients the importance of personal responsibility. Riverside staff did not delay requests for or impositions of sanctions; the process, however, took longer than it did in most other programs because of extensive state-mandated due process procedures. The Portland program staff also emphasized ways to solve problems related to nonparticipation rather than reductions in clients' grants. Staff in Riverside and, to a greater extent, in Portland were more willing to defer individuals from participation requirements than staff in either Columbus or Grand Rapids. Staff in Portland did, however, ultimately sanction noncompliant individuals.
The mandatory participation requirements in the Detroit and Oklahoma City programs were communicated less intensively to clients. As already mentioned, staff in these two sites focused on those who wanted to participate. Resource constraints kept staff from following up on nonparticipation, and staff tended to delay imposing sanctions.
Other Program Features
1. Child Care Supports for Participation and Work
How much support in the form of child care assistance could an individual expect for her participation in a program or subsequent employment?
For many welfare recipients with young children the major obstacle to working or attending an education or job training program is child care. All 11 programs studied in the evaluation provided this assistance to participants in the program (and to control group members who enrolled in activities on their own in the community) as well as transitional child care (TCC) for those who left welfare for work. However, the relative emphasis that the programs placed on making this assistance available and the messages that case managers sent to clients about the type of care they should choose varied by site (not by research group within site).
Participation-related child care. In the Atlanta LFA and HCD, Oklahoma, Portland, and Detroit programs, child care assistance was emphasized either by site staff or by the welfare department's organizational structure. In both Atlanta programs case managers actively promoted the availability of child care reimbursement as a benefit of program participation and even used it as an inducement for noncompliant clients to participate. In Oklahoma statewide emphasis on access to child care made assistance to clients readily available while they were in the program and after they left welfare for work. Oklahoma had no set caps on the amount of child care assistance that clients could receive. Atlanta and Oklahoma reimbursed only for care given by licensed providers.
Portland program caseworkers told clients that not having child care arrangements was not an acceptable reason for not participating in program activities. Staff often encouraged clients to have backup arrangements in case their regular provider fell through. Although case managers did not push specific types or locations of providers, they did emphasize the necessity for clients to make arrangements and assisted clients who were unable to make arrangements on their own.
Detroit program case managers reported that they spent much of their time on child care payment authorizations and that the priority placed on making child care payments took time away from employment and training counseling. Detroit staff would make referrals to licensed providers in the area on request, but the choice of provider (including choosing licensed child care or unlicensed care approved by the welfare department) was left to the client.
Both the Grand Rapids and Columbus programs would reimburse expenses from child care in licensed as well as unlicensed care, but expected clients to make their own arrangements. Referrals to licensed providers in the area could be made for clients at their request.
In all sites except Riverside, case managers said that child care providers were not difficult to come by. In Riverside, case managers noted that some area providers did not like working with the program or its participants because they did not approve of the reimbursement rates or procedures. These case managers encouraged clients to use low-cost, more informal arrangements, both to contain program costs and because they believed that clients would be more able to afford such arrangements after program or other government supports expired. Clients and case managers often clashed about the providers they wished to use, especially if clients chose more expensive care.
Transitional child care. In the Detroit, Portland, Columbus, and Oklahoma City programs, authorization for TCC payments did not appear to be difficult. In Portland, Columbus Integrated, and Oklahoma the use of integrated case managers, who are more likely to know both the welfare and employment information needed to determine if a client is eligible for TCC, may have made authorization in these sites easier for both program and control group members.
In the Atlanta, Grand Rapids, and Riverside LFA and HCD programs, few clients who began working received TCC; case managers in all three sites cited a lack of information about clients' welfare status when authorizing child care payments. Thus, TCC authorization in these sites would have been infrequent for both program and control group members.
2. Culture of Eligibility to Culture of Self-Sufficiency: Integrated Case Management
How likely was a welfare recipient to get a unified self-sufficiency message from the welfare department?
The eligibility-compliance culture of the welfare system (more common prior to FSA), in which contact between a client and an agency is focused solely on determining eligibility for staying on welfare, has been harshly criticized. Implementing a mandatory welfare-to-work program was one way that welfare offices hoped to change from an eligibility-compliance culture to a self-sufficiency culture, which would structure interactions and expectations around leaving welfare for work and preparation and supports for it. Yet this task is formidable; it requires the income maintenance and employment services staffs of the welfare offices to work together to send a unified message of the self-sufficiency goal to the client. If the sole responsibility for delivering the self-sufficiency message is remanded to the employment and training program, programs can be interpreted by clients and workers as requirements for continued receipt of assistance, or another element of compliance, instead of an overhaul of the philosophy of the welfare department. Implementing an integrated case management approach, in which one worker is responsible for both the eligibility determination and employment services functions, is one way that has been suggested to achieve a more unified culture.(26) Three of the programs in the NEWWS Evaluation used integrated case management, but these and the other eight programs met with different levels of success in coordinating the messages between their eligibility and employment preparation staffs and in refocusing the welfare department's interactions with clients on the road toward self-sufficiency.
As part of a specially formulated research experiment, the Columbus Integrated program used integrated case management. Staff had sufficient resources and small enough caseloads that they were able to perform both their income maintenance and self-sufficiency roles. Thus, an individual's case manager could both monitor her progress in becoming self-sufficient and verify her credentials for staying on welfare. As of the two-year follow-up point, this program had the largest effect on changing clients' minds about whether they agreed that the welfare office tried hard to get recipients employed or enrolled in school.
The Portland program was marked by a strong partnership between welfare-to-work staff (eligibility workers and integrated case managers) and case management staff contracted by the Portland welfare department. The division of labor was flexible between contractor staff and welfare department integrated case managers, with responsibility for case management services such as reassignment to activities and attendance monitoring, as well as a mission of promoting self-sufficiency, shared by both. Moreover, eligibility workers in Portland were among the most knowledgeable about the program and spent more time discussing the program with recipients than those in most other programs. These results suggest that together eligibility workers, integrated case managers, and contractor staff were able to send a unified self-sufficiency message to welfare recipients.
The Oklahoma City program also used integrated case management. However, limited resources and large caseloads led case managers to put little overall emphasis on the employment services function of their position; in fact, their performance evaluation benchmarks were primarily related to the accuracy of their eligibility duties. Like Portland, Oklahoma City supplemented its integrated case managers with some caseworkers who focused on employment-related services. However, owing to staffing constraints, not all clients received this added case management. The result was a program with little overall emphasis on self-sufficiency.
The Atlanta, Grand Rapids, Riverside, and Columbus Traditional programs all used a separated, or "traditional," case management structure, in which a client had two different case managers, one who specialized in determining eligibility and processing payments and another who focused on her participation and progress in a welfare-to-work program. Although the different staffs did not report any major problems in their working relationship, they mentioned that there was a lack of partnership between the two. Income maintenance workers knew little about the programs and most often discussed with clients the penalties for nonparticipation in the program, not the services it provided, suggesting that participation was cast as a compliance requirement and not a route to self-sufficiency.
The separation between the two staffs of the welfare department in the Detroit program was even more pronounced. Income maintenance workers knew little about the program and had almost no contact with clients regarding their participation; the welfare-to-work program case managers in Detroit handled some income-related functions related to program participation, such as child care payments, that income maintenance workers were responsible for in the other traditional sites. Staff mentioned that this separation was intentional, so that the welfare-to-work case managers would be able to communicate consistent messages and information. In short, the priorities of the two staffs were so dissimilar that an individual was likely to experience very different cultures during her contact with the department.
Program Changes in the Last Two Years of the Five-Year Follow-Up Period
The previous section described the 11 programs as they existed during the first three years of the follow-up period. In the last two years of the five-year follow-up period, however, the focus of many of the programs changed. Most commonly, some of the education-focused programs became more employment-focused, a change driven in part by the 1996 welfare reform law.(27) This shift was not incompatible with what might normally happen in an education-focused program. It would have occurred several years after most program group members started their participation in education or training activities. Most education-focused welfare-to-work programs involve periodic job search, particularly once individuals have improved their skills or achieved a credential. Therefore, it is unlikely that the shift to an employment-focused program would have greatly compromised the nature of the treatment studied in these originally education-focused programs. In addition, the changes in program emphases would have affected only those program group members still receiving welfare in the last two years of the five-year follow-up period. As discussed in the next section, which covers changes in control group members' treatment over time, this proportion varied by site. The following paragraphs highlight the most important program changes.
In Atlanta, the site (and state) implemented a strongly job search-focused "work first" program in December 1996. For LFA sample members, the new program continued their employment-focused treatment. For HCD sample members, the new program represented a change in program message and emphasized activities. At the same time, Georgia began the count of months toward a welfare time limit (which would have affected sample members in all three of the Atlanta research groups).
In Grand Rapids, a Work First program was implemented during the five-year follow-up period as well, in October 1994, but referral to this program was delayed for most LFA and HCD sample members. For individuals assigned to the LFA and HCD research groups in roughly the first half of the Grand Rapids random assignment period, referrals to the Work First program could be made three years after random assignment if these sample members were still receiving welfare. Reviews of Work First program databases, however, indicated that as of May 1996, which would have been about the fourth year of the five-year NEWWS follow-up period for most of these sample members, less than 10 percent of this group had, in fact, been referred to the new program. Individuals assigned to the LFA and HCD research groups in roughly the second half of the Grand Rapids random assignment period continued to be part of the original LFA and HCD programs throughout the five-year follow-up period. Grand Rapids also took part in another statewide initiative, Project Zero, which sought to drastically reduce the number of nonworking adults on the welfare caseload, but implementation of this program did not occur until almost all NEWWS sample members were beyond the end of their five-year follow-up period.
In Riverside, the LFA program focus and components generally carried over into years 4 and 5 of the five-year follow-up period. The HCD program, however, had more of an employment focus in years 4 and 5. Once HCD sample members completed their education activities in the last two years, it is likely that they would have been assigned, if still on welfare, to job search. If they did not find a job through job search, it is unlikely that they would have been assigned to more basic education; rather, they probably would have been assigned to do more job search or to attend group self-esteem-building sessions. Assignments to vocational training remained rare for both LFA and HCD sample members in the last two years of follow-up, as it had in the first three years. (The CalWorks program in California was implemented at a point that would have been, for most Riverside sample members, after the end of the five-year follow-up period examined in this report; its provisions are thus largely not reflected in the behavior of sample members examined here.)
In Columbus, several program changes took place in October 1997. First, all sample members in both the Integrated and Traditional program groups began to receive integrated case management, reflecting a county-wide shift to this type of arrangement. Second, the program became much more employment-focused than education-focused. Third, Ohio began the count of months toward a welfare time limit (which would have affected sample members in all three Columbus research groups).
Reflecting the Michigan changes outlined above for the Grand Rapids site, the Detroit program was transformed in October 1994 to the strongly employment-focused Work First program, operated by an agency other than the welfare department. In this site, unlike Grand Rapids, it was not possible to implement procedures whereby program group members could continue to be eligible for their "original" program. As a result, sample members still on welfare in late 1994 would have experienced an abrupt change in program type and focus, from an almost voluntary program emphasizing education to a strongly mandatory one emphasizing employment.(28)
The Oklahoma City program also became more employment-focused in years 4 and 5 of the five-year follow-up period. In addition, in October 1996 Oklahoma began the count of months toward a welfare time limit (which would have affected both program and control group members in the site).
In Portland, program group members would have continued to receive an employment-focused treatment in the last two years of the five-year follow-up period. Several other program changes, however, occurred in these years. First, over the course of the five years, the program made more use of job search and less use of education. In the last year of follow-up, for example, program staff were urged to reduce the number of assignments to GED classes and, when made, to limit the assignments to only three to six months. In 1999, GED and other education programs were viewed more as activities that could help working individuals retain and advance in jobs than as activities for initial work preparation. Along with this, life skills activities were still offered, but were compressed to two weeks, from the five to six weeks observed in the first three years of the follow-up. Second, in the last two years of follow-up, staff made more short-term training assignments (of three to six months) and worksite placements (a type of "supportive" work experience). Finally, toward the end of the follow-up period, Portland started to move away from a type of integrated case management and toward traditional case management.
Control Group Treatment Over Time
Part of the reliability of the findings from a random assignment social experiment rests on the assumption that control group members are not exposed to the specific program being evaluated. If control group members are exposed to the program, measured impacts could represent an underestimate of the true effects of the program. Rarely, however, is it possible to maintain this lack of exposure for a long period of time. In this regard, the NEWWS Evaluation was no exception.(29)
Agreements reached with the NEWWS Evaluation sites at the beginning of the study specified that all control group members were to be kept out of the welfare-to-work programs being studied in each site for at least three years from their random assignment date. (As noted earlier, control group members were free to enroll in other employment-related activities offered in their communities during this period and, as will be discussed below and in Chapter 3 of this report, many controls did, in fact, enroll in such activities.) Midway through the evaluation, program operators in Riverside and MDRC agreed to extend the embargo on providing welfare-to-work program services to control group members to five years. Program operators in Portland and Grand Rapids agreed to a similar extension, but only for a subsample of control group members. In Portland, about one-quarter of the control group was randomly selected (from every month of the random assignment period) to remain ineligible for welfare-to-work program services until the end of year 5; in Grand Rapids, the embargo on welfare-to-work program services was extended for all control group members who were randomly assigned during 1993, the last year of sample intake in the site. However, Atlanta, Columbus, and Oklahoma City could not continue barring control group members from welfare-to-work program services longer than three years because of the implementation of welfare time limits in their states; if NEWWS control group members could not be exempted from welfare time limits, then both HHS and MDRC felt it necessary to allow control group members to be eligible for some type of welfare-to-work program services once their time limit clock "starting ticking." Finally, in Detroit, it was not feasible to continue the control services embargo, after a different agency took over administration of the site's welfare-to-work program and entry points into the new program were so numerous and widespread that screening to identify NEWWS control group members would have been close to impossible.
Most probably program impacts on employment and earnings and other outcomes in the last two years of follow-up in a few of the five sites above would have been somewhat larger had some control group members not been exposed to welfare-to-work programs. For several reasons, however, lifting the control group embargo on services prior to the end of the five-year follow-up period in these five sites most likely had only a small effect on measured program impacts. As discussed in detail in Chapter 2, most control group members were not eligible to receive program services when the control group embargo ended, often because they had already left welfare. From one-quarter to one-half of control group members in these five sites were receiving welfare when their embargo on program services was lifted. In addition, after the embargo was lifted, some control group members were not contacted about enrolling in the program until after the end of the follow-up period, and others were assigned to a program orientation but did not show up. Moreover, the likely effects of ending the control group embargo were estimated for the sites where the embargo was lifted by calculating impacts for a subsample of control group members who were precluded from program services for four to five years. It was found that the patterns of impacts in years 4 and 5 resembled those for all sample members in those sites.
It should be emphasized that the control group situations described above do not affect the assessments in this report of the relative merits of the Labor Force Attachment and Human Capital Development approaches in welfare-to-work programs. As described in Chapter 2, the three-group random assignment designs in the three sites in which these two types of programs were simultaneously operated permit a direct comparison of these two approaches, that is, a comparison that does not need to take into account the services received by and the behavior of control group members.
A Brief Review of Two-Year Impact Results
Several reports have documented the range of effects achieved by all 11 NEWWS programs within a two-year follow-up period.(30) This report extends and expands on these two-year findings, lengthening the follow-up period to five years and including additional outcome measures. As a basis for understanding the long-term findings, this section recaps the two-year findings, noting which welfare-to-work program strategies were more or less successful in helping welfare recipients achieve self-sufficiency in the relatively short run.
All programs, regardless of their approach, increased participation in activities designed to promote employment during the two-year follow-up period. As expected, employment-focused programs increased participation primarily in job search activities, whereas education-focused programs increased levels primarily in basic education and vocational skills training classes. Very different patterns of participation impacts were found for individuals who entered the study with a high school diploma or GED certificate and for those who did not have these credentials. In most education-focused programs participation impacts were concentrated among those without a high school diploma or GED and resulted primarily from large increases in attendance in basic education; only small increases in attendance in post-secondary education or vocational training were found for the education-focused programs, and they were generally among only high school graduates or GED holders. In contrast, large impacts on participation in job search were achieved for both groups in the employment-focused programs.
Some education-focused programs, as well as the Portland program, were able to produce relatively large two-year impacts (about 10 percentage points) on GED attainment among sample members who did not have a high school diploma or GED certificate at study entry. Of the seven education-focused programs, the Grand Rapids HCD, Riverside HCD, and Columbus Traditional programs had this effect. The Portland program, in addition to boosting GED receipt, increased the rate at which those without education credentials obtained a trade license or certificate by 12 percentage points. For sample members with a high school diploma or GED certificate at study entry, only three programs (Atlanta LFA and HCD and Grand Rapids HCD) increased receipt of a trade license or certificate.
As expected, employment-focused programs produced larger gains in employment and earnings over the two-year follow-up period than education-focused programs. A majority of control group members in all sites except Riverside, the site with the most difficult labor market, found jobs on their own at some point within two years of random assignment and, as a group (including zeroes for nonearners), had average earnings during the second year of follow-up ranging from $2,127 (Oklahoma City) to $3,978 (Columbus). The Portland program attained the largest earnings increase of all programs, with members averaging more than $900 per year in earnings above control group members. Equally important, employment and earnings gains in Portland grew larger over time and reached their highest levels at the end of year 2. The other employment-focused programs produced moderate earnings increases, ranging from $400 to $650 per year, that grew smaller toward the end of year 2.
Several of the education-focused programs began to show moderate impacts in year 2. By the end of year 2 all but two of the education-focused programs had attained increases in employment and earnings that equaled or exceeded the gains achieved by all employment-focused programs except the Portland program. The two exceptions to this pattern, the Riverside HCD and Oklahoma City programs, did not raise employment or earnings levels in year 2.
All programs reduced two-year welfare dependency to some degree. Control group members in all but one site remained on welfare for an average of 16 to 20 months during the two-year follow-up period and received payments averaging between $3,624 (Oklahoma City) and $10,302 (Riverside HCD) during this period. Seven of the 11 programs, a mixture of employment- and education-focused approaches, decreased cumulative welfare expenditures by more than 10 percent, a historically large effect; welfare reductions in the other four programs were smaller. The Portland program produced a large decrease in welfare receipt that persisted at a high level throughout the follow-up period, showing a 12 percentage point decrease in welfare receipt during the last quarter of the two-year period; all other programs had reduced welfare receipt at this point by 3 to 7 percentage points. However, at least 40 percent of sample members in the programs were still relying to some extent on welfare at the end of two years.
Most programs increased sample members' reliance on earnings rather than welfare, but family net incomes were largely unchanged. As a result, within a two-year follow-up period the programs lifted few families above the poverty line. Impact estimates of reductions in welfare, Food Stamps, and other benefits generally matched or exceeded impact estimates of earnings gains. Including estimates of income from the Earned Income Tax Credit (EITC) produced little change in this finding for all programs except Portland's, which attained the largest and most consistent gain in total combined income ($238, or $425 including the EITC estimate, for year 2 of the follow-up) and also produced a small increase in the proportion with combined incomes above the poverty level (4 percentage points, or 7 percentage points including the EITC estimate, in year 2).
Although no programs had pervasive negative effects on sample members in the first two years of follow-up, some individuals were adversely affected. In year 2 of follow-up six programs (some employment-focused and some education-focused) produced small increases in the proportion of sample members with combined income from welfare, Food Stamps, and earnings equivalent to less than 50 percent of poverty levels. In addition, several programs (representing both types of approaches) increased the rate at which individuals left welfare without a job (although other persons in their household might have been working). Finally, some programs that increased employment also decreased family health insurance coverage (as reported by parents) and increased out-of-pocket child care expenditures.
The programs did not have widespread, large, or consistent two-year effects on the children of sample members, but positive and negative effects occurred in some programs. No programs in the evaluation provided direct services (with the exception of child care assistance) to children. Program-produced changes in the lives of sample members (virtually all mothers) may, nevertheless, influence the well-being of children. There is evidence that some of the programs affected the likelihood of at least one child in a family having behavioral, educational, or health and safety problems. There was not, however, a consistent pattern of benefit or harm to children. In addition, employment- and education-focused programs did not appear to affect children differently; there was no consistent evidence that one particular approach affected children more or less or was more likely to help or harm children.
Several employment- and education-focused programs attained at least moderate two-year employment and earnings gains for the "most disadvantaged" sample members. Five programs (Portland, Grand Rapids LFA and HCD, and Riverside LFA and HCD) increased employment and earnings for individuals who at study entry did not have a high school diploma or GED, had not worked in the prior year, and had been on welfare cumulatively for two years or more. These five programs and two others (Detroit and Columbus Integrated) also reduced the amount of time that the most disadvantaged individuals spent on welfare during the two-year follow-up period. Notably, for individuals who entered the study without a high school diploma or GED, the two-year employment and earnings impacts of the education-focused programs did not exceed those of the employment-focused programs.(31)
High enforcement programs did not produce the largest two-year impacts, but low enforcement programs resulted in only small effects. High enforcement programs, notably those in Grand Rapids and Columbus, did not necessarily produce the largest impacts. However, the two low enforcement programs Oklahoma City and, in its early stages, Detroit yielded only small impacts. It thus appears that a minimum level of enforcement by program staff is required to produce at least moderate earnings and welfare impacts, presumably because this extra "push" is needed in order to engage in program activities those who normally would not participate on their own initiative.
While many programs achieved positive two-year effects on employment, earnings, and reduced use of welfare, the Portland program stood out as unusually successful. The Portland program substantially increased employment and earnings, helped people to get good jobs, lowered welfare receipt, and achieved these outcomes for a cross section of sample members. The results are probably due to a combination of factors. While its employment message was strong, the program offered high-quality education and training services as well as job search, enforced a participation mandate, and had strong job development and placement services. In addition, contextual factors may have contributed to the program's success. In particular, it worked with a less disadvantaged welfare caseload (relative to the other studied programs) and operated within a good labor market with a relatively high state minimum wage.
Contents of This Report
This report builds and expands on the two-year findings. Chapter 2 and 3 give important background information about the NEWWS Evaluation and its participants and their activities. Chapter 2 describes the random assignment research design used to test the effectiveness of the programs, the definition and characteristics of the various samples included in this report, and the types and sources of data used. Chapter 3 describes the five-year effects of the programs on increasing participation in work-related activities. The chapter also documents whether programs increased the percentage of recipients who earned GEDs or other education credentials after random assignment. Importantly, the chapter expands the discussion in this chapter on the extent to which both control and program group members received welfare-to-work program services in the last two years of the five-year follow-up period.
Chapters 4, 5, 6, and 7 examine the five-year economic impacts of the programs, for all sample members as well as selected subgroups. These longer-term impacts could reflect an extension of the effects of services and experiences in the first two years of follow-up, the effect of new services and experiences in follow-up years 3 to 5, or a combination of both. Chapter 4 discusses the impacts of the programs on sample members' employment, earnings, job stability, and job quality. The chapter investigates whether employment- or education-focused programs fared better and what caused increases in average earnings: putting to work welfare recipients who would not have found jobs on their own, improving job quality for those who would have been employed anyway, or both. Chapter 5 presents impacts on welfare and Food Stamp receipt and payments, determining whether the programs achieved welfare savings and whether they did so by increasing the speed or frequency of welfare exits or by decreasing average grants for those on public assistance. Chapter 6 looks at earnings gains and welfare reductions from the perspective of sample members and presents impacts on individuals' combined income from earnings and benefits, level of self-sufficiency, and prospects for longer-term economic security. Chapter 7 determines the effects of alternative program strategies for different subgroups of welfare recipients. It explores the degree to which programs helped groups of the welfare population likely to have different capacities to find work on their own: those who had limited education credentials, those who were more disadvantaged (without recent work experience and who had been on welfare for two or more years), and those who were less disadvantaged.
Chapters 8, 9, and 10 are concerned with several noneconomic family outcomes. Chapter 8 examines the ways that programs affected sample members' health care coverage, for adults as well as children in the families studied. Chapter 9 discusses the effects of the programs on individuals' household and personal circumstances, examining impacts on marital status, the structure and composition of families and households, fertility, and barriers to work from and abuse by intimate partners and others. Chapter 10 looks at the effects of these welfare-to-work programs on sample members' work-related child care arrangements and on children's daily activities.
Chapters 11 and 12 examine the effects of the programs on children. Chapter 11 presents impacts on children of all ages in four of the sites using limited measures of child well-being. Chapter 12 presents impacts on a subset of children in three sites who were preschool-age at the start of the NEWWS Evaluation and generally between ages 8 and 10 at the five-year follow-up point; these analyses use in-depth child well-being measures, constructed from information supplied by parents, elementary school teachers, and the children.
Finally, Chapter 13 presents a benefit-cost analysis for each program studied, weighing benefits and costs from the perspectives of government and the sample members themselves.
1. The programs and individuals studied in this evaluation are drawn from the entire county (or counties) mentioned in parentheses after the city name; for ease of reference, in this report the sites will be referred to by the name of their corresponding urban area.
2. See Friedlander and Burtless, 1995; Bloom, 1997; and Riccio, Friedlander, and Freedman, 1994.
3. It is important to note that the studies of the programs in the education-focused category yield information about the effects of increasing welfare recipients' participation in basic education programs (including Adult Basic Education, GED preparation, and English as a Second Language classes) and, to a much lesser extent, in vocational skills training programs, but not in college. On their own, many welfare recipients enroll in various types of education or training classes and reap benefits from them; the education-focused programs in the evaluation, however, sought to increase participation in education or training activities beyond what would normally occur. As will be discussed in Chapter 3, most of the programs did indeed increase such participation, but the increases in enrollments were in basic education courses and, to some degree, in vocational training courses, but generally not in college-level ones.
4. See Scrivener and Walter, 2001, for a full discussion of the results of the direct test of Integrated versus Traditional case management in the Columbus site.
5. See Freedman et al., 2000a, p. ES-6; Hamilton and Scrivener, 1999, pp. 24-31; Bloom, 1997, p. 51; Kemple, Friedlander, and Fellerath, 1995, pp. ES-2 and ES-3; and Friedlander et al., 1987, pp. vii-x.
6. Control group members were eligible for child care assistance similar to that offered to program group members if they were participating in nonprogram activities in which they had enrolled on their own.
7. While comparisons of impacts across programs or sites are not as reliable as the impacts for each program or site in the evaluation, they are a much more accurate determination of which types of programs are high and low performers than simple comparisons of statistics, such as welfare caseload reductions, across localities or states.
8. For a description of the site selection process, see Hamilton and Brock, 1994, Appendix A.
9. The participation of single mothers in the labor market also increased dramatically during this time period. Rates of employment for single mothers with any children under age 18 increased from 57 percent in 1992 to 71 percent in 1999 and for those with a child under age 3 increased from 35 to 56 percent over this same time period (U.S. House of Representatives, 2000, pp. 1412-1427).
10. Data presented in this chapter are for the entire county (or counties) from which each site draws its sample members.
11. These amounts assume the receipt of the maximum welfare payment.
12. Greenberg, 1992.
13. The federal EITC is a credit against federal income taxes for taxpayers with annual earnings below a certain level.
14. See Hamilton et al., 1997; Scrivener et al., 1998; Farrell, 2000; Storto et al., 2000; and Scrivener and Walter, 2001.
15. See also Hamilton et al., 1997; Scrivener et al., 1998; Farrell, 2000; Storto et al., 2000; and Scrivener and Walter, 2001. In addition, Bos et al., 2001, provides a detailed description of how adult education programs were implemented in the Atlanta, Grand Rapids, and Riverside HCD programs.
16. Given Riverside's high grant level and earnings disregard policies, individuals could remain on welfare until they had a full-time wage of $8.35 per hour.
17. See also Hamilton and Brock, 1994, pp. 51-55, for a more detailed description of the sites' enrollment practices.
18. Exceptions for health reasons were typically few. In Columbus, for example, where detailed data on exemptions are available, the majority of exemptions were due to the youngest child being under age 3; less than 9 percent of the exempt single parents had a long-term illness or incapacitation and less than 3 percent were caring for an ill or incapacitated family member (including a child). See Hamilton, 1995.
19. U.S. House of Representatives, 1996, Table 8-32; percentage of all AFDC households with a child under age 3.
20. See Hamilton and Brock, 1994, and Knab et al., 2001, for a discussion of the length of time between referral to and enrollment in welfare-to-work programs and reasons for orientation nonattendance.
21. As described in more detail in Chapter 2, in most of the sites, random assignment to research groups occurred once individuals attended program orientations. Only in Columbus and Oklahoma City did random assignment occur earlier.
22. In Riverside, individuals working 30 hours per week or more were not enrolled in the program. In addition, program enrollees who were employed 15 to 29 hours per week were not assigned to additional, concurrent program activities.
23. See Bane and Ellwood, 1983; Pavetti, 1992; Gueron and Pauly, 1991; and Hamilton and Brock, 1994.
24. See Hamilton and Brock, 1994.
25. The financial sanctions in effect during the NEWWS Evaluation were not the "full-family" sanctions currently being implemented in numerous states. During the period under study, sanctions affected only the adult member of the welfare case; for a three-person family in 1993, for example, a sanction would reduce the welfare grant by approximately 20 percent, depending on the site. In addition, the financial penalty continued until the sanctioned individual complied with the program participation mandate, with a minimum sanction length of three months for the second "offense" and a minimum length of six months for the third offense (with no minimum length for the first offense). As a result, some sanctioned individuals endured a penalty for a short amount of time while some experienced a penalty for much longer. For example, almost half of those sanctioned in the two Grand Rapids programs were in this status for at least 12 months of a 24-month follow-up period. See Hamilton et al., 1997; Scrivener et al., 1998; Farrell, 2000; Storto et al., 2000; and Scrivener and Walter, 2001, for more details on sanction practices in each of the NEWWS Evaluation sites.
26. See Scrivener and Walter, 2001, pp. 3-5; and Bane and Ellwood, 1994, p. 127.
27. Three evaluation sites also implemented welfare time limits during the later part of the five-year follow-up period. These would have applied to both program and control group members. This means that the count of months toward their welfare limit would have started; no sample members actually would have reached their welfare time limit during the follow-up for this evaluation.
28. See Farrell, 2000, for details on how this change came about.
29. The authors know of only one large-scale social program evaluation the GAIN Evaluation, a study of California's late 1980s and early 1990s welfare-to-work program that, by design, kept control group members from exposure to the specific program being evaluated for a follow-up period as long as five years.
30. See Freedman et al., 2000a; McGroder et al., 2000; and Hamilton, 2000.
31. For an extension of these results into the third year of follow-up, see Michalopoulos and Schwartz, 2001.
Research Design, Sample Characteristics, Data Sources, and Analysis Issues
The primary aim of the NEWWS Evaluation was to test and compare the effectiveness of a variety of welfare-to-work approaches in different locales. This chapter describes the research designs employed, the samples of people studied and their characteristics, and the types of data used in the report. It concludes with guidelines for interpreting the results presented in the chapters that follow.
Research Design
To assess the effectiveness of different welfare-to-work strategies, the evaluation used a random assignment research design. In each of the seven sites in the evaluation, people who were required to participate in a welfare-to-work program were assigned, by chance, either to a program group, which had access to employment and training services and whose members were required to participate in the program, or to a control group, which received no program services and whose members were not subject to a participation requirement but could seek out similar services on their own in the community.(1) Program group members who did not comply with the participation mandate risked incurring a sanction, that is, having their welfare grant reduced. Control group members, in contrast, could not be sanctioned because of the control embargo that precluded them from participating in program activities. Throughout the report, the program and control groups are referred to as research groups and the people in them as sample members. The random assignment design ensured that there were no systematic differences between the background characteristics of program and control group members when they entered the study. Thus, any subsequent differences between the groups' outcomes (called impacts) can be attributed with confidence to the effects of the programs.
Sample members in each research group were tracked over a follow-up period of five years after their date of random assignment. Average outcomes for control group members (such as employment and welfare receipt) after random assignment represent what could be expected of welfare recipients had they never enrolled in a welfare-to-work program. Past studies have shown that many people to whom welfare-to-work programs are targeted will leave welfare and find work on their own, that is, without being assigned to a welfare-to-work program.
The differences between outcomes for the program and control groups represent the impacts or effects of each program. Unless otherwise noted, all "increases" and "decreases" reported in this document refer to such program-control differences.
Random Assignment Designs
Four of the sites implemented a three-way random assignment research design in order to test the effectiveness of two different program approaches. In the three-way design, each person was assigned, by chance, to one of two program groups or to a control group. This design is especially powerful because members of both research groups have the same background characteristics and face the same labor market conditions and other environmental factors that can affect a program's success in helping people find jobs and advance toward economic security. In the three-way sites, the relative effectiveness of the two programs can be assessed by comparing outcomes for the program groups with one another directly, that is, without taking the control group into account. To assess the programs' absolute effectiveness, however, it is necessary to compare outcomes for each program with the control group's outcomes.(2)
Three of these four sites (Atlanta, Grand Rapids, and Riverside) operated two programs that were designed for purposes of the evaluation to magnify the differences between the employment- and education-focused approaches described in Chapter 1: Labor Force Attachment (LFA) programs, which emphasize rapid job placement as the best way for welfare recipients to develop their work habits and skills, even if the job pays low wages; and Human Capital Development (HCD) programs, which emphasize that welfare recipients have to develop their "human capital," that is, their knowledge and basic skills, through education and training in order to have a better chance of finding and keeping jobs and advancing toward well-paid and secure employment. In each site, the two program models were implemented to maximize the contrast between them, thus making the differences between their effects easier to detect. Figure 2.1 illustrates the process by which welfare recipients and applicants in Atlanta and Grand Rapids were randomly assigned to the research groups.
Figure 2.1
Steps Leading to Random Assignment in Atlanta and Grand Rapids
The random assignment process differed in Riverside because California's welfare rules mandated that only people "in need of basic education" that is, people who lacked a high school diploma or GED, scored low on a welfare department math or reading literacy test, or required instruction in English as a Second Language (ESL) could be assigned to the HCD group. This constraint meant that whereas the HCD group included only people determined to need basic education, the LFA group included both such people and people determined not to need basic education. To facilitate direct comparisons between the Riverside LFA and HCD groups in this report, the results for the subgroup of LFA group members determined to need basic education are provided in addition to the results for the full LFA group.(3) A second conse-quence of this constraint in Riverside is that the only way to make direct comparisons between the results of the Riverside HCD program and those of the programs in other sites in the evaluation is to focus on the subgroups of people in those other programs who lacked a high school diploma or GED.(4) Figure 2.2 illustrates the process by which welfare recipients and applicants in Riverside were randomly assigned to the research groups.
Figure 2.2
Steps Leading to Random Assignment in Riverside
Unlike the goal in the other sites that used a three-way design, the goal in Columbus was to test and compare the effectiveness of two different case management models. In the Traditional model, one worker handled the welfare department's employment and training function and another worker handled welfare eligibility and payment issues often called "income maintenance." Both workers maintained relatively large caseloads. In the Integrated model, a single worker handled both the employment and training and income maintenance functions. In the Integrated model, the worker maintained a smaller caseload than either of the workers in the Traditional model since, on a per client basis, the worker was handling jobs "traditionally" done by two workers.
The remaining three sites in the evaluation (Oklahoma City, Detroit, and Portland) used a two-way random assignment design to test the effectiveness of program models already established in those sites. In other words, instead of implementing a program designed expressly for research purposes, as in the three-way sites, program administrators in each of the two-way sites determined their welfare-to-work program goals and practices and randomly assigned people to a group that entered the program or to a control group.(5) A summary of the research designs in all seven sites is presented in Table 2.1.
| Characteristic | Atlanta | Grand Rapids | Riverside | Columbus | Detroit | Oklahoma City | Portland |
|---|---|---|---|---|---|---|---|
| Type of random assignment | Three-way
(2 program groups, 1 control group) |
Three-way
(2 program groups, 1 control group) |
Three-way
(2 program groups, 1 control group) |
Three-way
(2 program groups, 1 control group) |
Two-way
(1 program group, 1 control group) |
Two-way
(1 program group, 1 control group) |
Two-way
(1 program group, 1 control group) |
| Type of study | Differential impacts of HCD and LFA approaches | Differential impacts of HCD and LFA approaches | Differential impacts of HCD and LFA approaches | Differential impacts of Integrated and Traditional case management strategies | Net impacts of established program | Net impacts of established program | Net impacts of established program |
| Sample composition | AFDC applicants and recipients | AFDC applicants and recipients; teen parents (ages 18 and 19) | AFDC applicants and recipients | AFDC applicants and recipients | AFDC applicants and recipients; teen parents (ages 18 and 19) | AFDC applicants; teen parents (ages 16 to 19) | AFDC applicants and recipients |
| Age of youngest child | 3 | 1 | 3 | 3 | 1 | 1 | 1 |
| Point of random assignment | Program orientation | Program orientation | Program orientation | Income maintenance office: application or redetermination | Program orientation | Income maintenance office: application only | Program orientation |
Random Assignment Periods and Procedures
In each site, sample members were randomly assigned over a period of approximately two years. Random assignment began in June 1991 in Riverside, California, and ended in December 1994 in Portland, Oregon (see Table 2.2). Thus, the results presented in this report cover the calendar period from June 1991 (the month of the first Riverside sample member's entry into the study) to December 1999 (the last month of the follow-up period for the last sample member in Portland to be randomly assigned). Throughout the report, the five years of the evaluation's follow-up period are labeled year 1, year 2, and so on. These labels refer not to calendar years but to years after random assignment; for example, year 1 refers to the first year after sample members were randomly assigned, regardless of whether year 1 began in 1991 or 1994 for individual sample members.
Site and Program | Full Impact Sample | Five-Year Client Survey Sample | Two-Year and Five-Year Client Survey Respondents | Five-Year Child Outcomes Study Sample | Five-Year Teacher Survey Sample |
|---|---|---|---|---|---|
Atlanta | |||||
| Random assignment period | 01/92-06/93 | 03/92-06/93 | 03/92-06/93 | 03/92-06/93 | 03/92-06/93 |
| Labor Force Attachment | 1,441 | 519 | 491 | 289 | 184 |
| Human Capital Development | 1,495 | 594 | 565 | 367 | 226 |
| Control | 1,497 | 552 | 527 | 311 | 193 |
| Full sample | 4,433 | 1,665 | 1,583 | 967 | 603 |
Grand Rapids | |||||
| Random assignment period | 09/91-01/94 | 03/92-01/94 | 03/92-01/94 | 03/92-01/94 | 03/92-01/94 |
| Labor Force Attachment | 1,557 | 535 | 506 | 214 | 144 |
| Human Capital Development | 1,542 | 547 | 511 | 196 | 120 |
| Control | 1,455 | 562 | 537 | 214 | 144 |
| Full sample | 4,554 | 1,644 | 1,554 | 624 | 408 |
Riverside | |||||
| Random assignment period | 06/91-06/93 | 09/91-05/93 | 09/91-05/93 | 09/91-05/93 | 09/91-05/93 |
| Labor Force Attachment | 3,384 | 499 | 424 | 185 | 108 |
| Human Capital Development | 1,596 | 376 | 323 | 208 | 131 |
| Control | 3,342 | 720 | 648 | 348 | 222 |
| Full sample | 8,322 | 1,595 | 1,395 | 741 | 461 |
Portland | |||||
| Random assignment period | 02/93-12/94 | 03/93-02/94 | 03/93-02/94 | ||
| Program | 3,529 | 281 | 238 | ||
| Control | 499 | 223 | 204 | ||
| Full sample | 4,028 | 504 | 442 | ||
Columbus | |||||
| Random assignment period | 09/92-07/94 | ||||
| Integrated | 2,513 | ||||
| Traditional | 2,570 | ||||
| Control | 2,159 | ||||
| Full sample | 7,242 | ||||
Detroit | |||||
| Random assignment period | 05/92-06/94 | ||||
| Program | 2,226 | ||||
| Control | 2,233 | ||||
| Full sample | 4,459 | ||||
Oklahoma City | |||||
| Random assignment period | 09/91-05/93 | ||||
| Program | 4,309 | ||||
| Control | 4,368 | ||||
| Full sampleV | 8,677 | ||||
| Full sample size | 41,715 | 5,408 | 4,974 | 2,332 | 1,472 |
| SOURCE: MDRC-created database. | |||||
The differences in the random assignment procedures used in different sites affected the composition of the site samples and, thus, the comparability of the results for different sites and programs.(6) In five of the seven sites, welfare applicants and recipients who were required to participate in a welfare-to-work program (because they met certain demographic criteria: for instance, had no children below the minimum age set by the program) were randomly assigned while attending a program orientation at their local employment and training office. In Columbus and Oklahoma City, in contrast, people were randomly assigned at their local income maintenance office before being assigned to an orientation.
Not everyone assigned to participate in a welfare-to-work program actually attends an orientation. Reasons for not attending include leaving welfare shortly after being referred to the program, having one's welfare application denied, or simply failing to show up.(7) As a result, the people who attend a program orientation may not be representative of everyone in their locale who is required to participate in a welfare-to-work program. For example, when the waiting list for orientation "slots" is long, the people who find jobs and exit welfare before being randomly assigned are likely to be more employable, on average, than people who do not. As a result, those who enroll in the program are disproportionately likely to be "disadvantaged." Data on how many people who were required to attend an orientation actually did so are available for three sites (Riverside, Grand Rapids, and Columbus). About 66 percent in the Riverside, Grand Rapids, and Columbus Traditional programs and about 83 percent in the Columbus Integrated program attended an orientation.(8) Because outcomes in this report are reported as averages for all sample members in a group including, in Columbus, those who did not ultimately attend an orientation the relative effects of the Columbus Integrated and Columbus Traditional programs on sample members' participation and subsequent employment, earnings, and welfare outcomes reflect not only differences in case management strategies but in the capacity to enroll people.
The Oklahoma City results should be interpreted with the following in mind: Oklahoma City, unlike the other sites, randomly assigned only welfare applicants (including those whose application for assistance was not yet approved) to the research groups.(9) Moreover, about 30 percent of sample members in Oklahoma City were denied cash welfare assistance shortly after being randomly assigned.(10) Therefore, the impacts for the Oklahoma City program are based on a sample that included a larger proportion than in the other sites of people who never received a welfare payment after random assignment for reasons unrelated to the program. In addition, past research has shown that welfare-to-work programs have different effects on welfare applicants than on recipients, most likely because recipients tend to be more disadvantaged than applicants.(11)
Analysis Samples and Sample Characteristics
Analysis Samples
Table 2.2 shows the dates of random assignment and sizes of the samples used in this report, by site and research group. All the analysis samples used in the report are listed and described below.
Full impact sample. The full impact sample includes 41,715 program and control group members from all seven sites, for whom five years of administrative records data were collected (Figure 2.3, box A).(12), (13)
Figure 2.3
Sample Size and Data Sources
Five-Year Client Survey sample. Additional data on outcomes for adults and children were collected by interviewing sample members around two years after their date of random assignment and, in four of the seven sites, around their five-year anniversary. This report focuses on outcomes from the Five-Year Client Survey (detailed findings from the Two-Year Client Survey can be found in the two-year report from the NEWWS Evaluation).(14) The Five-Year Client Survey sample (Figure 2.3, box B) includes 5,408 members of the full impact sample in Atlanta, Grand Rapids, Riverside, and Portland. In each site, survey selection took place during some, but not all, months of random assignment. The survey sample was drawn from members of the full impact sample who had earlier been selected to be interviewed at two years, whether or not they actually responded to the two-year survey (4,974 sample members answered both surveys, and 434 responded to only the five-year survey; see Table 2.2).(15) Those selected to be interviewed at two years were a stratified random sample of the full impact sample members who were randomly assigned during the months when the survey sample was selected.
Certain subgroups were intentionally oversampled to produce large enough samples for special analyses such as the Child Outcomes Study (COS) and an intensive examination of adult education.(16) Results from all programs in this report have been weighted to reflect the overall demographic characteristics of the larger sample. The survey response rates exceed 70 percent for all programs and research groups (and 80 percent in Atlanta and Grand Rapids).(17)
Child Outcomes Study sample. The COS, which is part of the NEWWS Evaluation, includes the families of 2,332 sample members who responded to the Five-Year Client Survey in Atlanta, Grand Rapids, and Riverside (Figure 2.3, box C).(18) Individuals in these three sites who were selected for the Two-Year Client Survey and who had at least one child aged 3 to 5 at random assignment were randomly selected to be part of the COS. Each family included a child aged 3 to 5 at random assignment who was randomly selected to serve as the focal child, that is, the child about whom the most extensive information was collected. Response rates to the COS survey in Atlanta and Grand Rapids exceeded 75 percent; response rates in Riverside were between 63 and 67 percent for different research groups.(19)
Teacher survey sample. In a study of children's school progress, which is also part of the evaluation, COS mothers who were interviewed at the five-year follow-up point were asked for permission to mail a survey to the focal child's current elementary school teacher that asked about outcomes such as academic performance. (For details, see Section III.) The teacher survey sample includes responses from 1,472 teachers of focal children (Figure 2.3, box D).(20) Response rates were lower for the teacher survey than for the other surveys, ranging from 37 to 57 percent.(21)
Sample Characteristics
Ethnicity. The racial and ethnic makeup of the full impact sample varied from site to site, reflecting general differences in the overall ethnic composition of the counties from which the samples were drawn. For example, whereas almost all sample members in Atlanta and Detroit were African-American, about one-half of sample members in Grand Rapids, Riverside, Columbus, and Oklahoma City and two-thirds of those in Portland were white. Only Riverside had a substantial proportion (one-third) of Hispanic sample members (see Table 2.3).
Characteristic | Atlanta | Grand Rapids | Riverside | Columbus | Detroit | Oklahoma City | Portland |
|---|---|---|---|---|---|---|---|
| Demographic characteristics | |||||||
Gender (%) | |||||||
| Male | 3.2 | 4.2 | 10.6 | 6.5 | 3.3 | 6.9 | 6.7 |
| Female | 96.8 | 95.8 | 89.4 | 93.5 | 96.7 | 93.1 | 93.3 |
| Age (%) | |||||||
| Under 19 | 0.0 | 5.4 | 0.9 | 0.2 | 2.9 | 9.8 | 0.0 |
| 19-24 | 9.5 | 34.2 | 15.5 | 12.9 | 26.2 | 27.0 | 23.1 |
| 25-34 | 54.9 | 40.6 | 49.7 | 55.7 | 43.3 | 42.9 | 50.8 |
| 35-44 | 29.6 | 16.5 | 27.8 | 26.8 | 22.7 | 17.3 | 22.2 |
| 45 or over | 6.0 | 3.3 | 6.1 | 4.4 | 5.0 | 2.9 | 3.9 |
| Average age (years) | 32.8 | 28.2 | 32.0 | 31.8 | 30.0 | 28.1 | 30.4 |
| Ethnicity (%) | |||||||
| White | 3.5 | 50.1 | 49.0 | 46.5 | 11.0 | 59.4 | 69.5 |
| Hispanic | 0.8 | 8.0 | 30.2 | 0.4 | 0.8 | 4.3 | 3.9 |
| Black | 95.2 | 39.3 | 16.7 | 52.0 | 87.3 | 28.9 | 20.1 |
| Black Hispanic | 0.1 | 0.3 | 0.0 | 0.0 | 0.0 | 0.3 | 0.3 |
| Native American/Alaskan Native | 0.1 | 1.5 | 1.4 | 0.1 | 0.2 | 6.4 | 3.0 |
| Asian/Pacific Islander | 0.2 | 0.4 | 2.7 | 0.8 | 0.2 | 0.6 | 2.2 |
| Other | 0.2 | 0.5 | 0.0 | 0.3 | 0.6 | 0.1 | 1.1 |
| Family status | |||||||
| Marital status (%) | |||||||
| Never married | 60.5 | 57.9 | 32.5 | 50.2 | 68.0 | 34.3 | 47.3 |
| Married, living with spouse | 1.4 | 3.3 | 8.1 | 8.2 | 2.7 | 3.8 | 1.7 |
| Separated | 20.7 | 18.4 | 31.4 | 22.3 | 15.8 | 35.7 | 21.8 |
| Divorced | 16.1 | 19.3 | 26.5 | 18.6 | 12.3 | 25.2 | 28.6 |
| Widowed | 1.4 | 1.0 | 1.5 | 0.8 | 1.2 | 1.0 | 0.6 |
| Number of children (%) | |||||||
| 1 | 36.7 | 46.9 | 38.6 | 40.0 | 43.7 | 50.6 | 40.6 |
| 2 | 33.2 | 35.3 | 32.0 | 33.4 | 29.6 | 30.5 | 33.6 |
| 3 or more | 30.1 | 17.9 | 29.4 | 26.6 | 26.7 | 18.9 | 25.8 |
| Average number of children | 2.1 | 1.8 | 2.0 | 2.0 | 2.0 | 1.7 | 2.0 |
| Age of children (%) | |||||||
| Any child aged 0-5 | 41.5 | 67.9 | 56.0 | 46.9 | 64.3 | 65.1 | 67.4 |
| Any child aged 6-11 | 63.0 | 38.3 | 56.2 | 57.3 | 44.3 | 40.5 | 47.6 |
| Any child aged 12-18 | 46.3 | 26.2 | 37.0 | 39.4 | 34.0 | 23.9 | 25.9 |
| Age of youngest child (%) | |||||||
| 2 or under | 0.3 | 46.3 | 6.2 | 1.8 | 39.3 | 41.4 | 40.2 |
| 3 to 5 | 41.2 | 21.6 | 49.8 | 45.1 | 25.0 | 23.8 | 27.3 |
| 6 or over | 58.5 | 32.1 | 44.0 | 53.1 | 35.7 | 34.9 | 32.6 |
| Had a child as a teenager (%) | 42.3 | 48.4 | 32.8 | 37.5 | 44.2 | 47.1 | 32.3 |
| Labor force status | |||||||
| Ever worked full time for 6 months or more | |||||||
| for one employer (%) | 71.4 | 63.8 | 71.0 | 42.5 | 48.1 | 68.8 | 76.9 |
| Any earnings in past 12 months (%) | 23.6 | 46.0 | 40.7 | 28.2 | 21.1 | 69.0 | 39.3 |
| Currently employed (%) | 6.9 | 11.4 | 11.2 | 4.0 | 6.8 | 8.6 | 9.6 |
| Education status | |||||||
| Received high school diploma or GED (%) | 59.7 | 59.0 | 56.2 | 57.4 | 56.5 | 55.1 | 67.3 |
Highest degree/diploma earned (%) | |||||||
| GED a | 5.4 | 8.2 | 9.2 | 7.0 | 10.7 | 11.3 | 21.5 |
| High school diploma | 46.7 | 45.9 | 41.8 | 44.6 | 37.0 | 38.2 | 34.5 |
| Technical/AA/2-year college | 6.6 | 4.1 | 4.3 | 4.6 | 8.0 | 4.3 | 9.7 |
| 4-year (or more) college | 1.3 | 0.9 | 0.9 | 1.6 | 1.1 | 1.6 | 1.9 |
| None of the above | 40.0 | 40.9 | 43.8 | 42.3 | 43.2 | 44.6 | 32.3 |
| Enrolled in education or training in past | |||||||
| 12 months (%) | 13.4 | 39.2 | 19.6 | 9.5 | 20.0 | 23.7 | 21.1 |
| Currently enrolled in education or training (%) | 8.4 | 34.8 | 14.1 | 7.8 | 28.2 | 12.9 | 13.5 |
| Public assistance status | |||||||
| Total prior AFDC receipt (%) b | |||||||
| None | 0.3 | 0.1 | 1.0 | 10.0 | 2.8 | 44.4 | 1.2 |
| Less than 1 year | 18.9 | 22.1 | 33.8 | 8.3 | 13.7 | 18.8 | 20.9 |
| 1 year or more but less than 2 years | 10.1 | 18.6 | 11.3 | 9.0 | 9.1 | 12.5 | 16.6 |
| 2 years or more but less than 5 years | 24.6 | 30.0 | 26.4 | 27.9 | 24.0 | 15.3 | 32.1 |
| 5 years or more but less than 10 years | 22.4 | 16.4 | 15.6 | 22.7 | 22.5 | 6.5 | 21.1 |
| 10 years or more | 23.7 | 12.8 | 11.8 | 22.1 | 27.9 | 2.5 | 8.2 |
| Raised as a child in a household receiving AFDC (%) | 26.9 | 32.8 | 19.5 | 27.0 | 40.1 | 21.7 | 23.8 |
| First spell of AFDC receipt (%) c | 7.2 | 27.9 | 23.5 | 9.6 | 4.1 | 42.0 | 7.2 |
Level of disadvantage | |||||||
| Most disadvantaged d | 24.2 | 15.1 | 24.7 | 19.0 | 25.1 | 4.9 | 15.3 |
| Housing status | |||||||
| Current housing status (%) | |||||||
| Public housing | 35.9 | 2.6 | 2.5 | 15.2 | 5.5 | 5.3 | 7.6 |
| Subsidized housing | 24.9 | 13.0 | 7.0 | 24.7 | 1.1 | 6.7 | 18.6 |
| Emergency or temporary housing | 0.7 | 2.4 | 1.4 | 1.4 | 0.8 | 14.4 | 3.7 |
| None of the above | 38.5 | 82.1 | 89.1 | 58.7 | 92.6 | 73.7 | 70.1 |
| Sample size | 4,433 | 4,554 | 8,322 | 7,242 | 4,459 | 8,677 | 4,028 |
| SOURCE: MDRC calculations from information routinely collected by welfare staff. NOTES: Distributions may not add to 100 percent because of rounding. a The GED credential is given to those who pass the GED test and is intended to signify knowledge of high school subjects. b This refers to the total number of months accumulated from at least one spell on an individual's own or spouse's AFDC case. It does not include AFDC receipt under a parent's name. c This does not mean that such individuals are new to the AFDC rolls, only that this is their first spell on AFDC. This spell, however, may have lasted several years. d The "most disadvantaged" subgroup consists of sample members who did not have a high school diploma or GED at random assignment, did not work for pay in the year prior to random assignment, and had received AFDC for two years or more (cumulatively) prior to random assignment. | |||||||
Family structure. Almost all the sample members in the evaluation were single parents. The "average" sample member was a 30-year-old single mother with two children.(22) She was likely to have had a preschool-age child at random assignment and to have had her first child as a teenager.
This characterization does not capture the diversity of the families who were subject to program participation mandates in these locales. In particular, it does not reflect important site differences in who was required to participate in a welfare-to-work program. Just under one-half of sample members in Grand Rapids, Detroit, Oklahoma City, and Portland where mothers with a child as young as age 1 were required to participate entered the program when their youngest child was under age 3. The remainder of the sample in these four sites and the full samples in the other three sites were evenly divided between mothers with a youngest child aged 3 to 5 and those with a youngest child aged 6 or over. In Grand Rapids, Detroit, and Oklahoma City, unlike in the other sites, teen parents are included in the full impact sample (see Table 2.1).
Educational attainment. Between 55 and 66 percent of sample members had a high school diploma or GED certificate when they entered the program, and some enrollees in all sites had some college or post-secondary schooling. On average, however, sample members had completed only 11 years of school before random assignment. Those sample members who had a high school diploma or GED certificate at random assignment are described in this report as graduates; those without a high school diploma or GED are described as nongraduates.
Employment history. Sample members' employment history varied by site. Less than one-half of sample members in all sites except Oklahoma City had worked at some point during the year before random assignment: from 21 percent (in Detroit) to 46 percent (in Grand Rapids). Not surprisingly, sample members in Oklahoma City, all of whom were welfare applicants, were far more likely to have worked in the year before entering the program (69 percent had done so).
In addition to having little recent work experience, less than one-half of the sample members in Columbus and Detroit had worked full time for six months or more for one employer at some point before random assignment; two-thirds to three-quarters of sample members in the other sites had done so.
Past welfare receipt. The majority of sample members in all sites except Oklahoma City had already received welfare for at least two years cumulatively before random assignment. Just 24 percent of those in Oklahoma City, compared with 54 to 74 percent in the other sites, had received cash assistance for two years or more. Excluding Oklahoma, between 28 and 50 percent of sample members had received welfare cumulatively for five years or more before random assignment.
"Most disadvantaged" status. The sample members considered to be the most disadvantaged were those who lacked a high school diploma or GED (or, in Riverside, who were determined to need basic education), lacked any work history in the year before random assignment, and had already received welfare for two years or more cumulatively before entering the program. The proportion of sample members in the most disadvantaged group ranged from 5 percent in Oklahoma City to 25 percent in Riverside and Detroit.
Housing status. The proportion of sample members who at random assignment were living in public housing developments or receiving housing subsidies through such programs as the Section 8 rental assistance program was highest in Atlanta (56 percent) and lowest in Detroit (7 percent). Some have argued that federal housing policies discourage people from working because from the standpoint of residents of public and subsidized housing, who pay rent on a sliding scale earnings increases mean rent increases. In addition, gross income limits on housing assistance eligibility could cause a newly employed person to lose her housing subsidy altogether.
Compared with people in the other sites, a fairly large proportion (14 percent) of people in the Oklahoma City sample lived in emergency or temporary housing that is, lived in a shelter or were homeless when they applied for welfare. Less than 3 percent of sample members in the other sites were experiencing this type of hardship at random assignment.
Data Sources
The outcomes and impacts presented in this report are drawn from four primary data sources: unemployment insurance, welfare, and Food Stamp administrative records; surveys of sample members that were conducted at the two-year and five-year follow-up points (the Two-Year Client Survey and the Five-Year Client Survey); a survey of sample members focused on outcomes for children (the Child Outcomes Study survey); and a teacher survey.
Client characteristic data. Standard personal data, such as educational background and welfare history, were collected by welfare staff during routine interviews at the time of random assignment and are available for all 41,715 heads of the single-parent families in the full impact sample.
Private Opinion Survey. Data on attitudes and opinions about welfare-to-work programs and employment prospects were collected through the Private Opinion Survey (POS), a brief, self-administered survey that was completed at program orientation in four of the sites (Atlanta, Grand Rapids, Riverside, and Portland), and are available for 18,461 respondents in these sites. These sample members represent 93 percent of those randomly assigned in the four sites during the periods when the POS was being administered.
Reading and math tests. Reading and math achievement tests were administered in four sites (Atlanta, Grand Rapids, Riverside, and Portland) at random assignment. Test scores are available for 20,577 sample members. These sample members represent about 93 percent of those randomly assigned in the four sites during the period when the tests were administered.(23)
Field research. MDRC staff observed all 11 programs in operation and interviewed enrollees, case managers, service providers, and program administrators in each site. Information was collected about a range of issues, such as management philosophy and structure, the degree to which the participation mandate was enforced, the nature of interactions between caseworkers and program participants, the extent to which the program was able to work with all those mandated to participate in it, the availability of services, and the relationships that program staff had established with outside service providers and income maintenance staff in the sites.
Unemployment insurance, welfare, and Food Stamp administrative records data. Most employment, earnings, and public assistance impacts were computed using automated county and state unemployment insurance (UI), welfare, and Food Stamp administrative records data. Five years of follow-up data from the UI system are available for all members of the full impact sample; five years of follow-up data from welfare and Food Stamp administrative records are available for all sample members in all sites except Oklahoma.
UI earnings, which are recorded statewide, provide unbiased measures of program impacts on employment and earnings. These data, however, do not include earnings from out of state; from jobs not usually covered by the UI system, such as self-employment, federal employment, or informal child care (all types of work that may have been "off the books"); or from employers who do not report earnings. Some of the earnings missed by the UI system may be captured by earnings and employment data collected through the two-year and five-year surveys.
In all sites except Riverside, welfare and Food Stamp payments were also recorded statewide, and payments are captured for all sample members except those who moved out of state. In Riverside (as everywhere in California), welfare and Food Stamp payments were recorded only within each county, which means that payments received by sample members who moved outside the county were not included in the analysis. Although this could lead to an underestimate of the payments received in the Riverside sample, it should not bias the impact estimates because there is no reason to expect the program and control groups to show different patterns of moving between counties.
UI earnings data are collected by calendar quarter: January through March, April through June, and so forth. For purposes of the evaluation, these data were reorganized so that the quarter during which a sample member was randomly assigned is always designated quarter 1, followed by quarter 2, and so forth. These quarters are then grouped into "years." Quarter 1 is not included in year 1 because it includes some income earned before random assignment, especially for sample members randomly assigned near the end of a calendar quarter. Thus, year 1 covers quarters 2 through 5, year 2 covers quarters 6 through 9, and so forth. Welfare and Food Stamp payments were recorded monthly but were grouped into quarters and years to align with the earnings data.
Two-Year Client Survey and Five-Year Client Survey. As noted in a previous section, this report includes the results of a survey administered at the five-year follow-up point and some results of a survey administered at the two-year follow-up point. Both the two-year and five-year surveys provide information about sample members' participation in training and education activities, attainment of education credentials, views of work and welfare, employment history, income, receipt of noncash benefits such as health coverage, child care use, living situations, and children's well-being.
Survey responses are the only source of information about many key outcomes, such as participation patterns for control group members, work hours and wages, income from other people in the household, and outcomes for children. For some outcomes, such as employment, respondents provided information that was also recorded from administrative data. It is possible for data from these two sources to differ. Because the five-year survey respondents represent a subsample of the full impact sample that was selected during a shorter period of random assignment months, the impact and survey samples may differ with respect to observed characteristics (such as educational attainment or prior work history) or with respect to unmeasured characteristics (such as assertiveness or learning style) that might have affected their ability to find and retain employment. (For more information on survey response bias and the degree to which the survey sample and full impact samples differ, see Appendix G.)
In some cases, administrative records data may be more accurate than the survey data. The client survey depends on people's ability to recall information about events or jobs that they may have held up to five years prior to being interviewed, and failures of memory can give rise to discrepancies between the dates of employment or amounts of earnings reported in the survey and reflected in administrative records. In addition, some respondents may have been reluctant to provide information on employment and income that could be found in administrative records or, alternatively, may have exaggerated their earnings and income. In other cases, however, survey data may be more accurate, such as when respondents were working off the books or in short-term employment. The survey may also have captured earnings that employers failed to report or reported inaccurately to the UI system. (For more information on the differences between UI-reported and survey-based measures of earnings, see Appendix H.)
Additional COS survey data. COS respondents provided information on focal children's academic functioning, social skills, and health and safety. In addition, mothers and the focal children themselves completed a Self-Administered Questionnaire (SAQ). Mothers' SAQ included questions about domestic abuse; children's SAQ included questions about academic functioning and social skills.
Teacher survey. Current teachers of focal children in the COS were asked to assess them with respect to their academic standing, academic progress, school engagement, behaviors requiring disciplinary action, and social skills. The teacher survey complements the data collected from mothers and the children themselves. Reports from teachers and mothers sometimes differ. Possible explanations include the following: The children behaved differently in the presence of mothers and teachers, mothers and teachers perceived the children's behavior differently, or mothers and teachers based their reports on different criteria.
Cost data. The cost analysis used data drawn from state, county, and local fiscal records, supportive service payment records, administrative records, the Two-Year Client Survey, the Five-Year Client Survey, and case file participation records.
Benefit-cost data. The benefit-cost analysis is based on administrative records data (UI-reported earnings, welfare, and Food Stamp payments), Two-Year Client Survey data, Five-Year Client Survey data, and published data.
Published data and agency reports. Published data and reports from government agencies were used to gather additional information about the environments in each of the sites, including unemployment rates, welfare caseloads, and welfare grant levels.
Analysis Issues
Calculating Impacts
As discussed above, control group outcomes in this evaluation represent outcomes expected in the absence of a welfare-to-work program. Program-control differences show the effect, or impact, of each program. In the sites that conducted side-by-side evaluations of alternative program approaches, differences between the outcomes for each program group represent the relative effects of each program.
Although random assignment minimizes the likelihood of the research groups' differing systematically at the outset, there can be small differences in their average characteristics at random assignment. To control for these differences, the outcomes for each research group were regression-adjusted using ordinary least squares in all the analyses presented in the chapters that follow.
In this report, a difference between the program and control groups with respect to a particular outcome is considered statistically significant if the result of a statistical test indicates that there is less than a 10 percent probability that the difference occurred by chance (that is, when the p-value, or level of significance, of the difference is under .10). Impacts are generally reported only if they are statistically significant. This rule is intended to keep researchers from inferring an impact where none exists.(24)
Many analysts have noted that the greater the number of analyses conducted (regardless of the outcomes or domains studied), the greater the likelihood of chance findings and, thus, one needs to take the number of outcomes examined into account. However, some argue that this is relevant only when outcomes are not theoretically independent from each other. There are stringent statistical tests of multiple dependent variables that automatically adjust for the (limited) number of theoretically related outcomes (that is, multivariate analysis of variance, or MANOVA), as well as post-hoc corrections to p-values that can be applied to results from multiple individual analyses of "similar" outcomes (for example, the Bonferroni correction).
For this report, we have not attempted to adjust for the number outcomes such as employment and AFDC receipt that are examined because many of these outcomes are so highly statistically significant that they would pass the most stringent statistical correction for the fact that many outcomes are being measured. By contrast, because we are less certain about whether the nontargeted (child and family) outcomes examined in Chapters 9, 11, and 12 are theoretically independent from one another (and, thus, whether we may in part be capitalizing on chance by examining multiple measures of the same or similar underlying constructs), we calculate and report the number of findings we might expect by chance as if they were independent from one another. The proportion of statistically significant impacts across all family outcome measures (in Chapter 9), and by selected categories of these measures, or across all child outcome measures (in each of Chapters 11 and 12) and across all relevant programs was calculated. Specifically, given that the experiment-wise Type I error rate was set at .10, any one result will emerge as significant 10 percent of the time owing to chance alone. The number of chance significant outcomes were calculated and noted in drawing any conclusions about the effects of these welfare-to-work programs.
Some might argue that a more stringent standard is needed, requiring that the number of significant impacts within each program must exceed chance levels or that the number of significant impacts within each domain of child development must exceed chance levels. Because there is a lack of consensus on this issue among statisticians, and given that a goal of the analyses on family and child outcomes was to provide a thorough examination of program impacts, we did not adhere to a more stringent standard.
All impact estimates are based on the entire research sample, including program group members who did not participate in program activities (it is likely that nearly all "nonparticipants" in the program group encountered the program messages and participation mandates, which may have affected their decision to look for work or to leave welfare). Because all sample members are included in the analyses, the impacts must be interpreted as being the results of the welfare-to-work programs as a whole, not only of participation in specific program services. By the same principle, calculations of average earnings and welfare payments which form the basis of many of the impact estimates include sample members who were not employed (that is, earned $0) or did not receive welfare (that is, received $0 in welfare). To the extent that a program turns nonearners into earners or encourages welfare recipients to leave welfare, excluding these $0 values from the program and control group averages would lead to seriously biased underestimates of program impacts. For example, previous research has shown that some welfare-to-work programs dramatically increased the proportion of people who have earnings without affecting the average earnings of those who work. These programs led to a relatively large impact on earnings when all sample members were included in the calculation. However, omitting people with $0 earnings from the analysis would have suggested that these programs had no impact on earnings.(25)
Some analyses in this report focus on subgroups of the full impact sample. In one such set of analyses, presented primarily in Chapter 7, each site sample is broken down by various background characteristics (such as previous work history) measured at the time of random assignment. The impacts found for these subgroups can be confidently attributed to the programs under study because they are based on characteristics measured before anyone entered the program and because program and control group members are similar in other respects; that is, the only difference between the program and control subgroups with a particular background characteristic is exposure to the program. In the language of evaluations, such impacts are experimental. However, because they are based on smaller samples of people, the impact estimates for subgroups are less likely to be statistically significant than those for the full sample. Other analyses in the report compare outcomes such as average hourly wages for program and control group members who shared characteristics (such as being employed) acquired after random assignment. These nonexperimental comparisons should be interpreted with caution because the research groups may differ with respect to measured or unmeasured background characteristics that affect employment and, in turn, hourly wages. The report presents findings from such nonexperimental comparisons to explore underlying trends in the experimental impact estimates.
Because data were unavailable, the results for sample members in Atlanta who were randomly assigned during the last six months of the random assignment period are not presented in this report. Similarly, because welfare and Food Stamp data were not available for years 4 and 5 for sample members in Oklahoma City, the year 4, year 5, and cumulative impacts on welfare receipt, Food Stamp receipt, and combined income are not reported for Oklahoma City.
As discussed in Chapter 1, changes in the labor market and the environments in which the programs operated during the follow-up period could have affected program impacts. In particular, the economic expansion that began in the mid 1990s created a strong demand for entry-level jobs nationwide. However, because program and control group members in each site experienced these changes, it is difficult to know ahead of time which group was most affected by them. On the one hand, program impacts on employment and earnings may diminish as more control group members find employment. On the other hand, during an economic expansion welfare-to-work programs may help people advance more quickly to higher-paying or more stable employment, resulting in increasing impacts over time.
In addition, most of the programs in the evaluation became more employment-focused over time. As a result, in the last two years of the follow-up period, people assigned to education-focused programs who remained on welfare received services and messages similar to those that people in the employment-focused programs were exposed throughout the follow-up period. However, people in the education-focused programs were probably little affected by this evolution in program approach because even under the original program model many of them would have received job search assistance if they had not found employment after completing education and training activities.(26)
Control Group Treatment Over Time
Another development that has direct consequences for assessing the impacts of the NEWWS programs studied concerns changes in the treatment of control group members over time. As discussed in Chapter 1, in five sites some control group members who were still receiving welfare after year 3 became eligible for and were required to participate in welfare-to-work program services. In other words, in these sites the control embargo was no longer in effect in year 4 and/or year 5. (In Riverside and Portland, the control embargo was in force for the entire five-year follow-up period for the control group samples analyzed in this report.) This section discusses the treatment of controls in years 4 and 5 of the follow-up period in detail, to assess the extent to which the early lifting of the control embargo in five sites affected the impact estimates in those sites, if it affected them at all.
Figure 2.4 presents a time line of control group members' eligibility for welfare-to-work program services in each site, by quarter of random assignment. In Atlanta, Columbus, and Oklahoma City, there was a fixed date on which the control group embargo on welfare-to-work program services was lifted. In these three sites, then, controls randomly assigned early in the sample intake period would have had little opportunity for exposure to the programs under study while those randomly assigned later in this period would have had more opportunity. In Grand Rapids, as noted above, controls randomly assigned in 1993 retained their embargo on welfare-to-work program services for a full five years; those randomly assigned prior to 1993 would have had their embargo lifted when they reached the end of their particular three-year follow-up period. In this site, as in Atlanta, Columbus, and Oklahoma City, control group members thus would have had differing amounts of time during the five-year follow-up in which to possibly be exposed to program services. In Detroit, the control group embargo on welfare-to-work program services was lifted when controls reached the end of their three-year follow-up period. Consequently, in this site, all controls would have had an equal amount of time during the full follow-up period for possible exposure to the programs under study.(27) All control group members in Riverside and a randomly selected group of control group members in Portland had a five-year embargo on welfare-to-work program services. (For ease of presentation, the randomly selected group of Portland controls rather than all control group members in the site is shown in Figure 2.4.)
Firgure 2.4
Time Line of Changes in Control Group Members' Eligibility for Welfare-to-Work Program Services over the Five-Year Follow-Up Period, by Quarter of Random Assignment and Site
Key: Random assignment quarter Not eligible for welfare-to-work program services Eligible for welfare-to-work program services
NOTE: Control group members shown in Portland are the randomly selected group of 499 for whom a full five-year embargo on welfare-to-work program services was in effect.
The extent to which control group members actually received welfare-to-work program services once their embargo ended is not known. However, for several reasons it is likely that relatively few control group members did so. First, only control group members still receiving welfare when their embargo on welfare-to-work program services was lifted in follow-up year 4 or 5 would have been informed of and required to participate in such services. (Once the control embargo was lifted, controls would have been considered to be mandatory for participation in a welfare-to-work program, as long as they did not meet any of the exemption criteria established under the Family Support Act.) The following proportions of all control group members in each site had four years of "no treatment" follow-up, that is, years with no months in which they were receiving welfare and were eligible for welfare-to-work program services: 86 percent in Atlanta, 67 percent in Grand Rapids, 91 percent in Columbus, and 37 percent in Detroit. Similarly, the following percentages of controls had five years of follow-up with no months in which they were receiving welfare and were eligible for welfare-to-work program services: 54 percent in Atlanta, 66 percent in Grand Rapids, 76 percent in Columbus, and 36 percent in Detroit. Additionally, for the samples used in this report, 100 percent of controls in Riverside and Portland had five years of follow-up with no months in which they were receiving welfare and were eligible for welfare-to-work program services. (Some of the control group members receiving welfare when the embargo was lifted may no longer have been considered mandatory for welfare-to-work program participation, as all would have been as of random assignment, and probably would not have been told about the program or a requirement to participate in it because of changes since study entry in their personal situations, such as illness or disability, employment of more than 20 hours per week, or the birth of child.)(28)
Second, welfare-to-work programs in most sites had a phase-in schedule for assigning people to the programs if they were required to participate in them. It is likely that control group members who were required to participate in the programs would have waited up to six months before being assigned to a program.
Finally, once assigned to a program, control group members would have needed to attend a program orientation, be assigned to a specific program activity, and participate in that activity in order to actually receive welfare-to-work program services. Typically, less than one-half of those assigned to welfare-to-work programs as mandatory participants end up actually participating in welfare-to-work program services.(29)
It should be kept in mind, however, that any encounters with welfare-to-work program staff would have represented control group members' first exposure to a mandatory welfare-to-work program after random assignment. These encounters and possible participation in welfare-to-work program activities may have given control group members an unanticipated boost to look for work, to pursue education or training, or to change other behaviors. While this boost would have occurred for program group members shortly after random assignment, and their exposure to program staff and participation in program activities would have continued for several years, the possible effects of this boost for control group members toward the end of the follow-up period cannot be ignored.
As discussed more extensively in Chapter 3, available data suggest that the proportion of control group members who actually received welfare-to-work program services in follow-up years 4 and 5 is likely to have been low. According to five-year survey data, which are available for four of the seven NEWWS sites, less than 6 percent of control group members subject to a five-year embargo received any welfare payments during the fifth year after random assignment and reported participating, at some point during the same year, in activities usually uniquely provided by welfare-to-work programs (job search workshops or work experience).(30) In comparison, participation rates measured in the same way were only slightly higher (between 7 and 12 percent, depending on the site) for control group members with a shorter embargo on program services.(31) Measured in a slightly different way, as the proportion of controls who ever received welfare in a "non-embargoed" month and participated in a job club or work experience activity in the same year, at most 15 percent of the Atlanta controls and 7 percent of Grand Rapids controls were likely to have received welfare-to-work program services during the five-year follow-up period.
While the level of participation of the control groups in welfare-to-work program services appears to be low, the behavior of control group members might have been affected by contact with program staff and by the messages they received about the advantages of working as opposed to receiving welfare.(32) In Portland, where a random sample of 499 control group members had an extension of the welfare-to-work program services embargo, it is possible to directly compare the employment and welfare behavior of control group members who had a full five-year embargo on welfare-to-work program services with those who had only a three-year embargo. As discussed more fully in Chapters 4 and 5, there were some differences in the behavior of these two groups. In follow-up year 3, employment levels were almost identical for the two Portland control groups. In years 4 and 5, employment increased much faster for Portland control group members whose embargo ended at the close of year 3 than for those whose embargo lasted the full five years. The employment rates of the former group were 6 percentage points higher in year 4 and 3 percentage points higher in year 5 than those of the latter group. Similarly, average earnings of the former group were $500 higher per year in these two follow-up years. Average amounts of welfare payments received by the two groups of controls in follow-up years 4 and 5 differed by year in direction and by a small amount ($57 less for the longer-term embargo group in year 4 and $186 more in year 5).
The Portland control group comparisons suggest that in this site the lifting of the embargo on welfare-to-work program services after year 3 did affect the behavior of some control group members. For this reason, the randomly selected 499 control group members, who had a five-year embargo on welfare-to-work program services, are used as the control group in all Portland analyses throughout the report.(33)
Unfortunately, a randomly selected alternative control group, similar to the one in the Portland site, does not exist in the other five sites where some control group members were eligible for welfare-to-work program services earlier than the end of the five-year follow-up period. In four of the sites (Atlanta, Grand Rapids, Columbus, and Oklahoma City) certain cohorts of control group members, that is, individuals who were randomly assigned in certain months, had much longer control service embargoes than did other cohorts. (See Figure 2.4.) In these four sites, there is no easy way to determine what effect the lifting of the embargo had on the behavior of controls toward the end of the five-year follow-up period, since the baseline demographics of sample members often differed between cohorts and the employment and welfare behavior observed for the various cohorts was often different even before the control embargo was lifted. In the fifth site, Detroit, all cohorts of controls had an equally long welfare-to-work program service embargo.
Program impacts on employment and earnings and other outcomes in the last two years of follow-up in a few of the five sites probably would have been somewhat larger had some control group members not been exposed to welfare-to-work programs. Impacts would likely have been affected more in follow-up year 5 than in year 4. However, as discussed in Chapter 4 of this report, many programs can continue to have effects long after control embargoes are lifted. Owing to the program group's early exposure to the programs, early gains in employment and earnings can continue in the later years of follow-up, reflecting a "head start" experienced by program group members.(34) This factor, combined with the findings of low year 5 control group welfare receipt and low year 5 control group use of program services, strongly suggests that ending the control group embargo earlier than the end of the five-year follow-up period did not change the impact findings very much. As a result, in this report all control group members in these five sites are included in the estimates of program impacts. Where appropriate, however, impact analyses for follow-up years 1 to 3 are separated from those for years 4 and 5.
Notably, the control group situations described above do not affect the assessments in this report of the relative merits of the Labor Force Attachment and Human Capital Development approaches in welfare-to-work programs. Owing to the research design in the three-way sites, the fact that the control group embargo ended after year 3 in several of these sites does not affect the estimates of the relative effectiveness of the LFA and HCD approaches over five years. Random assignment ensured that the background characteristics of LFA and HCD program group members did not differ systematically at the time of random assignment, which means that the outcomes for the two program groups can be compared directly with one another without taking the control group into account (Figures 2.1 and 2.2).(35)
1. As will be discussed in more detail later in this chapter, in some sites control group members became eligible for program services before the end of the five-year follow-up period.
2. The following hypothetical example of a side-by-side evaluation of two program approaches illustrates these points. Control group members earned a total of $40,000 on average over five years, compared with $40,000 for program group 1 and $35, 000 for program group 2. Direct comparisons of earnings for the two program groups suggest that the first program was relatively more effective than the second, because its members earned $5,000 more on average over five years. However, comparisons with the control group show that neither program was effective because neither raised average earnings above the control group level.
3. The Riverside design has implications for calculating the LFA program impacts. Whereas the outcomes for sample members in the other six sites are unweighted, in Riverside the outcomes are weighted averages of the outcomes for LFA group members found to need or not to need basic education at random assignment. This weighting scheme compensates for the overrepresentation of those determined not to need basic education in the LFA and control groups.
Owing to the Riverside program design, impacts cannot be correctly calculated in an unweighted regression model (that is, one that includes all the sample members in Riverside and gives all observations equal weight). Instead, the LFA impact is calculated as (Wneed * BLFAneed) + (Wnot * BLFAnot). In this equation, BLFAneed represents the impact for the "in-need" LFA group members and BLFAnot the impact for "not-in-need" LFA group members. Wneed, the weight for the in-need sample, equals the fraction of LFA group members, HCD group members, and control group members who were classified by program staff to be in need of basic education at random assignment, and Wnot, the weight for the not-in-need sample, equals 1 - Wneed.
The Riverside LFA impacts were generated using a regression model that included all Riverside sample members, whereas the Riverside HCD impacts were estimated using a regression model that included only LFA, HCD, and control group members determined to need basic education.
For many outcome measures, the report presents the range of control group averages across the seven sites. For Riverside, the average for the entire control group will be included in the range, and not the separate average for control group members in need of basic education that is used to estimate the impacts of the HCD program.
4. Nearly one-quarter of the people in the Riverside in-need subgroup actually had a high school diploma or GED. These people were determined to be in need of basic education because they scored low on the math or reading portion of the appraisal test or were judged by program staff to need English remediation. See also Hamilton et al., 1997.
5. See Hamilton and Brock, 1994, for a more detailed description of the research designs in the seven sites.
6. For a discussion of enrollment practices in the sites, see Chapter 1. See also Hamilton and Brock, 1994, pp. 51-55.
7. See Hamilton and Brock, 1994, for a discussion of the implications of orientation attendance. A separate experimental analysis of the deterrence effects of a participation mandate and reasons for nonattendance was conducted in Riverside and Grand Rapids for the NEWWS Evaluation. For this study people who attended a meeting at income maintenance to determine their eligibility for welfare benefits were randomly assigned when income maintenance workers determined they were subject to the participation mandate. They entered either a "pre-orientation program group" and were assigned to attend a program orientation or a "pre-orientation control group" and were not assigned. Members of the pre-orientation program group who showed up for their orientation during the sample intake period for this study were randomly assigned a second time to either a program or control group. Only those who were randomly assigned to a program or control group at program orientation in Riverside and Grand Rapids are included in the analyses presented in this report. See Knab et al., 2001, for estimates of the deterrence effects of assignment to a mandatory welfare-to-work program.
8. Brock and Harknett, 1998; Scrivener and Walter, 2001; and Knab et al., 2001.
9. Although Oklahoma City included nonapplicants in its participation mandate, recipients were not included in the evaluation because including them would have required significant alterations to existing welfare department procedures.
10. Storto et al., 2000.
11. Friedlander, 1988.
12. The sample includes only the 499 control group members in Portland who had a full five-year embargo on the receipt of program services (more information on the control group embargo is included at the end of this chapter). Also, the sample includes only sample members in Atlanta who were randomly assigned between January 1992 and June 1993, excluding those randomly assigned after June 1993.
13. Approximately 15,000 more people were randomly assigned than are in the full impact sample. Excluded from this report's analysis are people randomly assigned before they attended a program orientation as part of the deterrence study, two-parent (AFDC-UP) families, and teen parents in Riverside (who faced different program requirements than older sample members).
14. See Freedman et al., 2000a.
15. The Two-Year Client Survey was conducted in all seven NEWWS Evaluation sites and included 9,675 respondents. For more information, see Freedman et al., 2000a.
16. For the two-year results of the former, see McGroder et al., 2000; for the results of the latter, see Bos et al., 2001.
17. For specific response rates by site and research group, see Appendix Table G.1.
18. Mothers and focal children in 2,594 families responded to the Five-Year Client Survey. A total of 262 of these families were later dropped from the analysis sample. Of these, 203 families had moved out of the survey area by the time of the five-year survey and therefore were not administered the special in-person COS survey sections (a phone interview was conducted to obtain information for the sections of the survey that were administered to all five-year survey sample members; these sample members remain in the five-year survey sample). Fifty-seven families were dropped because the focal child was not the mother's biological child; one duplicate case was dropped; and one family was dropped because the focal child was deceased at the five-year follow-up point.
19. For specific response rates by site and research group, see Appendix Table G.1.
20. A total of 1,489 teachers responded to the teacher survey. Seventeen teacher respondents were dropped from the final analysis sample because they taught focal children who were among the 262 respondents dropped from the COS sample.
21. For specific response rates by site and research group, see Appendix Table G.1.
22. As shown in Table 2.3, single fathers, or the husbands of disabled spouses, make up from 3 to 11 percent of the full impact sample, depending on site. Female pronouns will be used hereafter to describe sample members because most of them are women.
23. Of those who did not take the tests, about one-third did not speak English; the rest were unable to remain on site to be tested, spoke English but were unable to read or write it, or did not take the test for other reasons.
24. However, inferring that there is no impact when an impact really exists is another error of concern. In an effort to guard against this type of error, impacts with a probability between 10 and 20 percent of having arisen by chance are also occasionally discussed in the report, though these findings are referred to as program-control differences rather than impacts. These program-control differences are discussed if they are comparable in magnitude to a statistically significant impact of another program on the same outcome or if the impact appears to be part of a pattern of increases or decreases relative to the control group.
25. See, for example, the discussion of two-year earnings impacts for Riverside LFA in Freedman et al., 2000a, pp. 61-63.
26. See Chapter 3 for a discussion of education-focused program group members' participation in job search activities.
27. While the research design in Detroit specified a full three-year embargo on welfare-to-work program services, 8 percent of all Detroit controls ended up participating in the new Work First program in follow-up year 3. See Farrell, 2000, for details.
28. For example, as discussed in Chapter 9, between 12 and 23 percent of control group members, depending on the site, reported at the five-year follow-up point that a new baby had been added to their household since random assignment; the Family Support Act exempted women with children under age 3 from a mandatory welfare-to-work program participation requirement (or, at state option, women with children under age 1).
29. For a complete description of this process in welfare-to-work programs, see Hamilton, 1995.
30. Several situations could account for this 6 percent. While it is unusual for community college or other non-welfare programs to offer such activities, control group members may have found these programs on their own and voluntarily enrolled in these activities, a practice permitted under the NEWWS research design. NEWWS field research suggests that, over time, more non-welfare agencies in the evaluation sites began to offer job search assistance, particularly in Portland. In addition, while site welfare-to-work program staff were very diligent in screening for control group members at points of welfare application or program enrollment, some of these control group members could represent exceptions, or a few cases where controls "slipped through" the screening process. Field research, however, as well as periodic welfare case file reviews, indicated that the screening procedures in almost all sites were tight and that outside of Detroit very few control group members slipped through them.
31. Expressing these numbers as the proportion of control group members who received any welfare in follow-up year 5 rather than as a proportion of all control group members, among sites or groups of controls with a full five-year embargo on welfare-to-work program services, the proportion participating at some point during year 5 in activities usually uniquely provided by welfare-to-work programs was 10 percent in Grand Rapids, 4 percent in Riverside, and 24 percent in Portland (but the denominator, or the number of Portland controls receiving any welfare in year 5, was small). In contrast, among sites or groups of controls where the embargo on welfare-to-work program services was lifted at some point during the last two years of the five-year follow-up period, the proportion of those receiving any welfare in year 5 who reported similar participation was 18 percent in Grand Rapids and 22 percent in Atlanta. While these two sets of percentages are much higher than those mentioned in the text, the difference between them again suggests that the level of contamination with welfare-to-work program services was only somewhat higher in sites or among groups of controls where the control embargo was lifted during follow-up year 4 or 5 than where the embargo was in effect for the full five-year follow-up.
32. Both control and program group members probably would have been affected by publicity about the 1996 welfare law and, in the three sites where the count toward a welfare time limit began during the five-year follow-up, by the messages conveyed by welfare staff about the urgent need to find a job and leave welfare. In the three sites where the count toward a welfare time limit began during the five-year follow-up (Atlanta, Columbus, and Oklahoma City), the count began at the same time that the control embargo on welfare-to-work program services was lifted.
33. Data that would show direct evidence that the differences in employment behavior between the two control groups in Portland are due to exposure to welfare-to-work programs (evidence such as differences in measured program participation rates for the two groups) are not available. Using the smaller, five-year embargoed control group to calculate impacts, however, provides the "safest" estimates of the true effects of the Portland program.
34. Prior studies of the long-term effects of welfare-to-work programs have demonstrated that even after a control embargo on welfare-to-work program services is lifted programs can continue to have impacts, though perhaps diminishing ones, stemming from a labor market "boost" received by program group members early in the follow-up period. (See, for example, the five-year effects of the 1980s SWIM program in San Diego, presented in Friedlander and Hamilton, 1993.) Many of the NEWWS Evaluation programs examined here provide similar examples.
35. As noted above, direct comparisons between the LFA and HCD programs in Riverside can be made only by comparing the HCD group with those members of the LFA group who lacked a high school diploma or basic skills at random assignment.
Impacts on Employment-Related Services and Degree Receipt
This chapter examines the extent to which the mandatory NEWWS employment- and education-focused programs actually increased sample members' participation in employment-related activities. The findings presented below demonstrate that participation differences between program and control group members (in all seven sites with five-year survey data) and between LFA and HCD program group members (in the three LFA-HCD sites) are substantial. This chapter therefore confirms that the NEWWS Evaluation represents a legitimate test of the relative effectiveness of different welfare-to-work program approaches. In addition to participation outcomes, the chapter also discusses whether education-focused programs increased the percentage of sample members who attained GED certificates or other education credentials after random assignment, a key impact measure for these types of programs. Results are presented for the full sample and for subgroups of people who were high school graduates ("graduates") or high school nongraduates ("nongraduates") as of random assignment. Data on participation, from the Five-Year Client Survey, are available for seven of the NEWWS programs: the LFA and HCD programs in Atlanta, Grand Rapids, and Riverside, and the Portland program.
Key Findings
- A majority of control group members in all four sites from 55 to 75 percent participated in an employment-promoting activity over the five-year follow-up period. Almost all of this activity was initiated by the control group members themselves. The most common activities in which controls participated were vocational training programs and post-secondary education. Despite the potential for controls to be subject to mandatory welfare-to-work programs at the end of the follow-up period in Atlanta and Grand Rapids, there is little evidence that much participation in such programs did, in fact, occur.
- Relative to participation levels found for control group members, all seven NEWWS programs increased participation in employment-promoting activities including job search, basic education, vocational training, and post-secondary education over the five-year follow-up period. For the most part, impacts on participation measured earlier in the follow-up were sustained at the five-year follow-up point: In all but two programs cumulative five-year impacts were as large as impacts found at the two-year mark.
- As expected, large impacts on job search participation (of approximately 30 percentage points) were found in all four employment-focused programs (the three LFA programs and the Portland program). All three education-focused programs (the three HCD programs) also increased job search participation, although generally to a lesser degree. Notably, the Riverside HCD program had a large 30 percentage point impact on job search participation, matching the impacts found in the employment-focused programs.
- All three HCD programs had large impacts on participation in education and training activities. For nongraduates, increases were particularly large in basic education, and the only other impact on participation found for this subgroup was an increase in post-secondary education in Grand Rapids. For graduates in the two HCD programs that enrolled them, a large impact on vocational training program participation was found in Atlanta, but not in Grand Rapids; both programs slightly increased basic education participation for this subgroup.
- The Portland program, with its employment focus and mix of initial program activity assignments, resulted in five-year increases in both job search and education participation. While both education subgroups in Portland had large impacts on job search participation, nongraduates also experienced a 10 percentage point increase in basic education participation and graduates had a 21 percentage point increase in post-secondary education.
- Impacts on the receipt of a high school diploma or GED for nongraduates, fostered by large increases in basic education participation, were found in all three HCD programs as well as in the Portland program. In addition, Portland had a notable increase in the proportion of nongraduates who obtained a high school diploma or GED and a second education or training credential. None of the LFA programs increased diploma receipt for this subgroup. Increases in the receipt of any type of education or training credential (generally a trade license or certificate) among graduates were found in the Atlanta LFA and HCD programs; the Grand Rapids LFA program led to a decrease in such attainment for this subgroup.
Analysis Issues
The analysis of sample members' levels of participation in potentially employment-promoting activities and degree attainment extends the discussion of program dimensions in Chapter 1. Participation levels for program group members and the types of activities in which people participate demonstrate how successfully employment- and education-focused programs implemented their strategies for self-sufficiency. For the seven programs for which Five-Year Client Survey data are available, this chapter will determine how consistently, over the full five-year follow-up period, programs increased participation levels or degree receipt beyond what welfare recipients would be expected to attain had they never enrolled in a mandatory welfare-to-work program. Results for control group members represent the alternative outcomes, and program-control group differences indicate the effect, or impact, of each program. It should be stressed that a program's effect on participation depends on the levels attained by members of both the program group and the control group. In previous evaluations of welfare-to-work programs, from 20 to 40 percent of control group members enrolled in education and training programs on their own over a two-year follow-up period.(1) As will be shown, rates of self-initiated participation for control group members in this study, where the follow-up is extended to five years, is significantly higher. This is important, as programs with similar rates of participation for program group members may have very different impacts, depending on how frequently their respective control group members engaged in employment-related activities on their own.
During the first two years of follow-up, all 11 NEWWS programs generated moderate to large impacts on participation,(2) and it was expected that these impacts would remain in later years as programs continued to enroll in activities individuals who finished their initial assignment without gaining employment and, in some cases, individuals who left welfare and then returned.(3) For participants in LFA programs, the intended program path started with job search activities such as job club, intended to last approximately five weeks, followed by short-term education and training only for those unable to find employment. The intended HCD sequence was longer-term education and training, generally lasting up to two years, after which clients were expected to test their gained skills in the labor market through some type of job search activity.(4) The Portland program employed a unique, mixed strategy: Program staff assigned most individuals to job search first, while the more disadvantaged members of the caseload were often first referred to education and training activities. (See Chapter 1 for a more in-depth discussion of the intended assignment patterns in and the specific activities offered by each NEWWS program.) As noted in Chapter 1, in all sites the program focus varied somewhat over the five-year follow-up period. Most notably among the sites included in the five-year participation analysis, the three HCD programs became more employment-focused over time, assigning more individuals to job search and work experience activities. To the extent that this occurred, over time impacts on job search participation should increase in these programs. In the LFA programs, program group members who did not find employment through job search and work experience could have been assigned to education and training, and thus over time education and training impacts can be expected to increase in these programs. Furthermore, as the more job-ready left welfare for work, programs were left with a more disadvantaged caseload at the end of the follow-up period. Thus, while impacts were expected to remain over the entire follow-up period, it was also expected that the patterns of impacts might change as the types of activities in which program group members participated changed over time.
The end of the embargoes on control group members receiving mandatory welfare-to-work program services in Atlanta and Grand Rapids two of the seven sites examined in this chapter is another factor that may have influenced long-term program impacts on participation. An analysis of welfare receipt and participation among controls who became eligible for welfare-to-work program services shows that most control group members in these four programs never received such services during the five-year follow-up period.
Participation levels presented in the chapter are estimated from survey responses. The analysis includes all instances of participation after random assignment, including activities that occurred outside the welfare-to-work programs. Most commonly, self-initiated or nonprogram participation among program group members occurred after sample members left the welfare or program-mandatory rolls; less commonly, they might have participated in a self-initiated activity while still enrolled in a welfare-to-work program that their case manager could not approve as a program activity because the type or intensity of the activity did not meet the program's standards.(5) Sample members are considered to have participated in an employment-related activity if they attended for at least one day. Most participants attended for a much longer period.(6) The Five-Year Client Survey asked about participation since random assignment and in follow-up year 5. Thus, cumulative five-year and year 5 participation data are available.
Review of Two-Year Findings
Two-year participation findings, available for all 11 NEWWS programs, provide the basis for the longer-term rates, during a time period in which program participation was at its most intense. Across all 11 programs, program group members' two-year participation rates (in any employment-related activity) ranged from 19 to 29 percent in Atlanta, Columbus, and Riverside to approximately 40 percent in Grand Rapids, Detroit, Oklahoma City, and Portland.(7) Two-year participation rates were comparable to participation rates attained in most welfare-to-work programs studied in previous evaluations. Program group members most often participated in job search and basic education, including Adult Basic Education (ABE), high school completion, and General Educational Development (GED) certificate preparation classes, with levels varying by program approach and educational attainment subgroup. On their own initiative, without a mandate to participate in a welfare-to-work program, control group members participated most often in education and training activities (including post-secondary education and vocational training), somewhat less frequently in basic education, and least often in job search, work experience, and on-the-job training. Control group participation levels were notably high in Detroit, Oklahoma City, and Grand Rapids.
The four employment-focused programs produced large gains in job search participation between 27 percentage points in Grand Rapids LFA and 32 percentage points in Portland for both graduates and nongraduates. All education-focused programs except for Detroit and Oklahoma City achieved large increases in basic education participation for nongraduate sample members; however, education-focused programs had little effect on increasing participation in employment-related training for graduates. The three HCD programs and Portland's employment-focused, varied first activity program produced moderate to large increases in the attainment of a GED certificate among welfare recipients who lacked this credential at random assignment.
In sum, the two-year participation levels indicate that all sites except Oklahoma City and Detroit, both of which were low enforcement, successfully implemented their self-sufficiency approach. All LFA programs generated large increases in job search, and all HCD programs generated large increases in education and training activities. Education-focused program impacts were concentrated among nongraduates, who were most often assigned to basic education. Portland's mixed approach achieved large increases in job search and education and training activities.
Sanction rates were also examined at the two-year follow-up. In general, there was no clear association between a program's level of sanctioning and participation rates or impacts. Among the three programs that had the smallest increases in participation in any activity, two were low enforcement, while the third Grand Rapids LFA sanctioned a larger percentage of sample members than any other program. The two programs with the highest participation rates, Riverside LFA and Portland, both had mid-level sanctioning rates.
For the remainder of the chapter, the focus will narrow to the seven sites with five-year survey data. When two- and five-year trends are discussed, the sample is further restricted to those sample members for whom both two- and five-year survey data are available.
Five-Year Control Group Participation Patterns
In a random assignment research design, control group participation levels reflect self-initiated patterns of activity in employment-promoting activities. Participation among controls does not represent a compromise to the experiment; rather it shows what program group members would have likely done in the absence of a mandatory welfare-to-work program. In Portland and Riverside, control group members were kept from receiving welfare-to-work program services for the entire five-year follow-up period. For these two sites the control group participation levels presented in Figure 3.1 (and in the tables throughout the chapter) represent entirely self-initiated participation in activities that control group members sought out and attended on their own in their communities. In Atlanta and Grand Rapids, some of the control group participation, specifically participation in job search and work experience activities toward the end of the five-year follow-up period among sample members who were still receiving welfare at that point, did in fact occur in the context of a mandatory welfare-to-work program, after embargoes on such services were lifted. In Atlanta, it is likely that as much as, but certainly no more than, 15 percent of all controls were "exposed" to a mandatory welfare-to-work program in year 4 or 5, while the corresponding estimate for Grand Rapids is 7 percent.(8) Thus, most controls were kept from re-ceiving program services. Control participation in mandatory welfare-to-work program activities was further limited in several respects: First, it was limited to the end of the follow-up period (as late as the last quarter of follow-up year 5 for some controls in Atlanta); second, it was limited to job search or work experience activities, since it is unlikely that controls would have been assigned to activities other than these once they became mandatory to participate.
Figure 3.1
Two-Year and Five-Year Participaiton Rate in Employment-Related Activities for Control Group Members
SOURCE: MDRC calculatios from the Two-Year and Five-Year Client Surveys.
NOTE: See Appendix A.2
Primarily on their own initiative, most control group members in the seven NEWWS programs participated in an employment-promoting activity during the five-year follow-up period, as shown in Figure 3.1. In all sites, controls mainly participated in education and training activities; across all sites from 35 to 55 percent of controls participated in education or training. More specifically, controls primarily enrolled themselves in vocational training programs and post-secondary education.(9) (Participation rates for these activities are presented in Appendix Table B.1.) More than 25 percent of control group members in every site except Atlanta took a post-secondary education course during the follow-up period. Participation levels were equally high for vocational training programs, ranging from 23 percent in Atlanta to 29 percent in Portland. For the most part, control group members' participation in education and training was consistent throughout the follow-up period, although in Atlanta participation was more common in years 3 to 5 than in years 1 and 2.
While education and training were the most common activities that controls participated in throughout the follow-up period, the largest increases in control group participation in years 3 to 5 were in job search. Job search was an activity in which very few controls had participated during the first two years of follow-up (two-year participation rates for this activity ranged from 4 percent in Atlanta to 7 percent in Portland). By the end of year 5, depending on site, from one-fifth to one-third of control group members had participated in job search. In Riverside and Portland, the two sites where control group participation levels represent entirely self-initiated participation, there were striking increases in job search participation in the last follow-up years. Such services were most likely provided to control group members by community colleges and other community-based organizations in the context of other types of education and training activities in which controls were participating, such as vocational training programs and courses at community colleges, after they had left welfare. In Portland, for example, the local community college system offered much-marketed job search services to low-income individuals independent of the welfare department-associated welfare-to-work program. In Atlanta and Grand Rapids, as previously noted, there is some evidence that some of the increase in job search participation was due to controls participating in mandatory welfare-to-work programs; however, even upper-bound estimates of likely program-related participation do not account for the bulk of the increase.
Five-Year Impacts on Participation
The previous section described control group members' patterns of participation in largely self-initiated employment-promoting activities over the five-year follow-up period. As noted, control participation levels represent the extent to which program group members would have likely participated in such activities in the absence of a mandatory welfare-to-work program. Participation impacts, or program-control differences, represent the degree to which program group members in the seven mandatory NEWWS programs examined in this chapter participated in employment-promoting activities above and beyond the rate at which they would have participated in the absence of a mandate to participate.
Impacts for the Full Sample
Impacts for the full sample on three different measures of participation participation in any activity, participation in job search or job club, and participation in any education or training activity are presented in Table 3.1. Appendix Table B.1 presents impacts separately on participation in basic education, post-secondary education, vocational training, and work experience.
|
Site and Program |
Sample Size | Program Group (%) | Control Group (%) | Difference (Impact) | Percentage Change (%) |
|---|---|---|---|---|---|
| Any activity | |||||
| Atlanta Labor Force Attachment | 1,071 | 74.9 | 54.8 | 20.1 *** | 36.7 |
| Atlanta Human Capital Development | 1,146 | 74.2 | 54.8 | 19.3 *** | 35.3 |
| Grand Rapids Labor Force Attachment | 1,097 | 77.7 | 63.9 | 13.7 *** | 21.5 |
| Grand Rapids Human Capital Development | 1,109 | 81.1 | 63.9 | 17.1 *** | 26.8 |
| Riverside Labor Force Attachment | 1,219 | 78.2 | 57.4 | 20.8 *** | 36.2 |
| Lacked high school diploma or basic skills | 657 | 77.0 | 55.1 | 21.9 *** | 39.7 |
| Riverside Human Capital Development | 778 | 82.0 | 55.1 | 26.8 *** | 48.6 |
| Portland | 504 | 83.8 | 75.0 | 8.8 ** | 11.8 |
| Job search or job club | |||||
| Atlanta Labor Force Attachment | 1,071 | 59.2 | 30.1 | 29.2 *** | 97.0 |
| Atlanta Human Capital Development | 1,146 | 43.7 | 30.1 | 13.6 *** | 45.4 |
| Grand Rapids Labor Force Attachment | 1,097 | 51.6 | 21.3 | 30.3 *** | 142.5 |
| Grand Rapids Human Capital Development | 1,109 | 39.5 | 21.3 | 18.2 *** | 85.6 |
| Riverside Labor Force Attachment | 1,219 | 53.5 | 20.4 | 33.2 *** | 162.9 |
| Lacked high school diploma or basic skills | 657 | 58.2 | 19.6 | 38.5 *** | 196.5 |
| Riverside Human Capital Development | 778 | 49.6 | 19.6 | 30.0 *** | 152.9 |
| Portland | 504 | 65.2 | 35.4 | 29.8 *** | 84.1 |
| Any education or training | |||||
| Atlanta Labor Force Attachment | 1,071 | 42.2 | 35.3 | 6.9** | 19.4 |
| Atlanta Human Capital Development | 1,146 | 61.2 | 35.3 | 25.8*** | 73.1 |
| Grand Rapids Labor Force Attachment | 1,097 | 54.4 | 55.2 | -0.7 | -1.3 |
| Grand Rapids Human Capital Development | 1,109 | 69.3 | 55.2 | 14.1*** | 25.6 |
| Riverside Labor Force Attachment | 1,219 | 50.1 | 48.9 | 1.3 | 2.6 |
| Lacked high school diploma or basic skills | 657 | 46.0 | 47.1 | -1.0 | -2.2 |
| Riverside Human Capital Development | 778 | 68.9 | 47.1 | 21.9*** | 46.4 |
| Portland | 504 | 67.5 | 55.0 | 12.5*** | 22.8 |
| SOURCE: MDRC calculations from the Five-Year Client Survey. NOTES: See Appendix A.2. |
|||||
Most program group members (from 74 percent in Atlanta HCD to 84 percent in Portland) participated in some type of employment-promoting activity over the five-year follow-up period. Overall participation rates (participation in "any activity") did not vary by program focus; for example, while the Portland employment-focused program achieved the highest overall participation rate, the Riverside HCD program achieved the second highest rate. All seven programs increased overall participation relative to control group levels. The magnitude of impacts varied somewhat by program: Most programs increased participation by approximately 20 percentage points or more, while the Portland program had a relatively small impact of only 9 percentage points.(10) The fact that the Portland program did not generate a larger increase in participation is likely due to substantial control group participation in the later follow-up years. Control group members in this site more actively participated in self-initiated activities, particularly in years 3 to 5, than controls in other sites.
As expected, job search was the most common activity among program group members in employment-focused programs; well over half of program group members in all four programs participated in this activity over the five-year follow-up period. All four of these programs increased participation in job search by approximately 30 percentage points or more. Work experience (including on-the-job training and unpaid work experience programs) was also a somewhat common activity in employment-focused programs, particularly the Portland and Atlanta LFA programs. All three LFA programs had moderate impacts on participation in this activity, while the Portland program did not increase participation relative to the control group.
Notably, job search participation among HCDs increased markedly in the last follow-up years. Across the three programs, five-year job search participation rates ranged from 40 to 50 percent.(11) As previously noted, these increases were anticipated for several reasons. It is natural that with the passage of time in a longer follow-up period, participation in all activities would increase. More specifically, over time in these programs additional program group members completed their initial education-focused activity assignments, and those who did so were then expected to test their skills in the labor market through some type of job search program. Finally, as discussed in Chapter 1, these programs became somewhat more employment-focused over time. As a result of all these factors, five-year impacts on job search participation were larger than two-year impacts in all three HCD programs. Despite these increases, cumulative five-year impacts on job search in the Atlanta and Grand Rapids HCD programs were still smaller than those in their LFA programs and in employment-focused programs in general, while the Riverside HCD program had a job search participation impact of 30 percentage points, roughly equal to the impacts found in the four employment-focused programs.
Education and training remained the most common type of activity for program group members in the three HCD programs throughout the five-year follow-up period. Participation levels in all three programs were in the 60 to 70 percent range on this aggregate measure. Two of the three HCD programs, Atlanta and Riverside, had impacts of greater than 20 percentage points on overall education and training, while the third, Grand Rapids, had a more modest impact of 14 percentage points.
The aggregate "education and training" measure includes participation in basic education, post-secondary education, and vocational training. Participation rates and impacts were also examined for each of these activities separately. (See Appendix Table B.1.) Notably, all three programs increased basic education participation for the full sample. The Riverside HCD program, which enrolled only nongraduates or those deemed by program staff to be in need of basic education at the time of study entry, increased basic education participation by 31 percentage points. The other two HCD programs, Atlanta and Grand Rapids, enrolled both graduates and nongraduates, and these programs had more modest gains when both subgroups are considered together. (Impacts are discussed separately for graduates and nongraduates, below.)
For the full sample, impacts on vocational training and post-secondary education were not as consistent across the HCD programs and, where they did occur, were not as large as impacts on participation in basic education. The HCD programs in Atlanta and Grand Rapids increased participation in vocational training for the full sample, although the latter impact was just below the threshold for statistical significance. No HCD program increased the percentage of sample members who took a course at a community, two-year, or four-year college.
While there was substantial education and training participation among LFAs roughly half of all program group members in the LFA programs participated in an education or training activity over the follow-up period impacts on education and training participation were much less common and, where they did occur, smaller in the LFA programs than in the HCD programs. Of the three LFA programs, only Atlanta increased overall participation in education and training activities. This impact was due to a 10 percentage point increase in basic education participation, as the program did not increase post-secondary education or vocational training participation.
The Portland program, while employment-focused, differed from the LFA programs in that case managers did not always assign individuals to job search as their first activity, and the program made heavy use of available short-term education and training programs. Portland staff, more so than case managers in the LFA programs, emphasized education and training as a means of building skills necessary to acquire higher-paying, stable jobs. This emphasis is reflected in program group members' participation: Over the five-year follow-up period, 42 percent of program group members took a course at a community, two-year, or four-year college, and 25 percent participated in a vocational training program. While the program did not increase vocational training participation relative to control group levels, the program did increase college participation by a notable 16 percentage points.
Impacts by Education Subgroup
Consistent with the findings for the full sample, all programs had substantial impacts on participation in any activity for both educational attainment subgroups, of roughly equal magnitude to those found for the full sample. These impacts were statistically significant in all sites except for Portland, where the lack of statistical significance is likely a product of small sample size.(12) All seven programs also had impacts on job search participation, equal in size to those found for the full sample, for both subgroups. (See Tables 3.2 and 3.3 for impacts on participation in any activity, job search, and any education or training activity, by educational attainment subgroup. Impacts on participation in basic education, post-secondary education, vocational training, and work experience are presented in Appendix Tables B.2 and B.3.) In the paragraphs that follow, impacts on participation in specific activities are discussed for both high school graduates and nongraduates.
| Site and Program | Sample Size | Program Group (%) | Control Group (%) | Difference (Impact) | Percentage Change (%) |
|---|---|---|---|---|---|
| Any activity | |||||
| Atlanta Labor Force Attachment | 407 | 70.1 | 49.8 | 20.3 *** | 40.7 |
| Atlanta Human Capital Development | 437 | 74.3 | 49.8 | 24.5 *** | 49.2 |
| Grand Rapids Labor Force Attachment | 434 | 77.0 | 64.1 | 13.0 *** | 20.2 |
| Grand Rapids Human Capital Development | 468 | 85.0 | 64.1 | 20.9 *** | 32.6 |
| Riverside Labor Force Attachment | 657 | 77.0 | 55.1 | 21.9 *** | 39.7 |
| Riverside Human Capital Development | 778 | 82.0 | 55.1 | 26.8 *** | 48.6 |
| Portland | 163 | 82.0 | 71.4 | 10.7 | 14.9 |
| Job search or job club | |||||
| Atlanta Labor Force Attachment | 407 | 56.6 | 27.7 | 28.9 *** | 104.4 |
| Atlanta Human Capital Development | 437 | 43.2 | 27.7 | 15.5 *** | 56.1 |
| Grand Rapids Labor Force Attachment | 434 | 51.1 | 22.6 | 28.5 *** | 126.4 |
| Grand Rapids Human Capital Development | 468 | 39.3 | 22.6 | 16.7 *** | 74.0 |
| Riverside Labor Force Attachment | 657 | 58.2 | 19.6 | 38.5 *** | 196.5 |
| Riverside Human Capital Development | 778 | 49.6 | 19.6 | 30.0 *** | 152.9 |
| Portland | 163 | 64.8 | 36.4 | 28.4 *** | 78.1 |
| Any education or training | |||||
| Atlanta Labor Force Attachment | 407 | 40.3 | 31.0 | 9.3 ** | 30.0 |
| Atlanta Human Capital Development | 437 | 65.4 | 31.0 | 34.4 *** | 111.3 |
| Grand Rapids Labor Force Attachment | 434 | 57.4 | 54.4 | 3.0 | 5.5 |
| Grand Rapids Human Capital Development | 468 | 75.9 | 54.4 | 21.5 *** | 39.5 |
| Riverside Labor Force Attachment | 657 | 46.0 | 47.1 | -1.0 | -2.2 |
| Riverside Human Capital Development | 778 | 68.9 | 47.1 | 21.9 *** | 46.4 |
| Portland | 163 | 68.3 | 58.0 | 10.3 | 17.8 |
| SOURCE: MDRC calculations from the Five-Year Client Survey. NOTES: See Appendix A.2. | |||||
Site and Program | Sample Size | Program Group (%) | Control Group (%) | Difference (Impact) | Percentage Change (%) |
|---|---|---|---|---|---|
| Any activity | |||||
| Atlanta Labor Force Attachment | 664 | 77.9 | 58.1 | 19.7 *** | 34.0 |
| Atlanta Human Capital Development | 709 | 73.8 | 58.1 | 15.7 *** | 27.0 |
| Grand Rapids Labor Force Attachment | 663 | 78.0 | 64.0 | 14.0 *** | 21.9 |
| Grand Rapids Human Capital Development | 641 | 79.0 | 64.0 | 15.0 *** | 23.5 |
| Riverside Labor Force Attachment | 562 | 79.8 | 60.4 | 19.3 *** | 32.0 |
| Portland | 334 | 84.2 | 77.1 | 7.1 | 9.2 |
| Job search or job club | |||||
| Atlanta Labor Force Attachment | 664 | 60.6 | 31.6 | 29.0 *** | 91.6 |
| Atlanta Human Capital Development | 709 | 44.1 | 31.6 | 12.4 *** | 39.3 |
| Grand Rapids Labor Force Attachment | 663 | 51.9 | 20.5 | 31.4 *** | 153.3 |
| Grand Rapids Human Capital Development | 641 | 39.5 | 20.5 | 19.0 *** | 92.6 |
| Riverside Labor Force Attachment | 562 | 47.3 | 21.2 | 26.0 *** | 122.7 |
| Portland | 334 | 64.2 | 36.0 | 28.2 *** | 78.2 |
| Any education or training | |||||
| Atlanta Labor Force Attachment | 664 | 43.8 | 38.1 | 5.7 | 15.0 |
| Atlanta Human Capital Development | 709 | 58.0 | 38.1 | 19.9 *** | 52.2 |
| Grand Rapids Labor Force Attachment | 663 | 52.9 | 55.9 | -3.0 | -5.4 |
| Grand Rapids Human Capital Development | 641 | 65.7 | 55.9 | 9.8 *** | 17.6 |
| Riverside Labor Force Attachment | 562 | 55.6 | 51.4 | 4.2 | 8.3 |
| Portland | 334 | 67.5 | 53.2 | 14.3 ** | 26.9 |
| SOURCE: MDRC calculations from the Five-Year Client Survey. NOTES: See Appendix A.2 | |||||
1. High School Nongraduates
All three HCD programs assigned nongraduate sample members to basic education as their first activity, and the majority of nongraduate HCDs (approximately 60 to 70 percent) in all three programs participated in basic education over the five-year follow-up period. These participation levels resulted in impacts ranging from 27 percentage points in Grand Rapids to 42 percentage points in Atlanta. A small increase in post-secondary education participation in Grand Rapids was the only impact found on participation in any education or training activity beyond basic education for this subgroup. (See Appendix Table B.2.)
Atlanta LFA and Portland, the only two employment-focused programs to increase education and training participation for the full sample, also increased this type of participation for nongraduates. The Atlanta LFA program had a 16 percentage point increase in basic education participation and the Portland program had a more modest 10 percentage point gain. None of the employment-focused programs increased participation in vocational training or post-secondary education for this subgroup.
2. High School Graduates
The HCD programs in Atlanta and Grand Rapids enrolled sample members with a high school diploma or GED at the time of study entry, and they often assigned sample members with these credentials to vocational training programs. Basic education was also an assigned activity for some sample members with a diploma or GED certificate if they had low math or reading skills. Although assignments to college were not made in HCD programs, in some instances case managers approved post-secondary education courses that program group members had already enrolled in on their own as fulfillment of the participation requirement.
In Atlanta, 44 percent of graduate HCDs participated in a vocational training program over the follow-up period, resulting in a large impact (17 percentage points) on the measure for this subgroup. Participation in post-secondary education was less common, and the program did not increase participation in this activity. In Grand Rapids, participation levels were high in both college and vocational training for graduates among both program and control group members, resulting in no impacts on either activity. Both HCD programs generated small increases in basic education participation for this subgroup. (See Appendix Table B.3.)
As was found for the full sample and for nongraduates, impacts on education and training activities for graduates in the three LFA programs were not widespread. None of the three LFA programs had a statistically significant impact on the aggregate education and training participation measure. The only participation impact found among these programs for this subgroup was a small increase in basic education participation in Atlanta.
The Portland program increased education and training participation by 14 percentage points for graduates, principally the result of a large 21 percentage point increase in college participation. More than half of the program group members in Portland took a course for credit at a two-year or four-year college, which was the only NEWWS program to increase college participation.
Impacts on Participation in Year 5
As noted, programs were not expected to continue to enroll individuals in employment-promoting activities at the same rate during follow-up years 3 to 5 as they did during years 1 and 2. If the NEWWS programs were achieving their goal of increasing sample members' self-sufficiency, program group members should have been leaving welfare for employment. As this occurred, participation levels should have decreased over the follow-up period, particularly in job search and work experience activities. Furthermore, changes in the welfare reform environment discussed in Chapter 1 might have led to a simultaneous increase in participation among control group members who were still on welfare in later follow-up years. All these factors would lead to small or no program impacts on participation in year 5.
As shown in Table 3.4, year 5 participation rates in "any activity" were similar for program and control group members in all sites (ranging from 18 to 26 percentage points), resulting in no statistically significant participation impacts. In year 5 program and control members most often participated in job search, post-secondary education, and vocational training. Very few sample members participated in basic education. (Results not shown in tables.)
|
Site and Program |
Sample Size | Program Group (%) | Control Group (%) | Difference (Impact) | Percentage Change (%) |
|---|---|---|---|---|---|
| Any activity | |||||
| Atlanta Labor Force Attachment | 1,018 | 24.6 | 22.5 | 2.1 | 9.2 |
| Atlanta Human Capital Development | 1,092 | 20.9 | 22.5 | -1.6 | -7.1 |
| Grand Rapids Labor Force Attachment | 1,043 | 19.0 | 18.7 | 0.4 | 2.0 |
| Grand Rapids Human Capital Development | 1,048 | 21.6 | 18.7 | 2.9 | 15.5 |
| Riverside Labor Force Attachment | 1,072 | 26.0 | 22.0 | 4.0 | 18.2 |
| Lacked high school diploma or basic skills | 582 | 24.4 | 20.5 | 3.8 | 18.6 |
| Riverside Human Capital Development | 685 | 19.6 | 20.5 | -0.9 | -4.4 |
| Portland | 442 | 20.3 | 21.9 | -1.6 | -7.2 |
| Job search or job club | |||||
| Atlanta Labor Force Attachment | 1,018 | 14.5 | 11.0 | 3.4 * | 31.1 |
| Atlanta Human Capital Development | 1,092 | 13.4 | 11.0 | 2.4 | 21.3 |
| Grand Rapids Labor Force Attachment | 1,043 | 8.9 | 5.6 | 3.3 ** | 59.2 |
| Grand Rapids Human Capital Development | 1,048 | 8.8 | 5.6 | 3.2 * | 58.2 |
| Riverside Labor Force Attachment | 1,072 | 10.0 | 3.9 | 6.1 *** | 157.7 |
| Lacked high school diploma or basic skills | 582 | 9.4 | 4.5 | 5.0 ** | 111.7 |
| Riverside Human Capital Development | 685 | 10.6 | 4.5 | 6.1 *** | 136.8 |
| Portland | 442 | 10.6 | 10.8 | -0.2 | -1.9 |
| Any education or training | |||||
| Atlanta Labor Force Attachment | 1,018 | 11.0 | 13.4 | -2.3 | -17.4 |
| Atlanta Human Capital Development | 1,092 | 10.8 | 13.4 | -2.6 | -19.5 |
| Grand Rapids Labor Force Attachment | 1,043 | 10.6 | 15.0 | -4.4 ** | -29.4 |
| Grand Rapids Human Capital Development | 1,048 | 13.7 | 15.0 | -1.3 | -8.8 |
| Riverside Labor Force Attachment | 1,072 | 16.0 | 18.8 | -2.7 | -14.5 |
| Lacked high school diploma or basic skills | 582 | 14.5 | 16.7 | -2.2 | -13.3 |
| Riverside Human Capital Development | 685 | 10.6 | 16.7 | -6.1 ** | -36.4 |
| Portland | 442 | 10.7 | 12.4 | -1.7 | -13.7 |
| SOURCE: MDRC calculations from the Five-Year Client Survey. NOTES: See Appendix A.2. |
|||||
While none of the seven programs increased overall participation in year 5, five of the programs (all except Atlanta HCD and Portland) continued to generate small increases in job search participation. None of the seven programs, however, increased participation in education or training activities in year 5, although there was a statistically significant decrease in the Riverside HCD program, probably because some Riverside HCD program group members who would have participated in this type of activity on their own were instead assigned to job search activities by their case managers.
Year 5 participation levels were substantially higher for sample members who received at least one welfare payment in year 5. In more than half of the programs year 5 participation rates (in "any activity") for year 5 welfare recipients exceeded 30 percent. Participation levels were particularly high in job search; in many programs year 5 job search participation rates for recipients were about twice as high as those found for the full sample. Year 5 participation levels for those still receiving welfare indicate that the programs were still actively working with the on-welfare caseload at the end of the five-year follow-up period.
In Atlanta and Portland, job search participation levels were high for control group members still receiving welfare in year 5 as well, with roughly one-fifth of control group welfare recipients in both sites participating in job search in year 5. As has been discussed, this level of control group participation in job search in Atlanta is likely at least partially the result of some control group members receiving welfare-to-work program services in year 5. In Portland, as noted, this level of control group job search participation is likely due to the marketing and reach of the local community college system, which offers job search services to low-income individuals independent of the welfare department-associated welfare-to-work program.
Year 5 participation rates did not vary by educational attainment subgroup. However, significant increases or impacts in job search participation were concentrated among nongraduates. (Results not shown.) This cluster of impacts is not surprising, given that nongraduates have longer stays on welfare and were thus more likely to still be on welfare and subject to a mandatory welfare-to-work program participation requirement at the end of the follow-up period.
Trends Over Time in Participation Impacts
Figure 3.2 presents two-year and five-year participation rates for program and control group members. For each program, the difference between the side-by-side black and white bars represents the program impact for the respective cumulative period. In all seven programs, as indicated by the stars representing statistical significance, program group participation levels were significantly higher than control group levels for both years 1 and 2 and years 1 to 5. Notably, two- and five-year cumulative impacts were similar. For the most part, impacts that were large after two years remained so after five years. Thus, during years 3 to 5, differences between program and control group participation rates were maintained; although, for the most part, impacts did not grow.
Figure 3.2
Two-Year and Five-Year Participation in Emploment-Related Activities, by Research Group
SOURCE: MDRC calculations from the Five-Year Client Survey.
NOTES: See Appendix A.2.
Five-year cumulative impacts were somewhat smaller than those found after two years in the Atlanta HCD and Portland programs. In all three HCD programs, five-year impacts on job search participation were larger than impacts found after two years. (Results not shown.)
Five-Year Impacts on Degree Receipt
Degree attainment can also play an important role in sample members' long-term labor market and welfare behavior. Individuals who received a GED or trade certificate during follow-up may have delayed entry into the labor market while they were attending school. In later years, however, those who attain new education credentials may have a better chance of finding a job or advancing to higher-paying and more stable employment. It is important to note that while research on the economic gains to welfare recipients of skill building, including degree attainment, has been rather inconclusive, a substantial body of research exists that shows the general importance of education credentials in the labor market. The outcomes described in this section thus may play an important role in the economic outcomes discussed in later chapters and in the future labor market success of program group members beyond the follow-up period covered in this report.(13)
In this analysis degree attainment was analyzed separately for high school graduates and nongraduates. For nongraduates, three measures were examined: receipt of a high school diploma or GED certificate, receipt of a trade license or certificate, and receipt of a high school diploma or GED certificate and a second education or training credential. For sample members with a high school diploma or GED at the time of program entry, two measures were analyzed: receipt of any education or training credential (either a trade license or an associate's, bachelor's, or graduate degree) and receipt of a trade license or certificate.
High School Nongraduates
In all three HCD programs a majority of high school nongraduate sample members participated in basic education, including GED preparation classes, and the degree to which these programs increased receipt of a high school diploma or GED certificate relative to control group levels is a key indicator of the relative success of the basic education component of these programs. As shown in Table 3.5, across all sites from 3 to 17 percent of all control group members in this subgroup received a high school diploma or GED during the follow-up period. All three HCD programs increased program group members' receipt of this credential. The largest impact 11 percentage points occurred in the Riverside HCD program, where nearly 20 percent of nongraduate program group members received this credential over the follow-up period. The Atlanta and Grand Rapids HCD programs had slightly smaller impacts, 7 and 9 percentage points, respectively. While the Atlanta HCD program had a moderate impact on this measure, absolute levels of degree receipt were quite low in this program for both program and control group members.
| Site and Program | Sample Size | Program Group (%) | Control Group (%) | Difference (Impact) | Percentage Change (%) |
|---|---|---|---|---|---|
| Received a high school diploma or GED | |||||
| Atlanta Labor Force Attachment | 407 | 4.1 | 3.0 | 1.2 | 39.0 |
| Atlanta Human Capital Development | 437 | 10.0 | 3.0 | 7.1*** | 237.6 |
| Grand Rapids Labor Force Attachment | 434 | 15.4 | 13.1 | 2.3 | 17.5 |
| Grand Rapids Human Capital Development | 468 | 22.5 | 13.1 | 9.4*** | 71.4 |
| Riverside Labor Force Attachment | 657 | 5.9 | 5.9 | 0.1 | 0.9 |
| Riverside Human Capital Development | 778 | 17.0 | 5.9 | 11.1*** | 188.5 |
| Portland | 163 | 25.9 | 16.8 | 9.1 | 54.2 |
| Received a trade license or certificate | |||||
| Atlanta Labor Force Attachment | 407 | 5.3 | 5.1 | 0.2 | 4.0 |
| Atlanta Human Capital Development | 437 | 6.0 | 5.1 | 0.9 | 17.1 |
| Grand Rapids Labor Force Attachment | 434 | 7.3 | 9.7 | -2.4 | -24.6 |
| Grand Rapids Human Capital Development | 468 | 9.7 | 9.7 | 0.0 | 0.4 |
| Riverside Labor Force Attachment | 657 | 11.7 | 10.7 | 1.0 | 9.4 |
| Riverside Human Capital Development | 778 | 11.6 | 10.7 | 0.9 | 8.5 |
| Portland | 163 | 16.6 | 4.4 | 12.2** | 277.5 |
| Received a high school diploma or GED and a second education or training credential | |||||
| Atlanta Labor Force Attachment | 407 | 1.3 | 1.1 | 0.2 | 20.3 |
| Atlanta Human Capital Development | 437 | 3.1 | 1.1 | 2.0 | 184.9 |
| Grand Rapids Labor Force Attachment | 434 | 2.6 | 3.6 | -1.0 | -28.0 |
| Grand Rapids Human Capital Development | 468 | 4.2 | 3.6 | 0.6 | 16.5 |
| Riverside Labor Force Attachment | 657 | 1.8 | 2.5 | -0.6 | -26.2 |
| Riverside Human Capital Development | 778 | 3.4 | 2.5 | 1.0 | 38.5 |
| Portland | 163 | 8.4 | 0.3 | 8.1** | 3171.6 |
| SOURCE: MDRC calculations from the Five-Year Client Survey. NOTES: See Appendix A.2. | |||||
Absolute levels of high school diploma and GED receipt were substantially lower for program group members in the three LFA programs than in the HCD programs, and none of the LFA programs significantly increased any measure of degree receipt for high school nongraduates.
The Portland program had the most noteworthy effects on degree attainment for nongraduates 25 percent of nongraduate program group members received a high school diploma or GED over the follow-up period, and the program produced an increase, though one above the standard cutoff used in this report for statistical significance, of 9 percentage points on the measure.(14) The program also had a large impact on the receipt of a trade license or certificate, and for this subgroup it was the only program to have such an effect. This finding is somewhat surprising given that the program did not have a statistically significant five-year impact on vocational training for nongraduates (see Section V). There are several possible explanations: The program may have increased the likelihood that program group members enrolled in training programs offering trade licenses or certificates rather than nondegree programs, while not necessarily increasing participation in training programs overall. The program also may have increased the likelihood that individuals completed credential-offering programs. Finally, it is possible that impacts on participation in training found at the two-year follow-up point drove this impact on trade license receipt.(15) The most notable finding on degree receipt in this program, however, is that it had a substantial impact on the percentage of high school nongraduate sample members who attained both a GED certificate or a high school diploma and a second education or training credential (such as a trade license or college degree) over the follow-up period. Nonexperimental research conducted as part of the NEWWS Evaluation has found that high school nongraduate participants in adult education have substantially stronger improvements in longer-term earnings and self-sufficiency if they also participate in skills training or college, making the Portland impact on this measure a particularly promising finding.(16)
High School Graduates
As shown in Table 3.6, education and training credential receipt was prevalent among control group members who already had a high school diploma or GED certificate at study entry, although less so in both Atlanta programs. Trade licenses were by far the most common type of credential earned for this subgroup; very few individuals obtained college degrees. Atlanta HCD, the only program to increase graduate sample members' vocational training participation, also increased trade license receipt for this subgroup. The only other program to have this effect was the Atlanta LFA program. In Grand Rapids, where graduate control group members' self-initiated participation in post-secondary education and vocational training was common throughout the follow-up period, the LFA program actually decreased credential receipt for graduates, likely by diverting to job search sample members who would have participated in education or training activities on their own (and thus possibly have obtained an education credential). This pattern is also reflected in the participation impacts discussed in Section V.
Site and Program | Sample Size | Program Group (%) | Control Group (%) | Difference (Impact) | Percentage Change (%) |
|---|---|---|---|---|---|
| Received any education or training credential | |||||
| Atlanta Labor Force Attachment | 664 | 20.4 | 13.2 | 7.2 ** | 54.7 |
| Atlanta Human Capital Development | 709 | 26.9 | 13.2 | 13.7 *** | 104.2 |
| Grand Rapids Labor Force Attachment | 663 | 18.3 | 29.8 | -11.5 *** | -38.7 |
| Grand Rapids Human Capital Development | 641 | 28.7 | 29.8 | -1.0 | -3.4 |
| Riverside Labor Force Attachment | 562 | 20.5 | 25.4 | -4.9 | -19.2 |
| Portland | 334 | 28.9 | 22.4 | 6.5 | 28.9 |
| Received a trade license or certificate | |||||
| Atlanta Labor Force Attachment | 664 | 16.8 | 12.0 | 4.9 * | 40.8 |
| Atlanta Human Capital Development | 709 | 23.6 | 12.0 | 11.6 *** | 97.3 |
| Grand Rapids Labor Force Attachment | 663 | 15.5 | 22.5 | -7.0 ** | -31.2 |
| Grand Rapids Human Capital Development | 641 | 26.1 | 22.5 | 3.6 | 16.1 |
| Riverside Labor Force Attachment | 562 | 18.1 | 22.9 | -4.8 | -21.1 |
| Portland | 334 | 23.6 | 18.6 | 5.0 | 26.8 |
| SOURCE: MDRC calculations from the Five-Year Client Survey. NOTES: See Appendix A.2. | |||||
Conclusions
All three LFA programs in NEWWS were job search first programs, for all individuals who were enrolled. Compared with what would have happened in the absence of these programs, the LFA programs increased participation in job search to quite a large degree for both graduates and nongraduates. For the most part, these programs did not increase education and training participation or degree receipt, although one program Atlanta LFA increased basic education participation for both subgroups and increased trade license or certificate receipt for graduates.
All HCD programs were education-focused, although activity assignments differed depending on the educational background and skills level of enrollees. Sample members without a high school diploma or GED as well as some sample members who had this credential but had low scores on math or reading achievement tests were most often assigned to basic education, and basic education participation rates for nongraduates were much higher than control group rates in all three programs. The two programs that enrolled graduates in HCD programs also increased their participation in this activity, although to a lesser degree. The HCD programs were more likely than the LFA programs to result in nongraduates obtaining a high school diploma or GED at some point during the follow-up period. For those with a high school diploma or GED, assignments to and participation in vocational training were most common, although only one of the two programs enrolling graduates increased participation in this activity relative to control group levels. The difference in assignment patterns for graduate and nongraduate sample members is likely explained by the fact that a high school diploma or GED, or in some cases a certain score on a reading or math achievement test, is required for entry into most vocational training programs. For the graduate subgroup, only the Atlanta HCD program resulted in more individuals obtaining an education or training credential (usually a trade license or certificate). Notably, the Atlanta LFA program had the same effect for this subgroup.
As is clear from the findings presented in this chapter, the Portland program was unique among these seven NEWWS programs in that program staff employed a mixed strategy, using both job search and short-term education and training programs as a first activity, to enhance individuals' self-sufficiency. Program staff encouraged short-term education and training as a means of enhancing employability specifically, as a means of obtaining jobs with higher wages and benefits for all sample members, both graduates and nongraduates. As a result, the program increased participation in job search for the full sample and for both subgroups, as well as increasing participation in education activities across both subgroups. For nongraduates, increases in education and training participation led to an increase, relative to control group levels, in the percentage of sample members who received a high school diploma or GED and a trade license or certificate during the follow-up period.
All seven programs generated impacts on participation of a sufficient magnitude to verify that the programs studied provided a good test of the relative effectiveness of different welfare-to-work program models, including the side-by-side test of LFA and HCD programs in the three sites that simultaneously operated both program types. The subsequent chapters in this report examine the economic effects of all 11 NEWWS programs on adults, as well as on households and children, that were produced by these changes in employment-promoting activities and in degree receipt.
1. See, for example, Riccio, Friedlander, and Freedman, 1994, Table 2.4; Kemple, Friedlander, and Fellerath, 1995, Table 3.5; Hamilton and Friedlander, 1989, Table 3.1.
2. See Freedman et al., 2000a, for details.
3. An awareness of welfare caseload dynamics is essential in understanding and interpreting welfare-to-work program participation rates. A number of studies have shown that many welfare recipients cycle on and off the welfare rolls, often leaving without any special intervention. For example, some people get jobs on their own or get married. To the extent that this occurs among individuals mandated for a welfare-to-work program before they enter their first program activity, a site's overall participation rate will be lowered. This rate will be further lowered to the extent that individuals obtain part-time employment, which, if it involves a specified number of hours per week, excuses clients from a program participation requirement. At the same time, welfare-to-work programs may induce some of these behavioral changes. For example, a desire to avoid a program participation requirement may lead some individuals to find employment or leave welfare sooner than they otherwise would have done, again lowering a site's participation rate if these actions are taken prior to starting an activity. Alternatively, some individuals might feel encouraged to remain on welfare longer to take advantage of a program's opportunities for education and training. Thus, participation rates, whether high or low, are influenced by normal welfare caseload turnover as well as by a welfare-to-work program's intervention. In any case, given welfare dynamics, participation rates in these programs should never be expected to reach 100 percent.
4. See Hamilton et al., 1997, Figure 3.1, Figure 3.2, and pp. 39-44, for a discussion of the sequencing of activities in both LFA and HCD programs.
5. In the benefit-cost analysis in Chapter 13, activities outside the programs are called "out-of-program" activities.
6. For detailed descriptions of the length and intensity of participation, including total hours of participation, in all 11 NEWWS programs, see the following: Hamilton et al., 1997, Tables 5.5 and 6.4 (for the LFA and HCD programs in Atlanta, Grand Rapids, and Riverside); Scrivener et al., 1998, Table 3.4 (for Portland); Storto et al., 2000, Table 2.2 (for Oklahoma City); Farrell, 2000, Table 3.5 (for Detroit); and Scrivener and Walter, 2001, Table 3.4 for Columbus.
7. See Freedman et al., 2000a, Appendix A.1, for complete two-year participation rates and impacts.
8. The estimates of the proportion of control group members in Atlanta and Grand Rapids who were "exposed" to welfare-to-work program services after embargoes on such services were lifted are considered upper-bound ones for a number of reasons, but primarily because of client survey data limitations regarding the timing of participation. The Five-Year Client Survey did not ask respondents to specify the exact month (as opposed to year) when they participated in various activities, so months of post-embargo welfare receipt cannot be lined up directly with participation spells. The method of calculating these estimates thus erred on the side of counting as "exposure" some participation that, in fact, may have occurred while an individual was no longer receiving welfare.
9. "Post-secondary education" includes courses taken at community colleges or two-year and four-year colleges, including but not limited to courses taken for credit toward an associate's or bachelor's degree. "Vocational training" includes classes taken for training in a specific job, trade, or occupation, and does not include courses taken at community colleges or two-year and four-year colleges.
10. Five-year participation rates for program and control group members subject to mandatory welfare-to-work programs are generally not available, and thus statistics comparable to those presented in this section largely do not exist. Five-year participation rates were calculated in an evaluation of San Diego's Saturated Work Initiative Model (SWIM), a program operated in the mid to late 1980s. Among single parents in SWIM (comparable to the NEWWS samples analyzed in this report), 66 percent of program group members participated in activities intended to increase their employment during a five-year follow-up compared with 42 percent of controls, resulting in an impact of 24 percentage points. This measure includes participation in welfare-to-work program activities as well as self-initiated participation. However, control group members became mandatory for California's GAIN welfare-to-work program after three years if they were still receiving welfare, so SWIM's estimates of control group participation levels and participation impacts are not directly comparable to those in most NEWWS sites.
11. Specifically, the Atlanta HCD program's five-year job search participation rate was 3.6 times greater than its two-year job search participation rate. In the Grand Rapids and Riverside HCD programs, five-year job search participation rates were 2.9 and 2.7 times greater, respectively, than two-year rates.
12. For both subgroups, the impacts on participation in any activity are just above the threshold for statistical significance. The sample sizes for these subgroups, including program and control group members, are small: The nongraduate subgroup consists of only 163 sample members, while the graduate subgroup consists of 334 sample members.
13. Chapter 4 in Bos et al., 2001, presents a synthesis of findings related to this topic and of research on welfare populations. In particular, see Mincer, 1974; Polachek and Siebert, 1993; and Sum, Taggart, and Fogg, 1995.
14. The 9.1 percentage point increase has a p-value of .22, and the lack of statistical significance may be a product of small sample size (see footnote 12).
15. For a more detailed discussion of the types of training programs offered by the Portland program, and two-year participation rates and impacts, see Scrivener et al., 1998, Chapter 3.
16. Among nongraduate sample members who participated in basic education during an initial two-year follow-up period, those who went on to participate in post-secondary education or training had an additional $1,542 in earnings in the third year of follow-up compared with those who participated only in basic education and did not go on to participate in post-secondary education or training (Bos et al., 2001, Table 6.4).
Impacts on Employment and Earnings
Welfare-to-work programs have two primary goals: reducing welfare use and increasing employment. This chapter explores the success of the programs studied in the NEWWS Evaluation in achieving the second of these goals by presenting their effects on employment and earnings over the five years after people were randomly assigned. The chapter also investigates whether employment-focused programs had larger initial effects than education-focused programs, whether the effects of education-focused programs had surpassed those of employment-focused programs by the five-year point, and which approach was more successful for high school graduates and nongraduates. Special attention is given to a direct comparison of results for the LFA and HCD programs in Atlanta, Grand Rapids, and Riverside, because these findings provide the most reliable evidence of the relative benefits of each approach.
Key Findings
- Over five years, nearly every program increased employment and earnings relative to what was achieved by their control groups, but the size of the effects varied over time and by program.
- The employment-focused programs produced effects almost immediately, while the education-focused programs did not generally have effects until more than a year after random assignment. In the middle of the follow-up period most of the programs increased employment and earnings. Effects of both types of programs generally diminished during the last two years and were statistically insignificant for most programs at the end of the follow-up period.
- Portland produced the largest, most consistent effects by far: over the five-year period, program group members worked 1.6 quarters more than control group members a 21 percent increase in employment and their earnings were about $5,000 greater on average. Portland's success may be due to its unique combination of a focus on employment, use of both job search and education, and emphasis on finding good jobs.
- The employment-focused LFA programs in Atlanta, Grand Rapids, and Riverside also increased employment and earnings above control group levels, but less than the program in Portland. Earnings gains (the amount by which earnings for the program group exceeded earnings for the control group) ranged from about $1,500 in Grand Rapids to about $2,500 in Atlanta and Riverside. Employment gains ranged from 0.7 quarter in Grand Rapids to 1.1 quarters in Riverside.
- Overall effects were smaller for the education-focused programs than for the employment-focused programs. Neither of the two programs with low enforcement of the participation mandate (Detroit and Oklahoma City) significantly raised employment above control group levels. Among the other five education-focused programs, employment gains over five years ranged from 0.3 to 0.8 quarter and earnings gains ranged from about $800 to about $2,000.
- Side-by-side comparisons of the LFA and HCD programs in Atlanta, Grand Rapids, and Riverside indicate that the LFA programs had larger effects than the HCD programs in all sites in the year after random assignment. The two approaches had similar effects after the first year; however, impacts for HCD programs did not surpass the effects of the LFA programs at the end of the follow-up. As a result, HCD programs led to smaller cumulative employment and earnings impacts over five years than LFA programs and appear unlikely to close the gap in later years.
- Differences between the LFA and HCD programs were concentrated among high school nongraduates, suggesting that job search is a better method than education for increasing employment and earnings of nongraduates, but not for high school graduates. Portland, which used a mix of job search and education, had the largest effects for both groups, however.
Expected Effects
The programs pursued three different strategies to attain the goals of increased employment and earnings: job search first, education and training first, and a mix of job search and education as initial activities.
Employment-focused, job-search-first programs (Atlanta, Grand Rapids, and Riverside LFA are examples of this approach) encourage rapid entry into the labor market and are therefore expected to boost employment and earnings quickly. Proponents of this approach expect people to work their way up to better jobs by gaining experience and skills on the job. If this occurs, then initial employment gains may persist or increase over time, and earnings increases may grow larger in later years. On the other hand, effects of job-search-first programs may grow smaller over time, as control group members begin finding work on their own or if program group members work at low-quality and unstable jobs that they quickly lose.
Education-focused programs (they include Atlanta, Grand Rapids, and Riverside HCD; Columbus Integrated and Traditional; Detroit; and Oklahoma City) aim to increase enrollees' skills and credentials before they seek employment. Employment and earnings gains may be delayed while recipients participate in education and training activities. Education-focused programs may even reduce employment and earnings initially if people would work in the absence of the programs, but enroll in education and training activities instead. For these reasons, effects are likely to be smaller for the education-focused programs than for job-search-first programs in the short term. Toward the end of follow-up, however, the effects of education-focused programs may catch up to or surpass the effects of job-search-first programs, if program group members make up for initial forgone earnings by obtaining higher-quality jobs than they would have under the job-search-first programs. Employment and earnings gains may never occur, however, if enrollees drop out of education and training activities or if area employers have little demand for the skills and credentials that enrollees obtain.
Employment-focused programs that use a mix of initial activities (Portland used this approach) try to combine the best features of the job-search-first and education-focused approaches. For more job-ready individuals, they use the same approach as other employment-focused programs by trying to move people into jobs relatively quickly. Case managers, however, have more discretion to assign some people to short-term skill-building activities first. If this strategy is successful, boosts in employment should occur early in the follow-up, as people required to look for work find jobs. Initial gains may be smaller than they would be for job-search-first programs, however, because some enrollees participate in education or training activities before looking for work. These programs could achieve especially large gains in employment and earnings later in the follow-up by moving a large portion of the caseload into higher-quality jobs. Specifically, more job-ready participants (who are asked to look for work) may be able to advance to better jobs, and less job-ready individuals (who are allowed to enroll in education and training activities) may be able to use their new skills to find better jobs than they would have found if they had looked for work initially. If neither element of the employment-focused, varied first activity approach is effective, however, or if activities are targeted at the wrong people, effects on employment and earnings may be small and should not increase.
This discussion assumes that welfare-to-work programs affect employment and earnings by providing helpful services. It is also possible, however, that sanctions used to enforce the mandates encourage people to go to work in order to avoid participating in program activities. If this is true, then highly mandatory programs may have similar effects initially regardless of their self-sufficiency approach. If education-focused programs have smaller effects than employment-focused programs, it may indicate that services are responsible for the program impacts rather than enforcement of the mandates.
Impacts on Employment and Earnings
Employment and Earnings Over Five Years
Table 4.1 provides the first indication of how much the different programs affected the number of people who worked, the number of quarters they worked, and their total earnings over the five-year follow-up period for each of the 11 programs in the NEWWS Evaluation. In addition, results are shown for sample members in need of basic education for the Riverside LFA program so they can be compared with results from the Riverside HCD program, which included only people in need of basic education.(1) (All tables and figures in this chapter show results based on quarterly reports by employers to state unemployment insurance systems.)
Table 4.1 shows that most control group members worked at some point in the follow-up period, even though most of them were never eligible for program services and none of them were eligible for services during the first three years after random assignment. For example, about 88 percent of control group members in Grand Rapids and 86 percent in Columbus worked at some point. Except for Riverside (66 percent), the Oklahoma City control group was the least likely to work: about 79 percent of control group members worked at some point. The fact that so many control group members worked left little room for the programs to increase the number of people who ever worked during the follow-up period. In fact, only three of the programs increased the number of people who worked by more than 3 percentage points. Moreover, the two programs with the largest effects were both in Riverside, where control group members were the least likely to work.
|
Site and Program |
SampleSize | ProgramGroup | Control Group | Difference(Impact) | Percentage Change (%) |
|---|---|---|---|---|---|
| Ever employed (%) | |||||
| Atlanta Labor Force Attachment | 2,938 | 83.0 | 80.3 | 2.8 ** | 3.4 |
| Atlanta Human Capital Development | 2,992 | 81.2 | 80.3 | 1.0 | 1.2 |
| Grand Rapids Labor Force Attachment | 3,012 | 89.8 | 88.3 | 1.5 | 1.7 |
| Grand Rapids Human Capital Development | 2,997 | 89.5 | 88.3 | 1.2 | 1.4 |
| Riverside Labor Force Attachment | 6,726 | 74.5 | 66.1 | 8.4 *** | 12.7 |
| Lacked high school diploma or basic skills | 3,125 | 70.5 | 61.1 | 9.4 *** | 15.4 |
| Riverside Human Capital Development | 3,135 | 66.9 | 61.1 | 5.8 *** | 9.5 |
| Columbus Integrated | 4,672 | 87.9 | 86.0 | 1.9 ** | 2.2 |
| Columbus Traditional | 4,729 | 87.0 | 86.0 | 1.0 | 1.2 |
| Detroit | 4,459 | 85.5 | 83.0 | 2.5 ** | 3.0 |
| Oklahoma City | 8,677 | 78.8 | 79.2 | -0.5 | -0.6 |
| Portland | 4,028 | 85.8 | 81.7 | 4.1 ** | 5.0 |
| Average number of quarters employed | |||||
| Atlanta Labor Force Attachment | 2,938 | 8.5 | 7.8 | 0.8 *** | 9.7 |
| Atlanta Human Capital Development | 2,992 | 8.3 | 7.8 | 0.5 ** | 6.9 |
| Grand Rapids Labor Force Attachment | 3,012 | 9.8 | 9.1 | 0.7 *** | 8.0 |
| Grand Rapids Human Capital Development | 2,997 | 9.5 | 9.1 | 0.4 * | 3.9 |
| Riverside Labor Force Attachment | 6,726 | 6.8 | 5.6 | 1.1 *** | 20.3 |
| Lacked high school diploma or basic skills | 3,125 | 6.0 | 4.7 | 1.3 *** | 26.9 |
| Riverside Human Capital Development | 3,135 | 5.5 | 4.7 | 0.8 *** | 16.6 |
| Columbus Integrated | 4,672 | 10.2 | 9.8 | 0.4 ** | 4.4 |
| Columbus Traditional | 4,729 | 10.1 | 9.8 | 0.3 * | 3.1 |
| Detroit | 4,459 | 8.3 | 8.0 | 0.2 | 3.0 |
| Oklahoma City | 8,677 | 6.6 | 6.7 | -0.1 | -1.3 |
| Portland | 4,028 | 9.4 | 7.8 | 1.6 *** | 21.1 |
| Average total earnings ($) | |||||
| Atlanta Labor Force Attachment | 2,938 | 19,838 | 17,380 | 2,459 *** | 14.1 |
| Atlanta Human Capital Development | 2,992 | 19,397 | 17,380 | 2,017 ** | 11.6 |
| Grand Rapids Labor Force Attachment | 3,012 | 22,322 | 20,770 | 1,552 * | 7.5 |
| Grand Rapids Human Capital Development | 2,997 | 21,616 | 20,770 | 846 | 4.1 |
| Riverside Labor Force Attachment | 6,726 | 17,438 | 14,889 | 2,549 *** | 17.1 |
| Lacked high school diploma or basic skills | 3,125 | 13,193 | 10,912 | 2,281 *** | 20.9 |
| Riverside Human Capital Development | 3,135 | 12,273 | 10,912 | 1,361 * | 12.5 |
| Columbus Integrated | 4,672 | 27,621 | 25,566 | 2,055 *** | 8.0 |
| Columbus Traditional | 4,729 | 26,977 | 25,566 | 1,410 * | 5.5 |
| Detroit | 4,459 | 21,968 | 20,508 | 1,460 * | 7.1 |
| Oklahoma City | 8,677 | 12,868 | 12,752 | 115 | 0.9 |
| Portland | 4,028 | 26,041 | 20,891 | 5,150 *** | 24.7 |
| SOURCE: MDRC calculations from state and county administrative records. NOTES: See Appendix A.1. |
|||||
In contrast to their generally small effects on whether people ever worked, the programs generally increased how long people worked and how much they earned (see the middle and lower panels of Table 4.1). In 9 of the 11 programs, the program group worked more quarters on average than the control group; moreover, a slightly different combination of nine programs increased average earnings above the control group level.
The Portland program had the largest effect by far on number of quarters employed and earnings. Whereas control group members worked 7.8 quarters on average during the 20-quarter follow-up period, program group members worked 9.4 quarters on average, for an impact of more than 1.6 quarters, or more than 20 percent.(2) The program also increased earnings by more than $5,000 over five years, from just under $21,000 on average for the control group to just over $26,000 on average for the program group. The magnitude of effects in Portland should not be understated: Among welfare-to-work programs studied using random assignment, only the Riverside GAIN effects have been as large as the effects of Portland.(3) Portland's success may be due to its strong focus on employment, its emphasis on finding good jobs, and its mixed use of services that assigned some people to job search and some to education and training. Since it is the only program studied in NEWWS that had an employment focus with a mix of activities, it is difficult to say how important a factor that was in its success. However, in previous evaluations the most effective welfare-to-work programs have generally used similar mixed approaches to determining people's initial activities.(4) Portland's unusual success may also have been bolstered by its strong economy, although other programs that were operated in areas of high employment growth and low unemployment led to smaller impacts on employment and earnings (see Table 1.2). It should also be noted that case managers in Portland were less likely than program staff in other sites to refer people with serious personal or family problems or extremely low motivation levels. These individuals did not attend a program orientation and were not included in the research sample.(5)
The LFA programs, with their requirement that most participants look for work initially, had the next largest impacts as a group. Increases in the average number of quarters of work ranged from 0.7 quarter (Grand Rapids) to 1.1 quarters (Riverside). In addition, increases in total earnings over five years above the control group average ranged from about $1,500 per LFA (Grand Rapids) to about $2,500 (Atlanta and Riverside).
The education-focused programs led to smaller impacts on cumulative measures of employment and earnings. Columbus Integrated and Atlanta HCD were the most effective at increasing total earnings above control group levels. These two programs led to five-year earnings impacts of about $2,000 per program group member, which exceeded the increase for only one of the four employment-focused programs: Grand Rapids LFA. Every other education-focused program led to earnings gains of less than $1,500 over five years (or $300 per year) a historically small increase. Moreover, the program-control group difference in total earnings was not statistically significant in Grand Rapids and Oklahoma City.
As shown in Table 4.1, the Oklahoma City program was the least effective among the 11 programs in the NEWWS Evaluation, probably reflecting the de facto voluntary nature of the program. During much of the follow-up period, the Oklahoma City program was hampered by large caseloads and limited resources. The program placed a priority on working with enrollees who expressed the greatest interest in participating. In addition, its integrated case management program staff spent most of their time performing financial functions (such as helping enrollees to find child care) and devoted little time to monitoring participation in employment-related activities. Detroit, which faced similar problems, also ran a de facto voluntary program for much of the period. Detroit did not significantly raise employment, although it did increase earnings of program group members by more than $1,400 over five years relative to the control group.(6)
Employment by Year
As discussed earlier, employment-focused programs were expected to boost employment and earnings immediately, and effects of education-focused programs were expected to occur more slowly, after enrollees completed their education and training activities. Table 4.1 shows that the cumulative impacts of education-focused programs fell short of the impacts of most employment-focused programs. It is possible, however, that the skills obtained through education and training helped program members find such good jobs that the impacts of the education-focused programs were continuing to grow at the end of the five-year follow-up period relative to the impacts of the employment-focused programs. If this pattern did occur, education-focused programs might have larger impacts than employment-focused programs after year 5 and larger cumulative gains over a longer follow-up period. Figure 4.1 and Appendix Table C.1 show, for each of the 11 programs for the five years of follow-up, the proportion of program and control group members who worked for pay.(7) In addition, the figure and table can be used to examine separately the period when all control group members were embargoed from receiving control group services and the later period. As discussed in Chapter 1, control group members could not receive program services in any of the sites during the first three years after random assignment. In Riverside and Portland, control group members in the report sample were embargoed from receiving services for all five years. It is important to keep in mind, however, that most evidence indicates that ending the embargo is unlikely to have changed the effects of most programs on employment and earnings very much (as discussed at the end of this chapter).
Figure 4.1
Impacts on Emloyment in Year 1 to 5
SOURCE: MDRC calculations from state and county administrative records.
NOTES: See Appendix A.1.
The two Atlanta programs exemplify the basic differences in the effects of employment-focused and education-focused programs.(8) In the LFA program, the attempt to move people into jobs quickly produced an immediate payoff. During the first year after random assignment, about 49 percent of the program group worked compared with 44 percent of the control group, a 5 percentage point difference that was statistically significant. In contrast, the HCD program did not significantly increase employment in the first year, as was expected from the fact that many program group members were enrolled in education and training. However, the effects of the HCD program increased after year 1, and both the LFA and the HCD programs had similar effects on employment in years 2 and 3.
If education-focused programs are to "make up" the ground lost to employment-focused programs in the first year, then they must eventually have larger effects than the employment-focused programs. Figure 4.1 shows that, in fact, the Atlanta LFA program continued to significantly increase employment in year 4, but the HCD program did not. In year 5, neither program significantly affected employment. This finding suggests that it is unlikely that the HCD program will eventually catch up to or surpass the LFA program with longer follow-up.
The other employment-focused programs led to even larger impacts on percentage employed in year 1. Riverside LFA, arguably the most strongly employment-focused program in the evaluation, produced the largest initial increase in employment. (See Appendix Table C.1.) In Riverside, about 51 percent of LFAs worked for pay some time during year 1 compared with 34 percent of the control group. Impacts of the Grand Rapids LFA and Portland programs averaged about 10 percentage points above the control group level in year 1.
As discussed above, the positive effects of employment-focused programs may persist or grow larger over time. This pattern did not occur for the LFA programs in Grand Rapids and Riverside, however. For both programs, impacts on employment declined after year 1, although for different reasons. In Grand Rapids, LFA employment levels increased between years 1 and 2 and again between years 2 and 3, but control group employment levels increased even faster. This result suggests that the Grand Rapids LFA program succeeded initially by encouraging people who would have worked eventually to take jobs faster. In contrast, Riverside LFA employment levels diminished sharply between years 1 and 2, suggesting that a number of LFAs were encouraged to find work by the program but left employment quickly and had trouble finding another job. These differences are also consistent with the fact that Riverside had a fairly weak economy during the early part of the follow-up period.
Most other education-focused programs had small effects initially. Only two of the seven education-focused programs Grand Rapids HCD and Riverside HCD led to significant impacts on employment in year 1. For all of the education-focused programs except Oklahoma City (where impacts were statistically insignificant in each year), impacts were somewhat larger in year 2 or 3 than in year 1. Disappointingly, the effects of most education-focused programs declined after year 3. For all education-focused programs except Riverside HCD, employment levels were not significantly higher for program group members than for control group members in year 5. (Section V of this chapter presents further evidence that the end of the control group embargo is not responsible for declining impacts in most programs in years 4 and 5.)
Results in Grand Rapids and Riverside confirm that, as in Atlanta, it is unlikely that education-focused programs would surpass employment-focused programs with longer follow-up. In Grand Rapids, neither the LFA or HCD program increased employment levels above the control group levels in years 4 and 5, while in Riverside, the LFAs in need of basic education had slightly larger employment impacts than the HCDs.
The Portland program led to the most successful year-by-year pattern of employment impacts: It immediately increased employment (like the LFA programs) but continued to increase employment after year 1 (like the HCD programs). In fact, the year 3 impact on percentage employed (of 12 percentage points) was more than twice as large in Portland as any other program. Like both types of programs, its effects declined after year 3; however, the Portland program again led to relatively large effects (of 8 percentage points) and was one of only three programs to still have a positive effect on employment in year 5. This pattern suggests that Portland combined the best features of employment-focused and education-focused programs, assigning job-ready individuals to job search but allowing others to get short-term education or training initially.
Nonetheless, the decline in employment impacts for all programs after year 3 underscores the limitations of both employment- and education-focused welfare-to-work strategies that rely primarily on pre-employment services and mandates. More recently, policymakers and program administrators have attempted to strengthen the effects of welfare-to-work programs by adding a variety of post-employment services and financial incentives aimed at helping people stay employed and advance to better jobs. It remains to be seen whether such strategies accomplish these goals.
Earnings by Year
Figure 4.2 and Appendix Table C.2 show the average annual earnings of program and control group members during years 1 to 5. These earnings comparisons provide the most comprehensive measure of the effects of the programs because they reflect differences in the number of people who went to work, how long they worked, and their average wages.
Figure 4.2
Impacts on Employment in Years 1 to 5
SOURCE: MDRC calculations from state and county administrative records.
NOTES: See Appendix A.1.
The trends in program effects on earnings are similar to the effects on employment. Employment-focused programs had larger initial effects than education-focused programs, but most programs of both types significantly raised earnings in years 2 and 3 (the exception was Oklahoma City). Earnings impacts for both types of programs declined after year 3 and, especially, after year 4. By the end of the follow-up period, only the Riverside LFA and Portland programs continued to significantly raise earnings above the control group average.
The pattern of earnings suggests once again that employment-focused programs will continue to produce larger cumulative impacts than education-focused programs. Figure 4.2 and Appendix Table C.2 indicate that the LFA and HCD programs had similar effects on earnings in year 5, and, if anything, earnings impacts were slightly higher for the LFAs.
Results for Atlanta LFA and Portland deserve special attention. Like the other employment-focused programs (Grand Rapids LFA and Riverside LFA), Atlanta LFA and Portland significantly raised earnings in year 1. Unlike the other two programs, however, effects on earnings were greater in years 2 and 3 than in year 1. In other words, the pattern of their effects on earnings over time looked more like the education-focused programs and probably reflects these programs' greater emphasis on short-term education and training than the employment-focused programs in Grand Rapids and Riverside.
The findings for Portland are especially noteworthy. Portland continued to produce unusually large earnings impacts (of $1,200 and $900) in years 4 and 5 well above the effects of any other program. As discussed earlier, the Portland program allowed some people considered in need of education to enroll initially in short-term education and training and encouraged job search participants to seek out relatively well-paying jobs with an opportunity for advancement.
Employment Stability and Earnings Growth
As discussed above, in theory both employment- and education-focused programs can lead to impacts on measures of employment stability and earnings growth. Participants in employment-focused programs are expected to start working quickly and acquire valuable skills and experience on the job that lead to employment stability and advancement. In contrast, education-focused programs aim to increase job-related skills and education credentials before employment, thereby helping people to obtain stable and high-paying jobs. Table 4.2 shows the effects of the 11 programs on the proportion of sample members who worked in all four quarters of year 5, and the proportion who earned $10,000 or more in year 5. Appendix Table C.6 shows the effects of the programs on two related measures: whether sample members worked in 9 of the 12 quarters in years 3 to 5 and whether they experienced earnings growth during the follow-up period.(9)
|
Site and Program |
Sample Size | Program Group (%) | Control Group (%) | Difference (Impact) | Percentage Change (%) |
|---|---|---|---|---|---|
| Employed in all four quarters of year 5 | |||||
| Atlanta Labor Force Attachment | 2,938 | 36.9 | 36.6 | 0.3 | 0.8 |
| Atlanta Human Capital Development | 2,992 | 36.5 | 36.6 | -0.1 | -0.3 |
| Grand Rapids Labor Force Attachment | 3,012 | 39.4 | 38.7 | 0.7 | 1.7 |
| Grand Rapids Human Capital Development | 2,997 | 38.3 | 38.7 | -0.4 | -1.0 |
| Riverside Labor Force Attachment | 6,726 | 26.3 | 23.2 | 3.1 *** | 13.5 |
| Lacked high school diploma or basic skills | 3,125 | 23.1 | 18.8 | 4.3 *** | 22.9 |
| Riverside Human Capital Development | 3,135 | 21.6 | 18.8 | 2.7 * | 14.5 |
| Columbus Integrated | 4,672 | 40.7 | 39.4 | 1.3 | 3.3 |
| Columbus Traditional | 4,729 | 40.4 | 39.4 | 1.1 | 2.7 |
| Detroit | 4,459 | 37.4 | 37.1 | 0.3 | 0.7 |
| Oklahoma City | 8,677 | 19.4 | 18.7 | 0.7 | 3.9 |
| Portland | 4,028 | 38.2 | 33.6 | 4.6 ** | 13.6 |
| Earned $10,000 or more | |||||
| Atlanta Labor Force Attachment | 2,938 | 25.7 | 25.0 | 0.6 | 2.6 |
| Atlanta Human Capital Development | 2,992 | 24.8 | 25.0 | -0.2 | -0.8 |
| Grand Rapids Labor Force Attachment | 3,012 | 26.0 | 25.4 | 0.6 | 2.3 |
| Grand Rapids Human Capital Development | 2,997 | 25.3 | 25.4 | -0.0 | -0.1 |
| Riverside Labor Force Attachment | 6,726 | 18.6 | 16.6 | 2.0 ** | 12.2 |
| Lacked high school diploma or basic skills | 3,125 | 14.3 | 12.0 | 2.3 ** | 19.6 |
| Riverside Human Capital Development | 3,135 | 12.7 | 12.0 | 0.7 | 5.9 |
| Columbus Integrated | 4,672 | 34.1 | 31.6 | 2.5 * | 7.9 |
| Columbus Traditional | 4,729 | 32.2 | 31.6 | 0.6 | 2.1 |
| Detroit | 4,459 | 28.4 | 27.1 | 1.4 | 5.0 |
| Oklahoma City | 8,677 | 13.9 | 13.2 | 0.7 | 5.3 |
| Portland | 4,028 | 30.4 | 26.6 | 3.8 * | 14.4 |
| SOURCE: MDRC calculations from state and county administrative records. NOTES: See Appendix A.1. |
|||||
In general, there is little indication that either the employment-focused or the education-focused programs improved stable employment or high earnings in year 5. Only three programs Riverside LFA, Riverside HCD, and Portland resulted in significantly more people working in all four quarters of year 5. Likewise, only three programs Riverside LFA, Columbus Integrated, and Portland resulted in significantly more people earning at least $10,000 in year 5. Moreover, effects for all programs were relatively small that is, less than 5 percentage points on these measures.
A slightly different pattern emerges when stable employment and earnings growth are measured over more years of follow-up. (See Appendix Table C.6.) All four employment-focused programs, but only two of seven education-focused programs led to an impact on the measure of stable employment during years 3 to 5. However, on the measure of earnings over time, only the Riverside LFA and Portland programs, along with the Riverside HCD program, led to increases over the control group. The Portland program led to the largest impacts on both outcomes, which helps explain why it continued to produce earnings impacts in the last years of follow-up.
High School Graduates and Nongraduates
The effects of employment- and education-focused programs may vary according to education credential. Many proponents of skill-building activities have argued that education and training are particularly important for people who have not graduated from high school because nongraduates typically earn much less than graduates. Proponents of education-focused programs have asserted that job search programs will likely be ineffective for people with low skills and without education credentials, because these problems prevent many people from finding a job, even when assisted by a welfare-to-work program. Moreover, many nongraduates who do find employment will likely work at unstable jobs with low salaries and few if any benefits. On the other hand, people who had not graduated from high school may not attend long enough to benefit from the additional education provided in the education-focused programs; or, alternatively, local employers may have little demand for the skills they acquired.
There is also debate over whether employment- or education-focused programs provide greater benefit to enrollees who had already attained a high school diploma or GED certificate before entering a welfare-to-work program. According to one argument, many high school graduates have previous work experience and only require assistance in finding a job, or perhaps a referral to a better job than they could have found on their own. In these instances, employment-focused programs (particularly the job-search-first programs) should lead to larger effects than education-focused programs, which require people to forgo work. However, high school graduates in education-focused programs often take college courses or enroll in vocational training. If these types of education and training provide valuable skills, then education-focused programs may lead to more substantial effects for high school graduates than employment-focused programs.
The relative success of Portland's employment-focused program that used a mix of job search and education depends largely on its ability to determine who would benefit from education and training and who would benefit from job search. If identification of the two groups was quite successful, then the program might have had the largest effects over a long period for both high school graduates and nongraduates. It would have been able to increase earnings quickly for those capable of finding good jobs quickly and increase earnings later for those who would not have found good jobs quickly but who could have done so after short-term skills training.
Figure 4.3 shows how much the programs benefited high school graduates and nongraduates by showing the effects of the 11 programs on earnings over five years for these subgroups. (The next section of this chapter provides a more detailed comparison of the LFA and HCD programs for graduates and nongraduates.)
Figure 4.3
Impacts on Total Earnings in Years 1 to 5 for Sample Members With and Without a High School Diploma or GED at Random Assignment
SOURCE: MDRC calculations from state and county administrative records.
NOTES: See Appendix A.1.
Only 4 of the 10 programs significantly increased earnings for high school graduates.(10) While 3 of these 4 programs were employment-focused suggesting that job search is more effective for this group there were no large differences in either Atlanta or Grand Rapids in the effect of the LFA and HCD programs for graduates. Moreover, the largest impact on total earnings for high school graduates was in Portland, which allowed some high school graduates to obtain education and training and which resulted in a substantial increase in the use of vocational training and post-secondary education by high school graduates.
Only 5 of the 11 programs significantly increased earnings for nongraduates. Again, results were more positive for employment-focused programs than for education-focused programs. Three of the 4 employment-focused programs significantly raised earnings for nongraduates compared with only 2 of the 7 education-focused programs. Moreover, in each site the LFA program had a larger effect on earnings than the HCD program. However, the Portland employment-focused program, which used a mix of initial activities, had the largest effect for nongraduates.
There is also no clear indication that the welfare-to-work programs as a whole were more effective for one group or the other.(11) Five of the programs had larger effects for high school graduates than for nongraduates, but five had larger effects for nongraduates. These differences appear to be related more to site than to program approach. For example, both Atlanta programs had larger effects for high school graduates than for nongraduates, but both Grand Rapids programs had larger effects for high school nongraduates than for graduates.
Comparing Results for the LFA and HCD Programs
In comparing effects of the 11 programs, the section above provided informal comparisons of job search and education as alternative approaches to encouraging employment and increasing earnings. Fortunately, NEWWS was set up to allow a more formal comparison of the two approaches. In Atlanta, Grand Rapids, and Riverside (for those in need of basic education), people were assigned at random to either the LFA program, which required most participants to look for work intially, or the HCD program, which required most to enroll in education or training initially. Because people were assigned at random to the two programs, the difference in outcomes between individuals in the two programs provides a reliable indication of which approach was more effective. Table 4.3 makes this comparison for number of quarters employed and earnings over the five-year follow-up period.
| Site and Program | Sample Size | LFA | HCD | Difference (Impact) | p-Value |
|---|---|---|---|---|---|
| Average number of quarters employed | |||||
| Full impact sample | |||||
| Atlanta | 2,936 | 8.5 | 8.3 | 0.2 | 0.34 |
| Grand Rapids | 3,099 | 9.8 | 9.5 | 0.4 | 0.07 |
| Riverside | 4,980 | 6.8 | n/a | n/a | n/a |
| No high school diploma or GED | |||||
| Atlanta | 1,190 | 7.0 | 6.4 | 0.7** | 0.04 |
| Grand Rapids | 1,268 | 8.7 | 7.8 | 0.9*** | 0.00 |
| Riverside | 3,182 | 6.0 | 5.5 | 0.5** | 0.02 |
| High school diploma or GED | |||||
| Atlanta | 1,742 | 9.5 | 9.6 | -0.1 | 0.73 |
| Grand Rapids | 1,827 | 10.6 | 10.6 | 0.0 | 0.96 |
| Riverside | 1,798 | 7.8 | n/a | n/a | n/a |
| Total earnings in years 1 to 5 ($) | |||||
| Full impact sample | |||||
| Atlanta | 2,936 | 19,838 | 19,397 | 442 | 0.59 |
| Grand Rapids | 3,099 | 22,323 | 21,616 | 706 | 0.39 |
| Riverside | 4,980 | 17,438 | n/a | n/a | n/a |
| No high school diploma or GED | |||||
| Atlanta | 1,190 | 13,439 | 12,344 | 1,095 | 0.26 |
| Grand Rapids | 1,268 | 16,243 | 14,299 | 1,945** | 0.04 |
| Riverside | 3,182 | 13,193 | 12,273 | 920 | 0.18 |
| High school diploma or GED | |||||
| Atlanta | 1,742 | 24,163 | 24,111 | 52 | 0.97 |
| Grand Rapids | 1,827 | 26,496 | 26,729 | -233 | 0.85 |
| Riverside | 1,798 | 23,019 | n/a | n/a | n/a |
| SOURCE: MDRC calculations from state and county administrative records. NOTES: See Appendix A.1. |
|||||
For the full sample, Atlanta and Grand Rapids LFAs worked and earned about as much as HCDs.(12) For example, in Atlanta LFAs worked 8.5 quarters on average during the 20-quarter follow-up period and HCDs worked 8.3 quarters. Likewise, people in both programs earned between $19,000 and $20,000 on average.
Any significant differences between the two programs were in favor of the LFA approach. In particular, in Grand Rapids the LFA group worked 9.8 quarters on average and the HCD group worked 9.5 quarters. Other differences between outcomes for the full samples in the two sites were not statistically significant; however, employment and earnings were always higher in the LFA programs.
For people who had not graduated from high school, the results are clearer: Nongraduates in the LFA programs were more likely to work than their counterparts in the HCD programs. In all three sites the LFA group also had higher average earnings than the HCD group. Although the difference in earnings between the LFA and HCD groups was statistically significant for high school nongraduates only in Grand Rapids, the average difference across the three sites was statistically significant at the 1 percent significance level.
For high school graduates, the two program approaches had virtually the same effect. For example, sample members worked about 9.5 quarters on average in both Atlanta programs and about 10.6 quarters on average in both Grand Rapids programs. There were similarly small differences in earnings in both sites.
Although the job-search-first approach was sometimes more effective than the education-first approach at encouraging work and increasing earnings, it is important to keep in mind that the most effective program by far among the NEWWS programs was Portland, a program that used both job search and education and training with a focus on employment. Portland's success implies that a "one size fits all" approach is probably not the best approach and that using both job search and education may be better if sites can find strategies that can determine which individuals within a group might benefit most from school.
Effects on Employment and Earnings Impacts by the End of the Control Group Embargo
This section considers whether the NEWWS Evaluation underestimated the longer-term effects on employment and earnings in sites where the embargo on control group services ended after year 3. It concludes that the effect of removing the embargo was likely to be small except in Portland, but that most programs' effects on employment and earnings would likely have diminished in years 4 and 5 even if the control group embargo had remained in place. Because the embargo did appear to have an effect in Portland, results in this report were limited in Portland to 499 control group members who were prohibited from receiving program services for five years.
The likely effect of ending the control group embargo was examined in four sites where some control group members gained access to program services in year 4 and some did not gain access until later. In those four sites Atlanta, Columbus, Grand Rapids, and Portland control group members were divided into a "short-term embargo" group whose members became eligible for program services in year 4 and a "longer-term embargo" group, whose members remained ineligible for program services for at least four years.
These comparisons were not made in Riverside, Oklahoma City, and Detroit. In Riverside, the test was unnecessary because all control group members remained ineligible for services for five years after random assignment. In Oklahoma City, the fact that the program did not significantly affect employment or earnings during years 1 to 3 when all control group members remained ineligible for services suggests that ending the embargo did not alter the program's effects. In Detroit, the test could not be performed because all control group members became eligible for services after three years of follow-up. It is possible that the ending the embargo caused impacts in the program to decline after year 3, but there is no way to know whether that was true.
In comparing the short-term and longer-term embargo groups, the rate of decline of impacts for the two groups in year 4 and the size of impacts for the longer-term embargo group in year 4 were examined. If impacts declined faster for the short-term embargo group than for the longer-term embargo group in year 4, it is evidence that ending the embargo did reduce impacts. In contrast, if a program did not significantly affect outcomes for the longer-term embargo group in year 4 even though the group was still embargoed from receiving services it is evidence that ending the embargo did not have a large effect because the effectiveness of the program ended before the embargo.
Portland provided the most reliable comparison. Several years into the Portland evaluation, 499 control group members were chosen at random from the control group and remained ineligible for services for five years after random assignment. This group of 499 constituted the longer-term embargo group while the short-term embargo group consisted of all other control group members. Because the groups were chosen at random, they did not differ systematically from each other in background characteristics, and both groups resembled the program group.
A comparison of the short-term and longer-term embargo groups in Portland suggested that ending the control group embargo did contribute to a decline in impacts there. The effect of the program on employment was about the same in year 3 whether it was calculated with the short-term or longer-term embargo group (12 versus 11 percentage points). In year 4, however, the effect was only 2.6 percentage points when measured with the short-term embargo group but 8.5 percentage points when measured with the longer-term embargo group. In year 5, the effect had largely disappeared when measured with the short-term embargo group but was 3.8 percentage points when measured with the longer-term embargo group. Differences in the trends in earnings impacts were similar to differences in the trends in employment impacts during years 3 to 5. Because of these differences, MDRC decided to include only members of the longer-term embargo control group in estimating impacts of the Portland program.
Comparisons were conducted less rigorously in the other three sites. Control group members in the short-term and longer-term embargo groups were randomly assigned at different times and therefore might have differed systematically from each other at the time of random assignment. Consequently, one cannot rule out the possibility that variation in impacts between the two groups resulted from differences in background characteristics, labor markets, or other external factors that may have affected employment.
Despite these measurement problems, it appears that ending the control group embargo was not responsible for the small effect on employment and earnings in Grand Rapids at the end of the follow-up period. For both programs, the effect on employment and/or earnings declined during year 3, when all control group members were ineligible for services. This downward trend in impacts continued in years 4 and 5 and was seen in both the short-term and longer-term cohorts for both the LFA and the HCD programs.
In Atlanta, the trend in impacts after year 3 suggests that the end of the control group embargo had only a limited effect. For the longer-term embargo group, the effect of the LFA program on employment and earnings was no longer statistically significant by the second half of year 4 even though control group members were not yet eligible for program services. In other words, it appears that the effectiveness of the program had largely disappeared before control group members were allowed to receive program services. A similar pattern was seen for the HCD program in Atlanta.(13)
The evidence is similar for the Columbus Integrated program. During year 3, the estimated effect of the program on employment was 4.3 percentage points and statistically significant for the longer-term embargo group. In year 4, however, the effect on employment had decreased to 2.4 percentage points and was not statistically significant. The program's effect on earnings also began to decline during year 3 and continued downward slightly during year 4.
Ending the embargo may have made more of a difference for the Columbus Traditional program. Impacts on earnings for the first three years after random assignment were similar whether calculated for the short-term or the longer-term embargo groups. In years 4 and 5, however, the estimated impact on earnings was about $950 higher when calculated with the longer-term embargo group. On the other hand, Columbus Traditional had only a negligible effect on employment for either sample, either before or after year 3.
Whether or not allowing the control group to receive program services diminished the effect of the programs at the end of the follow-up period, the essential story is clear from the first three years of follow-up: The Portland program had the largest and most consistent effects by far, employment-focused job-search-first programs had the next largest effects, and education-focused programs had the smallest effects. Moreover, issues surrounding the control group embargo do not affect the comparison of LFA and HCD programs in Atlanta, Grand Rapids, and Riverside. That comparison indicates that the two program approaches were equally effective for graduates, that the LFA programs were more effective for nongraduates, and that there was no evidence that the cumulative effect of the HCD programs would surpass the cumulative effect of the LFA programs with longer follow-up.
1. As discussed elsewhere, results for Portland use only the 499 control group members who were prohibited from receiving program services for five years.
2. Program impacts on average number of quarters employed can be greatly affected by the total number of quarters of follow-up during which employment is measured. For this reason, some researchers prefer to convert this measure to a standard format by dividing average number of quarters employed by the total number of follow-up quarters. The result (usually expressed as a percentage) is called the "average quarterly employment rate." For example, in Portland, program group members worked for pay for an average of 9.4 of the 20 quarters of follow-up, whereas control group members were employed for 7.8 quarters on average. These averages are equivalent to quarterly employment rates of 47 percent (9.4 / 20 x 100) and 39 percent (7.8 / 20 x 100), respectively. Similarly, in Portland the impact on average number of quarters employed (of 1.6 quarters) is equivalent to an impact of 8 percentage points in the average quarterly employment rate (1.6 / 20 x 100; or 47 percent - 39 percent).
3. Riccio, Friedlander, and Freedman, 1994; Friedlander and Burtless, 1995; Bloom and Michalopoulos, 2001. The Riverside GAIN program increased employment by 9.9 percentage points and earnings by about $1,000 per year (Freedman et al., 1996; Hotz, Imbens, and Klerman, 2000). Two other programs came close to Portland over a three-year period: the Canadian Self-Sufficiency Project (Michalopoulos and Berlin, 2001), which increased employment by 7.1 percentage points and earnings by about $725 per year, and the Connecticut Jobs First program, which increased employment by 7.8 percentage points but increased earnings by only about $425 per year (Bloom et al., 2000b).
4. Bloom and Michalopoulos, 2001; Michalopoulos and Schwartz, 2001.
5. This finding is based on field research. On the other hand, results in Chapter 7 demonstrate that Portland's approach led to large earnings impacts for sample members who faced at least one serious barrier to employment.
6. Unlike Oklahoma City, Detroit operated a traditional case management system, in which separate Income Maintenance staff determined recipients' eligibility for public assistance and calculated their monthly welfare and Food Stamp grants. However, the Detroit welfare-to-work program was not fully staffed during the first years of the evaluation, and its case managers reported spending most of their limited time working with clients who were participating on their own initiative and who had requested child care and other supportive services.
7. Results over time are also shown in Appendix Tables C.2 (impacts on total earnings in each year of follow-up), C.3 (impacts on cumulative measures of employment and earnings in years 1 to 3), and C.4 (impacts on employment and earnings in year 5). Appendix Table C.5 shows employment and full-time employment at the end of year 5 according to responses to the Five-Year Client Survey. Results in those tables confirm that few programs continued to have significant effects in year 5.
8. The two Atlanta programs are discussed first because they appear side by side in Figure 4.1 and because they used different self-sufficiency approaches. Section IV of this chapter presents a statistical comparison of employment and earnings in the LFA and HCD programs in Atlanta, Grand Rapids, and Riverside.
9. In addition, Freedman, 2000, describes results on employment stability and earnings growth through four years in much greater detail, and Michalopoulos, 2001, compares the effects of the NEWWS programs with effects from programs in Minnesota and Canada that supplemented earnings to encourage work.
10. The few high school graduates who were assigned to the Riverside HCD program were considered in need of basic education and included in the group of nongraduates shown.
11. Michalopoulos and Schwartz, 2001, report that impacts over a three-year follow-up period were larger for high school graduates than for nongraduates. Among the programs they studied, three of the four with the largest differences in earnings impacts between high school graduates and nongraduates were in the Alameda, Riverside, and San Diego GAIN programs, all of which used a mix of initial activities and none of which were in the NEWWS Evaluation. Even among the 20 programs that they studied, however, impacts were larger for high school graduates in 11 programs but larger for nongraduates in 9 programs.
12. Recall that people not in need of basic education were not randomly assigned to the Riverside LFA program, so that a comparison of the two approaches could not be made for the full sample in Riverside.
13. For unknown reasons, the effects of the HCD program were larger in years 4 and 5 when calculated with the short-term embargo group than the longer-term embargo group.
Impacts on Public Assistance
Welfare-to-work programs have two goals: increasing employment and earnings and reducing welfare receipt and benefit amounts. Chapter 4 discussed which programs were most successful with the work part of the transition. The Portland program had the largest and most persistent effects. Employment-focused programs increased employment quickly, and both employment- and education-focused programs increased employment in the intermediate term. For people who had not graduated from high school, however, job search appeared more effective than education at increasing earnings and employment.
This chapter explores the effects of the programs on welfare and Food Stamp receipt and benefit amounts over five years, estimated from automated state and county payment records. In addition to showing the effects of the programs on these outcomes for all programs for the full report sample, the chapter compares the effects of the programs for high school graduates and nongraduates and compares the effects of the LFA and HCD programs in Atlanta, Grand Rapids, and Riverside.
Key Findings
- Over five years, program group members in all programs spent less time on welfare and received smaller welfare payments on average than control group members. Welfare savings were larger and more persistent than earnings gains: Few programs continued to affect employment and earnings in year 5, but most programs continued to generate welfare savings at the end of year 5.
- Welfare savings were generally larger for programs that had larger effects on earnings, but they varied for other reasons as well. Total payments were reduced more in higher-grant sites such as Riverside, Portland, and Grand Rapids and less in low-grant sites such as Atlanta. Benefits were reduced more in sites such as Columbus and Grand Rapids that strictly enforced mandates than in sites such as Detroit that did not.
- Direct comparisons between the LFA and HCD programs showed that LFA programs generally resulted in lower welfare use and expenditures. However, reductions were found for only some programs: Atlanta and Grand Rapids LFA for nongraduates and Grand Rapids LFA for graduates.
- The programs had similar effects for high school graduates and nongraduates. Most programs produced welfare savings for both groups, and there was little evidence that the effects were larger for one group than the other: In five programs welfare savings were larger for high school graduates and in five programs welfare savings were larger for nongraduates.
- Over five years, program group members in all programs spent less time on Food Stamps and received smaller Food Stamp payments on average than control group members. Effects were generally smaller for Food Stamps than for welfare payments, however, because some program group members appropriately continued to receive Food Stamps after they left welfare.
Expected Effects
How much a welfare-to-work program reduces welfare use will be influenced by a number of factors, including how much the program increased employment and earnings, how strictly it enforced participation mandates by using sanctions, and the generosity of the state's welfare benefits.
In all of the NEWWS sites, as family earnings increased, welfare payments decreased. Therefore, it is reasonable to expect programs that led to the largest increases in earnings over five years particularly Portland to also have produced the largest decreases in welfare receipt and expenditures.
Sanctions (described in Chapter 1) also have a direct effect on welfare use. As a result, tough, enforcement-oriented programs like Grand Rapids HCD could have reduced welfare payments much more than they increased employment or earnings. Because the programs studied in NEWWS did not use "full-family sanctions," which remove all of a family's welfare grant,(1) the direct effect of sanctions would be to reduce payment amounts rather than time on welfare. Nevertheless, a family whose benefits were reduced through sanctions might have decided to leave welfare altogether, so that sanctions might have indirectly resulted in fewer people receiving welfare. Requiring people to participate in welfare-to-work services might also encourage them to leave welfare even if it does not increase their employment and earnings. For example, some people might already be working but not reporting their earnings to the welfare system. Going to school or attending a job club might interfere with their ability to perform their job or might be too great a burden on top of working, and they might choose to keep their job and stop receiving benefits.
Differences in maximum welfare grant levels by site and in welfare earnings disregards (described in Chapter 1) could also influence the effects that programs have on welfare use. Other things being equal, savings in welfare expenditures will be larger in high-grant states simply because there are more dollars to save. On the other hand, reductions in months on welfare will be larger in low-grant states because people lose their eligibility for cash assistance at a lower level of earnings. Likewise, reductions in both welfare payment amounts and welfare receipt will be smaller in states with relatively more generous earnings disregards (that is, those that allow working welfare recipients to keep more of their welfare payments), like California and Georgia.
Site-by-site differences in background characteristics of sample members may also be related to program effects on welfare use. Welfare savings may be greater in sites where most sample members had long stays on welfare before random assignment or faced other severe barriers to employment, because control group members in these sites were likely to remain on assistance for a long time. If long-term welfare recipients have severe barriers that keep them from working and that are not ameliorated by the programs, however, then sites with a more disadvantaged caseload might have smaller effects on welfare use. (Chapter 7 examines this question by investigating whether the programs increased the earnings of long-term welfare recipients and reduced their welfare benefit levels.)
In short, we should expect relatively high reductions in welfare receipt and payment amounts in Portland, which increased earnings the most and had relatively generous benefits. We might also expect high reductions in Riverside LFA, which had the second highest impact on earnings and had the most generous benefits, although California's generous earnings disregard might have diminished the program's effect on welfare use. We should expect higher reductions in welfare use in Grand Rapids and Columbus than would be indicated by their impacts on employment and earnings because these sites had some of the toughest sanction policies, but smaller reductions in Detroit than would be indicated by their impacts on employment and earnings because Detroit did not strictly enforce participation mandates for much of the follow-up period. Finally, we should expect relatively small reductions in welfare benefit amounts in Atlanta even though its two programs increased earnings because it had such low grant levels and had a more generous earnings disregard policy than any site except Riverside.
The effects of the programs on Food Stamp receipt and benefit amounts are harder to predict. Regarding cash assistance, as earnings increase, Food Stamp benefits decrease. However, additional earnings generally reduce Food Stamps less than welfare: Each additional dollar of earnings reduces Food Stamp amounts by less than a dollar. In addition, Food Stamp grant calculations count a dollar of earnings less than a dollar of welfare, so a person who replaces welfare dollars with earnings may experience a net increase in Food Stamps.(2) It is also possible that recipients gave up Food Stamps after they left welfare for employment or other reasons, even if they still qualified for them, either because they wanted to leave public assistance entirely or because they did not know they were still eligible for noncash assistance.
Welfare Receipt and Payments
Welfare Receipt and Payments Over Five Years
Welfare Receipt by Year
Table 5.1 shows the average number of months that program and control group members received cash assistance and the payment amounts they received on average. Both of these outcomes were measured over the five-year follow-up period from administrative records collected from state and county welfare systems. The table also shows the impact of the programs (as always, measured as the difference in average outcome levels between the program and control groups) and their levels of statistical significance.(3) Because welfare administrative records were available in Oklahoma City for only three years, results for Oklahoma City are not shown in most tables and figures in this chapter.
| Site and Program | Sample Size | Program Group | Control Group | Difference (Impact) | PercentageChange (%) |
|---|---|---|---|---|---|
| Average number of months of welfare receipt in years 1 to 5 | |||||
| Atlanta Labor Force Attachment | 2,938 | 34.4 | 37.2 | -2.9 *** | -7.7 |
| Atlanta Human Capital Development | 2,992 | 35.3 | 37.2 | -1.9 *** | -5.2 |
| Grand Rapids Labor Force Attachment | 3,012 | 26.9 | 31.1 | -4.2 *** | -13.6 |
| Grand Rapids Human Capital Development | 2,997 | 28.2 | 31.1 | -2.9 *** | -9.2 |
| Riverside Labor Force Attachment | 6,726 | 27.8 | 31.0 | -3.2 *** | -10.5 |
| Lacked high school diploma or basic skills | 3,125 | 30.1 | 33.3 | -3.2 *** | -9.6 |
| Riverside Human Capital Development | 3,135 | 30.0 | 33.3 | -3.3 *** | -9.8 |
| Columbus Integrated | 4,672 | 23.3 | 27.2 | -3.9 *** | -14.4 |
| Columbus Traditional | 4,729 | 24.7 | 27.2 | -2.5 *** | -9.2 |
| Detroit | 4,459 | 36.1 | 37.7 | -1.6 *** | -4.3 |
| Oklahoma City | n/a | n/a | n/a | n/a | n/a |
| Portland | 4,028 | 19.8 | 25.3 | -5.6 *** | -21.9 |
| Average total welfare payments received in years 1 to 5 ($) | |||||
| Atlanta Labor Force Attachment | 2,938 | 9,064 | 9,946 | -881 *** | -8.9 |
| Atlanta Human Capital Development | 2,992 | 9,236 | 9,946 | -710 *** | -7.1 |
| Grand Rapids Labor Force Attachment | 3,012 | 10,414 | 12,966 | -2,552 *** | -19.7 |
| Grand Rapids Human Capital Development | 2,997 | 11,199 | 12,966 | -1,767 *** | -13.6 |
| Riverside Labor Force Attachment | 6,726 | 15,584 | 18,294 | -2,710 *** | -14.8 |
| Lacked high school diploma or basic skills | 3,125 | 17,171 | 20,126 | -2,955 *** | -14.7 |
| Riverside Human Capital Development | 3,135 | 17,176 | 20,126 | -2,949 *** | -14.7 |
| Columbus Integrated | 4,672 | 7,481 | 9,005 | -1,523 *** | -16.9 |
| Columbus Traditional | 4,729 | 7,899 | 9,005 | -1,105 *** | -12.3 |
| Detroit | 4,459 | 15,686 | 16,247 | -561 ** | -3.5 |
| Oklahoma City | n/a | n/a | n/a | n/a | n/a |
| Portland | 4,028 | 8,940 | 11,686 | -2,746 *** | -23.5 |
| SOURCE: MDRC calculations from state and county administrative records. NOTES: See Appendix A.1. |
|||||
Table 5.1 shows that all 10 programs in which welfare payments were available had a significant effect on the number of months that people received welfare. The effect ranged from 1.6 months (over a five-year, or 60-month, period) in Detroit to 5.6 months in Portland. All 10 programs also reduced welfare payments over the five-year period (relative to control group levels), with welfare savings ranging from $561 in Detroit to $2,949 in Riverside HCD (although the percentage change was largest in Portland: 23 percent).
Results by site are generally consistent with the expectations outlined in the prior section. First, the program with the largest effects on employment and earnings Portland also had the largest reduction in welfare use. These reductions are especially impressive considering the Portland control group's relatively low welfare use. On average, control group members in Portland received welfare for about 25 months over the 60-month period. In comparison, program group members in Portland received welfare for a little less than 20 months, for a reduction of 5.6 months, or nearly half a year during the five-year follow-up period. Portland also had the second-largest reduction in dollars spent on welfare an average of $2,746 per program group member.
Second, the site with the most generous grant amounts Riverside also generated relatively large welfare savings.(4) Over the five-year period, HCD program group members received nearly $3,000 less than control group members on average while LFA program group members received about $2,700 less than control group members.(5) The fact that the two programs reduced benefits by similar amounts is somewhat surprising considering that the effect of the LFA program on earnings was twice as large as the effect of the HCD program. This finding suggests that the HCD program discouraged a number of people from receiving benefits without helping them get a job. If that occurred, the HCD program will have left individuals with less total income on average than they would have had without the program, an issue addressed in Chapter 6.
Third, reductions in welfare use were fairly large in programs that had tough sanction policies (Columbus Integrated and both programs in Grand Rapids) and fairly small in the site with the least strict sanction policy (Detroit). This comparison is especially revealing because Detroit and Grand Rapids, which are both in Michigan, had similar grant levels and earnings disregards and because the effect on earnings over five years was about as large in Detroit as in Grand Rapids LFA and nearly twice as large as in Grand Rapids HCD. Detroit had the smallest reductions in payment amounts ($561 over five years) and months on welfare (1.6 months on average). In comparison, Grand Rapids LFA reduced time on welfare by 4.2 months and average welfare payments by more than $2,500, and Grand Rapids HCD reduced time by 2.9 months and payments by $1,800.
Finally, reductions in welfare use were relatively small for both programs in Atlanta. Time on welfare was reduced by 2.9 months in the LFA program and 1.9 months in the HCD program, while cash payments over five years were reduced by about $900 in the LFA program and $700 in the HCD program. Decreases in welfare were smaller than in Grand Rapids or Riverside, even though the effects on earnings in Atlanta rivaled those in the other two sites. The relatively low level of welfare savings in Atlanta may reflect the low level of benefits available; people who lost welfare when they went to work could not lose very much.(6) In addition, the site's use of fill-the-gap budgeting allowed working welfare recipients to keep relatively more of their grant than they could in the other sites.
As discussed above, there are two direct causes of welfare savings: increased earnings and use of sanctions. Chapter 4 showed that program services had immediate effects in the employment-focused programs, but delayed effects in the education-focused programs. If earnings were primarily responsible for reductions in welfare use, then reductions in welfare use should also have been larger initially in employment-focused programs than in education-focused programs (particularly in the same sites). Although sanctions did not stop people from receiving welfare in any of the sites, sanctions may have given people a reason to voluntarily stop receiving benefits. If reductions in welfare receipt were due to sanctions (indirectly) or to people voluntarily leaving welfare rather than complying with programs requirements, welfare savings should have been fairly close initially in the two types of programs.
Figure 5.1 and Appendix Table D.2 show the proportion of program and control group members who received welfare in the last quarter of each of the five years of follow-up. By showing results by year, the figure and table also can be used to examine results for the first three years, before any control group members were allowed to receive program services. They also indicate how quickly people left welfare and how many remained on welfare at the end of year 5 a number that may be important in a world of time-limited welfare.
Figure 5.1
Impacts on Welfare Reciept in the Last Quarter of Years 1 to 5
SOURCE: MDRC calculations from state and county administrative records.
NOTES: See Appendix A.1.
According to Figure 5.1 and Appendix Table D.2, many people left welfare in each site, even in the absence of welfare-to-work services. In Atlanta, as discussed in earlier chapters, virtually everyone was receiving welfare at the time of random assignment (that is, the quarter prior to year 1). By the end of year 1, however, about 83 percent of control group members were still on welfare. The decline in welfare receipt continued throughout the follow-up period. By the last quarter of year 5, less than 40 percent of the control group in Atlanta still received welfare.
Trends in program impacts (that is, the difference in welfare receipt between the program and control groups) suggest that increased employment and earnings were primarily responsible for reductions in welfare receipt at least during the early years of follow-up. The two Atlanta programs illustrate the basic points.(7) At the end of year 1, Atlanta LFA significantly reduced welfare receipt, but Atlanta HCD did not. This is consistent with the finding in Chapter 4 that the LFA program had a larger initial effect on employment and earnings than the HCD program. Moreover, the magnitude of the impacts on welfare receipt at the end of the first year were similar to the impacts on employment shown in Figure 4.1.
The impact of Atlanta HCD on welfare receipt increased after year 1 and was statistically significant in years 2 and 3. This trend paralleled Atlanta HCD impacts on employment and earnings. (See Chapter 4 that) Similarly, Atlanta LFA increased employment and earnings above control group levels in years 2 and 3 and continued to decrease welfare receipt during these years. These patterns suggest that reductions in welfare receipt were driven largely by increases in employment and earnings.
Impacts of the two Atlanta programs on welfare receipt declined toward the end of the follow-up period. However, reductions continued even after these programs ceased to raise employment levels above the control group. In particular, Atlanta LFA continued to significantly reduce welfare receipt at the end of year 5, even though it did not significantly increase employment during year 5. Likewise, Atlanta HCD significantly reduced welfare receipt at the end of year 4, even though it did not significantly increase employment during year 4.
The trends for the other employment-focused and education-focused programs were similar to the patterns for the Atlanta programs. First, many people left welfare in each site, even in the absence of welfare-to-work services. Control group members left the welfare rolls particularly fast in Columbus and Portland. In both sites, fewer than 20 percent of control group members were still receiving welfare at the end of year 5. In comparison, about 35 percent of control group members in Atlanta, Detroit, and Riverside remained on welfare.
The trends in impacts were also similar to the patterns for the two Atlanta programs. During the first year, the employment-focused programs led to larger decreases in welfare receipt than the education-focused programs. All four employment-focused programs significantly reduced welfare receipt, and three of these programs (Grand Rapids and Riverside LFA and Portland) produced the largest decreases (of 7 to 9 percentage points) among the 11 programs. In contrast, five of the seven education-focused programs reduced welfare receipt in year 1, and decreases ranged from 1 to 4 percentage points for most programs.(8)
Like Atlanta HCD, the other education-focused programs led to larger reductions in welfare receipt after year 1. This trend was most noticeable in Detroit, where impacts were not statistically significant in year 1 but became statistically significant in years 2 and 3 (just as Detroit's effects on employment and earnings became statistically significant in years 2 and 3).
Like Atlanta LFA, the other employment-focused programs continued to reduce welfare receipt in years 2 and 3, a trend that (for the most part) mirrored their effects on employment and earnings. For example, Riverside LFA impacts on welfare receipt declined only slightly between years 1 and 2, just as its effects on earnings declined slightly, while Portland impacts on welfare receipt increased between years 1 and 2, just as its effects on employment increased somewhat between the two years.(9)
The impacts on welfare receipt decreased over time in all of the programs, but most programs continued to significantly reduce welfare receipt at the end of year 5. This result is somewhat surprising, given that few programs increased employment and earnings above control group levels in year 5. This pattern is especially striking for Grand Rapids LFA, which decreased receipt below the control group by 3 percentage points at the end of year 5, but led to a similar reduction in percentage employed during that year. This finding implies that some program group members who exited welfare for employment earlier in the follow-up did not return to assistance after leaving employment, even though they were eligible to, a pattern that may reflect the national climate in the aftermath of the federal welfare reform legislation of 1996.
Comparing High School Graduates and Nongraduates
As discussed in Chapter 4, employment- and education-focused programs were expected to have different effects on employment and earnings depending on whether people were job-ready or in need of more education. There are also reasons to expect welfare-to-work programs to produce different amounts of welfare savings for the two groups. High school nongraduates received higher welfare benefits on average than graduates, probably because they have a harder time finding work.(10) Welfare savings may therefore be larger for high school nongraduates than for graduates. When they do go to work, however, graduates are likely to earn more than nongraduates, and are probably more likely to find jobs that pay enough to help them leave welfare. A program that is just as likely to help high school graduates as nongraduates find work may therefore produce larger welfare savings for graduates.
Figure 5.2 explores these possibilities by comparing program effects on welfare benefits over five years separately for high school graduates and nongraduates. According to the figure, there is little evidence that the programs reduced benefits more for one group than the other (just as there was little evidence in Chapter 4 that the programs were systematically affecting earnings more for one group than the other). Over five years, virtually all of the programs reduced welfare payments for both subgroups. Moreover, in five of the nine programs where effects could be measured over five years, they were larger for high school nongraduates than graduates (both Grand Rapids programs, both Columbus programs, and Riverside LFA), but in the other four programs effects were larger for graduates than for nongraduates (both Atlanta programs, Detroit, and Portland).
Figure 5.2
Impacts on Total Welfare Payments in Years 1 to 5 for Sample Members With and Without a Hign school Diploma or GED at Random Assignment
SOURCE: MDRC calculations from state and county administrative records.
NOTES: See Appendix A.1.
Food Stamp Payments and Receipt
Table 5.2 examines the effects of the programs on the number of months that sample members received Food Stamps and Food Stamp expenditures for the five-year follow-up period. (Appendix Table D.3 shows results on the same outcomes for the first three years of follow-up, which is the period before any control group members could have received program services. Appendix Table D.4 shows results for the last quarter of year 5.)
|
Site and Program |
Sample Size | Program Group | Control Group | Difference (Impact) | Percentage Change (%) |
|---|---|---|---|---|---|
| Average number of months of Food Stamp receipt in years 1 to 5 | |||||
| Atlanta Labor Force Attachment | 2,938 | 42.1 | 43.4 | -1.4 ** | -3.2 |
| Atlanta Human Capital Development | 2,992 | 42.2 | 43.4 | -1.2 ** | -2.8 |
| Grand Rapids Labor Force Attachment | 3,012 | 30.5 | 33.8 | -3.3 *** | -9.9 |
| Grand Rapids Human Capital Development | 2,997 | 31.7 | 33.8 | -2.1 *** | -6.2 |
| Riverside Labor Force Attachment | 6,726 | 25.3 | 29.0 | -3.6 *** | -12.6 |
| Lacked high school diploma or basic skills | 3,125 | 27.7 | 31.2 | -3.5 *** | -11.2 |
| Riverside Human Capital Development | 3,135 | 27.4 | 31.2 | -3.8 *** | -12.3 |
| Columbus Integrated | 4,672 | 27.9 | 31.2 | -3.4 *** | -10.7 |
| Columbus Traditional | 4,729 | 29.1 | 31.2 | -2.1 *** | -6.6 |
| Detroit | 4,459 | 38.8 | 40.3 | -1.5 *** | -3.8 |
| Oklahoma City | n/a | n/a | n/a | n/a | n/a |
| Portland | 4,028 | 29.4 | 32.5 | -3.1 *** | -9.4 |
| Average total Food Stamps received in years 1 to 5 ($) | |||||
| Atlanta Labor Force Attachment | 2,938 | 10,661 | 11,089 | -428 ** | -3.9 |
| Atlanta Human Capital Development | 2,992 | 10,930 | 11,089 | -159 | -1.4 |
| Grand Rapids Labor Force Attachment | 3,012 | 6,351 | 6,966 | -615 *** | -8.8 |
| Grand Rapids Human Capital Development | 2,997 | 6,580 | 6,966 | -387 *** | -5.6 |
| Riverside Labor Force Attachment | 6,726 | 4,981 | 5,870 | -888 *** | -15.1 |
| Lacked high school diploma or basic skills | 3,125 | 5,577 | 6,504 | -928 *** | -14.3 |
| Riverside Human Capital Development | 3,135 | 5,492 | 6,504 | -1,013 *** | -15.6 |
| Columbus Integrated | 4,672 | 7,160 | 8,185 | -1,025 *** | -12.5 |
| Columbus Traditional | 4,729 | 7,537 | 8,185 | -648 *** | -7.9 |
| Detroit | 4,459 | 9,186 | 9,519 | -334 ** | -3.5 |
| Oklahoma City | n/a | n/a | n/a | n/a | n/a |
| Portland | 4,028 | 6,926 | 7,753 | -827 *** | -10.7 |
| SOURCE: MDRC calculations from state and county administrative records. NOTES: See Appendix A.1. |
|||||
In all sites except Riverside, control group members received Food Stamps for more months than they received welfare payments. Most likely, some control group members in these sites received too much income from earnings or other sources to receive cash assistance but remained eligible for Food Stamps. The difference between Food Stamp and welfare receipt was largest in Portland, where control group members received Food Stamps for 32 months on average during years 1 to 5, while they received welfare for only about 25 months.(11) Similarly, in Atlanta, which is the only site where the federal maximum Food Stamp allotment exceeded the maximum welfare grant, control group members received Food Stamps for an average of 43 months over five years (the most among the six sites), about six months longer than they received welfare.
The importance of Food Stamps as an income supplement for control group members varied from place to place. These differences largely reflect the generosity of welfare benefits and Food Stamp rules, which reduce Food Stamps by a certain amount for each dollar of welfare benefits. At one extreme, Atlanta's maximum welfare grant is so low that people received more on average from Food Stamps about $11,000 from Food Stamps compared with nearly $10,000 from welfare over the five-year period. In Riverside, in contrast, control group members received less than $6,000 in Food Stamp benefits on average, compared with welfare benefits of $18,000. Over five years, all programs significantly reduced total months of Food Stamp receipt and all but one program Atlanta HCD significantly reduced Food Stamp expenditures. Although impacts were smaller for Food Stamps than for welfare, programs that reduced welfare the most also tended to be the programs that reduced Food Stamp use the most. For example, five programs reduced welfare receipt by more than three months: Portland, Grand Rapids LFA, both Riverside programs, and Columbus Integrated. (See Table 5.1.) The same five programs reduced Food Stamp receipt by more than three months. Likewise, Atlanta HCD and Detroit reduced both welfare and Food Stamp receipt by less than two months. Because of the complex interaction between welfare and Food Stamp payment amounts, the extent to which programs reduced Food Stamp payment amounts below control group levels was less clear-cut. For example, the Columbus Integrated program, which had effects on welfare benefits in the midrange of the programs being studied, had the largest reduction in Food Stamp payments. However, the programs that reduced welfare payments the most Portland and the two Riverside programs also had among the largest effects on Food Stamp payments over five years.
Comparing the Effects of the LFA and HCD Programs
As discussed in earlier chapters, NEWWS was set up to allow a rigorous comparison of LFA and HCD programs in Atlanta, Grand Rapids, and Riverside (for those in need of basic education). According to Chapter 4, differences were concentrated among high school nongraduates, for whom LFA programs generally resulted in higher employment and earnings than HCD programs (see Table 4.3).
Table 5.3, which shows the effects of the two self-sufficiency approaches on welfare receipt and benefit amounts, tells a similar story. In some sites, there was virtually no difference between the two approaches. Where there were differences, however, the LFA programs resulted in lower welfare use than the HCD programs over five years.
| Site and Program | Sample Size | LFA | HCD | Difference (Impact) | p-Value |
|---|---|---|---|---|---|
| Total welfare payments received in years 1 to 5 ($) | |||||
| Full impact sample | |||||
| Atlanta | 2,936 | 9,064 | 9,236 | -172 | 0.36 |
| Grand Rapids | 3,099 | 10,414 | 11,199 | -785 *** | 0.00 |
| Riverside | 4,980 | 15,584 | n/a | n/a | n/a |
| No high school diploma or GED | |||||
| Atlanta | 1,190 | 10,209 | 10,740 | -531 * | 0.07 |
| Grand Rapids | 1,268 | 12,009 | 12,914 | -905 ** | 0.03 |
| Riverside | 3,182 | 17,171 | 17,176 | -6 | 0.99 |
| High school diploma or GED | |||||
| Atlanta | 1,742 | 8,278 | 8,252 | 27 | 0.91 |
| Grand Rapids | 1,827 | 9,297 | 10,024 | -727 ** | 0.03 |
| Riverside | 1,798 | 13,504 | n/a | n/a | n/a |
| Total months of welfare receipt in years 1 to 5 | |||||
| Full impact sample | |||||
| Atlanta | 2,936 | 34.4 | 35.3 | -0.9 | 0.17 |
| Grand Rapids | 3,099 | 26.9 | 28.2 | -1.4 ** | 0.02 |
| Riverside | 4,980 | 27.8 | n/a | n/a | n/a |
| No high school diploma or GED | |||||
| Atlanta | 1,190 | 38.0 | 40.5 | -2.5 ** | 0.02 |
| Grand Rapids | 1,268 | 30.5 | 32.1 | -1.5 | 0.11 |
| Riverside | 3,182 | 30.1 | 30.0 | 0.1 | 0.92 |
| High school diploma or GED | |||||
| Atlanta | 1,742 | 31.9 | 31.9 | -0.0 | 0.97 |
| Grand Rapids | 1,827 | 24.3 | 25.6 | -1.3 * | 0.10 |
| Riverside | 1,798 | 24.7 | n/a | n/a | n/a |
| SOURCE: MDRC calculations from state and county administrative records. NOTES: See Appendix A.1. |
|||||
For the full sample, Grand Rapids LFAs received welfare for an average of just under 27 months compared with just over 28 months for Grand Rapids HCDs. Although this difference was fairly small, it was statistically significant and produced a statistically significant welfare savings of nearly $800 per person. Atlanta LFAs were also less likely to receive welfare than Atlanta HCDs and received slightly less in welfare payments, but these differences were not statistically significant.
At the end of year 5 (not shown in the table) the LFA programs in Atlanta and Grand Rapids continued to have lower welfare use than the HCD programs, although the differences were not statistically significant. Thus, HCD programs are unlikely to produce lower cumulative welfare use than the LFA programs with a longer follow-up period.
In Atlanta and Grand Rapids, results for high school nongraduates were more consistent than results for the full sample. In these two sites, nongraduates in the LFA program received significantly less in welfare payments than nongraduates in the HCD program. However, for nongraduates in Riverside, LFAs and HCDs received welfare benefits for the same number of quarters and averaged the same amount of welfare payments over five years. Still, when averaged across the three sites, LFAs received nearly $500 less in welfare payments over five years than HCDs (($531+$905+$6)/3; the difference was statistically significant). Likewise, a simple average across the three sites indicates that the high school nongraduates in the LFA programs received welfare for 1.3 months less on average than nongraduates in the HCD programs, a difference that was again statistically significant.
For high school graduates, the LFA program produced greater welfare savings in Grand Rapids, but not in Atlanta. In Grand Rapids, LFAs received welfare for an average of 24.3 months and HCDs for 25.6 months. Although the effect was fairly small, the difference was statistically significant, and it led to statistically significant welfare savings of more than $700 per person. In contrast, high school graduates in the two Atlanta programs were about equally likely to receive welfare and received similar benefit amounts on average.
Although the job-search-first approach sometimes resulted in less welfare use than the education-first approach, it is important to keep in mind that (for the full sample) the greatest proportional savings among the NEWWS programs was in Portland, which also generated the largest gains in employment and earnings. The fact that Portland used job search, education, and training with a focus on employment suggests that the combination of approaches is better than either one alone.
1. Pavetti and Bloom, 2001.
2. The Food Stamp benefit level equals the maximum benefit level minus 30 percent of a household's countable income. Countable income includes welfare payments plus 80 percent of earnings, so a sample member who replaces welfare with earnings could lower her countable income and thus increase her Food Stamp payments (Ohls and Beebout, 1993).
3. Appendix Table D.1 shows results on the same outcomes for the first three years of follow-up. Appendix Table D.2 shows results on welfare receipt for the last quarter of each follow-up year.
4. Welfare spending was also the highest for control group members in Riverside. The notion that savings might be greater in high-grant states does not mean that a state should raise its welfare grant to generate savings.
5. As shown in Table 5.1, Riverside LFA and HCD led to similarly large reductions in welfare expenditures for sample members determined to need basic education.
6. An alternative possibility is that Atlanta's low grants meant that control group members left welfare quickly, which was not the case. The average control group member in Atlanta received welfare for 37 months during the five-year follow-up period, longer than in all other sites except Detroit.
7. The Atlanta programs are discussed here because they appear side by side on the figure and used different self-sufficiency approaches. Section V of this chapter explicitly compares the LFA and HCD programs in Atlanta, Grand Rapids, and Riverside and indicates where the programs resulted in statistically significant outcome differences in outcomes.
8. The small program-control group difference (of 2 percentage points) for Atlanta LCD had a p-value of .014, just above the 10 percent level for statistical significance.
9. As discussed above, impacts on welfareon welfare receipt may be larger or smaller than expected, given a program's impact on employment and earnings. For instance, Riverside LFA employment impacts decreased sharply between years 1 and 2, but reductions in welfare receipt changed very little. Similarly, Portland impacts on total earnings increased dramatically between years 1 and 2, but reductions in welfare receipt increased to a lesser extent.
10. Michalopoulus and Schwartz, 2001. Among control group members in the NEWWS sites, the proportion of high school graduates and GED recipients who ever worked for pay during years 1 to 5 exceeded the proportion of nongraduates by 3 to 11 percentage points (results not shown).
11. Furthermore, in the last quarter of year 5, twice as many control group members in Portland (34.0 percent versus 17.1 percent) received Food Stamps as were receiving welfare. In addition, more than one-half of control group members in Atlanta were still receiving Food Stamps in the last quarter of year 5 compared with a little more than one-third who received a welfare check. See Appendix Tables D.2 and D.4.
Impacts on Income and Self-Sufficiency
This chapter considers whether employment- and education-focused programs helped program group members reach a higher level of self-sufficiency and economic security than control group members. First, the chapter analyzes whether programs led to impacts in combined income over five years from earnings (net of payroll taxes), Earned Income Tax Credits (EITC), welfare, and Food Stamps. Next, the chapter considers whether program group members were more likely to leave welfare for employment than control group members. Impacts on these measures were calculated with administrative data. Finally, the chapter looks at program effects on receipt of income from earnings and other sources by sample members and other members of their household at the end of year 5, based on responses to the Five-Year Client Survey.
Promoting self-sufficiency is an important goal for welfare-to-work programs, particularly in the current welfare environment. Programs that increase employment and raise income reduce the likelihood that families will return to welfare and/or experience long-term joblessness and hardshipВ a possibility under time-limited welfare if families exhaust their eligibility for assistance.
The findings in this chapter may also foreshadow results on other outcomes for families and children discussed in Chapters 9 through 12. Recent research has indicated that programs that increase employment and raise income are likely to benefit children, whereas programs that increase employment without increasing income typically do not. Moreover, programs that increased income resulted in less stress for parents and less domestic violence.(1)
Key Findings
- During years 1 to 5, program group members in all sites became more self-sufficient than control group members by spending less time on welfare and receiving more of their total income in the form of earnings.
- Most programs did not raise program group members' income above control group levels, and some programs decreased income relative to the control group. Effects on income varied by site rather than program approach.
- With some exceptions, programs led to more favorable effects (larger increases or smaller decreases) for sample members who had received a high school diploma or GED certificate before random assignment than for those who had not.
- At the end of year 5, virtually all sample members reported having some source of income, either of their own or from other household members. However, programs had little effect on income sources other than sample members' earnings and public assistance.
Analysis Issues
Defining Self-Sufficiency
All welfare-to-work programs seek to increase self-sufficiency, although this concept may take on different meanings. Attaining self-sufficiency can mean ending welfare dependency, the primary goal of many welfare-to-work programs. As discussed in Chapter 5, all programs in the evaluation achieved this goal to some extent, although reductions in welfare receipt were small for some programs. Likewise, a program may promote self-sufficiency by increasing welfare recipients' reliance on earnings rather than welfare benefits. This goal may be accomplished by encouraging welfare recipients to combine work and welfare (at least in the short term) or by helping them to leave welfare for employment. As discussed in Chapter 4, most programs resulted in higher earnings, implying that they achieved this goal as well.
Single mothers must obtain adequate resources to be truly self-sufficient, however. Therefore, a more comprehensive measure of self-sufficiency is income. In the context of an experimental evaluation, a program successful at improving self-sufficiency would reduce welfare use but leave program group members with more income than they would have had in the absence of the program.
It should also be recognized that focusing solely on how programs affected sample members' income might lead to inaccurate conclusions about whether sample members benefited from enrolling in a welfare-to-work program. As will be explored at the end of the chapter, income from a spouse, partner, or other friends and family members may have contributed significantly toward the well-being of sample members and their children.
Direct and Indirect Effects on Self-Sufficiency and Income
Welfare-to-work programs can directly increase self-sufficiency by increasing employment and decreasing welfare receipt. They may also increase people's income by helping them qualify for or find relatively well-paying jobs with benefits. The amount of effort that program staff devote to helping recipients who enter employment apply for the EITC, maintain eligibility for Food Stamps, and obtain medical coverage and child care assistance may also help participants maintain or increase their income (or forgo costly expenditures for medical care and other necessities). Programs may not increase income, however, if they encourage people to take jobs that pay less than welfare, impose financial sanctions on those who have difficulty in finding work, or induce them to leave welfare without employment.
Welfare-to-work programs can also indirectly affect self-sufficiency and income by changing welfare recipients' social and family networks once they begin working. For example, welfare recipients who work for pay may be more likely to find spouses or partners who work, or may find job leads for other members of their household, thereby increasing the household's income and economic security. Increases in employment and income may also improve people's ability to purchase goods (like cars and clothing) and services (like reliable day care and health care) that support job retention and advancement and may help them obtain credit or save for the future. Alternatively, welfare-to-work programs that increase unstable employment or encourage recipients to leave welfare without employment may lead to immediate hardship and decrease people's ability to maintain the social and material supports needed to find employment in the future. Later chapters in this report discuss the effects of the programs on marriage, household composition, and material hardship.
Measurement Issues
This chapter estimates effects on income and self-sufficiency using administrative records and data from the Five-Year Client Survey. As discussed in Chapter 2, each of these data sources has its advantages and limitations. Measures of employment and welfare status and income are calculated from statewide unemployment insurance (UI) earnings records and welfare and Food Stamp payment records for all sample members. Estimates based on these administrative records are useful because they include everyone, cover the entire follow-up period, and likely include the primary sources of income received by most sample members.(2) However, these estimates leave out other potentially important sources of income, including earnings not reported to the UI system, child support, other types of transfer payments, income from other household members (from earnings, public assistance, or other sources), or income from family and friends who live outside the household. Therefore, on the basis of administrative records alone some sample members may be incorrectly classified as not employed or as having no income. The Five-Year Client Survey includes these other sources of earnings and income, but for a much smaller group of sample members and for only one month at the end of year 5. Further, like all survey-based data, reported earnings and income are subject to recall error, nonreporting, and exaggeration.
Finally, it should be kept in mind that this analysis does not consider program effects on work-related expenses (such as out-of-pocket expenses for child care and transportation), which decrease the income of sample members and their households.(3)
Impacts on Income
Combined Income Over Five Years
The upper panel of Table 6.1 shows average income over five years for program and control group members in each program, as well as the impact of each program, measured as usual as the difference between the program and control groups. For this analysis, sample members' income is the sum of their earnings, welfare payments, Food Stamps, and estimated Earned Income Tax Credits (EITCs) less estimated payroll taxes.(4)
|
Site and Program |
Sample Size | Program Group | Control Group | Difference (Impact) | Percentage Change (%) |
|---|---|---|---|---|---|
|
Average combined income in years 1 to 5 ($) |
|||||
| Atlanta Labor Force Attachment | 2,938 | 41,138 | 39,987 | 1,152 | 2.9 |
| Atlanta Human Capital Development | 2,992 | 41,120 | 39,987 | 1,133 | 2.8 |
| Grand Rapids Labor Force Attachment | 3,012 | 40,739 | 42,172 | -1,433* | -3.4 |
| Grand Rapids Human Capital Development | 2,997 | 40,925 | 42,172 | -1,247 | -3.0 |
| Riverside Labor Force Attachment | 6,726 | 38,929 | 39,804 | -875 | -2.2 |
| Lacked high school diploma or basic skills | 3,125 | 37,030 | 38,311 | -1,280 | -3.3 |
| Riverside Human Capital Development | 3,135 | 35,924 | 38,311 | -2,387*** | -6.2 |
| Columbus Integrated | 4,672 | 44,037 | 44,478 | -441 | -1.0 |
| Columbus Traditional | 4,729 | 44,232 | 44,478 | -246 | -0.6 |
| Detroit | 4,459 | 48,256 | 47,685 | 571 | 1.2 |
| Oklahoma City | 8,677 | n/a | n/a | n/a | n/a |
| Portland | 4,028 | 43,677 | 41,807 | 1,870 | 4.5 |
|
Earnings as a percentage of combined income (%) |
|||||
| Atlanta Labor Force Attachment | 2,938 | 36.0 | 32.4 | 3.5 *** | 10.9 |
| Atlanta Human Capital Development | 2,992 | 35.4 | 32.4 | 3.0 *** | 9.1 |
| Grand Rapids Labor Force Attachment | 3,012 | 43.7 | 39.8 | 3.9 *** | 9.9 |
| Grand Rapids Human Capital Development | 2,997 | 42.0 | 39.8 | 2.3 ** | 5.7 |
| Riverside Labor Force Attachment | 6,726 | 34.1 | 28.6 | 5.5 *** | 19.1 |
| Lacked high school diploma or basic skills | 3,125 | 28.5 | 22.8 | 5.7 *** | 25.0 |
| Riverside Human Capital Development | 3,135 | 26.3 | 22.8 | 3.5 *** | 15.2 |
| Columbus Integrated | 4,672 | 48.6 | 44.3 | 4.3 *** | 9.8 |
| Columbus Traditional | 4,729 | 46.9 | 44.3 | 2.6 *** | 5.8 |
| Detroit | 4,459 | 35.8 | 33.1 | 2.7 *** | 8.0 |
| Oklahoma City | 8,677 | n/a | n/a | n/a | n/a |
| Portland | 4,028 | 45.5 | 38.7 | 6.8 *** | 17.7 |
| SOURCE:В MDRC calculations from state and county administrative records. NOTES: See Appendix A.1. |
|||||
Over five years, control group members in five sites received between $40,000 and $45,000 in combined income, or about $8,000 to $9,000 per year. Control group members in Detroit received a slightly larger amount: $9,500 per year (Table 6.1, upper panel).(5) Control group members differed in the proportion of their combined income that was made up by earnings: about 30 percent of combined income in Atlanta, Detroit, and Riverside over five years, compared with about 40 percent in Columbus, Grand Rapids, and Portland (Table 6.1, lower panel).
Programs in the NEWWS Evaluation generally did not increase sample members' income relative to the control group. More often, they helped program group members substitute earnings and EITC for welfare and Food Stamps but left them with about the same level of income as control group members. This finding applies to LFA and HCD programs and, more generally, to employment- and education-focused programs.
There were some differences in program effects on combined income, but impacts varied by site rather than program approach. In Atlanta LFA and HCD and Portland, program group members received from 2.8 to 4.5 percent more in combined income than control group members, although these program-control group differences were just above the 0.10 level of statistical significance.(6) (See Table 6.1.) In contrast, both programs in Grand Rapids and Riverside reduced combined income over five years by 2.2 to 6.2 percent below control group levels.(7) The programs in Columbus and Detroit affected income very little.
The general inability of welfare-to-work services and mandates to increase income is not specific to programs studied in the NEWWS Evaluation. Results from San Diego's Saturation Work Initiative Model (SWIM), California's Greater Avenues for Independence (GAIN) program, Los Angeles County's Jobs First GAIN program, and Florida's Project Independence were similar: even when programs increased earnings, they seldom increased income very much.(8)
Combined Income by Year
Figure 6.1 and Appendix Table E.1 display the combined income of program and control group members during each year of follow-up. Year-by-year trends in income varied considerably by site. In Grand Rapids, Detroit, and Atlanta, control group members' average combined income increased by about $900 to nearly $1,800 between years 1 and 5. In contrast, control group members' combined income decreased during these years in Portland and Riverside by about $700 and $1,050, respectively. Control group members in Columbus received about the same amount of income during years 1 and 5. Trends for program group members were similar.(9)
Figure 6.1
Impacts on Combined Income in Years 1 to 5
SOURCE: MDRC calculations from state and county administrative records.
NOTES: See Appendix A.1.
Programs may have affected income in particular years, but these impacts can be obscured when impacts are calculated for the entire five-year follow-up. For example, it would be important to know whether programs increased income during the first three years of follow-up when no control group members had access to program services. Likewise, increases or decreases in income during year 5 would suggest future trends.(10)
According to Figure 6.1 and Appendix Table E.1, program impacts on combined income were concentrated during certain follow-up years. For example, during the first three years after random assignment, the Atlanta LFA program led to significantly greater income on average for program group members than for control group members. Similarly, the Atlanta HCD program achieved statistically significant increases in combined income in years 2 to 4. However, neither Atlanta program significantly increased income in year 5 of the follow-up period. Portland also increased program group members' income above control group levels in years 2 to 5 by $400 to $600 per program group member.(11) These three programs were not the norm, however. Grand Rapids LFA and HCD were more typical. The reductions in income stemming from the two Grand Rapids programs were spread out over the entire five-year period, even though the differences were statistically significant only in years 1 and 2 for Grand Rapids LFA and years 1 and 4 for Grand Rapids HCD.
Perhaps the most troubling results were in the Riverside HCD program, which left program group members with significantly less income than control group members in each year of the follow-up period. Reductions in income were especially large in years 2 to 4, when the average program group member had between $460 and $560 less in combined income than the average control group member.
Impacts on Combined Income for Educational Attainment Subgroups
Figure 6.2 shows program effects on combined income over five years for high school graduates and nongraduates. It appears that income was more adversely affected for high school nongraduates. Of the nine programs for which results can be compared for the two groups, six either increased income more for high school graduates than nongraduates or reduced income less for graduates than for nongraduates. The differences are most striking in Atlanta, where the average high school graduate program group members had about $1,500 to $2,000 more income than comparable control group members, but average nongraduate program group members had about the same income as comparable control group members. The main exception was Portland, which increased combined income for nongraduates by nearly $2,700 (or 7 percent) above the control group mean, but led to a much smaller gain for graduates. However, the program-control group difference for nongraduates was not statistically significant (p-value = .18).
Figure 6.2
Impacts on Combined Income in Years 1 to 5, by Hign School Diploma or GED Status
SOURCE: MDRC calculations from state and county administrative records.
NOTES: See Appendix A.1.
There is not much indication that program approach produced important differences in income gains or losses. Both Atlanta programs had virtually no effect on income for high school nongraduates, and both had much larger effects for high school graduates. Likewise, both Grand Rapids programs resulted in lower combined income for program group members than control group members in each subgroup. On the other hand, the income losses from the Grand Rapids LFA program were more concentrated among high school graduates, whereas the income losses from the Grand Rapids HCD program were more concentrated among high school nongraduates.
Impacts on Employment and Welfare Status Over Five Years
Income is perhaps the most comprehensive measure of economic well-being, but welfare-to-work programs can also help welfare recipients attain greater self-sufficiency by helping them find jobs and leave public assistance. Table 6.2 shows the degree to which the NEWWS programs succeeded in this regard by showing outcomes and impacts for four composite measures: (1) the proportion of people who were working and not receiving welfare, (2) the proportion who combined work and welfare, (3) the proportion who were on welfare and not working, and (4) the proportion who were neither working nor receiving welfare.(12) The table reports the percentage of the five-year (or 20-quarter) follow-up that people spent on average in each employment-welfare status.
Most programs led to small increases (of less than 5 percentage points) in the likelihood that people would work without receiving welfare (Table 6.2, first panel). The Portland program produced the largest impact, an increase of 7.3 percentage points above the control group level. Except in Riverside, programs did not affect the likelihood that sample members would combine work and welfare (Table 6.2, second panel). The increase for the Riverside programs probably reflects their overall gains in employment and the relatively high grant levels and generous earnings disregards in California.(13)
|
Site and Program |
Sample Size | Program Group (%) | Control Group (%) | Difference (Impact) | Percentage Change (%) |
|---|---|---|---|---|---|
|
Employed and not on welfare |
|||||
| Atlanta Labor Force Attachment | 2,938 | 24.2 | 21.4 | 2.8 *** | 13.1 |
| Atlanta Human Capital Development | 2,992 | 23.8 | 21.4 | 2.4 ** | 11.2 |
| Grand Rapids Labor Force Attachment | 3,012 | 29.4 | 27.1 | 2.3 ** | 8.6 |
| Grand Rapids Human Capital Development | 2,997 | 28.6 | 27.1 | 1.5 | 5.4 |
| Riverside Labor Force Attachment | 6,726 | 19.5 | 17.3 | 2.2 *** | 12.9 |
| Lacked high school diploma or basic skills | 3,125 | 15.8 | 13.8 | 2.1 *** | 15.2 |
| Riverside Human Capital Development | 3,135 | 14.8 | 13.8 | 1.1 | 7.8 |
| Columbus Integrated | 4,672 | 35.1 | 30.3 | 4.8 *** | 15.8 |
| Columbus Traditional | 4,729 | 33.2 | 30.3 | 2.9 *** | 9.4 |
| Detroit | 4,459 | 21.6 | 19.7 | 1.9 *** | 9.5 |
| Oklahoma City | 8,677 | n/a | n/a | n/a | n/a |
| Portland | 4,028 | 35.1 | 27.8 | 7.3 *** | 26.4 |
|
Employed and on welfare |
|||||
| Atlanta Labor Force Attachment | 2,938 | 18.5 | 17.5 | 1.0 | 5.5 |
| Atlanta Human Capital Development | 2,992 | 17.8 | 17.5 | 0.3 | 1.7 |
| Grand Rapids Labor Force Attachment | 3,012 | 19.7 | 18.4 | 1.3 ** | 7.1 |
| Grand Rapids Human Capital Development | 2,997 | 18.7 | 18.4 | 0.3 | 1.6 |
| Riverside Labor Force Attachment | 6,726 | 14.3 | 10.8 | 3.5 *** | 32.2 |
| Lacked high school diploma or basic skills | 3,125 | 14.0 | 9.7 | 4.2 *** | 43.5 |
| Riverside Human Capital Development | 3,135 | 12.6 | 9.7 | 2.8 *** | 29.0 |
| Columbus Integrated | 4,672 | 16.1 | 18.7 | -2.7 *** | -14.2 |
| Columbus Traditional | 4,729 | 17.4 | 18.7 | -1.3 *** | -7.2 |
| Detroit | 4,459 | 19.8 | 20.5 | -0.7 | -3.2 |
| Oklahoma City | 8,677 | n/a | n/a | n/a *** | n/a |
| Portland | 4,028 | 12.1 | 11.2 | 0.9 | 7.9 |
|
Not employed and on welfare |
|||||
| Atlanta Labor Force Attachment | 2,938 | 42.0 | 47.4 | -5.4 *** | -11.4 |
| Atlanta Human Capital Development | 2,992 | 44.1 | 47.4 | -3.3 *** | -6.9 |
| Grand Rapids Labor Force Attachment | 3,012 | 29.4 | 37.2 | -7.8 *** | -21.0 |
| Grand Rapids Human Capital Development | 2,997 | 32.4 | 37.2 | -4.8 *** | -12.9 |
| Riverside Labor Force Attachment | 6,726 | 35.0 | 43.7 | -8.6 *** | -19.8 |
| Lacked high school diploma or basic skills | 3,125 | 39.3 | 48.5 | -9.3 *** | -19.1 |
| Riverside Human Capital Development | 3,135 | 40.4 | 48.5 | -8.1 *** | -16.7 |
| Columbus Integrated | 4,672 | 26.4 | 29.7 | -3.3 *** | -11.1 |
| Columbus Traditional | 4,729 | 27.3 | 29.7 | -2.4 *** | -8.2 |
| Detroit | 4,459 | 43.4 | 45.5 | -2.1 ** | -4.5 |
| Oklahoma City | n/a | n/a | n/a | n/a *** | n/a |
| Portland | 4,028 | 24.9 | 34.8 | -9.9 *** | -28.4 |
|
Not employed and not on welfare |
|||||
| Atlanta Labor Force Attachment | 2,938 | 15.4 | 13.7 | 1.6 * | 12.0 |
| Atlanta Human Capital Development | 2,992 | 14.3 | 13.7 | 0.6 | 4.3 |
| Grand Rapids Labor Force Attachment | 3,012 | 21.5 | 17.3 | 4.2 *** | 24.2 |
| Grand Rapids Human Capital Development | 2,997 | 20.3 | 17.3 | 3.0 *** | 17.5 |
| Riverside Labor Force Attachment | 6,726 | 31.2 | 28.2 | 2.9 *** | 10.4 |
| Lacked high school diploma or basic skills | 3,125 | 30.9 | 28.0 | 2.9 *** | 10.5 |
| Riverside Human Capital Development | 3,135 | 32.2 | 28.0 | 4.2 *** | 15.0 |
| Columbus Integrated | 4,672 | 22.4 | 21.2 | 1.1 | 5.4 |
| Columbus Traditional | 4,729 | 22.1 | 21.2 | 0.9 | 4.2 |
| Detroit | 4,459 | 15.2 | 14.4 | 0.8 | 5.9 |
| Oklahoma City | n/a | n/a | n/a | n/a *** | n/a |
| Portland | 4,028 | 27.9 | 26.3 | 1.6 | 6.3 |
| SOURCE:В MDRC calculations from state and county administrative records. NOTES: See Appendix A.1. |
|||||
The largest change for most programs was in the likelihood that people would rely on welfare without working (Table 6.2, third panel). In general, employment-focused programs produced the largest reductions in the proportion of follow-up quarters not employed and on welfare. The Portland employment-focused program produced the largest impact among all programsВ 9.9 percentage points below the control group level. The three LFA programs led to reductions in this status of between 5.4 percentage points (Atlanta) and 8.6 percentage points (Riverside). Among education-focused programs, only Riverside HCD led to a comparable reduction. All other education-focused programs (including Atlanta and Grand Rapids HCD) decreased the proportion of follow-up quarters not employed and on welfare by less than 5 percentage points.
Finally, five programs (the three LFA programs and Grand Rapids and Riverside HCD) led to small but statistically significant increases in the proportion of follow-up quarters not employed and not on welfare. These programs probably encouraged some people to leave welfare for reasons other than employment. Notably, the LFA and HCD programs in Grand Rapids and Riverside also led to the largest reductions in combined income.
The proportion of follow-up quarters that program group members spent neither working nor receiving welfare (Table 6.1, fourth panel) ranged from about 15 percent in Atlanta and Detroit to more than 30 percent in Riverside. These program group members may have experienced severe financial hardship during the follow-up period. However, it is important to keep in mind the measure includes only income from earnings and welfare. It does not include income from Food Stamps(14) or other forms of public assistance, income of other household members, or earnings from jobs that were out of state or not reported to the state's UI earnings system.(15)
Comparing the Effects of the LFA and HCD Programs on Income
So far, the chapter has made informal comparisons of the effects of employment- and education-focused programs on measures of income and self-sufficiency. This section summarizes the results of a more precise test of the relative effects of the LFA and HCD programs in Atlanta, Grand Rapids, and Riverside on combined income over five years. (See Table 6.3.) Not surprisingly, there was no statistically significant LFA-HCD difference in combined income for the full samples in Atlanta and Grand Rapids. As noted above, both programs in each site led to similar effects on income: an increase for Atlanta LFA and HCD relative to the control group and a reduction for Grand Rapids LFA and HCD (Table 6.1).
|
Site and Program |
Sample Size | LFA ($) | HCD ($) | Difference (Impact) | p-value |
|---|---|---|---|---|---|
|
Full impact sample |
|||||
| Atlanta | 2,936 | 41,138 | 41,120 | 19 | 0.98 |
| Grand Rapids | 3,099 | 40,739 | 40,925 | -186 | 0.81 |
| Riverside | 4,980 | 38,929 | n/a | n/a | n/a |
|
No high school diploma or GED |
|||||
| Atlanta | 1,190 | 37,490 | 36,890 | 599 | 0.52 |
| Grand Rapids | 1,268 | 37,015 | 35,835 | 1,180 | 0.23 |
| Riverside | 3,182 | 37,030 | 35,924 | 1,106 | 0.17 |
|
High school diploma or GED |
|||||
| Atlanta | 1,742 | 43,596 | 43,994 | -397 | 0.70 |
| Grand Rapids | 1,827 | 43,249 | 44,489 | -1,240 | 0.26 |
| Riverside | 1,798 | 41,433 | n/a | n/a | n/a |
| SOURCE:В MDRC calculations from state and county administrative records. NOTES: See Appendix A.1. |
|||||
The small overall difference in income generated by the two approaches masks larger differences for educational attainment subgroups. For high school nongraduates, the LFA program group had higher income than the HCD program group in each site. Although the difference was not statistically significant in any one site, a simple average of the impacts across the three sites indicates that the three LFA programs as a group resulted in nearly $1,000 more income over five years than the HCD programs. Moreover, this LFA-HCD difference for high school nongraduates was statistically significant. These results, along with the LFA programs' larger effects on employment and earnings, suggest that job search was a better approach for helping welfare recipients with low educational attainment than the type of mandatory adult basic education services provided by the HCD programs. It should be recalled, however, that both the LFA and HCD programs in Grand Rapids and Riverside decreased the income of nongraduates relative to the control group.
Among high school graduates and GED recipients, on the other hand, HCDs in Atlanta and Grand Rapids received more income over five years than LFAs. In neither site, however, was the difference statistically significant; the same result occurred when LFA and HCD impacts were averaged across the two sites.
Impacts on Respondent and Household Income at the End of Year 5
Additional Sources of Income for Respondents
Survey respondents were asked whether they had received income from earnings, welfare, Food Stamps, child support payments, Supplemental Security Income (SSI) payments, Social Security benefits, or "other income" at the end of year 5. (See Appendix Table E.4, middle panels.) The last four types of income were not recorded from administrative records and not included in the measures of combined income presented above. Under some circumstances, programs may affect levels of receipt of child support payments. For instance, program group members were likely to have had more contact with the welfare department than control group members and may have received more help in seeking a child support order. Similarly, programs may have increased SSI receipt if program group members with disabilities had greater contact with the welfare department than control group members.(16)
It turns out, however, that programs had only small and scattered effects on receipt of these income sources at the end of year 5. Only Atlanta HCD increased receipt of child support paymentsВ by 4.2 percentage points above the control group level of 13.7 percent. Two other programs, Grand Rapids HCD and Riverside LFA, produced a similar increase in SSI receipt. Finally, Riverside HCD reduced "other income" by 4.5 percentage points.
Household Income
As noted above, it is important to consider whether other household members received income from work or other sources and may have contributed toward the purchase of food, clothing, shelter, and other necessities for respondents and their children and helped respondents to experience a higher standard of living than they could have attained from their income alone.(17) Higher household income may also be associated with favorable outcomes for children, an issue explored in Chapters 11 and 12.
Respondents to the Five-Year Client Survey reported whether other members of their household received income from employment, welfare, Food Stamps, child support, SSI, social security, or other sources at the end of year 5. (See Appendix Table E.4, lower panels.) About 55 percent of control group respondents in Grand Rapids, Portland, and Riverside and about 40 percent in Atlanta reported living with someone who received income from one or more of these sources. Most programs did not affect the likelihood of living with others who received income, although the decrease for Riverside LFA (of 4.2 percentage points) was just above the 10 percent level of statistical significance.(18)
A large majority of control group respondents reported that they were living in a household in which at least one person worked for pay: from 65.5 percent in Atlanta to 80.3 percent in Grand Rapids (Appendix Table E.4, upper panels). Furthermore, a fairly large proportion of control group membersВ 42 to 43 percent in Grand Rapids, Portland, and Riverside and 28 percent in AtlantaВ lived with at least one person who was working for pay (Appendix Table E.4, lower panels). Fewer than half of these respondents (or from 10 to 19 percent of all control group members) reported that they were jobless but other household members were working for pay (Appendix Table E.5, lower panels).
Programs did not affect the likelihood of living with another person who was the sole wage-earner in the household or living in a household in which the respondent and another household member were working for pay. Similarly, programs had no impact on the proportion of sample members who lived in a household in which monthly earnings for other household members totaled $1,000 or more (results not shown).
1.В Morris et al., 2001; Bos et al., 1999; Knox, Miller, and Gennetian, 2000.
2.В See, for example, Appendix Table E.4, which displays income receipt at the end of year 5, based on survey data. A relatively small percentage of respondents reported receipt of income from sources other than earnings, welfare, and Food Stamps. The main exceptions are child support payments in Grand Rapids and Portland.
3.В See Chapter 10 for a discussion of out-of-pocket expenses for child care.
4.В It was beyond the scope of this evaluation to measure the EITC and payroll taxes directly. Instead, these outcomes were estimated on the basis of sample members' measured earnings, rules for calculating the EITC and taxes, and assumptions about the percentage of sample members who applied for the EITC on their federal income tax return. For each sample member, EITC calculations for years 1 to 5 use the tax rules for the calendar years in which the quarters 4, 8, 12, 16, and 20 occurred. Tax years range from 1992 through 1996 to 1995 through 1999, depending on when sample members were randomly assigned. Calculations assume an 80 percent take-up rate (see Scholz, 1996). Specifically, sample members' EITCs were estimated for each year of follow-up and then multiplied by 0.8. Estimated payroll taxes were calculated by multiplying earnings by 7.65 percent, the tax rate during most of the follow-up period.
5.В As noted above, it was assumed that 80 percent of sample members with earnings received EITCs. To test the effects of this assumption, sample, sample members' combined income was recalculated, assuming that 100 percent of sample members with earnings received thereceived the EITC. The higher rate made the impact estimates on combined income more positive by about $170 for Portland and $90 for90 for Riverside LFA. However, impact estimates for other programs changed verychanged very little. (Results not shown.)
6.В P-values were 0.11 for Atlanta LFA and HCD and 0.15 for Portland.
7.В The decreases for Grand Rapids LFA (-$1,433) and Riverside HCD (-$2,387) were statistically significant. The decreases for Grandfor Grand Rapids HCD (-$1,247, p-value = 0.11) and) and Riverside LFA (-$875, p, p-value = 0.16) were) were just above the 0.1 level of statistical significance.
8.В See Riccio, Friedlander, and Freedman, 1994, for GAIN; Hamilton and Friedlander, 1989, for SWIM; Freedman et al., 2000b, for Jobs-First GAIN; and Kemple, Friedlander, and Fellerath, 1995, for Project Independence. See Bloom and Michalopoulos, 2001, for a synthesis of these and other studies.
9.В Trends in average combined income for the program or control group may be misleading because they do not include other sources of income for sample members and their households. However, it is reasonable to assume that excluding these sources did not bias the cumulative or yearly estimates of impacts on combined income.
10.В Appendix Table E.2 shows the cumulative effects of the programs on income from earnings and public assistance (that is, without estimated EITC payments and payroll taxes) over the first three years and for the last quarter of year 5.
11.В For Portland, only the impact for year 4 was statistically significant, although p-values for years 2 (.139) and 3 (.102) were just above the .1 level of statistical significance. The increase in year 5 of $470 had a p-value of .245. Detroit's program also led to higher income during years 2 to 4, but the differences were small and not statistically significant.
12.В Appendix Table E.3 shows similar results for the last quarter of year 5. Since the programs in general had small effects on employment during the last quarter of year 5, they also had relatively small effects on these outcomes. However, several, several of the programs continued to significantly reduce the proportion of people who were on welfare and not working. Notably, Portland, Portland increased the proportion of sample members who were working and not receiving welfare by 5.8 percentage points relative to the control group, by far the largest effect on this measure of self-sufficiency.
13.В The other three employment-focused programsIn addition, Atlanta and Grand Rapids LFA both led to 1 percentage point gains in this measure. The fact that people were not able to leave welfare when they went to work implies that they earned fairly little, perhaps because they worked part time or earned close to the minimum wage. In addition, sample, sample members may have received welfare and earnings at different times during a quarter as they transitioned from welfare to work or from work back to welfare. MDRC did not investigate these issues.
14.В Across all sites, program and control group members received income from Food Stamps only (no earnings or welfare payments) during 2 to 7 percent of follow-up quarters. Programs did not affect the likelihood of having income from Food Stamps only.
15.В Appendix GH compares employment levels and impacts calculated with UI earnings and survey data. As discussed in Appendix G, survey respondents in Riverside and Portland reported considerably more employment than was recorded on UI earnings records. Furthermore, as will be discussed in greater detail below, according to responses to the Five-Year Client Survey, virtually all sample members had some type of income in the month prior to completing the survey. (See Appendix Table E.4.)
16.В Some respondents also reported earnings that had not been recorded by the UI system. See Appendix H.
17.В Some household members who received income may not have contributed financial support to sample members and their children. For instance, people receiving SSI or other types of assistance may have neededhave needed to use all of their income to meet their own needs.
18.В A larger reduction for LFA nongraduates was statistically significant. The Portland program led to a program-control group difference of +6.0 percentage points that was not statistically significant (p-value = .21). The p-value for the Riverside LFA impact was .13.
What Works Best for Whom: Economic Effects by Subgroup
Chapters 4-6 revealed the overall effects on economic outcomes of the NEWWS programs studied. With regard to earnings, for example, employment-focused programs had larger immediate effects than education-focused programs, Portland had large and persistent effects, and education-focused programs in Detroit and Oklahoma City had relatively small effects. This chapter investigates whether some groups were affected more or less than others. In particular, program effects on average earnings, welfare benefits, and income are compared for long-term and short-term welfare recipients; those who had worked in the year prior to random assignment and those who had not; by race and ethnicity; and for groups defined by whether they faced multiple barriers to work as long-term welfare recipients, high school dropouts, or long-term unemployed.
Knowing how welfare-to-work services affect various subgroups can help in deciding where to target new resources or where to develop new services. For example, recipients who are more disadvantaged and who are likely to have the most difficulty finding a job will be particularly at risk of losing income if they lose eligibility for benefits under TANF. Programs that showed positive effects for these recipients may serve as good models under time-limited welfare.
Results for more job-ready recipients are also of interest. Programs may only have assisted these individuals to secure jobs more quickly than they would have otherwise, but have had little long-term effect. If that was the case, targeting them might have been an inefficient use of resources. On the other hand, the programs might have helped job-ready recipients find higher-quality jobs, which could very well have had substantial positive effects on their long-term earnings. The results for job-ready subgroups can inform this debate, which is likely to become more heated as states continue to try different strategies under TANF.
Key Findings
- The programs generally had larger effects on earnings for people who had not worked in the year prior to random assignment than for those who had worked, and larger effects on earnings for long-term recipients than for short-term recipients. Programs that increased earnings more also tended to reduce cash assistance more. As a result, effects on cash assistance also were generally larger for those who had not worked recently and for long-term recipients.
- There were often substantial differences in effects on earnings by level of disadvantage, but those differences were not consistent across programs. Some programs had much larger effects for the most disadvantaged than for the moderately disadvantaged. Other programs had much smaller effects for the most disadvantaged. Site may be more important than self-sufficiency approach. For example, both Grand Rapids programs had larger effects for the moderately disadvantaged, but both Atlanta programs had larger effects for the most disadvantaged.
- In general, program effects on earnings were larger for Hispanic and African-American sample members than for white sample members. An important exception was the Portland program, which increased earnings of white sample members by more than $6,000 over five years but did not change earnings of African-American sample members. It is unclear what caused this large difference, but the result should be interpreted with a great deal of caution since the Portland sample contains few African-American control group members.
- Although the more disadvantaged groups had higher earnings because of the programs, they still earned very little. For example, sample members without recent work experience had only about half as much in earnings as those with recent work experience, and the most disadvantaged group earned about half as much as the moderately disadvantaged group.
- The programs did not systematically change income (from earnings, public assistance, and projected EITC payments net of payroll taxes) for any subgroups. This reflects the fact that welfare recipients who went to work lost some of their benefits, whether they were in the program or control group, while sanctions for those who failed to comply with program mandates may have reduced public assistance benefits by more than earnings for some people.
- People in the LFA programs generally had higher earnings than those in the HCD programs. However, differences between the two approaches were generally small for most subgroups.
- The Portland program, which was employment-focused but used a mix of job search and Adult Basic Education, had the largest or close to the largest effects on earnings for most subgroups. This suggests that the Portland approach was effective for a wide range of people.
Analysis Issues
Subgroups were organized around three barriers to employment: high school education (high school graduates compared with nongraduates), recent work experience (those who had worked in the year prior to random assignment compared with those who had not), and welfare history (those who had ever been on welfare at least two years prior to random assignment compared with those who had not). Results by high school credential were presented in Chapters 4-6. This chapter presents results for the other barriers. In addition, the three individual barriers to employment were used to define three mutually exclusive subgroups based on relative disadvantage. The "most disadvantaged" were sample members who did not have a high school diploma or GED prior to random assignment, did not work in the year prior to random assignment, and were on welfare for two years or more prior to random assignment. "Moderately disadvantaged" sample members faced only one or two of the three barriers, while the "least disadvantaged" faced none. Finally, results are presented in this chapter for racial and ethnic groups.
Subgroups were identified using information collected just before individuals were randomly assigned. Because these groups were defined by pre-existing characteristics observed at study enrollment, control and program group members in the subgroup should be comparable at the time of random assignment, and any systematic differences that emerge between the two groups can be reliably attributed to the programs being studied.
The chapter presents results for three outcomes: (1) earnings, (2) cash assistance, and (3) combined income from earnings, cash assistance, Food Stamps, and the federal Earned Income Credit (EITC) net of payroll taxes. The three outcomes represent three different perspectives. Many policymakers want to encourage welfare recipients to work; for them, the "best" program may be the one that increases earnings the most. Other policymakers may be primarily interested in reducing spending on welfare; for them the best program may be the one that reduces cash assistance the most. Welfare recipients and policymakers concerned about child and family poverty may care most about their total income; for them, the best program may be the one that increases income the most.
For each outcome, the chapter focuses on cumulative dollar amounts over a five-year follow-up period. Although use of program services by control group members might have reduced the effects of some programs in years 4 and 5, five years of follow-up are used for two reasons. First, the program effects over five years are generally similar to their effects over three years. Second, an earlier report presented a detailed analysis of subgroup impacts over three years.(1)
In analyzing subgroups, several types of comparisons are made, with each comparison answering a different question. The first question is whether there is evidence that the welfare-to-work programs taken as a whole without regard to the approach they used or where they operated tended to affect a particular subgroup. For example, with welfare time limits, administrators probably want to make sure that long-term welfare recipients are able to find work and leave welfare; if the programs were not particularly effective at benefiting long-term recipients, policymakers might want to target them for more resources or devise different and better services.
A second question is whether the programs tended to have larger effects for one subgroup than another. Again, the answer to this question can help policymakers think about how to use their precious resources or whether new services should be developed. Suppose that long-term recipients were generally affected by the programs being studied. Suppose, however, that they were affected less than short-term recipients. This might suggest that more effort or different services should be considered for long-term recipients to increase the effectiveness of welfare-to-work services.
A third question is whether one program approach benefited a subgroup more than another approach or whether it benefited one subgroup more than another subgroup. Because there are fewer programs of each type, however, statements about the effects of particular program models might be more speculative. For example, Portland is the only NEWWS program that was employment-focused with varied first activities. Although its impacts on earnings were by far the largest and this chapter shows that the effects were also the largest for most subgroups it cannot be determined whether this is a consequence of Portland's approach, the way sample members were chosen, the Portland economy (or other local factors), or unobserved differences between welfare recipients in Portland and in the other sites.
Since the number of people in a subgroup is, obviously, less than the number in the full sample, it is consequently more difficult to confidently say that an individual program had an effect for a subgroup than it is for the full sample, and it is more difficult to say whether the estimated effects are bigger for one group than another because of the program rather than by chance. However, the pattern of impacts across programs can provide statistical evidence that the programs taken as a whole had a particular effect, even if no individual program had a statistically significant effect. For example, suppose the question is whether welfare-to-work programs have a larger impact on earnings for long-term welfare recipients or for short-term welfare recipients. If 9 or more programs had a larger impact for long-term recipients than for short-term recipients, say, the hypothesis that the impacts are the same for the two groups can be rejected at the 10 percent significance level even if no single program had a statistically significant different effect for long-term recipients than for short-term recipients. Likewise, if 10 or more programs have an impact in the same direction, we can reject the hypothesis of no difference at the 5 percent significance level, and if all 11 programs have an impact in the same direction, we can reject the hypothesis of no difference at the 1 percent significance level.(2)
The most rigorous means of examining the effects of program models is to compare the effects of the three LFA programs with their HCD counterparts. The chapter consequently devotes a section to this comparison. With only three programs of each type, however, it can be difficult to draw firm conclusions about the relative benefits of the two approaches by subgroup. As mentioned above, it is harder to find statistically significant effects for a subgroup than for the full sample and unlikely that the impacts between two subgroups will be statistically significantly different in any specific site. Moreover, three sites are too few to use only the pattern of results to draw conclusions about the relative effectiveness of the two approaches unless the differences in impacts between the two approaches are large. If, for example, the LFA and HCD approaches are equally effective, then the chance that all three LFA programs would have larger impacts than all three HCD programs would be 12.5 percent, or greater than the usual threshold for drawing conclusions based on statistical significance. However, it is extremely unlikely that all three LFA programs would increase earnings significantly more than all three HCD programs simply by chance, and statistically significant differences in all three sites would be enough to draw solid conclusions based on the statistical evidence.
Impacts by Subgroup
Welfare History
The Family Support Act (FSA) of 1988, the provisions of which helped shape the programs studied in this report, was designed to help individuals who were most likely to become long-term welfare recipients. The FSA required states to target this group for welfare-to-work resources and to offer services that were thought to provide them the greatest benefit. An important question about the NEWWS programs, therefore, is whether the programs had positive effects for long-term recipients.
Table 7.1 shows impacts on earnings; cash assistance payments; and income from earnings, cash assistance, Food Stamps, and the federal EITC net of payroll taxes, all measured over the five years following random assignment. Impacts are shown for each program for two subgroups: long-term welfare recipients (those who had been on welfare for two years or more prior to random assignment) and short-term welfare recipients and welfare applicants (those who had been on welfare for less than two years prior to random assignment). To allow comparisons between the two Riverside programs to be made, results for the Riverside LFA program are presented both for the entire sample and for sample members considered in need of basic education (the only group assigned to the Riverside HCD program).
Long-term recipients. Table 7.1 shows that the NEWWS programs did increase earnings for long-term recipients. In all of the programs, long-term recipient program group members had higher earnings over three years than long-term recipient control group members. Impacts were highest in Portland, at nearly $6,000 over five years, with most programs having impacts between $1,000 and $3,700 per person. In all cases but three, moreover, the differences were statistically significant.
|
Site and Program |
SampleSize | Average Total Earnings in Years1 to 5 ($) | Average Welfare Payments in Years 1 to 5 ($) | Combined Income in Years 1 to 5 ($) |
|---|---|---|---|---|
|
On welfare for two years or more |
||||
| Atlanta Labor Force Attachment | 2,063 | 2,522*** | -1,137*** | 1,002 |
| Atlanta Human Capital Development | 2,100 | 2,059** | -856*** | 1,080 |
| Grand Rapids Labor Force Attachment | 1,791 | 1,221 | -2,931*** | -2,206** |
| Grand Rapids Human Capital Development | 1,775 | 205 | -2,121*** | -2,341** |
| Riverside Labor Force Attachment | 3,510 | 3,657*** | -3,427*** | -568 |
| Lacked high school diploma or basic skills | 1,831 | 3,016*** | -3,302*** | -901 |
| Riverside Human Capital Development | 1,841 | 2,582*** | -3,018*** | -1,177 |
| Columbus Integrated | 3,392 | 2,523*** | -1,686*** | -176 |
| Columbus Traditional | 3,415 | 1,424* | -1,154*** | -129 |
| Detroit | 3,313 | 1,710* | -860*** | 370 |
| Oklahoma City | 2,076 | 699 | n/a | n/a |
| Portland | 2,443 | 5,859*** | -2,905*** | 2,415 |
|
On welfare for less than two years |
||||
| Atlanta Labor Force Attachment | 840 | 2,262 | -227 | 1,623 |
| Atlanta Human Capital Development | 847 | 1,540 | -419 | 850 |
| Grand Rapids Labor Force Attachment | 1,219 | 2,125 | -2,002*** | -280 |
| Grand Rapids Human Capital Development | 1,215 | 1,603 | -1,241*** | 169 |
| Riverside Labor Force Attachment | 3,101 | 1,506 | -1,979*** | -1,111 |
| Lacked high school diploma or basic skills | 1,248 | 1,550 | -2,565*** | -1,624 |
| Riverside Human Capital Development | 1,238 | -415 | -2,855*** | -4,066*** |
| Columbus Integrated | 806 | -843 | -1,468*** | -3,362 |
| Columbus Traditional | 793 | 1,291 | -1,018*** | -917 |
| Detroit | 1,015 | 1,298 | -16 | 1,300 |
| Oklahoma City | 2,683 | -236 | n/a | n/a |
| Portland | 1,494 | 4,410* | -2,638*** | 1,538 |
| SOURCES: MDRC calculations from unemployment insurance (UI) earnings records and AFDC records. NOTES: Impacts on earnings were significantly different across subgroups in Riverside HCD. Impacts on AFDC were significantly different across subgroups in Atlanta LFA and Grand Rapids LFA. N/a = not applicable. |
||||
In all 10 programs for which welfare benefits could be measured, long-term recipients in the program group received less in cash assistance than long-term recipients in the control group. In part, this is a natural consequence of going to work. In all sites, welfare recipients' cash benefits were reduced somewhat when their earnings increased. In a number of programs, however, impacts on cash assistance were greater than impacts on earnings. In addition, as discussed in Chapter 5, sanctions would have resulted in welfare savings over and above the program effects on earnings, particularly in Grand Rapids and Columbus Integrated. Alternatively, some people may have left welfare because they were unable or unwilling to comply with program requirements, and others may not have returned to welfare when they lost their jobs.
Because the programs generally resulted in higher earnings but less cash assistance for long-term recipients than would have occurred otherwise, they generally had relatively small effects on income from earnings, cash assistance, Food Stamps, and projected EITC payments net of payroll taxes. For programs outside Grand Rapids, the impacts on income were small enough that they could not reliably be attributed to the programs. Moreover, the estimated impact was negative in six programs and positive in four, providing a further indication that these programs did not systematically affect income. Nevertheless, the results in Grand Rapids may be considered troubling. The reductions in income that were seen overall (in Chapter 6) appear to be concentrated among long-term recipients.
Short-term recipients. Table 7.1 indicates that the programs also generally had effects for short-term welfare recipients, although not as consistently as for long-term recipients. In all cases, short-term recipient program group members had lower cash assistance payments than their control group counterparts (though the difference was statistically significant in only 7 of the 10 programs). In only 8 of the 11 programs, however, did they have higher earnings and the earnings impact was statistically significant for short-term recipients only in Portland. Particularly troubling are the impacts for the Riverside HCD short-term recipients whose earnings were virtually unchanged, but whose cash assistance was reduced by more than $2,800 over five years. As a result of the program, short-term recipient program group members received more than $4,000 less in income from earnings, cash assistance, and Food Stamps than short-term recipient control group members. The Riverside HCD program was not alone: The Columbus Integrated program reduced income by more than $3,300 for program group members.
Comparing short-term and long-term recipients. Should long-term recipients, short-term recipients, or both be targeted for new services? One means of addressing the question is to look at the relative effects of the programs on the two groups. Even though many of the programs were effective for both groups, long-term recipients were generally helped more than short-term recipients. In 9 of the 11 programs (the only exception was in Grand Rapids), the impact on earnings for long-term recipients was greater than the impact on earnings for short-term recipients. This suggests that the approach of the programs studied in this report was effective at increasing the earnings of long-term recipients.
If a goal of welfare reform is to increase income, however, then these programs were generally equally ineffective for long-term recipients and short-term recipients. If they are not already doing so, states should consider further supplementing the earnings of welfare recipients or recent welfare recipients through enhanced earnings disregards, state Earned Income Tax Credits, or other means, to make it more likely that the programs that encourage them to work also help them to obtain greater financial resources.(3)
Another means of asking whether one group or another should be targeted for future services is to look at their outcomes. If the earnings levels of long-term recipients remained low despite the fact that the programs were generally effective for this group it might suggest the need for more or different services to further ameliorate their barriers to work. Across the 11 programs there is generally a large gap in the earnings levels of the two groups: earnings of long-term recipients were generally about 60 to 75 percent of earnings for short-term recipients (result not shown in Table 7.1). For example, in the Atlanta LFA program, earnings for long-term recipients in the program groups were about $16,500 over five years compared with nearly $28,000 for short-term recipients.
Recent Work Experience
Welfare-to-work programs are likely to have small effects if they offer services only to people whose barriers to work are so serious that they cannot take advantage of the services. People without much work experience could represent such a group; the fact that they have not worked much may indicate the presence of barriers that are keeping them from working (if not the possibility that they would rather raise their children than work). Alternatively, welfare-to-work services could have small effects if control group members are so likely to work on their own (that is, without the benefit of program services) that services are unable to improve their outcomes. People who have worked recently clearly have the ability to find work and may represent such a group.
Recent work history does predict future earnings well and may help identify groups that could benefit from welfare-to-work services and groups that have less need to benefit. Across the 11 programs people who had worked in the year prior to random assignment earned about twice as much as people who had not.(4) The most extreme differences occurred in the Riverside HCD program, where control group members who had worked in the year prior to random assignment earned about $19,000 on average in the five years after random assignment, and control group members who had not worked earned less than $7,000. (In Portland, however, those who had worked recently earned only about 50 percent more than those who had not. This could reflect Portland's very strong economy, the removal of the most disadvantaged participants from the study through up-front screening, or some other factor.)
Table 7.2 addresses whether these differences in ability to earn without employment services translated into differential impacts for the two groups. The table offers evidence but far from overwhelming evidence that those who have not worked recently do benefit more from welfare-to-work services, at least in terms of increasing their earnings.
Site and Program | Sample Size | Average Total Earnings in Years 1 to 5 ($) | Average Welfare Payments in Years 1 to 5 ($) | Combined Income in Years 1 to 5 ($) |
|---|---|---|---|---|
Did not work in year prior to random assignment | ||||
| Atlanta Labor Force Attachment | 1,869 | 3,763*** | -928*** | 2,323*** |
| Atlanta Human Capital Development | 1,937 | 3,110*** | -780*** | 2,020** |
| Grand Rapids Labor Force Attachment | 1,527 | 1,630 | -3,103*** | -2,157** |
| Grand Rapids Human Capital Development | 1,489 | 543 | -2,112*** | -1,896* |
| Riverside Labor Force Attachment | 4,010 | 3,024*** | -3,027*** | -751 |
| Lacked high school diploma or basic skills | 2,074 | 1,986*** | -3,286*** | -1,978** |
| Riverside Human Capital Development | 2,065 | 1,863*** | -2,969*** | -1,950** |
| Columbus Integrated | 2,143 | 2,925*** | -1,506*** | 531 |
| Columbus Traditional | 2,160 | 2,641*** | -1,225*** | 938 |
| Detroit | 2,978 | 904 | -400 | 260 |
| Oklahoma City | 3,910 | 245 | n/a | n/a |
| Portland | 2,317 | 6,276*** | -3,620*** | 2,035 |
Worked in year prior to random assignment | ||||
| Atlanta Labor Force Attachment | 1,069 | 370 | -782** | -691 |
| Atlanta Human Capital Development | 1,055 | 238 | -666** | -402 |
| Grand Rapids Labor Force Attachment | 1,485 | 1,622 | -2,007*** | -586 |
| Grand Rapids Human Capital Development | 1,508 | 1,240 | -1,478*** | -565 |
| Riverside Labor Force Attachment | 2,716 | 1,788 | -2,22*** | -1,073 |
| Lacked high school diploma or basic skills | 1,051 | 2,958* | -2,320*** | 202 |
| Riverside Human Capital Development | 1,070 | 403 | -2,723*** | -2,962* |
| Columbus Integrated | 2,529 | 1,374 | -1,533*** | -1,227 |
| Columbus Traditional | 2,569 | 486 | -987*** | -1,160 |
| Detroit | 1,481 | 2,592* | -907** | 1,155 |
| Oklahoma City | 4,767 | 13 | n/a | n/a |
| Portland | 1,711 | 3,522 | -1,620*** | 1,408 |
| SOURCES: MDRC calculations from unemployment insurance (UI) earnings records and AFDC records. NOTES: Impacts on earnings were significantly different across subgroups in Atlanta LFA. Impacts on AFDC were significantly different across subgroups in Grand Rapids LFA and Portland. Impacts on total income were significantly different across subgroups in Atlanta LFA. N/a = not applicable. | ||||
Those without recent work experience. The programs generally led to higher earnings among those without recent work experience, which implies that the services helped those who had not worked recently find jobs (although there are many in this group who did not work much). In all 11 programs, earnings were higher for program group members who had not worked recently than for their control group counterparts.
Once again, however, positive impacts on earnings did not generally translate into positive impacts on income. The most positive findings were in Atlanta, where the programs resulted in income gains of more than $2,000 compared with what would have occurred otherwise. This might reflect Atlanta's fill-the-gap budgeting, which allowed working welfare recipients to keep more of their welfare checks than they kept in other sites, or the relatively low welfare benefit levels in Georgia.
Those with recent work experience. In all 11 programs, earnings were higher for program group members who had worked recently than for their control group counterparts. However, the difference was statistically significant only in Detroit, and the impacts were typically not very big. Only in Detroit and Portland did the impact on earnings exceed $2,000. Despite the relatively small gains in earnings, the programs universally led to reduced welfare benefits for this group. In fact, the modest earnings gains combined with systematic reductions in welfare benefits led to the most systematic reductions in income for any subgroup: In 8 of the 10 programs for which income could be measured, program group members with recent work experi-ence ended up with lower income than their control group counterparts, though only in the Riverside HCD program was this difference large enough to be statistically significant.
Comparing those with and without recent work experience. The differences between those who had worked and those who had not are somewhat greater than the differences between long-term and short-term recipients. In the two Atlanta programs, for example, earnings impacts were virtually the same for long-term and short-term recipients, but were about $3,000 to $3,500 greater for those without recent work experience than for those with recent work experience. In Portland, the impact on earnings for the group without recent work experience was nearly twice as large as the impact for those with recent work experience. In 9 of the 11 programs, in fact, impacts on earnings were larger for those who had not worked recently than for those who had. All of this suggests that people who have not worked in a while benefit the most (in terms of increasing their earnings and self-sufficiency, but not in terms of increasing their income) from these types of welfare-to-work programs.
Composite Level of Disadvantage
In summarizing results from several welfare-to-work programs from the 1980s, Friedlander found that earnings gains were concentrated neither among groups of long-term recipients, who are expected to have a hard time finding work and leaving welfare, nor among groups such as new welfare applicants, who are most likely to work without assistance from a welfare-to-work program. Instead, Friedlander found the largest earnings gains among a middle group of welfare applicants who had spent some but not a great deal of prior time on welfare. In contrast, welfare savings came primarily from long-term recipients, especially those without a high school diploma or with little recent work experience.(5)
Do the NEWWS sites provide evidence that their approach changed these patterns by making the impacts for long-term recipients closer to those for short-term recipients? Results already presented in this chapter differ from Friedlander's, in that impacts on earnings were generally larger for the more disadvantaged group than for the less disadvantaged group. Table 7.3 shows impacts for three groups defined by three barriers to work. (As noted above, the most disadvantaged did not have a high school diploma or GED prior to random assignment, did not work in the year prior to random assignment, and were on welfare for two or more years prior to random assignment. The moderately disadvantaged faced only one or two of the three barriers, while the least disadvantaged faced none.)
|
Site and Program |
Sample Size | Average Total Earningsin Years1 to 5 ($) | Average Welfare Payments in Years1 to 5 ($) | Combined Income in Years 1 to 5 ($) |
|---|---|---|---|---|
|
Most disadvantaged |
||||
| Atlanta Labor Force Attachment | 698 | 1,946* | -766** | 1,285 |
| Atlanta Human Capital Development | 734 | -290 | -243 | -452 |
| Grand Rapids Labor Force Attachment | 458 | 3,994*** | -4,028*** | -828 |
| Grand Rapids Human Capital Development | 453 | 1,566 | -2,886 | -2,136 |
| Riverside Labor Force Attachment | 1,362 | 2,938*** | -3,592*** | -1,395 |
| Riverside Human Capital Development | 1,362 | 3,279*** | -2,973*** | -445 |
| Columbus Integrated | 911 | 2,206* | -2,047*** | -1,055 |
| Columbus Traditional | 901 | 302 | -1,198*** | -1,318 |
| Detroit | 1,119 | 1,187 | -1,219** | -827 |
| Oklahoma City | 429 | 740 | n/a | n/a |
| Portland | 617 | 3,162 | -2,862** | 491 |
|
Moderately disadvantaged |
||||
| Atlanta Labor Force Attachment | 1,887 | 3,406*** | -1,020*** | 1,772* |
| Atlanta Human Capital Development | 1,911 | 2,617** | -904*** | 1,489 |
| Grand Rapids Labor Force Attachment | 2,123 | 270 | -2,396*** | -2,510** |
| Grand Rapids Human Capital Development | 2,078 | 934 | -1,749*** | -1,187 |
| Riverside Labor Force Attachment | 4,298 | 3,035*** | -2,547*** | -204 |
| Riverside Human Capital Development | 3,049 | -1,307 | -2,185*** | -3,902*** |
| Columbus Integrated | 3,155 | 2,663** | -1,432*** | 235 |
| Columbus Traditional | 3,236 | 1,790* | -1,060*** | 253 |
| Detroit | 3,018 | 2,173** | -271 | 1,740* |
| Oklahoma City | 6,170 | 255 | n/a | n/a |
| Portland | 2,803 | 5,960*** | -3,318*** | 1,935 |
|
Least disadvantaged |
||||
| Atlanta Labor Force Attachment | 353 | -1,121 | -119 | -1,347 |
| Atlanta Human Capital Development | 347 | 709 | -514 | 161 |
| Grand Rapids Labor Force Attachment | 431 | 1,998 | -1,145* | 665 |
| Grand Rapids Human Capital Development | 466 | 1,869 | -408 | 1,664 |
| Riverside Labor Force Attachment | 1,066 | 516 | -1,871*** | -2,162 |
| Columbus Integrated | 606 | -3,413 | -991*** | -4,916* |
| Columbus Traditional | 592 | -1,951 | -1,184*** | -4,376 |
| Detroit | 322 | -894 | -1,201 | -2,483 |
| Oklahoma City | 2,078 | -448 | n/a | n/a |
| Portland | 608 | 2,430 | -159 | 1,883 |
| SOURCES: MDRC calculations from unemployment insurance (UI) earnings records and AFDC records. NOTES: Impacts on earnings were significantly different across subgroups in Grand Rapids LFA, Riverside LFA, Riverside HCD, and Columbus Integrated. Impacts on AFDC benefits were significantly different across subgroups in Grand Rapids LFA and HCD and Portland. Impacts on total income were significantly different across subgroups in Riverside HCD, Columbus Traditional, Columbus Integrated, and Detroit. N/a = not applicable. |
||||
This method of defining the most and the least disadvantaged does a good job of finding groups that would fare well and poorly on their own, in terms of their earnings levels. In the five years after random assignment, average earnings for the most disadvantaged control group members ranged from less than $4,000 in Oklahoma City to about $11,500 in Detroit (not shown in Table 7.3). In contrast, earnings levels for the least disadvantaged control group members were at least three times higher in all sites, ranging from about $20,000 in Oklahoma City to nearly $45,000 in Columbus (and more than $40,000 in Detroit).
Examining subgroups by level of disadvantage may also help find more precisely defined groups who are helped most and least by the programs. This may help policymakers and welfare administrators decide whether the more disadvantaged or the less disadvantaged sample members benefit more from employment-focused or education-focused activities. It may also help them determine whether new services are needed for the more disadvantaged or the less disadvantaged sample members. In this sense, results by level of disadvantage are in the spirit of recent work on "profiling."(6)
Impacts for the most disadvantaged. These welfare-to-work programs generally increased earnings for the most disadvantaged, but their effects for this group were not as strong as for either long-term welfare recipients or those who had not worked in the year prior to random assignment. In 10 of the 11 programs, the most disadvantaged program group members earned more than their control group counterparts. Because there were relatively few people in this group, the impacts on earnings were statistically significant in only five of the programs. While several programs were moderately successful, increasing earnings by about $3,000 or more, just as many were not very effective, with impacts on earnings close to zero.
Impacts on welfare benefits were more systematic, with significant reductions in 9 of the 10 programs for which they could be measured. This same pattern was seen for the other subgroups discussed above: more systematic changes in welfare payments than in earnings levels. As a result, the most disadvantaged program group members had lower income than their control group counterparts in 8 of the 10 programs (though the impact was not statistically significant in any of the programs).
These results are somewhat different than the results for the two barriers described above, where earnings impacts were quite strong for those who had not worked in the year prior to random assignment and for long-term welfare recipients. This could imply that it is particularly difficult to assist people with three barriers to work rather than possibly just one. On the other hand, it could indicate an important drawback to this method of counting barriers to work: Not all barriers are equal. In particular, Michalopoulos and Schwartz indicate that earnings gains were generally larger for people with a high school credential than for those who lacked one.(7) Thus, two of the barriers appear to be related to greater effects of the welfare-to-work programs, while a third barrier appears to be related to smaller effects of the programs. In such circumstances, understanding the effects of multiple barriers requires more sophisticated methods than the categorization shown in Table 7.3
Comparing impacts by level of disadvantage. If Friedlander's 1988 findings on pre-FSA programs hold for programs in the NEWWS Evaluation, then Table 7.3should indicate that impacts are larger for the moderately disadvantaged than for the most disadvantaged. It does not. The impacts of the NEWWS programs on earnings were generally more broadly distributed than the impacts of the programs studied by Friedlander. Seven of the 11 programs had significant impacts on earnings for the moderately disadvantaged group, but 5 programs also had significant impacts on earnings for the most disadvantaged group. In addition, in both Grand Rapids programs and the Riverside HCD program earnings impacts were much larger for the most disadvantaged than for the moderately disadvantaged, while in the two Atlanta programs and the Portland program the opposite was true.
In one way, however, results in Table 7.3 are similar to results from the pre-FSA programs studied by Friedlander: The programs had little effect on earnings for the least disadvantaged. Impacts on cumulative earnings were not statistically significant for any of the programs. Moreover, just as many programs left the least disadvantaged program group members with higher earnings than their control group counterparts as left them with lower earnings. Despite the modest effects on earnings, each of the programs reduced cash assistance amounts four of them significantly so although the programs did not have a consistently negative effect on income.
Race and Ethnicity
The final subgroups discussed in this chapter are defined by race and ethnicity. Results are presented for three groups: white, African-American, and Hispanic.(8) If minority sample members are faring much worse under these programs than white sample members, it might be a signal to policymakers and administrators that something is preventing minority sample members from fully participating in or benefiting from the programs. It may mean that the services offered were not enough to overcome additional barriers to employment often faced by minority group members, such as living far from available jobs, having language barriers, and being discriminated against by employers.
Results in Table 7.4 provide little reason to be concerned. In general, impacts on earnings were larger for African-American and Hispanic sample members than for white sample members. In seven of the nine programs where comparisons between white and African-American sample members could be made (Atlanta had too few white sample members to make reliable comparisons), impacts on earnings were larger for African-American sample members. For example, the Grand Rapids LFA program increased earnings by nearly $2,400 over five years for African-American sample members, but had virtually no effect on earnings for white sample members. In all programs except Portland and Detroit, in fact, the average impact on earnings for African-American sample members exceeded the average impact for white sample members by more than $1,100 over five years.
|
Site and Program |
Sample Size | Average Total Earnings in Years 1 to 5 ($) | Average Welfare Payments in Years1 to 5 ($) | Combined Income in Years1 to 5 ($) |
|---|---|---|---|---|
|
White, non-Hispanic |
||||
| Atlanta Labor Force Attachment | n/a | n/a | n/a | n/a |
| Atlanta Human Capital Development | n/a | n/a | n/a | n/a |
| Grand Rapids Labor Force Attachment | 1,470 | -275 | -2,302*** | -3,168*** |
| Grand Rapids Human Capital Development | 1,515 | -394 | -1,569*** | -2,467** |
| Riverside Labor Force Attachment | 3,464 | 1,565* | -2,350*** | -1,426 |
| Lacked high school diploma or basic skills | 1,245 | 2,158* | -2,818*** | -1,148 |
| Riverside Human Capital Development | 1,208 | 623 | -2,482*** | -2,430* |
| Columbus Integrated | 2,161 | 1,762 | -1,489*** | -734 |
| Columbus Traditional | 2,204 | 73 | -829*** | -1,104 |
| Detroit | 481 | 2,794 | -1,129 | 819 |
| Oklahoma City | 5,109 | -241 | n/a | n/a |
| Portland | 2,754 | 6,343*** | -2,971*** | 2,730* |
|
Black, non-Hispanic |
||||
| Atlanta Labor Force Attachment | 2,791 | 2,385*** | -831*** | 1,180 |
| Atlanta Human Capital Development | 2,838 | 1,828** | -718*** | 998 |
| Grand Rapids Labor Force Attachment | 1,214 | 2,367** | -2,462*** | -162 |
| Grand Rapids Human Capital Development | 1,158 | 999 | -1,472*** | -445 |
| Riverside Labor Force Attachment | 1,121 | 3,775** | -2,248*** | 934 |
| Lacked high school diploma or basic skills | 501 | 1,743 | -2,580** | -1,233 |
| Riverside Human Capital Development | 510 | 1,576 | -2,578** | -1,614 |
| Columbus Integrated | 2,414 | 2,528** | -1,528*** | 8 |
| Columbus Traditional | 2,431 | 2,395** | -1,265*** | 390 |
| Detroit | 3,836 | 1,375 | -557** | 500 |
| Oklahoma City | 2,484 | 1,238* | n/a | n/a |
| Portland | 798 | -278 | -1,740* | -1,754 |
|
Hispanic |
||||
| Atlanta Labor Force Attachment | n/a | n/a | n/a | n/a |
| Atlanta Human Capital Development | n/a | n/a | n/a | n/a |
| Grand Rapids Labor Force Attachment | 244 | 4,973* | -4,211*** | -464 |
| Grand Rapids Human Capital Development | 249 | 6,270** | -4,002*** | 999 |
| Riverside Labor Force Attachment | 1,858 | 4,357*** | -3,732*** | -420 |
| Lacked high school diploma or basic skills | 1,210 | 3,639*** | -3,669*** | -981 |
| Riverside Human Capital Development | 1,240 | 3,018*** | -3,536*** | -1,682 |
| Columbus Integrated | n/a | n/a | n/a | n/a |
| Columbus Traditional | n/a | n/a | n/a | n/a |
| Detroit | n/a | n/a | n/a | n/a |
| Oklahoma City | 392 | -546 | n/a | n/a |
| Portland | n/a | n/a | n/a | n/a |
| SOURCES: MDRC calculations from unemployment insurance (UI) earnings records and AFDC records. NOTES: Impacts on earnings were significantly different across subgroups in Grand Rapids LFA and HCD, Riverside LFA and HCD, and Portland. Impacts on AFDC benefits were significantly different across subgroups in Grand Rapids LFA and HCD. N/a = not applicable. |
||||
Despite these differences in the impacts on earnings, the impacts on welfare benefits were fairly similar for African-American and white sample members. In the Grand Rapids LFA program, for example, the impact on welfare benefits was about $2,300 for white sample members and about $2,500 for African-American sample members. As a result, both programs in Riverside and Grand Rapids significantly reduced income for white sample members but not for African-American sample members.
In the five programs with enough Hispanic sample members to allow impacts to be reliably measured, program effects on earnings were generally larger for Hispanic sample members than for white or African-American sample members, although comparisons can be made across the three groups for only the four programs in Grand Rapids and Riverside because most of the sites had too few Hispanic sample members to provide reliable estimates of program impacts. The large earnings gains for Hispanic sample members were accompanied by large welfare reductions.
One exception to the positive results for African-American sample members was in Portland, where earnings increased by more than $6,000 for white sample members, but were little changed for African-American sample members. This result should be interpreted with a great deal of caution, however. The Portland sample used in this report contains only 101 African-American control group members, and the estimated effects for African-American sample members are consequently quite imprecise. In examining impacts through three years, Michalopoulos and Schwartz were able to use the entire Portland control group and found that the program increased earnings for African-American sample members by about $2,000 over three years.(9)
The reasons for the large discrepancy in the Portland effects are unclear. One possible explanation is that African-American control group members had higher rates of job search, employment, and earnings than white control group members, which made it relatively harder to generate employment and earnings gains for African-Americans. According to the Five-Year Client Survey, about 47 percent of African-American control group members reported receiving some sort of job search assistance after entering the study compared with about 32 percent of white control group members. Perhaps as a result of their greater efforts to look for work, African-American control group members worked more often than white control group members (9.2 quarters on average compared with 7.4 quarters) and they earned considerably more than white control group members (about $24,000 over five years compared with less than $20,000, shown in Appendix Table F.4). However, it is unclear why African-American control group members had such high rates of employment.
Perhaps because of the high rates of participation in job search by African-American control group members in Portland, the program effect on job search participation was somewhat lower for African-Americans than for whites (a 22 percentage point increase for African-Americans compared with a 30 percentage point increase for whites in the five years following random assignment). The smaller effect on job search assistance was offset somewhat by larger effects on any education and training for African-American sample members (a 24 percentage point increase for African-Americans compared with an 11 percentage point increase for whites). In addition, the program affected different types of education and training activities for the two groups, with the program's effect on vocational training concentrated among African-American sample members, its effect on basic education concentrated among white sample members, and its effect on post-secondary education about the same for white and African-American sample members. It is important to note that the program increased both job search assistance and education and training for both white and African-American sample members, so these results do not imply that lack of job search assistance or undue reliance on education were the cause of the low earnings gains for African-Americans.
Comparing the LFA and HCD Programs
In comparing the LFA and HCD programs in Atlanta, Grand Rapids, and Riverside, it is difficult to draw strong conclusions about the relative effectiveness of the job-search-first and education-first approaches over a five-year period. Across the 10 subgroups discussed in this chapter, only three impacts on earnings were numerically larger in an HCD program, of 27 comparisons that were made. This suggests that the LFA programs were somewhat more effective than the HCD programs at increasing earnings.
However, differences between the impacts of the two approaches were generally fairly small and generally not statistically significant. Moreover, for no subgroup was the evidence compelling. The largest differences between the two approaches were for the most disadvantaged sample members. In Atlanta, the earnings impact was more than $2,000 greater for the LFA program than for the HCD program ($1,946 compared with -$290; see Table 7.3). In Grand Rapids, the difference was nearly $2,500 ($3,994 compared with $1,566). However, in Riverside, the impact was slightly greater for the HCD program than for the LFA program.
What Has Been Learned?
Overall, the results suggest that these welfare-to-work programs, both education-focused and employment-focused, increased earnings and decreased welfare receipt for a wide range of subgroups. In particular, they suggest that the approach of the Family Support Act of 1988 was reasonably successful: The welfare-to-work programs did generally result in higher earnings for long-term welfare recipients.
On the other hand, for the least disadvantaged (those sample members who have a high school diploma, have recent work experience, and have little prior welfare history) none of the programs increased earnings significantly, and the programs as a whole were just as likely to increase earnings as to reduce them. This may reflect, again, the fact that the programs were forced by the provisions of the Family Support Act to concentrate resources on the most disadvantaged. Alternatively, it could be interpreted as evidence that concentrating services on the least disadvantaged is an inefficient use of resources. Perhaps more advanced training, training that builds on skills already in place, is needed for the least disadvantaged.
Impacts by subgroup bolster evidence from the full sample regarding the most successful programs and program approaches. LFA programs tended to have larger effects on earnings and welfare benefits for most subgroups, although differences were typically fairly small. The Portland program was the most effective for the broadest range of subgroups. Of the seven subgroups compared (not counting those by race and ethnicity), Portland had the largest effect on earnings for six of the subgroups. This may reflect the program's unique use of both job search and education as initial program activities, but it might also reflect the program's use of job development, its experience running job search programs, its willingness to exempt welfare recipients who were perceived to be the hardest-to-employ from welfare-to-work services, or the interaction of these features with the city's robust economy during this period.
1. Michalopoulos and Schwartz, 2001.
2. The number of programs that produced a larger effect for one group than for another follows a binomial distribution. Significance levels were determined using this distribution, under the hypothesis that the programs had equally large effects for all subgroups. For example, if welfare-to-work programs do not affect the earnings of long-term welfare recipients, then the chance that all 11 programs would have had higher earnings for long-term recipient program group members than for long-term recipient control group members is 0.1 percent. Likewise, the chance that 10 or more of the programs would have had higher earnings for long-term recipient program group members than for long-term recipient control group members is 1.17 percent, and the chance that 9 or more of the programs would have had higher earnings for long-term recipient program group members than for long-term recipient control group members is 6.54 percent.
3. Michalopoulos and Berlin, 2001.
4. Michalopoulos and Schwartz, 2001.
5. Friedlander, 1988.
6. For example, Eberts, 1997; and Rangarajan, Schochet, and Chu, 1998.
7. Michalopoulos and Schwartz, 2001.
8. A few sample members were not part of any of these three groups. Because only Oklahoma City and Riverside had more than a handful of these people, results for them are not shown in Table 7.4.
9. Michalopoulos and Schwartz, 2001.
Impacts on Health Care Coverage
The programs in the NEWWS Evaluation were designed to move recipients from welfare to work, and the earlier chapters showed that many were successful in doing this. However, moving from welfare to work leads to a loss in health coverage for some recipients if they cannot replace their lost Medicaid with private coverage. Many low-wage workers do not have employer-provided coverage, either because their employers do not offer it or they cannot afford the high premiums.
This chapter examines how the NEWWS programs affected the health coverage of respondents and their children. Data on health coverage from the Five-Year Client Survey are available for 7 of the 11 programs. Health coverage status is an important aspect of family well-being, because it affects families' access to care and the quality of care they receive. Children who are uninsured, for example, are much less likely to see doctors than their insured counterparts, are less likely to have preventative care, and are more likely to have unmet health care needs.(1) Increasingly, families are without coverage: In 1997, for example, nearly half of all working poor parents were uninsured.(2)
Key Findings
- The majority of control group respondents, ranging from 72 percent in Atlanta to 81 percent in Portland, had health coverage at the end of year 5. Most control group members had coverage through public sources, such as Medicaid or other public programs, rather than employer-provided or other private sources. Children in the control group were somewhat more likely to have coverage than adults, owing to the greater number of public health programs available to low-income children. Coverage rates for children ranged from 80 percent in Portland to 85 percent in Atlanta.
- Employment did not guarantee coverage: About 20 to 30 percent of individuals in the control groups who were working did not have health coverage. Among those who were working and did have coverage, between 40 and 60 percent obtained it from their employers.
- Between 20 and 30 percent of adults in the control groups did not have health care coverage at the end of year 5. This indicates that many of those who left welfare for work were not able to replace Medicaid with private coverage once their transitional benefits expired. Control group respondents without coverage were just as likely as those with coverage to have been working at the end of the five-year follow-up, but they were much less likely to have been offered coverage through their jobs.
- None of the programs had a statistically significant impact on coverage for adults or children. There is some evidence that because the programs increased employment, they may have led to a shift from public to private coverage. In all seven programs, program group respondents were more likely than those in the control group to have private health care coverage at the end of year 5 and less likely (in six of seven programs) to have publicly funded coverage, although these differences were not statistically significant.
- Not all families eligible for Transitional Medicaid received it. Only about 60 to 75 percent of control group members who left welfare for work received Transitional Medicaid during the five-year follow-up period. Most of the programs increased the use of Transitional Medicaid, although only the impacts in Riverside LFA and HCD were statistically significant. The effect in these two programs is due to the fact that they increased the number of individuals who were eligible (those who left welfare for work) and slightly increased use among those who were eligible.
Expected Effects
At random assignment, all sample members and their children had health care coverage because they were receiving AFDC and thus automatically covered under Medicaid. In general, Medicaid receipt should fall over time for both the program and control groups since most of them left welfare during the five-year follow-up period. By the last quarter, for example, welfare receipt ranged from 20 to 40 percent across the sites. Health care coverage in general should also fall over time if some sample members cannot replace their lost Medicaid coverage. Coverage might also fall more rapidly for parents than for their children. Most children under age 18 are now eligible for Medicaid if their family's income is less than 100 percent of the poverty level. They may also be eligible, depending on the state in which they live, for coverage through the Children's Health Insurance Program (CHIP). As a result, it is not uncommon for coverage status to vary within the same family. The mother might not have coverage or might be covered through her employer, while her children might be covered through CHIP or Medicaid.
The primary way that the programs might affect health coverage is through their effects on welfare receipt and employment. When people leave welfare for work, they run the risk of losing coverage because they will either immediately or eventually lose their public coverage and cannot often find private coverage to replace it. Transitional Medicaid is the main source of public coverage and is available to families for up to one year after leaving welfare.(3) However, recent research finds that its use has been slow to take hold: According to one study, over 600,000 recipients lost Medicaid coverage when their welfare case was closed, even though the majority still met Medicaid eligibility standards.(4) There are a number of possible reasons why individuals eligible for Medicaid do not receive it. One reason is that recently employed recipients, believing they are no longer eligible for welfare, often fail to respond to eligibility verification notices they receive from welfare caseworkers. Caseworkers typically close the cases of those who do not respond, which includes not only welfare but Medicaid and Food Stamps.(5)
Once Transitional Medicaid expires the family must find other insurance, mostly from private sources. This has become increasingly difficult for many low-income workers. There has been a decrease over the past decade in the number of workers with employer-sponsored coverage, especially among those in low-wage jobs.(6) With rising health care costs, many employers have been faced with the choice of dropping coverage for their employees or passing along the costs to them in the form of higher premiums. As a result, many former welfare recipients are either not offered insurance or cannot afford it.
In terms of program impacts, increases in employment and reductions in welfare receipt might lead to reductions in coverage, particularly in those sites with the largest reductions in welfare receipt. Because better jobs are more likely to come with employer-provided insurance, programs that may have encouraged recipients to take lower-quality jobs than they would have otherwise particularly programs with an emphasis on quick employment may have decreased coverage. In contrast, if program group members moved into better jobs than control group members over time, they may have increased coverage. Average wages, one measure of job quality, were higher for program group members in some sites (Atlanta LFA, Riverside HCD, and Riverside LFA all significantly increased hourly wages). Program impacts on earnings might also affect coverage, particularly coverage for children, since families with higher earnings may be less likely to qualify for public insurance programs.
The programs might also affect coverage in ways that are not related to their impacts on employment or welfare receipt. For example, program group members have more contact than control group members with program staff, making it more likely that they will be made aware of and receive help in obtaining Transitional Medicaid or other kinds of public coverage. Programs with a strictly enforced participation mandate might also affect coverage, probably reducing it, if they lead many individuals to leave welfare before they have found jobs.
Health Care Coverage at the End of Year 5
Data on health coverage come from the Five-Year Client Survey. Since coverage is measured at only one point, the month before the five-year survey, it is not possible to measure the continuity of coverage. Respondents were considered covered if they reported receiving Medicaid, if they were working and had accepted their employer's health insurance plan, or if they reported receiving coverage from another private source. If they were receiving welfare or SSI, they were assumed to be covered through Medicaid.
Employment and Employer-Provided Coverage
As discussed in Chapter 4, UI records data show that most of the programs began increasing employment within a year or two of random assignment, and the biggest effects were in Riverside LFA and HCD, Grand Rapids LFA, and Portland. The employment impacts also lessened over time. By the end of year 5, only Riverside LFA and HCD and Portland continued to show significant positive impacts. This pattern also holds with the survey data. The upper panel of Table 8.1 shows impacts on employment at the time of the five-year survey. In Riverside LFA, Riverside HCD, and Portland, more individuals in the program group than the control group were working at the five-year point, although the difference in Portland is not statistically significant.
The middle panel of Table 8.1 shows the percentage of program and control group members who were employed with health care coverage, either public or private. A comparison of this panel with the upper panel shows that a significant proportion of those working (ranging from 20 to 30 percent across sites) did not have health coverage. For example, 54 percent of control group members in Atlanta were employed, but only 37 percent were employed and had coverage, meaning that about 70 percent of those working had coverage. Control group rates were highest in Riverside and Portland. In Riverside, the high welfare benefit levels meant that more welfare recipients could work and still remain eligible for some welfare and, therefore, Medicaid.(7) In Portland, the high coverage rate may reflect the existence of the state's public health program the Oregon Health Plan (OHP).(8)
The lower panel of Table 8.1 shows the percentage of program and control group members who were employed with employer-provided coverage. Between 15 and 30 percent of control group members were working and had employer-provided coverage. A comparison of these numbers with the upper and middle panels highlights several points. First, only about one-third to one-half of those who were employed had employer-provided coverage. In Atlanta, for example, 54 percent of the control group worked, but only 19 percent of the control group worked and had employer-provided coverage, meaning that only 35 percent of workers had coverage through their jobs. Second, 40 to 60 percent of workers with coverage had employer-provided coverage (compare the middle and lower panels), showing that other programs were an important source of coverage among low-wage workers. Other sources of coverage, for example, are Transitional Medicaid and coverage from a spouse (as shown in Chapter 9, from 10 to 20 percent of respondents were married at the five-year point). The percentage with employer-provided coverage is especially low in Riverside, where 40 percent of controls were working and had coverage, but only 15 percent had employer-provided coverage, meaning that 38 percent of insured workers had coverage through their jobs.
One reason for the relatively low rates of employer-provided coverage is that some respondents were offered but declined to enroll in their employer's plan. In Riverside and Atlanta, about 70 percent of those offered coverage accepted it compared with about 80 percent of those in Grand Rapids and Portland. The low employer-provided coverage rates in Riverside may reflect the fact that relatively more workers were still eligible for welfare and Medicaid. Individuals who have Medicaid are probably more likely to turn down an employer's offer of coverage.
|
Site and Program |
Sample Size | Program Group (%) | Control Group (%) | Difference (Impact) |
Percentage Change (%) |
|---|---|---|---|---|---|
|
Employed at interview |
|||||
| Atlanta Labor Force Attachment | 1,071 | 57.2 | 54.3 | 2.9 | 5.4 |
| Atlanta Human Capital Development | 1,146 | 52.9 | 54.3 | -1.4 | -2.5 |
| Grand Rapids Labor Force Attachment | 1,097 | 65.1 | 67.7 | -2.6 | -3.8 |
| Grand Rapids Human Capital Development | 1,109 | 66.2 | 67.7 | -1.5 | -2.2 |
| Riverside Labor Force Attachment | 1,219 | 55.0 | 48.9 | 6.1** | 12.4 |
| Lacked high school diploma or basic skills | 657 | 48.9 | 43.0 | 5.9 | 13.6 |
| Riverside Human Capital Development | 778 | 51.1 | 43.0 | 8.0** | 18.7 |
| Portland | 504 | 61.7 | 58.3 | 3.4 | 5.9 |
|
Employed with health care coverage at interview |
|||||
| Atlanta Labor Force Attachment | 1,071 | 39.1 | 36.9 | 2.2 | 6.0 |
| Atlanta Human Capital Development | 1,146 | 38.0 | 36.9 | 1.1 | 3.1 |
| Grand Rapids Labor Force Attachment | 1,097 | 49.4 | 51.8 | -2.4 | -4.6 |
| Grand Rapids Human Capital Development | 1,109 | 51.9 | 51.8 | 0.1 | 0.2 |
| Riverside Labor Force Attachment | 1,219 | 40.9 | 39.6 | 1.4 | 3.4 |
| Lacked high school diploma or basic skills | 657 | 38.5 | 33.8 | 4.6 | 13.7 |
| Riverside Human Capital Development | 778 | 40.6 | 33.8 | 6.8* | 20.0 |
| Portland | 504 | 47.8 | 47.7 | 0.1 | 0.2 |
|
Employed with employer-provided health care coverage at interview |
|||||
| Atlanta Labor Force Attachment | 1,071 | 22.2 | 19.0 | 3.2 | 16.6 |
| Atlanta Human Capital Development | 1,146 | 21.2 | 19.0 | 2.2 | 11.5 |
| Grand Rapids Labor Force Attachment | 1,097 | 30.5 | 30.0 | 0.5 | 1.7 |
| Grand Rapids Human Capital Development | 1,109 | 30.4 | 30.0 | 0.4 | 1.5 |
| Riverside Labor Force Attachment | 1,219 | 17.5 | 15.0 | 2.5 | 16.6 |
| Lacked high school diploma or basic skills | 657 | 12.0 | 10.9 | 1.1 | 9.8 |
| Riverside Human Capital Development | 778 | 14.8 | 10.9 | 3.9 | 35.5 |
| Portland | 504 | 31.6 | 27.3 | 4.4 | 16.0 |
| SOURCE: MDRC calculations from the Five-Year Client Survey. NOTES: See Appendix A.2. |
|||||
Only Riverside HCD produced a statistically significant increase in the number of respondents who were employed and had coverage (of any type) at the end of year 5. None of the programs had a statistically significant impact on being employed and having employer-provided coverage. However, the impacts on employer-provided coverage are similar in size to the impacts on employment in Atlanta, Grand Rapids, and Portland, although somewhat smaller in Riverside. This suggests that the program group members who were encouraged to work had jobs at the end of year 5 that were at least as likely to have insurance as the jobs held by control group members.
Public Versus Private Coverage for Respondents
Table 8.2 presents coverage more generally, not as it relates to employment status. Respondents were considered to be covered by public health insurance if they reported receiving Medicaid in the month prior to interview or if they were receiving welfare or SSI. They were considered to be covered by private health care if they had accepted their employer's plan or had coverage from another private source.
|
Site and Program |
Sample Size | Program Group (%) | Control Group (%) | Difference (Impact) | Percentage Change (%) |
|---|---|---|---|---|---|
|
Has health care coverage |
|||||
| Atlanta Labor Force Attachment | 1,071 | 71.0 | 72.4 | -1.4 | -1.9 |
| Atlanta Human Capital Development | 1,146 | 74.0 | 72.4 | 1.6 | 2.2 |
| Grand Rapids Labor Force Attachment | 1,097 | 75.1 | 77.7 | -2.6 | -3.3 |
| Grand Rapids Human Capital Development | 1,109 | 77.8 | 77.7 | 0.1 | 0.2 |
| Riverside Labor Force Attachment | 1,219 | 78.3 | 80.3 | -2.0 | -2.5 |
| Lacked high school diploma or basic skills | 657 | 82.7 | 80.0 | 2.7 | 3.4 |
| Riverside Human Capital Development | 778 | 80.3 | 80.0 | 0.3 | 0.4 |
| Portland | 504 | 74.7 | 80.6 | -6.0 | -7.4 |
|
Has public health care coverage |
|||||
| Atlanta Labor Force Attachment | 1,071 | 46.5 | 51.7 | -5.2* | -10.1 |
| Atlanta Human Capital Development | 1,146 | 50.3 | 51.7 | -1.4 | -2.7 |
| Grand Rapids Labor Force Attachment | 1,097 | 41.2 | 42.5 | -1.2 | -2.9 |
| Grand Rapids Human Capital Development | 1,109 | 43.2 | 42.5 | 0.7 | 1.7 |
| Riverside Labor Force Attachment | 1,219 | 55.8 | 59.4 | -3.6 | -6.1 |
| Lacked high school diploma or basic skills | 657 | 65.3 | 66.4 | -1.1 | -1.7 |
| Riverside Human Capital Development | 778 | 63.4 | 66.4 | -3.1 | -4.6 |
| Portland | 504 | 43.2 | 47.0 | -3.7 | -7.9 |
|
Has private health care coverage |
|||||
| Atlanta Labor Force Attachment | 1,071 | 28.6 | 24.1 | 4.5* | 18.5 |
| Atlanta Human Capital Development | 1,146 | 27.4 | 24.1 | 3.3 | 13.5 |
| Grand Rapids Labor Force Attachment | 1,097 | 41.8 | 40.7 | 1.1 | 2.7 |
| Grand Rapids Human Capital Development | 1,109 | 41.3 | 40.7 | 0.5 | 1.3 |
| Riverside Labor Force Attachment | 1,219 | 26.9 | 25.7 | 1.2 | 4.7 |
| Lacked high school diploma or basic skills | 657 | 20.9 | 18.0 | 2.9 | 16.4 |
| Riverside Human Capital Development | 778 | 23.9 | 18.0 | 5.9* | 33.1 |
| Portland | 504 | 39.0 | 37.8 | 1.2 | 3.2 |
| SOURCE: MDRC calculations from the Five-Year Client Survey. NOTES: See Appendix A.2. |
|||||
As the upper panel of Table 8.2 shows, coverage levels for the control group ranged from 72 percent in Atlanta to 81 percent in Portland, meaning that 20 to 30 percent of respondents were uninsured at the end of year 5. The percentage uninsured is higher than it is for the nation as a whole but lower than it is for low-income individuals. A recent study reported that 16 percent of all adults with children were uninsured; among those with incomes below poverty, 42 percent were uninsured.(9) A later section in this chapter will look more closely at the respondents who no longer had coverage at the time of the survey.
The middle and lower panels of Table 8.2 show that the majority of the coverage is from public sources. In Portland, for example, 80 percent of control group members had any coverage, 47 percent had public coverage, and 38 percent had private coverage. This extent of public coverage makes sense, considering that 20 to 40 percent of sample members were still on welfare at the five-year point and that those who left welfare shortly before that were probably still receiving Transitional Medicaid (data on the take-up of Transitional Medicaid are shown in a later section). In Riverside, for example, nearly 40 percent of control group members were still receiving welfare at the end of year 5, which explains why the extent of public coverage is relatively high in this site. In Portland, the extent of public coverage is also related to Oregon's OHP, since about 20 percent of control group members were receiving welfare at the five-year point.
None of the programs produced a statistically significant impact on coverage (upper panel of Table 8.2). Levels of coverage were about the same for program group members as for control group members. This result is encouraging since the expectation was that many welfare recipients would lose insurance as they went from welfare to work.
The impacts on types of coverage (middle and lower panels of Table 8.2) show that the programs may have led to a shift from public to private sources. Although few of the impacts are statistically significant, the general pattern is that program group members were less likely to have public coverage and more likely to have private coverage than control group members. For the Atlanta LFA program, this shift in coverage is statistically significant; the program reduced public health care coverage by 5 percentage points and increased private coverage by 5 percentage points. This is consistent with the program's moving more welfare recipients to work.
Coverage for Children
Table 8.3 presents coverage for respondents' children. Respondents reported the coverage status of each of their children at the end of year 5. This analysis is limited to respondents' dependent children aged 18 or younger at the five-year interview date.(10) Children were considered to be covered if respondents reported that children had coverage from Medicaid or from a private insurer or if respondents reported receiving welfare or SSI benefits. Although data were collected for each child, the following analysis examines whether all children in the family were covered.(11)
|
Site and Program |
Sample Size | Program Group (%) | Control Group (%) | Difference (Impact) | Percentage Change (%) |
|---|---|---|---|---|---|
|
All dependent children have health care coverage |
|||||
| Atlanta Labor Force Attachment | 974 | 85.1 | 84.5 | 0.6 | 0.7 |
| Atlanta Human Capital Development | 1,057 | 83.4 | 84.5 | -1.1 | -1.3 |
| Grand Rapids Labor Force Attachment | 1,005 | 78.8 | 81.8 | -3.0 | -3.6 |
| Grand Rapids Human Capital Development | 1,026 | 79.3 | 81.8 | -2.5 | -3.1 |
| Riverside Labor Force Attachment | 1,120 | 81.9 | 83.2 | -1.3 | -1.6 |
| Lacked high school diploma or basic skills | 614 | 83.8 | 82.1 | 1.8 | 2.2 |
| Riverside Human Capital Development | 743 | 85.2 | 82.1 | 3.2 | 3.8 |
| Portland | 451 | 75.5 | 80.2 | -4.7 | -5.8 |
|
All dependent children have public health care coverage |
|||||
| Atlanta Labor Force Attachment | 974 | 63.7 | 69.1 | -5.3* | -7.7 |
| Atlanta Human Capital Development | 1,057 | 66.6 | 69.1 | -2.5 | -3.6 |
| Grand Rapids Labor Force Attachment | 1,005 | 49.7 | 51.3 | -1.6 | -3.2 |
| Grand Rapids Human Capital Development | 1,026 | 52.1 | 51.3 | 0.8 | 1.5 |
| Riverside Labor Force Attachment | 1,120 | 57.6 | 63.9 | -6.2** | -9.8 |
| Lacked high school diploma or basic skills | 614 | 66.0 | 70.9 | -4.9 | -6.9 |
| Riverside Human Capital Development | 743 | 67.8 | 70.9 | -3.1 | -4.4 |
| Portland | 451 | 44.1 | 52.1 | -8.0 | -15.3 |
|
All dependent children have private health care coverage |
|||||
| Atlanta Labor Force Attachment | 974 | 19.8 | 16.0 | 3.9* | 24.2 |
| Atlanta Human Capital Development | 1,057 | 16.0 | 16.0 | 0.0 | 0.1 |
| Grand Rapids Labor Force Attachment | 1,005 | 31.5 | 32.8 | -1.3 | -4.0 |
| Grand Rapids Human Capital Development | 1,026 | 29.2 | 32.8 | -3.6 | -10.8 |
| Riverside Labor Force Attachment | 1,120 | 22.9 | 19.9 | 3.1 | 15.5 |
| Lacked high school diploma or basic skills | 614 | 16.4 | 12.4 | 4.0 | 32.3 |
| Riverside Human Capital Development | 743 | 19.1 | 12.4 | 6.7** | 54.2 |
| Portland | 451 | 30.2 | 28.6 | 1.7 | 5.8 |
| SOURCE: MDRC calculations from the Five-Year Client Survey. NOTES: See Appendix A.2. |
|||||
A comparison of Table 8.3 with Table 8.2 shows that children were somewhat more likely than adults to have health care coverage at the end of year 5. In Grand Rapids, for example, 82 percent of control group children were covered compared with 78 percent of respondents. This is perhaps not surprising given the range of public programs that specifically cover low-income children. Nonetheless, the rates of noncoverage (15 to 20 percent) suggest that these programs are not serving all eligible children.
The greater number of public programs for children also is reflected in the fact that children were more likely than adults to be covered through public sources and less likely through private sources. Only 33 percent of control group children in Grand Rapids, for example, had private insurance compared with 41 percent of adults. Employers may not have offered to cover children or the adults may have turned down this coverage if it was too expensive.
Finally, none of the programs had statistically significant effects on coverage for children. They did lead to a slight shift from public to private sources, which mirrors the effects found for adults. It is encouraging that the programs, by leading more families from welfare to work, did not negatively affect children's health care coverage.
Transitional Medicaid Use During the Five-Year Follow-Up
For many welfare recipients, the potential loss of health coverage can create a disincentive to leave welfare and go to work. Transitional Medicaid reduces this disincentive by allowing working individuals to continue coverage, for themselves and their children, for up to one year after leaving welfare.(12) Receipt of this benefit is not automatic, however, and eligible recipients must request it and receive caseworker approval.(13) Data on the receipt of Transitional Medicaid are available from the survey: Respondents were asked whether they had ever received this benefit during the five-year follow-up period.
Table 8.4 shows the percentage of respondents who were eligible for and used Transitional Medicaid. The upper panel shows the percentage of program and control group members who worked and were off welfare at some point during the follow-up period and indicates the number of respondents who would have been eligible to receive Transitional Medicaid. Between 56 and 68 percent of control group members worked and left welfare during the follow-up period in Atlanta, Grand Rapids, and Portland. A much smaller percentage of controls (40 percent) were eligible for Transitional Medicaid in Riverside, probably because fewer in this site worked and when they did many were still eligible for welfare. Both programs in Riverside produced a significant increase in the percentage working and off welfare.
|
Site and Program |
Sample Size | Program Group (%) | Control Group (%) | Difference (Impact) | Percentage Change (%) |
|---|---|---|---|---|---|
|
Ever employed and off welfare during follow-up |
|||||
| Atlanta Labor Force Attachment | 1,071 | 59.7 | 56.2 | 3.5 | 6.2 |
| Atlanta Human Capital Development | 1,146 | 59.6 | 56.2 | 3.3 | 5.9 |
| Grand Rapids Labor Force Attachment | 1,097 | 71.3 | 67.7 | 3.6 | 5.4 |
| Grand Rapids Human Capital Development | 1,109 | 67.9 | 67.7 | 0.3 | 0.4 |
| Riverside Labor Force Attachment | 1,219 | 46.9 | 39.6 | 7.2*** | 18.3 |
| Lacked high school diploma or basic skills | 657 | 37.7 | 28.3 | 9.4*** | 33.3 |
| Riverside Human Capital Development | 778 | 36.6 | 28.3 | 8.2** | 29.1 |
| Portland | 504 | 67.8 | 66.8 | 1.0 | 1.6 |
|
Ever covered by Transitional Medicaid during follow-up |
|||||
| Atlanta Labor Force Attachment | 1,071 | 42.0 | 38.1 | 3.8 | 10.1 |
| Atlanta Human Capital Development | 1,146 | 42.7 | 38.1 | 4.6 | 12.1 |
| Grand Rapids Labor Force Attachment | 1,097 | 53.5 | 50.8 | 2.7 | 5.3 |
| Grand Rapids Human Capital Development | 1,109 | 50.2 | 50.8 | -0.6 | -1.2 |
| Riverside Labor Force Attachment | 1,219 | 32.3 | 25.8 | 6.5** | 25.3 |
| Lacked high school diploma or basic skills | 657 | 25.6 | 17.9 | 7.7** | 42.8 |
| Riverside Human Capital Development | 778 | 26.2 | 17.9 | 8.2** | 45.9 |
| Portland | 504 | 55.5 | 50.8 | 4.6 | 9.1 |
|
For those ever off welfare and employed,ever covered by Transitional Medicaid |
|||||
| Atlanta Labor Force Attachment | 70.3 | 67.8 | 2.5 | 3.6 | |
| Atlanta Human Capital Development | 71.7 | 67.8 | 4.0 | 5.8 | |
| Grand Rapids Labor Force Attachment | 75.0 | 75.1 | -0.1 | -0.1 | |
| Grand Rapids Human Capital Development | 73.9 | 75.1 | -1.2 | -1.6 | |
| Riverside Labor Force Attachment | 68.8 | 65.0 | 3.9 | 6.0 | |
| Lacked high school diploma or basic skills | 67.9 | 63.3 | 4.6 | 7.2 | |
| Riverside Human Capital Development | 71.6 | 63.3 | 8.2 | 13.0 | |
| Portland | 81.8 | 76.2 | 5.7 | 7.4 | |
|
SOURCE: MDRC calculations from the Five-Year Client Survey. |
|||||
The middle panel of Table 8.4 shows that not all who were eligible received Transitional Medicaid. This can be seen more easily in the lower panel, which shows receipt of Transitional Medicaid by those who were eligible (this is a nonexperimental comparison). Data for the control groups show that about 65 to 76 percent of eligible respondents received this benefit. In Atlanta, for example, 56 percent of control group members were eligible for Transitional Medicaid and 38 percent received it, meaning that only 68 percent of eligible respondents received it. Data from the two-year survey (not shown) indicate that the use of this benefit increased over time, probably as more families became eligible. At the two-year point, between 10 and 25 percent of control group members reported having used Transitional Medicaid compared with a much higher percentage at the end of year 5.
The programs generally increased Transitional Medicaid use although only Riverside LFA and Riverside HCD produced statistically significant increases because they increased the percentage who were eligible to receive it, but also because they increased its use among those eligible. The similarity of the impacts on the number of respondents who were employed and off welfare and on the number who used Transitional Medicaid suggests that most of those individuals who were induced by the program to work and leave welfare did, in fact, receive Transitional Medicaid at some point. This is also reflected in the nonexperimental numbers in the lower panel of Table 8.4. Of program group members who were eligible in Portland, for example, 82 percent received the benefit compared with only 76 percent of control group members. This finding is consistent with one of the hypotheses raised earlier that more involvement by caseworkers would lead to greater use of available benefits.
Loss of Coverage by the End of Year 5
Section III of this chapter showed that 20 to 30 percent of respondents no longer had health coverage at the end of year 5. Were these families less likely to work, did they work in lower-quality jobs, or did they earn too much to qualify for public programs? Although an extensive analysis of the uninsured is beyond the scope of the chapter, this section presents a comparison of adults who had coverage and those who did not have coverage at the end of the five-year follow-up.
Figure 8.1 presents coverage status for respondents at the end of year 2 and at the end of year 5.(14) The figure shows that between 70 and 80 percent of respondents had coverage at the end of year 5. Also, among those who were uninsured at that point, most had lost coverage after year 2.
Figure 8.1
Health Care Coverage for Respondents Over Five Years
SOURCES: MDRC calculations from the Two-Year Client Survey and Five-Year Client Survey.
NOTES: See Appendix A.2.
Table 8.5 compares uninsured and insured control group members. The first two rows show that the uninsured were equally if not more likely to have been working at the end of the five-year follow-up than the insured, but they were much less likely to have been offered coverage by their employers. In Grand Rapids, for example, 69 percent of the uninsured were working at the end of the follow-up, but only 20 percent were working and were offered employer-provided coverage. In contrast, 66 percent of insured control group members were working at the end of the follow-up and 43 percent of them were offered coverage through their jobs.
| Site | Respondent Covered at End of Year 5 | Respondent Not Covered at End of Year 5 |
|---|---|---|
|
Atlanta |
||
| Employed at end of year 5 (%) | 54.4 | 67.8 |
| Employed and offered coverage from employer (%) | 36.5 | 19.9 |
| Earnings in last quarter ($) | 1,729 | 1,597 |
| Ever received Transitional Medicaid (%) | 39.4 | 50.7 |
| Covered at end of year 2 (%) | 92.1 | 83.8 |
| All children covered at end of year 5 (%) | 95.5 | 54.3 |
| Sample size | 406 | 146 |
|
Grand Rapids |
||
| Employed at end of year 5 (%) | 65.7 | 68.9 |
| Employed and offered coverage from employer (%) | 42.7 | 19.7 |
| Earnings in last quarter ($) | 2076.5 | 1364.6 |
| Ever received Transitional Medicaid (%) | 48.2 | 54.1 |
| Covered at end of year 2 (%) | 89.2 | 77.7 |
| All children covered at end of year 5 (%) | 90.9 | 48.2 |
| Sample size | 440 | 122 |
|
Riverside |
||
| Employed at end of year 5 (%) | 49.4 | 47.5 |
| Employed and offered coverage from employer (%) | 25.4 | 9.2 |
| Earnings in last quarter ($) | 1,181 | 1058.9 |
| Ever received Transitional Medicaid (%) | 26.4 | 22.7 |
| Covered at end of year 2 (%) | 92.2 | 76.5 |
| All children covered at end of year 5 (%) | 94.2 | 32.5 |
| Sample size | 579 | 141 |
|
Portland |
||
| Employed at end of year 5 (%) | 60.8 | 59.5 |
| Employed and offered coverage from employer (%) | 43.1 | 9.5 |
| Earnings in last quarter ($) | 1745.0 | 1165.1 |
| Ever received Transitional Medicaid (%) | 50.3 | 50 |
| Covered at end of year 2 (%) | 92.1 | 85 |
| All children covered at end of year 5 (%) | 86.5 | 43.2 |
| Sample size | 181 | 42 |
| SOURCE: MDRC calculations from the Five-Year Client Survey. NOTES: See Appendix A.2. |
||
The uninsured on average earned less than the insured in the last quarter of follow-up. This difference is consistent with the lower rate of employer-provided coverage and suggests that the uninsured may have been working in lower-quality jobs, including more part-time work. The lower earnings also suggest that the lack of coverage is not due to the fact that these workers were earning too much to qualify for public programs serving low-income families. In Portland, for example, average earnings in the last quarter of year 5 were $1,165 for the uninsured, which is less than $5,000 on an annual basis. The OHP provides coverage to families who are not eligible for Medicaid but have incomes below the poverty line. Although coverage rates were generally higher in Portland than in the other sites (see Table 8.2), many eligible people probably did not receive benefits. The low earnings also suggest that the children of uninsured respondents were probably eligible for Medicaid or other public health programs. Nonetheless, only between one-third and one-half of them were covered. In Riverside, for example, only 33 percent of uninsured adults had coverage for their children.
Finally, the uninsured were at least as likely as the insured to have received Transitional Medicaid during the follow-up period. This suggests that they were not more likely to immediately lose coverage when they made the transition from welfare to work, but that they were unable to replace Medicaid when their transitional benefits expired.
Conclusions
A potential effect of the NEWWS programs was the loss of health coverage for participants as they moved from welfare to work. Many low-wage workers either are not offered or do not accept employer-provided insurance, making it difficult to replace their lost Medicaid with private coverage. It is encouraging that the programs did not lead to a loss in coverage for the adults or their children.
As expected, levels of health care coverage decreased for both program and control group members over time, though a large majority of respondents and their children were still insured at the end of the five-year follow-up. Since the evaluation started, states have continued to address the issue of coverage for low-income families by creating or expanding public programs. These additional benefits were not a program component and were available to members of both research groups. The findings presented in this chapter indicate that the sites were generally successful in continuing to provide coverage for families who left welfare. Nonetheless, 20 to 30 percent of adults were not covered, suggesting that states should continue in their efforts to reach the uninsured. Because access to health care has been expanding, the findings presented here on the effects of the programs might have been different had the evaluation taken place in more recent years.
1. Mullahy and Wolfe, 2000.
2. Guyer and Mann, 1999.
3. Kaplan, 1997.
4. Families USA Foundation, 1999.
5. Quint and Widom, 2001.
6. Farber and Levy, 2000.
7. In Riverside, about 16 percent of control group respondents (and one-third of those employed) combined work and welfare at the end of year 5, compared with less than 10 percent of control group members in the other three sites.
8. The OHP provides publicly financed health care coverage to residents of the state of Oregon who do not qualify for Medicaid but whose income is below the poverty level.
9. See Holahan and Brennan, 2000.
10. About 8 percent of respondents were excluded from this analysis either because none of their children was 18 or under at interview or because they reported having no children.
11. In very few families between 3 and 7 percent of control group families were only some (but not all) of the children in the family covered.
12. In 1998 the State of California increased Transitional Medicaid coverage for up to two years. This change would have affected only the sample members in Riverside who entered the program near the end of the random assignment period.
13. Recipients who find jobs and stop communicating with caseworkers might lose transitional coverage, since caseworkers might close their cases.
14. For more detailed information about health care coverage at the end of year 2, see Freedman et al., 2000a, Chapter 8.
Impacts on Household and Personal Circumstances
This chapter examines the effects of employment- and education-focused programs on a variety of outcomes meant to capture the household and personal circumstances of families in the NEWWS Evaluation, such as sample members' household composition, relationships with other household members and with people outside the household (such as colleagues at work), and housing status.
Policymakers and researchers have increasingly come to recognize that welfare recipients' household and personal circumstances may influence the long-term impacts of welfare-to-work programs as well as mediate the impacts of these programs on children. For example, as welfare recipients go to work or increase their hours of employment, a spouse, partner, or extended family member may provide important support, particularly by caring for young children. The financial contribution of other members of the family or household may also play an essential role in moving a family out of poverty and into long-term self-sufficiencyВ particularly since, as demonstrated in earlier chapters, these programs do not necessarily increase respondents' total income. The quality of personal and other relationships and housing status can also influence employment and other economic outcomes. For instance, welfare recipients who must contend with verbal or physical threats or who are worried about leaving their children alone at home in an unsafe neighborhood may have difficulty getting a job and keeping it.
At the two-year follow-up, few of the 11 welfare-to-work programs evaluated were found to have effects on marriage or additional births (that is, the number of children born since random assignment).(1) This chapter examines whether these programs produced impacts on these and related outcomes five years after study entry. In addition, for six programs in three sites, it examines program impacts on measures not examined at the two-year pointВ specifically, measures of physical and nonphysical employment-related and domestic abuse by intimate partners or others.
Earlier chapters in this report presented impacts on employment, earnings, welfare receipt, and income for the full sample in the evaluation. This chapter examines outcomes and impacts for the Five-Year Client Survey sample, which included sample members in Atlanta, Grand Rapids, Riverside, and Portland (for details, see Chapter 2). In general, as discussed in Appendix I, the pattern of impacts for the client survey sample and the administrative records sample were similar for the LFA and HCD programs in Atlanta and Grand Rapids. The Riverside LFA program produced larger reductions in welfare and, consequently, larger losses in income for the client survey sample than for the full sample, and the Riverside HCD program produced larger increases in earnings, and, consequently, smaller losses in income for the client survey sample than for the full sample. The effects of the Portland program were also much less positive for the client survey sample than for the full sample. Thus, generalizations concerning how the Portland program effects on economic outcomes, especially those documented in earlier chapters, may have affected the outcomes described in this chapter should be made with considerable caution. Furthermore, it is important to remember that many of the family outcomes examined in this chapter were measured at the time of the five-year survey or during the last year of follow-up, by which time the control group embargo had been lifted in Atlanta and Grand Rapids.
The outcomes presented in this chapter depict important aspects of sample members' livesВ as a representative subsample of welfare recipientsВ and, thus, are noteworthy whether or not program impacts occurred. For this reason, hypotheses are followed by a descriptive analysis of selected outcomes that are of particular interest in terms of welfare and employment policy: marriage, living with unrelated adults and reports of domestic abuse. The chapter discusses program impacts and, briefly, links between effects on these aspects of sample members' lives and program practices and program effects on employment.
Key Findings
These welfare-to-work programs generally had few effects on household and personal circumstances. The lack of dramatic changes in outcomes such as marriage and fertility are not surprising since these programs were structured primarily to alter employment behavior rather than aimed at decreasing additional births or affecting marriage. Even so, the lives of program participants were affected in ways that are reassuringly positive: Some program group members moved to get better housing, one program increased home ownership, and nearly all LFA and HCD programs reduced reports of physical abuse. Furthermore, there were more effects on these latter outcomes than would be expected by chance. All program effects on household and personal circumstances are summarized in Table 9.1.
- Employment- and education- focused programs produced no significant effects on marriage. However, there is some evidence that program group members engaged in less-formal relationships. The Riverside LFA program increasing cohabitation by 4.6 percentage points (or by 43 percent). All other programs showed a pattern of increased rates of cohabitation, though the effects of these other programs were not statistically significant.
- There were few program impacts and no differences in effects by program approach on measures of fertility and household composition.
- Program impacts on moving and housing were concentrated in Grand Rapids. Program group members in both Grand Rapids programs were more likely to move and more than once since random assignment, primarily for better housing. The Grand Rapids HCD program also increased home ownership at the five-year follow-up. There were no significant differences between program approaches on these outcomes.
- Although there were few program impacts on many aspects of the quality of relationships, such as ever experiencing employment-related discouragement, harassment or deterrence, or nonphysical abuse, there were fewer reports of experiencing any physical abuse during the last year of follow-up for program group members than for control group members. A similar proportion of program and control group members reported experiencing harassment, abuse (physical or otherwise), or other types of employment-related discouragement or deterrence at any time in their lives. However, there were fewer reports of physical abuse, by about 4 to 6 percentage points, during the last year of follow-up for program group members than for control group members in all sites, with these effects being statistically significant in the Atlanta LFA, Grand Rapids HCD, and Riverside LFA programs. There is some evidence that program effects on increasing employment, especially effects on employment that occurred early in the follow-up period (for example, by increasing self-esteem, ameliorating family stress, or simply reducing the amount of time individuals spent with partners), and enhanced caseworker attention to support services contributed to the fewer reports of experiencing physical abuse among program group members. These effects were measured only for mothers of children who were preschool-age at study entry in the six LFA and HCD programs (the Child Outcomes Study sample).
| Atlanta | Grand Rapids | Riverside | Portland | |||||
|---|---|---|---|---|---|---|---|---|
| LFA | HCD | LFA | HCD | LFA Full | LFA In-need a | HCD | ||
| Marital status b | ||||||||
| Married | ||||||||
| Cohabiting | I | I | ||||||
| Separated or divorced c | D | |||||||
| Never married | ||||||||
Fertilityd | ||||||||
| Presence of a new baby | I | |||||||
Household composition b | ||||||||
| Lives alone | D | D | I | |||||
| Lives with other adult and/or children e | ||||||||
Moving and housing status | ||||||||
| Ever moved d | I | I | ||||||
| Moved more than once d | I | I | I | I | ||||
| Owns a home b | I | |||||||
| Other housing b, f | I | |||||||
| Domestic abuse in prior year g,h | D | D | D | D | ||||
NOTES: "I" indicates that the program produced a statistically significant increase in the outcome. "D" indicates a statistically significant decrease in the outcome. Blank spaces indicate that there were no impacts. | ||||||||
Measurement Issues
The outcomes examined in this chapter generally fall into the following categories: marital status, household composition, moving and housing status, and employment-related and domestic abuse by intimate partners and others. The relevant time periods for which these outcomes are measured also vary, with some capturing circumstances at the time of the survey interview, the month prior to the interview, during the last year of the follow-up, during the entire five-year follow-up period, or at any point during an individual's life.
Marital Status and Household Composition
Marital status was measured at one point in time: the month prior to the survey interview. Dynamic or cumulative patterns of marital status are therefore not captured. Five mutually exclusive categories of marital status were created: married and living with a spouse, cohabiting, separated, divorced, widowed, and never married.
Measures of household composition were constructed from a grid that questioned the respondent about all household members who stayed in her home for at least two nights a week during the month prior to the survey interview. For all measures of household composition, it is assumed that household members in the respondent's generation or older are adults(2) and that household members in a younger generation are children.(3) Adult extended family members include parents, siblings, grandparents, aunts and uncles, cousins, and other adult relatives. For comparability with census definitions of "living with unrelated adults," the measure of living with unrelated adults was constructed based on information about household members and marital status.(4) If respondents listed unrelated adults as household members or reported not being married and living as a couple with a boyfriend or partner, they were considered to be living with unrelated adults. "Living with unrelated adults" includes cohabiting partners who may have fathered at least one child in the household.(5) All marital status categories were coded independently from household composition outcomes. For example, some respondents could have reported living with a "partner" or a "spouse" but may not have reported being "married" or "cohabiting."(6) Specific information about whether or not the father of at least one child lived in the household was not collected for all client survey respondents. Thus, respondents who reported being married or cohabiting may include marriage or cohabitation with the father of at least one child in the household. Presence of a new baby during the five-year follow-up period was measured by comparing the birth dates of each biological, legally adopted, or step child with the date of random assignment (plus nine months). To evaluate changes in outcomes over time and compare these patterns with national figures (described in the final section of this chapter), comparable outcomes were created for sample members who had information collected at both the two-year follow-up and the five-year follow-up.
Moving and Housing Status
Measures of moving reflect any moves that occurred during the five-year follow-up period. The proportion of respondents who moved is calculated as the number who reported moving at least once since the time of random assignment over all respondents. Measures of housing status reflect only status during the month prior to the survey interview. Respondents were considered to be renting a home or room if they reported renting their own home or room or living with family or friends and contributing to part of the rent. Respondents were considered to be living in other housing arrangements if they reported living in a group shelter, in jail, alone and rent free, in another unspecified housing arrangement, or being homeless.(7) Living in public or subsidized housing includes living in housing owned or operated by a local housing authority or other government agency as well as Section 8 housing.
Employment-Related and Domestic Abuse
Measures of employment-related and domestic abuse were collected at the five-year follow-up for sample members who were part of the Child Outcomes Study(8) via a self-administered questionnaire, a method intended to preserve the privacy of the interviewee and increase valid response rates to sensitive questions. Average rates of nonresponse on these outcomes ranged from about 4 to 10 percent for questions about employment-related abuse and from about 7 to 15 percent for more specific questions about domestic abuse.(9) Multiple measures depicting the quality of employment-related and other relationships were created, including job-related abuse that occurred at any time in a respondent's life, abuse (physical or nonphysical) by intimate partners or others at any time in a respondent's life, and abuse that occurred during the year prior to the five-year follow-up interview.
Employment-related abuse (including job discouragement, job harassment, and job deterrence) is measured over a respondent's lifetime (though individuals are likely to have experienced these employment-related aspects of abuse only as an adult when they were actually employed). Respondents were considered to have experienced job discouragement at any time in their life if they reported at least one of the following: someone ever tried to discourage them from finding a job or going to work; someone ever made them feel guilty about going to work; someone ever refused to help them or went back on promises to help with child care, transportation, or housework; someone ever made it difficult for them to attend or complete programs or classes that would help them get a good job. The measure of job harassment includes being harassed at the workplace over the telephone and/or in person. Respondents were considered to have experienced job deterrence at any time in their life if they reported that someone had ever caused them to quit or lose a job and/or someone ever prevented them from finding a job. These variables were coded as "0" for respondents reporting that they did not experience job discouragement, harassment over the phone or in person, or job deterrence.
Because individuals in program and control groups are similar at the time of random assignment in their observed and unobserved characteristics, including experience with abuse prior to random assignment, it is likely that program impacts on these "lifetime" measures will capture effects that occurred during the five-year follow-up period. Furthermore, program group members may have been more likely to experience these types of employment-related abuse during this period since these welfare-to-work programs targeted and increased employment. As a result, fewer reports of employment-related abuse for program group members than for control group members will be especially difficult to detect. Nonetheless these measures are important in providing a general picture of rates of employment-related abuse in welfare populations.
Measures of experiencing any physical or nonphysical domestic abuse were created for two points in time: at any time in the respondent's life and during the year prior to the five-year follow-up interview. Information about the timing of the most recent incident of abuse (for example, this week, a week ago, a month ago, six months ago, a year ago, or more than a year ago) was collected only for those respondents who reported any domestic abuse. Respondents were generally asked if they had experienced any of the following types of domestic abuse: being yelled at or put down on purpose, controlled, insulted, or sworn at; threatened with physical harm; hit, slapped, kicked, or otherwise physically harmed; or if "none of these things have ever happened to me." Measures of more recent domestic abuse were also created for those individuals who reported these same types of abuse in the year prior to the survey interview (for example, the most recent incident happened either the week of the survey interview or the week, month, six months, or year prior to the survey interview). For these measures of recent abuse, respondents who did not report any domestic abuse or who reported that the most recent incident occurred more than a year ago were coded as "0." The measures of more recent physical domestic abuse include being hit, slapped, kicked, or otherwise physically harmed; the measures of more recent nonphysical domestic abuse include all other types of abuse. Measures of physical abuse and nonphysical abuse are not mutually exclusive.
Effects of Welfare-to-Work Approaches on Marital Status
The policies followed in the welfare-to-work programs examined in this report are not expected to directly affect, in this case increase or preserve, the likelihood of marriage.(10) Given the effects on economic outcomes such as employment, earnings, and income, how may these welfare-to-work programs affect the marital status of survey respondents? Increased employment may increase the likelihood of marriage by expanding a single mother's social network or increasing her self-esteem, and, perhaps, her attractiveness to a potential partner. Or increased employment may decrease the likelihood of marriage because less time is available to search for a partner or socialize with a partner.(11)Income may also affect marriage in distinct ways by decreasing financial pressure to get married or, perhaps, by increasing the financial security of a respondent who would otherwise have less family income.
At the time of study entry all survey respondents were single mothers with children. How many of these mothers got married in the absence of the program? Figure 9.1 shows control group levels of marriage for the four survey sites using the sample of respondents who were in both the two-year and five-year surveys. This figure shows that across all sites between 8 percent and just over 20 percent of survey respondents reported being married at either the two-year or the five-year follow-up. This outcome was created to roughly capture a measure of "ever being married" during a five-year time period. This figure also shows that in Atlanta and Portland the majority of respondents in the control group who got married did so relatively late in the follow-up period, at some time during the last three years (that is, they were married at the five-year follow-up but not married at the two-year follow-up).
Figure 9.1
Control Group Levels of Marriage
SOURCE: MDRC calculations from the Five-Year Client Survey.
NOTES: See Appendix A.2.
Owing to missing values, sample sizes may vary.
Table 9.2 shows that none of these welfare-to-work programs had an impact on marriage. Although all programs showed a consistent pattern of increasing cohabitation, only the Riverside LFA program increased cohabitation significantly ¯ by 4.6 percentage points, or 43 percent. It may be the case that more program group members are engaging in less-formal relationships than control group members, who are more likely to get married. Interestingly, in support of this hypothesis, program group members in Portland were 6.0 percentage points, or 43 percent, more likely to cohabit, and 6.2 percentage points, or 26 percent, less likely to get married than control group members. These effects approached statistical significance (p = 0.11). Thus, it appears that in Portland control group members were more likely to get married and program group members were more likely to cohabit. However, this hypothesized pattern of more program group members engaging in less-formal relationships than control group members is not clear in Atlanta and Grand Rapids.
|
Site and Program |
Sample Size | Program Group (%) | Control Group (%) | Difference (Impact) | Percentage Change (%) |
|---|---|---|---|---|---|
|
Married, living with spouse |
|||||
| Atlanta Labor Force Attachment | 1,060 | 9.8 | 8.4 | 1.3 | 15.8 |
| Atlanta Human Capital Development | 1,135 | 6.9 | 8.4 | -1.5 | -18.2 |
| Grand Rapids Labor Force Attachment | 1,090 | 22.8 | 20.5 | 2.3 | 11.1 |
| Grand Rapids Human Capital Development | 1,102 | 20.3 | 20.5 | -0.2 | -1.0 |
| Riverside Labor Force Attachment | 1,213 | 20.6 | 22.0 | -1.4 | -6.5 |
| Lacked high school diploma or basic skills | 654 | 18.6 | 18.1 | 0.5 | 2.9 |
| Riverside Human Capital Development | 773 | 21.8 | 18.1 | 3.7 | 20.5 |
| Portland | 501 | 17.4 | 23.6 | -6.2 | -26.1 |
|
Cohabiting |
|||||
| Atlanta Labor Force Attachment | 1,060 | 7.1 | 6.7 | 0.4 | 5.8 |
| Atlanta Human Capital Development | 1,135 | 6.7 | 6.7 | 0.1 | 1.0 |
| Grand Rapids Labor Force Attachment | 1,090 | 17.4 | 15.9 | 1.5 | 9.4 |
| Grand Rapids Human Capital Development | 1,102 | 16.2 | 15.9 | 0.3 | 2.0 |
| Riverside Labor Force Attachment | 1,213 | 15.3 | 10.7 | 4.6** | 42.6 |
| Lacked high school diploma or basic skills | 654 | 15.9 | 11.1 | 4.9* | 43.9 |
| Riverside Human Capital Development | 773 | 13.7 | 11.1 | 2.6 | 23.6 |
| Portland | 501 | 19.9 | 13.9 | 6.0 | 43.0 |
|
Separated, divorced, or widowed |
|||||
| Atlanta Labor Force Attachment | 1,060 | 34.8 | 37.2 | -2.4 | -6.5 |
| Atlanta Human Capital Development | 1,135 | 36.1 | 37.2 | -1.1 | -3.0 |
| Grand Rapids Labor Force Attachment | 1,090 | 29.5 | 33.7 | -4.1* | -12.2 |
| Grand Rapids Human Capital Development | 1,102 | 32.7 | 33.7 | -0.9 | -2.8 |
| Riverside Labor Force Attachment | 1,213 | 45.0 | 45.8 | -0.8 | -1.8 |
| Lacked high school diploma or basic skills | 654 | 44.1 | 46.4 | -2.3 | -5.0 |
| Riverside Human Capital Development | 773 | 42.1 | 46.4 | -4.3 | -9.2 |
| Portland | 501 | 38.8 | 38.8 | 0.0 | 0.0 |
|
Never married |
|||||
| Atlanta Labor Force Attachment | 1,060 | 48.4 | 47.7 | 0.7 | 1.5 |
| Atlanta Human Capital Development | 1,135 | 50.3 | 47.7 | 2.6 | 5.4 |
| Grand Rapids Labor Force Attachment | 1,090 | 30.3 | 30.0 | 0.4 | 1.2 |
| Grand Rapids Human Capital Development | 1,102 | 30.8 | 30.0 | 0.8 | 2.7 |
| Riverside Labor Force Attachment | 1,213 | 19.1 | 21.4 | -2.3 | -10.9 |
| Lacked high school diploma or basic skills | 654 | 21.3 | 24.4 | -3.1 | -12.5 |
| Riverside Human Capital Development | 773 | 22.3 | 24.4 | -2.1 | -8.5 |
| Portland | 501 | 23.9 | 23.7 | 0.2 | 0.7 |
| SOURCE: MDRC calculations from the Five-Year Client Survey. NOTES: See Appendix A.2. Owing to missing values, sample sizes may vary. |
|||||
Only one other impact on marital status was found. Program group members in the Grand Rapid LFA program were less likely to report being separated, divorced, or widowed by 4.1 percentage points, or 12 percent. This latter effect is a result of either control group members moving into marriage or cohabitation or program group members getting married and then separating or divorcing by the time of the five-year follow-up. The lack of more pervasive impacts on these point-in-time measures of marital status is not surprising for two reasons. As noted above, these programs were not intended to affect marital behavior. Also, point-in-time measures of marital status will not capture effects on marital status changes, and program effects on employment, for example, that occurred earlier in the follow-up period may have been more likely to affect the timing of marriage. Also, a respondent could have married after the two-year survey and divorced by the time of the five-year survey, and this change would not be measured in the survey.
Effects of Welfare-to-Work Approaches on Fertility and Household Composition
Fertility
While these welfare-to-work programs were not intended to alter fertility behavior, single mothers who are engaged in employment or training and who face a higher cost of becoming pregnant may be less likely to have another child. Or, because of age exemptions for the youngest child, they may have an incentive to have a baby in order to avoid any employment or training requirements.
Table 9.3 (first panel) shows impacts on fertility. There were few significant impacts on the presence of a new baby in the household during the five-year follow-up period. Although there were no program impacts for program group members in the Riverside HCD program, LFA in-need sample members (a subgroup without a high school diploma or GED at study entry) were significantly more likely to have a new baby in the household, and, in fact, a similar effect that approaches statistical significance exists for all program group members in the Riverside LFA program (p = 0.12). (Because respondents were not specifically asked about whether or not they had a baby, this measure captures new births as well as new additions of an infant through marriage, adoption, or foster care.)
| Site and Program | Sample Size | Program Group (%) | Control Group (%) | Difference (Impact) | Percentage Change (%) |
|---|---|---|---|---|---|
|
Since random assignment |
|||||
| Presence of a new baby | |||||
| Atlanta Labor Force Attachment | 1,069 | 11.6 | 12.4 | -0.8 | -6.7 |
| Atlanta Human Capital Development | 1,145 | 12.5 | 12.4 | 0.1 | 0.7 |
| Grand Rapids Labor Force Attachment | 1,097 | 22.6 | 21.7 | 0.9 | 4.3 |
| Grand Rapids Human Capital Development | 1,109 | 22.2 | 21.7 | 0.5 | 2.4 |
| Riverside Labor Force Attachment | 1,218 | 25.5 | 22.1 | 3.4 | 15.5 |
| Lacked high school diploma or basic skills | 656 | 28.2 | 23.1 | 5.1* | 22.2 |
| Riverside Human Capital Development | 777 | 24.1 | 23.1 | 1.0 | 4.5 |
| Portland | 504 | 17.3 | 22.7 | -5.3 | -23.6 |
|
During month prior to five-year interview |
|||||
| Lives alone | |||||
| Atlanta Labor Force Attachment | 1,071 | 2.2 | 5.8 | -3.6*** | -62.7 |
| Atlanta Human Capital Development | 1,146 | 3.2 | 5.8 | -2.6** | -44.4 |
| Grand Rapids Labor Force Attachment | 1,097 | 3.9 | 3.5 | 0.5 | 13.1 |
| Grand Rapids Human Capital Development | 1,109 | 4.1 | 3.5 | 0.6 | 17.4 |
| Riverside Labor Force Attachment | 1,219 | 3.7 | 2.4 | 1.3 | 52.8 |
| Lacked high school diploma or basic skills | 657 | 3.2 | 1.2 | 2.0 | 159.9 |
| Riverside Human Capital Development | 778 | 3.6 | 1.2 | 2.3* | 190.4 |
| Portland | 504 | 2.8 | 3.2 | -0.4 | -11.6 |
| Lives with children only | |||||
| Atlanta Labor Force Attachment | 1,071 | 62.9 | 61.8 | 1.1 | 1.7 |
| Atlanta Human Capital Development | 1,146 | 65.6 | 61.8 | 3.8 | 6.1 |
| Grand Rapids Labor Force Attachment | 1,097 | 45.1 | 49.6 | -4.5 | -9.1 |
| Grand Rapids Human Capital Development | 1,109 | 49.2 | 49.6 | -0.3 | -0.7 |
| Riverside Labor Force Attachment | 1,219 | 46.0 | 46.6 | -0.6 | -1.2 |
| Lacked high school diploma or basic skills | 657 | 50.3 | 50.1 | 0.1 | 0.3 |
| Riverside Human Capital Development | 778 | 47.2 | 50.1 | -2.9 | -5.9 |
| Portland | 504 | 48.6 | 45.2 | 3.5 | 7.7 |
| Lives with partner/spouse and children | |||||
| Atlanta Labor Force Attachment | 1,071 | 14.4 | 12.3 | 2.1 | 16.9 |
| Atlanta Human Capital Development | 1,146 | 11.3 | 12.3 | -1.0 | -8.3 |
| Grand Rapids Labor Force Attachment | 1,097 | 34.4 | 31.5 | 2.9 | 9.0 |
| Grand Rapids Human Capital Development | 1,109 | 31.7 | 31.5 | 0.2 | 0.6 |
| Riverside Labor Force Attachment | 1,219 | 31.2 | 28.5 | 2.6 | 9.3 |
| Lacked high school diploma or basic skills | 657 | 29.4 | 25.5 | 3.9 | 15.2 |
| Riverside Human Capital Development | 778 | 29.9 | 25.5 | 4.4 | 17.1 |
| Portland | 504 | 28.5 | 32.7 | -4.2 | -12.9 |
| Lives with adult extended family and any children | |||||
| Atlanta Labor Force Attachment | 1,071 | 17.1 | 16.7 | 0.4 | 2.7 |
| Atlanta Human Capital Development | 1,146 | 18.2 | 16.7 | 1.5 | 8.8 |
| Grand Rapids Labor Force Attachment | 1,097 | 13.4 | 12.5 | 1.0 | 7.9 |
| Grand Rapids Human Capital Development | 1,109 | 9.8 | 12.5 | -2.7 | -21.7 |
| Riverside Labor Force Attachment | 1,219 | 14.3 | 16.4 | -2.0 | -12.4 |
| Lacked high school diploma or basic skills | 657 | 13.8 | 17.1 | -3.3 | -19.3 |
| Riverside Human Capital Development | 778 | 15.8 | 17.1 | -1.3 | -7.9 |
| Portland | 504 | 12.8 | 12.7 | 0.1 | 0.4 |
| Lives with an unrelated adult | |||||
| Atlanta Labor Force Attachment | 1,060 | 10.0 | 8.4 | 1.6 | 18.7 |
| Atlanta Human Capital Development | 1,135 | 8.8 | 8.4 | 0.3 | 3.7 |
| Grand Rapids Labor Force Attachment | 1,090 | 20.8 | 19.5 | 1.3 | 6.4 |
| Grand Rapids Human Capital Development | 1,102 | 19.5 | 19.5 | -0.1 | -0.3 |
| Riverside Labor Force Attachment | 1,213 | 19.7 | 17.5 | 2.2 | 12.5 |
| Lacked high school diploma or basic skills | 654 | 18.9 | 17.3 | 1.6 | 9.0 |
| Riverside Human Capital Development | 774 | 17.2 | 17.3 | -0.1 | -0.7 |
| Portland | 502 | 25.9 | 21.0 | 4.9 | 23.3 |
|
SOURCE: MDRC calculations from the Five-Year Client Survey. |
|||||
Household Composition
In the context of these programs, the immediate presence of other adults or family members may facilitate the transition to employment or may ease the burden of being employed during rotating or weekend hours because of available child care support. Thus, welfare-to-work programs may encourage single mothers to live with extended family. Decreases in income may also encourage program group members to live with extended family or a partner. Alternatively, increases in income may allow program group members to live independently, without the sup-port of relatives or other adults. Because it is often assumed that other household members contribute to family income, one outcome of particular policy interest is the proportion of single mothers who live with an unrelated adult. Unrelated adults who are not cohabiting partners may not contribute to family resources in the same way as spouses or cohabiting partners. Figure 9.2 shows the proportion of control group members who lived with unrelated adults, for the sample of respondents who were in both the two-year and five-year surveys. Approximately 10 percent (in Atlanta) to 25 percent (in Riverside) of the control group members lived with an unrelated adult at the two-year follow-up or the five-year follow-up (again, a best estimate of "ever" occurring over a five-year follow-up period). These outcome levels are somewhat comparable to recent figures showing that 24 percent of single mothers in the poorest income decile lived with an unrelated male in 1995 and 25 percent lived with an unrelated male in 1997.(12)
Figure 9.2
Control Group Levels of Living with an Unrelated Adult
SOURCE: MDRC calculations from the Five-Year Client Survey.
NOTES: See Appendix A.2.
Owing to missing values, sample sizes may vary.
The lower panels of Table 9.3 show impacts on living with an unrelated adult and various other measures of household composition. Overall, there were few program impacts on household composition, except for impacts on living alone. The proportion of sample members who reported living alone at the five-year follow-up was less than 6 percent. The Atlanta LFA and HCD programs decreased the proportion of program group members who reported living alone by 3.6 and 2.6 percentage points, respectively; however, the Riverside HCD program increased the proportion by 2.3 percentage points. Although it is difficult to determine without further analysis, these effects may suggest that program group families in Atlanta are more likely to live with their children and program group families in the Riverside HCD program are less likely to live with their children. There is some, albeit rough, support for this hypothesis. As discussed in Chapter 11, very young children of program group members in Atlanta are less likely than children of control group members to have not lived with their mother because she could not care for them, and children of program group members in Riverside are more likely than children of control group members to have not lived with their mother because she could not care for them.
Effects of Welfare-to-Work Approaches on Housing Status
Program effects on employment and income may also affect the likelihood of moving or seeking better or different housing. For example, a decrease in income may encourage or force program group members to move into less expensive housing, whereas an increase in income may allow them to move into better housing or own a home.
Table 9.4 shows impacts on moving and housing. A high proportion of sample members 66 to 86 percent of the control group moved at least once during the five-year follow-up period. There was greater variation across sites in the proportion of control group members who moved more than once during the follow-up period: Nearly 30 percent moved more than once in Atlanta whereas up to 60 percent moved more than once in Riverside. Of the seven welfare-to-work programs, impacts on ever moving were produced only in Grand Rapids, where the LFA program increased the likelihood of moving by 7.6 percentage points, or 9.8 percent, and the HCD program increased the likelihood of moving by 5.4 percentage points, or 7 percent. Although the Riverside LFA program did not increase the likelihood of moving once during the follow-up, it did increase the likelihood of moving more than once by 7.9 percentage points, or 13 percent. A significant increase in moving more than once was also found for the Riverside in-need sample (respondents without a high school diploma or GED at study entry).
|
Site and Program |
Sample Size | Program Group (%) | Control Group (%) | Difference (Impact) | Percentage Change (%) |
|---|---|---|---|---|---|
| Since random assignment | |||||
| Ever moved | |||||
| Atlanta Labor Force Attachment | 1,070 | 67.0 | 66.2 | 0.8 | 1.2 |
| Atlanta Human Capital Development | 1,145 | 67.7 | 66.2 | 1.5 | 2.3 |
| Grand Rapids Labor Force Attachment | 1,097 | 85.6 | 78.0 | 7.6*** | 9.8 |
| Grand Rapids Human Capital Development | 1,109 | 83.4 | 78.0 | 5.4** | 6.9 |
| Riverside Labor Force Attachment | 1,219 | 86.4 | 84.0 | 2.4 | 2.8 |
| Lacked high school diploma or basic skills | 657 | 83.6 | 81.8 | 1.8 | 2.2 |
| Riverside Human Capital Development | 778 | 81.8 | 81.8 | -0.0 | -0.0 |
| Portland | 504 | 84.8 | 85.7 | -0.9 | -1.0 |
| Moved more than once | |||||
| Atlanta Labor Force Attachment | 1,068 | 30.6 | 29.7 | 0.9 | 3.1 |
| Atlanta Human Capital Development | 1,144 | 31.1 | 29.7 | 1.4 | 4.7 |
| Grand Rapids Labor Force Attachment | 1,096 | 62.0 | 53.9 | 8.2*** | 15.2 |
| Grand Rapids Human Capital Development | 1,108 | 62.1 | 53.9 | 8.3*** | 15.3 |
| Riverside Labor Force Attachment | 1,219 | 67.5 | 59.6 | 7.9*** | 13.3 |
| Lacked high school diploma or basic skills | 657 | 63.4 | 56.2 | 7.1** | 12.6 |
| Riverside Human Capital Development | 778 | 59.1 | 56.2 | 2.8 | 5.0 |
| Portland | 504 | 60.2 | 59.4 | 0.8 | 1.3 |
| At five-year interview | |||||
| Owns home | |||||
| Atlanta Labor Force Attachment | 1,062 | 7.5 | 5.3 | 2.2 | 42.3 |
| Atlanta Human Capital Development | 1,133 | 6.6 | 5.3 | 1.3 | 24.5 |
| Grand Rapids Labor Force Attachment | 1,080 | 20.6 | 18.2 | 2.4 | 13.3 |
| Grand Rapids Human Capital Development | 1,091 | 22.3 | 18.2 | 4.1* | 22.4 |
| Riverside Labor Force Attachment | 1,208 | 10.7 | 10.2 | 0.5 | 4.9 |
| Lacked high school diploma or basic skills | 650 | 8.9 | 9.1 | -0.2 | -2.5 |
| Riverside Human Capital Development | 766 | 12.9 | 9.1 | 3.8 | 41.8 |
| Portland | 495 | 7.7 | 9.4 | -1.6 | -17.3 |
| Lives in public/subsidized housing | |||||
| Atlanta Labor Force Attachment | 1,062 | 49.1 | 53.6 | -4.5 | -8.5 |
| Atlanta Human Capital Development | 1,133 | 50.8 | 53.6 | -2.9 | -5.3 |
| Grand Rapids Labor Force Attachment | 1,080 | 20.1 | 21.9 | -1.8 | -8.3 |
| Grand Rapids Human Capital Development | 1,091 | 18.3 | 21.9 | -3.6 | -16.5 |
| Riverside Labor Force Attachment | 1,208 | 15.0 | 14.6 | 0.4 | 2.6 |
| Lacked high school diploma or basic skills | 650 | 18.1 | 16.2 | 1.9 | 11.9 |
| Riverside Human Capital Development | 766 | 17.6 | 16.2 | 1.4 | 8.7 |
| Portland | 495 | 29.2 | 30.0 | -0.8 | -2.7 |
| Rents home or room | |||||
| Atlanta Labor Force Attachment | 1,062 | 42.7 | 40.6 | 2.1 | 5.1 |
| Atlanta Human Capital Development | 1,133 | 40.9 | 40.6 | 0.3 | 0.7 |
| Grand Rapids Labor Force Attachment | 1,080 | 58.2 | 58.1 | 0.1 | 0.1 |
| Grand Rapids Human Capital Development | 1,091 | 57.1 | 58.1 | -1.1 | -1.8 |
| Riverside Labor Force Attachment | 1,208 | 72.4 | 73.7 | -1.3 | -1.7 |
| Lacked high school diploma or basic skills | 650 | 71.4 | 73.2 | -1.8 | -2.4 |
| Riverside Human Capital Development | 766 | 67.0 | 73.2 | -6.2* | -8.4 |
| Portland | 495 | 59.8 | 59.4 | 0.4 | 0.7 |
| Other housing | |||||
| Atlanta Labor Force Attachment | 1,062 | 0.6 | 0.4 | 0.2 | 50.4 |
| Atlanta Human Capital Development | 1,133 | 1.7 | 0.4 | 1.3** | 302.5 |
| Grand Rapids Labor Force Attachment | 1,080 | 1.1 | 1.8 | -0.7 | -37.9 |
| Grand Rapids Human Capital Development | 1,091 | 2.3 | 1.8 | 0.6 | 33.7 |
| Riverside Labor Force Attachment | 1,208 | 1.9 | 1.5 | 0.4 | 25.1 |
| Lacked high school diploma or basic skills | 650 | 1.6 | 1.5 | 0.1 | 5.9 |
| Riverside Human Capital Development | 766 | 2.4 | 1.5 | 0.9 | 62.4 |
| Portland | 495 | 3.3 | 1.3 | 2.0 | 158.2 |
|
SOURCE: MDRC calculations from the Five-Year Client Survey. |
|||||
Why did survey respondents move? For program group members (across sites) who moved, the most important reasons were that they felt that their current housing was old or bad, that is, the neighborhood was unsafe; they wanted to leave a bad relationship; or the space was too small. The next most important reason for moving was that respondents wanted and could afford a better place to live. And these reasons were more commonly cited by program group members who moved than by control group members who moved, which suggests that program impacts on moving can be interpreted as a positive outcome for respondents' well-being.
Table 9.4 also shows that there were few program effects on home ownership or other housing situations, such as renting or living in public or subsidized housing. One impact of note, given the impacts on moving previously discussed, was that the Grand Rapids HCD program increased the likelihood of owning a home at the time of the five-year follow-up by 4.1 percentage points, or 22 percent. It is striking that rates of home ownership among control group members are nearly twice as high in Grand Rapids as in all other sites. This can be partly attributed to relatively low housing costs and relatively little access to apartments or public/subsidized housing compared with the other sites.
Effects of Welfare-to-Work Approaches on Employment-Related and Domestic Abuse
Current debates about welfare policy include whether or not and how to protect welfare recipients from abusive relationships. Although these welfare-to-work programs did not test the effects of time limits or special exemptions for victims of domestic abuse, increased employment and earnings may indirectly affect the incidence of employment-related or domestic abuse. On the one hand, increased employment may enhance self-esteem and encourage program group members to leave abusive relationships, or increased hours of employment may simply reduce contact with intimate partners or family members who are abusive or remove individuals from abusive situations. On the other hand, increased employment may exacerbate abuse that is occurring in a current relationship as a partner or spouse negatively or violently reacts to the enhanced independence and self-sufficiency that accompanies employment.(13) In addition, because welfare benefits are replaced by earned income, increased employment in the context of these programs may not lead to increased personal resources or, possibly, the ability to leave an abusive relationship.
As noted, information about job-related and domestic abuse was collected only for respondents in the Child Outcomes Study sample in Atlanta, Grand Rapids, and Riverside. Employment-related abuse is measured over a respondent's lifetime. However, because information about the timing of domestic abuse is available, these outcomes are presented for two time periods at any time during a respondent's life and in the year prior to the five-year follow-up interview.
Figure 9.3 shows the proportion of control group members who experienced domestic abuse at any time in their life (for example, had been threatened, yelled at, insulted, or physically harmed) and who experienced domestic abuse by an intimate partner in the year prior to the five-year follow-up interview. Nearly 50 to 70 percent of control group members reported ever being abused. Control group rates of any domestic abuse by intimate partners in the year prior to the five-year follow-up interview ranged from 15 to 20 percent. This is roughly 9 percentage points lower than the abuse rates documented for long-term recipients with young school-age children in the Minnesota Family Investment Program (MFIP) and for recipients and applicants in Florida's Family Transition Program (FTP).(14) However, state and national estimates suggest that approximately 20 percent of the welfare population currently experiences domestic abuse and from 40 to 70 percent experienced domestic abuse at any time during their life.(15)
Figure 9.3
Control Group Levels of Having Experienced Any Domestic Abuse
SOURCE: MDRC calculations from the Five-Year Client Survey.
NOTES: See Appendix A.2.
Owing to missing values, sample sizes may vary.
Impacts on employment-related abuse and domestic abuse that occurred at any time during a respondent's life are shown in Table 9.5. Less that 10 percent of control group members reported experiencing any job harassment (being interrupted by phone or in person by someone) and up to approximately 30 percent reported experiencing job deterrence (being forced to quit or prevented from taking a job). The first three panels of Table 9.5 show that none of the programs in the three sites affected reports of any lifetime experience of having been discouraged from taking a job, harassed while holding a job, or deterred from getting a job. The relatively low levels and lack of impacts on these outcomes are quite encouraging given recent literature that finds that conflicts with intimate partners or others serve as an important barrier to work faced by former welfare recipients.(16) Table 9.5 also shows reported any lifetime experience of domestic abuse. With one exception, none of programs affected reports of any lifetime experience of domestic abuse. Program group members in the Grand Rapids LFA program were 11 percentage points, or 17 percent, more likely to report any lifetime experience of domestic abuse. Further analyses suggest that this experience occurred before random assignment or during the first few years of follow-up and, for the most part, was nonphysical abuse by someone other than an intimate partner, such as a friend or family member (not shown). There were significant differences between the LFA and HCD program approaches on this outcome in Grand Rapids.
|
Site and Program |
Sample Size | Program Group (%) | Control Group (%) | Difference (Impact) | Percentage Change (%) |
|---|---|---|---|---|---|
|
Any lifetime experience of having been discouraged from taking a job a |
|||||
| Atlanta Labor Force Attachment | 573 | 29.1 | 29.6 | -0.5 | -1.8 |
| Atlanta Human Capital Development | 646 | 30.7 | 29.6 | 1.0 | 3.5 |
| Grand Rapids Labor Force Attachment | 417 | 36.1 | 39.5 | -3.4 | -8.6 |
| Grand Rapids Human Capital Development | 398 | 33.0 | 39.5 | -6.5 | -16.4 |
| Riverside Labor Force Attachment | 502 | 36.7 | 36.5 | 0.2 | 0.6 |
| Lacked high school diploma or basic skills | 317 | 31.6 | 36.7 | -5.1 | -13.9 |
| Riverside Human Capital Development | 418 | 34.5 | 36.7 | -2.2 | -6.1 |
|
Any lifetime experience of having been harassed while holding a job a |
|||||
| Atlanta Labor Force Attachment | 565 | 6.0 | 8.1 | -2.1 | -25.6 |
| Atlanta Human Capital Development | 639 | 6.2 | 8.1 | -1.9 | -23.9 |
| Grand Rapids Labor Force Attachment | 415 | 8.5 | 9.9 | -1.4 | -14.1 |
| Grand Rapids Human Capital Development | 398 | 8.0 | 9.9 | -1.9 | -19.4 |
| Riverside Labor Force Attachment | 496 | 9.9 | 8.6 | 1.3 | 15.3 |
| Lacked high school diploma or basic skills | 311 | 11.3 | 7.2 | 4.1 | 56.7 |
| Riverside Human Capital Development | 413 | 7.2 | 7.2 | -0.1 | -0.8 |
|
Any lifetime experience of having been deterred from getting or holding a job a |
|||||
| Atlanta Labor Force Attachment | 575 | 20.9 | 20.8 | 0.1 | 0.4 |
| Atlanta Human Capital Development | 654 | 20.2 | 20.8 | -0.6 | -3.1 |
| Grand Rapids Labor Force Attachment | 417 | 21.2 | 29.0 | -7.8* | -26.8 |
| Grand Rapids Human Capital Development | 400 | 23.4 | 29.0 | -5.6 | -19.3 |
| Riverside Labor Force Attachment | 501 | 27.0 | 24.6 | 2.3 | 9.5 |
| Lacked high school diploma or basic skills | 315 | 23.2 | 23.7 | -0.5 | -2.1 |
| Riverside Human Capital Development | 415 | 24.5 | 23.7 | 0.8 | 3.6 |
|
Any lifetime experience of domestic abuse |
|||||
| Atlanta Labor Force Attachment | 541 | 51.5 | 49.6 | 1.9 | 3.8 |
| Atlanta Human Capital Development | 610 | 48.7 | 49.6 | -1.0 | -1.9 |
| Grand Rapids Labor Force Attachment | 396 | 75.8 | 65.0 | 10.8** | 16.6 |
| Grand Rapids Human Capital Development | 379 | 62.8 | 65.0 | -2.2 | -3.5 |
| Riverside Labor Force Attachment | 471 | 61.4 | 65.1 | -3.7 | -5.7 |
| Lacked high school diploma or basic skills | 289 | 55.8 | 61.2 | -5.4 | -8.8 |
| Riverside Human Capital Development | 380 | 64.0 | 61.2 | 2.8 | 4.6 |
|
SOURCE: MDRC calculations from the Five-Year Client Survey. |
|||||
Table 9.6 shows impacts on measures of domestic abuse in the year prior to the five-year follow-up interview. Although programs had few effects on measures of employment-related or domestic abuse that occurred at any point in a respondent's life, the welfare-to-work programs did affect one important aspect of the quality of more recent relationships. In particular, in all of the LFA and HCD programs, program group members reported fewer incidences of experiencing physical domestic abuse (for example, hitting) during the year prior to the five-year follow-up interview, by 3 to 6 percentage points, than control group members, and nearly all of these program-control group differences achieved or approached statistical significance.(17)
|
Site and Program |
Sample Size | Program Group (%) | Control Group (%) | Difference (Impact) | Percentage Change (%) |
|---|---|---|---|---|---|
|
Experienced any abuse in prior year |
|||||
| Atlanta Labor Force Attachment | 541 | 21.4 | 20.1 | 1.3 | 6.4 |
| Atlanta Human Capital Development | 610 | 16.4 | 20.1 | -3.7 | -18.3 |
| Grand Rapids Labor Force Attachment | 396 | 24.5 | 22.2 | 2.3 | 10.2 |
| Grand Rapids Human Capital Development | 379 | 17.5 | 22.2 | -4.7 | -21.2 |
| Riverside Labor Force Attachment | 471 | 15.6 | 19.2 | -3.5 | -18.4 |
| Lacked high school diploma or basic skills | 289 | 16.0 | 18.9 | -2.9 | -15.6 |
| Riverside Human Capital Development | 380 | 19.2 | 18.9 | 0.3 | 1.4 |
|
Experienced physical abuse in prior year |
|||||
| Atlanta Labor Force Attachment | 541 | 3.7 | 7.4 | -3.7** | -50.2 |
| Atlanta Human Capital Development | 610 | 4.4 | 7.4 | -2.9 | -39.7 |
| Grand Rapids Labor Force Attachment | 396 | 10.9 | 13.7 | -2.8 | -20.4 |
| Grand Rapids Human Capital Development | 379 | 7.9 | 13.7 | -5.8* | -42.4 |
| Riverside Labor Force Attachment | 471 | 7.5 | 13.1 | -5.6** | -42.5 |
| Lacked high school diploma or basic skills | 289 | 6.0 | 12.3 | -6.3** | -51.4 |
| Riverside Human Capital Development | 380 | 7.0 | 12.3 | -5.2 | -42.6 |
|
Experienced nonphysical abuse in prior year |
|||||
| Atlanta Labor Force Attachment | 541 | 21.0 | 19.7 | 1.3 | 6.6 |
| Atlanta Human Capital Development | 610 | 15.8 | 19.7 | -3.9 | -20.0 |
| Grand Rapids Labor Force Attachment | 396 | 24.5 | 21.7 | 2.8 | 13.0 |
| Grand Rapids Human Capital Development | 379 | 17.5 | 21.7 | -4.2 | -19.2 |
| Riverside Labor Force Attachment | 471 | 15.6 | 18.8 | -3.2 | -16.9 |
| Lacked high school diploma or basic skills | 289 | 16.0 | 18.4 | -2.4 | -13.0 |
| Riverside Human Capital Development | 380 | 19.2 | 18.4 | 0.9 | 4.7 |
|
SOURCE: MDRC calculations from the Five-Year Client Survey. |
|||||
Links Between Effects on Household and Personal Circumstances and Particular Program Practices or Program Effects on Employment
Of all of the outcomes and impacts examined in this chapter, only one may be differentiated by a particular program practice: The effect of the Grand Rapids LFA program on having experienced domestic abuse at any point during a respondent's lifetime is significantly different from the effect of the HCD program. Although puzzling, it is unlikely that this effect depicts a general difference between LFA and HCD program approaches since these differences did not occur in Atlanta or Riverside, or on measures that were confined to the year prior to the five-year follow-up interview. These welfare-to-work programs had few effects on measures of any lifetime experience of employment-related or domestic abuse. However, a striking and consistent pattern of impacts was found on the measure of quality of relationships in the prior year, with fewer program group members than control group members reporting experiences with physical domestic abuse (for example, by an intimate partner) in the year prior to the five-year follow-up interview. At the same time, there is no evidence that these programs altered respondents' living arrangements at the time of the survey interview.
The effects on physical domestic abuse are reassuringly positive and suggest that employment can play a role in influencing some aspects of the quality of relationships. Unlike policies that include an enhanced earned income disregard, increased employment, in the context of these welfare-to-work programs, did not always lead to enhanced personal resources or income that ultimately may help domestic abuse victims to leave their relationships. For example, recent MFIP findings suggest that MFIP's financial incentives, rather than the added effects of MFIP's participation requirements, produced the effects on decreased domestic abuse.(18) Although these financial incentives contributed to increased employment, their most pronounced effects were to increase income and decrease poverty. The welfare-to-work programs evaluated in this report do not have a similar policy component, and certainly do not mirror MFIP's effects on income. In fact, as previously mentioned, cumulative effects on average combined income varied considerably across sites. The differences in program effects on average combined income that also exist during the final year of follow-up and similarities in program effects on domestic abuse across sites suggest that some other program effect may be contributing to decreased reported rates of physical abuse.
Research suggests that women who have experienced recent abuse are as likely to be employed as those who have not had a similar experience with abuse and that most women on welfare want to work and have work experience whether or not they have been victims of abuse.(19) Program effects on employment somewhat support these research conclusions; that is, though most programs increased employment at some point during the follow-up period some programs did not necessarily increase employment during the last year of follow-up, yet these same programs led to fewer reports of physical abuse. Thus, in some cases current employment may have played an important role, while in other cases early boosts in employment may have played an important role. Employment may not have always led to increased financial resources, but it may have increased self-esteem or self-efficacy, giving respondents the courage to leave abusive relationships, or may have simply reduced contact with abusive partners or situations.
Additional analyses were conducted to examine the relationship, if any, between effects on employment attributed to the LFA and HCD programs and effects on domestic abuse. These analyses do not reveal whether or not employment led to less physical abuse or vice versa, but can reveal whether or not effects on employment or program practices were at all related to effects on physical abuse: Did the same program group members who experienced increased employment also report less physical abuse? This hypothesis was examined by estimating impacts on joint outcomes depicting combinations of "being employed at any time during the follow-up period" and "experiencing physical domestic abuse in the year prior to the five-year follow-up interview" and "being employed in the year prior to the five-year follow-up interview" and "experiencing physical domestic abuse in the year prior to the five-year follow-up interview." Effects on these joint outcomes showed that both Riverside programs and the Grand Rapids HCD program significantly increased the likelihood of being employed and reporting no physical abuse in the year prior to the five-year follow-up interview. The pattern of effects was similar, though not statistically significant, for both Atlanta programs. Similar effects were not found for the joint measure of "employed and reporting no physical domestic abuse in the year prior to the five-year follow-up interview," suggesting that early effects on employment may have contributed to decreased reports of physical abuse later in the follow-up.(20)
Research suggests that abuse by intimate partners may be a critical barrier to employment: Abusers sabotage women's employment efforts, often leaving them with little choice except to quit or miss work or schooling.(21) In this regard, it may have been the mandate to engage in employment-related activities and the very real threat of losing portions of the welfare benefit that helped women overcome any partner interference in their training or employment.(22)
Other possible explanations for observed effects on physical domestic abuse that occurred later in the follow-up period include program practices by caseworkers early in the follow-up period. Did program group members who reported less physical abuse later in the follow-up period also report that JOBS staff helped them with particular problems or provided them with particular services early in the follow-up period? Program group members who reported physical abuse in the last year of follow-up were more likely than program group members who reported no abuse to also report at the two-year follow-up that JOBS staff were highly likely to help them with problems that made it difficult to participate in JOBS. Again, these differences were especially pronounced among sample members in both Riverside program and the Grand Rapids HCD program. Thus, some of the impact on decreased physical abuse later in the follow-up period may be attributed to effects on "increased attention" by JOBS staff to deal with problems, such as domestic abuse, that might have made participation in employment or employment-related activities difficult.
The TANF program of PRWORA paid special attention to women who experienced or were at imminent risk of serious partner violence and intimidation. As a result, states were given the option to engage in procedures to identify and assess risk, provide services, and, if necessary, temporarily waive program requirements for victims of abuse. The findings of this study are encouraging because they highlight the fact that for some battered women, employment and/or caseworker attention to special services may lead to greater safety. Perhaps these women were able to make changes in their relationships or separate from their abusers, or, through employment, were able to physically remove themselves from abusive situations. Thus, the belief that most battered women need to be relieved of employment requirements is not born out by this study. However, employment at a time of risk may not have such positive results for other abused women, particularly those who are at imminent risk and may require temporary relief from employment requirements. Identifying these women and their needs is important so that various approaches may be developed and implemented to best assist all battered women to move into employment and off welfare.(23) The impacts on physical domestic abuse noted in this chapter are quite striking and merit further analysis and attention. Questions for further research include: How does employment and the characteristics of employment affect employment-related and domestic abuse? Does employment-related and domestic abuse affect employment stability? Are effects on employment-related and domestic abuse linked to effects on marital status or household composition?
1. Freedman et al., 2000a, Appendix C.
2. This category includes spouse, partner, father, mother, brother, sister, grandfather, grandmother, uncle, aunt, cousin, other adult male or female relative, and other unrelated male or female adult.
3. This category includes son, daughter, nephew, niece, grandson, granddaughter, and other unrelated male or female child.
4. U.S. Bureau of the Census, 2000.
5. Including cohabitors who may have fathered at least one child in the household is consistent with recent work by Primus et al., 1999, that documents the proportion of single mother families who live with unrelated adults using data from the Current Population Survey. Information about whether or not a father lives in the household is available for a subset of client survey respondents (those who were also in the Child Outcomes Study sample). However, even this information is relevant only for the focal child in the family.
6. The percentage of respondents who reported these types of discrepancies is small: 2.6 percent of respondents reported being married or cohabiting and did not list a spouse or partner as a household member; 0.7 percent of respondents reported not being married or cohabiting but listed a spouse or partner as a household member.
7. The incidences of each of these categories alone were negligible less than 5 percent of the client survey sample reported living in any one of these arrangements.
8. For more detailed information on the Child Outcomes Study sample, see Chapter 12.
9. Gallup-Black, 1999, discusses in detail the quality of data on domestic abuse and domestic barriers to work when collected via self-administered questionnaire (SAQ) and Computer-Assisted Self-Interview (CASI). CASI users sit alone with a computer and headphones and type their answers directly into the computer. Gallup-Black compared SAQ-based NEWWS data with CASI-based Minnesota Family Investment Program (MFIP) data. She found that NEWWS respondents were less likely to answer questions about sensitive items than MFIP respondents.
10. An example of a policy that can be expected to affect marriage is streamlining eligibility rules for two-parent families on welfare by excluding any restrictions about the number of hours a spouse may work, often called the "100-hour" rule.
11. See Harknett and Gennetian, 2001, for detailed hypotheses about how welfare and employment programs, or changes in employment and income, may affect marital behavior.
12. Primus et al., 1999, using data from the Current Population Survey.
13. Riger and Krieglstein, 2000.
14. Gennetian and Miller, 2000; Bloom et al., 2000a.
15. Allard et al., 1997; Raphael and Tolman, 1997.
16. Danziger et al., 1999.
17. The effect for the Atlanta HCD program is significant at the p = 0.12 level and the effect for the Riverside HCD program is significant at the p = 0.11 level.
18. Gennetian and Miller, 2000.
19. Lyon, 2000.
20. According to a similar analysis, the puzzling effects on ever experiencing abuse in the Grand Rapids LFA program do not appear to be linked with employment at all.
21. Riger, Ahrens, and Blickenstaff, 2001, Chapter 7. Some examples of sabotage include turning off alarm clocks, failing to fulfill child care responsibilities, destroying textbooks, and administering beatings so that a woman has highly visible bruises. Raphael, 1996.
22. Unfortunately, the survey collected only information about whether or not a respondent "ever" experienced various barriers to employment by intimate partners and others, and, thus, these measures cannot be confined to the same period of time as the abuse measures.
23. The survey questions in the five-year interview in this study were not structured to identify women at imminent risk (or the effects of any exemptions for these women), but rather to capture whether or not personal or other relationships affected women's ability to participate in program requirements or employment, and whether or not services by program staff, enhanced employment, or other resources, including income, can alter the quality of relationships.
Impacts on Child Care and Child Activities
This chapter examines the impacts of seven of the NEWWS programs on child care use and, for a subgroup of school-age children ("focal" children), on activities during after-school hours from two perspectives: (1) as supporting mothers' employment and transition from welfare and (2) as providing a context for children's development.
A primary goal of the NEWWS Evaluation was to consider the extent to which mothers assigned to a welfare-to-work program moved from welfare to employment and to understand the supports that were important in bringing about such a transition. Child care was assumed to be of importance as a support for the transition to employment in that it provides for the supervision of children so that mothers can participate in work-related activities. Child care benefits were provided to both program and control group mothers who were participating in appropriate activities and were available while mothers were receiving welfare and for a year after they left welfare because of earnings.(1) Thus, a key question addressed in the first section of this chapter is whether there were program impacts on use of child care for purposes of employment and whether program group mothers made greater use of child care benefits than control group mothers.
National survey data indicate that child care arrangements are reported by low-income mothers who are not employed as well as by mothers who are employed (albeit at a lower rate).(2) Nonemployed mothers appear to use child care in order to expose their children to stimulating early childhood education environments and to support their own participation in education and training activities.(3) Within the context of the JOBS welfare-to-work programs studied here, the possibility exists that children's exposure to child care in the program groups might increase not only because of employment, but also as mothers increase their participation in education or training activities and perhaps come to value exposure to educational experiences and want these experiences for their children as well as for themselves.
The second section of this chapter explores the extent and nature of children's exposure to nonmaternal care for any purpose, not only as a support for employment. The underlying question is whether mothers' assignment to one of the welfare-to-work programs results in changes in the amount and type of nonmaternal care that children experience. The section that explores child care as a support for employment takes into account child care for all the children in the family under age 13 while the section that explores child care for any purpose focuses on the focal children in the Child Outcomes Study, who were aged 8 to 10 years at the five-year follow-up. This section presents impacts on both what mothers describe as regular child care arrangements for this school-age child and on supervision and activities during the after-school hours.
Key Findings
- In general, there were very few program impacts on child care at the time of the survey. This is perhaps not surprising, given that there was no child care component specific to the JOBS program and that the impacts of these programs on employment tended to decline by the end of the follow-up period.
- Most programs increased mothers' use of child care after leaving welfare because of earnings across the full follow-up period. However, only three of seven programs increased use of transitional child care benefits over this period. Impacts on transitional benefits may be more limited owing to site variation in the take-up rate of this benefit.
- Child care impacts for a recent period occurred only in the few programs with sustained employment impacts. Portland's program had an impact on child care use during a recent spell of employment. In addition, both of Riverside's programs changed time use "on a recent weekday," reducing time spent with the mother and increasing time with another adult. At the same time, in the Riverside LFA program, a smaller proportion of program group families than control group families used formal child care as their regular main arrangement.
- There were no impacts on the extent of participation in what mothers described as regular child care arrangements for their children aged 8 to 10 at the time of the final follow-up. None of the programs affected total number of hours in child care, use of a regular arrangement, or use of multiple arrangements.
- Self-care among children aged 8 to 10 generally did not increase when mothers had been assigned to a welfare-to-work program, and there were no program impacts on focal children's participation in an after-school activity or lesson. Although there were no impacts on self-care "in the past week," the proportion of control group children in self-care ranged from 9 percent in Riverside to 12 percent in Grand Rapids. For the subgroup of mothers with limited education, the Riverside LFA program increased the proportion of children who spent any time "on a recent weekday" unsupervised by adults The programs examined here did not increase (or decrease) children's exposure to potentially beneficial after-school activities and lessons.(4)
Analysis Issues
Mothers' Employment and Child Care Use
The child care questions asked of the client survey sample (families in four research sites) provide the basis for the discussion below of child care use as a support for employment. For the full follow-up period, respondents were asked about the use of child care after leaving welfare because of earnings, and about receipt of transitional child care benefits.(5) Unlike the survey information that was collected at the two-year follow-up point, the five-year survey did not ask about receipt of child care subsidies other than transitional child care benefits. For the more limited time period of the "most recent spell of employment," respondents were asked about use of, and out-of-pocket expenses for, child care while employed.(6)
Children of client survey sample members were between ages 1 and 18 at random assignment and between ages 6 and 23 at the time of the five-year follow-up. The questions about child care as a support for employment generally were asked about all children in the family who were under age 13 at the time when the care was used. This could have included child care for children who joined the family after random assignment.(7) These analyses were performed with the family as the unit of analysis, because the child care information could not be linked to specific children in each family. Although it is possible to conduct analyses by the age of the youngest child in the household to capture roughly the different needs and circumstances for children across the age range, especially for those with toddlers and preschoolers rather than only school-age children, this type of analysis was beyond the scope of the chapter.
It is critical to emphasize that child care benefits were available to both program and control group mothers based on their engagement in work-related activities or their employment following an exit from welfare. That is, there was no distinctive child care treatment specific to the program groups within the NEWWS Evaluation (such as provision of more extensive subsidies, special help when there were problems with child care arrangements, or access to particular high-quality child care). Accordingly, any impacts on child care participation and use of child care benefits for purposes of employment are related to program impacts on mothers' work-related activities or to other program features, such as the programs' mandatory nature, that might have led directly to increased employment. Thus, it is important to ask whether and to what extent families made greater use of child care and of child care benefits for purposes of employment when they had been assigned to a JOBS welfare-to-work program.
Findings presented in Chapter 4 of this report regarding program impacts on employment underscore the importance of distinguishing between measures for the full follow-up period and measures for a recent and more limited time period. While impacts on average number of quarters employed and on total earnings were widespread across the full follow-up period, impacts were more circumscribed for measures of employment and earnings at the time of the five-year follow-up, and in a number of sites differences were no longer statistically significant at this point. As noted in Chapter 4, the narrowing of differences by the five-year point between the program and control groups on employment outcomes reflects, in part, the fact that control group mothers increasingly sought employment on their own over the follow-up period and the fact that some program group members left employment over time. In addition, in Atlanta and Grand Rapids some or all control group families became subject to new state welfare-to-work initiatives in the final years of the evaluation, following passage of the 1996 welfare law.(8) Thus, in looking at program impacts on child care as a support for employment and the transition from welfare, it seems reasonable to predict more widespread impacts on measures looking across the full follow-up period than on measures pertaining to the recent spell of employment.
It will also be important to keep in mind the ages of the children in the evaluation sample. While some mothers bore children over the course of the five-year follow-up, children who had already been born at the start of the follow-up period increasingly moved into the school-age and adolescent age groups as the follow-up period proceeded. Findings from recent national surveys indicate that while child care use increases with child age during the preschool years (that is, for children aged 5 or under but not yet in kindergarten),(9) child care use begins to decline once children enter school, though substantial numbers of school-age children continue to be in regular child care arrangements.(10) Thus, by the time of the final follow-up, a higher proportion of the children in the NEWWS Evaluation might be in an age range for which employed mothers no longer report regular child care arrangements for their children. Against this backdrop of diminished overall child care use, program impacts might also be less likely to occur.
In sum, across the full five-year follow-up as well as at the end of the follow-up, it is important to consider the extent to which child care played a role in supporting mothers' employment and departure from welfare receipt. Yet because the child care treatment did not differ for program and control groups, because the pattern of employment impacts narrowed in a number of sites over the course of the follow-up, and because of the increasing ages of children in the evaluation sample, the possibility exists that impacts will not be strong or widespread, particularly when focusing on child care use during mothers' most recent spell of employment.
Children's Experiences in Child Care and Activities
The child care questions included in the more extensive interview for the Child Outcomes Study sample (families in three of the sites) provide the basis for the discussion below of children's recent experiences in child care whether used as a support for employment or not. These questions focus specifically on the focal child rather than on all children in the family, and they focus exclusively on the time period of the final follow-up. Focal children were between about ages 8 and 10 at the time of the five-year follow-up. Regarding child care "in the past week," respondents were asked about use of any self-care, whether there was a regular child care arrangement for the focal child, type of regular arrangement used, whether multiple arrangements were used, hours in child care across all regular arrangements, and reliance on a caregiver aged 17 or under. To address the possibility that mothers may not describe the supervision they use for their school-age children as child care, mothers were also asked about whom the focal child was with in half-hour time periods from 3 P.M. to midnight "on a recent weekday." To address the possibility of program impacts on participation in activities or lessons, mothers' reports of participation in such activities on the recent weekday afternoon and evening was also analyzed. Self care was a category in the questions both about child care in the past week and about supervision on the recent weekday afternoon and evening. While measures about a recent afternoon and evening provide useful information, such measures likely underestimate activities that occur more sporadically.
A substantial body of research indicates that children's experiences in child care are related to their development.(11) Much of this research has focused on variations in child care quality and in children's development and confirms that across both formal and informal types of child care, higher-quality care is associated with more advanced cognitive development and more positive social behavior.(12) In addition, however, there is also emerging evidence that type of child care is related to development. While quality varies widely within both formal and informal types of child care, recent findings suggest that beyond variation in quality, participation in more formal, center-based programs is related to children's cognitive school-readiness.(13) This may be, for example, because of the greater likelihood that a formal curriculum is used in center-based programs or that there is a greater availability of books and other cognitively stimulating materials in such settings.
In keeping with this finding from the larger research literature, earlier analyses of children's development in the NEWWS Child Outcomes Study have linked participation in formal child care settings with better concurrent developmental outcomes (higher scores on assessments of cognitive school-readiness).(14) Accordingly, in the analyses presented here, a key issue will be not only whether mothers' assignment to a JOBS welfare-to-work program affected the overall extent of children's exposure to nonmaternal care, but also whether there were shifts in type of care, especially between formal and informal types.(15),(16)
Some concern has been expressed that mandatory participation in a welfare-to-work program might push some mothers to leave school-age children to care for themselves while the mothers engage in work or work-preparation activities or that children might be left in the care of young caregivers (for example, older siblings). National data indicate that self-care as a primary child care arrangement increases during the early school years, from about 5 percent for children aged 6 to 9 to about 24 percent for those aged 10 to 12 with employed mothers.(17) While findings linking self-care with children's development appear to differ by age and characteristics of the child, there is some evidence that regularly spending time in unstructured and unsupervised settings is deleterious for children's development.(18) Given these concerns, this chapter also examines whether there were program impacts on families' use of self-care and care by a young caregiver.
At the time of the five-year follow-up, the focal children studied were aged 8 to 10. As noted above, in this age range a decreasing proportion of mothers report child care use. The possibility exists, however, that mothers are nevertheless making arrangements for the supervision of their school-age children, but not describing these arrangements as child care. To capture a range of possible supervisory situations that school-age children might be in, outcomes examined include not only child care arrangements that the focal children participated in "in the past week," but also a broader description of supervision of the children "on a recent weekday" (specifically whether they were with the mother, with another adult, with only a young peer or sibling, or in self-care).
After-school activities (such as participation in clubs, sports, or lessons) might also be providing supervision in the mother's absence and yet not be labeled child care. Accordingly, this section also includes after-school activities during a recent weekday afternoon. Indeed, in a recent evaluation of a New Hope work incentive and support program, program impacts on use of after-school child care and participation in activities were found, especially for boys. This program also had favorable impacts on the social and academic outcomes of school-age boys. The researchers hypothesize that participation in after-school care and activities might have protected the school-age boys in the study from exposure to street activity in dangerous neighborhoods and exposed them instead to positive, sometimes academically oriented, supervised activities.(19)
Child Care Use as a Support for Employment
This section describes findings on the use of child care as a support for employment and on the use of transitional child care benefits. Findings on control group levels are presented first, followed by findings on program impacts.
Control Group Levels of Child Care
Across the full follow-up period, the use of child care arrangements after leaving welfare because of earnings varied considerably for control group members in the client survey sample across the sites. As Table 10.1 shows, control group members in Atlanta and Riverside reported the lowest levels of child care use (17 to 24 percent), whereas control group members in Portland reported the highest level (44 percent). Receipt of transitional child care benefits also varied considerably among control group members across the sites, ranging from approximately 5 percent in Riverside to 31 percent in Portland. Among those control group members who used child care after leaving welfare because of earnings, receipt of transitional child care benefits ranged from 20 percent in Riverside to 70 percent in Portland.
Site and Program | Sample Size | Program Group | Control Group | Difference (Impact) | Percentage Change (%) | p-Value |
|---|---|---|---|---|---|---|
Used child care after leaving AFDC because of earnings (%) | ||||||
| Atlanta Labor Force Attachment | 1,054 | 35.8 | 24.3 | 11.5*** | 47.4 | 0.00 |
| Atlanta Human Capital Development | 1,123 | 29.9 | 24.3 | 5.7** | 23.4 | 0.02 |
| Grand Rapids Labor Force Attachment | 1,084 | 46.0 | 38.1 | 7.9*** | 20.7 | 0.00 |
| Grand Rapids Human Capital Development | 1,098 | 41.7 | 38.1 | 3.7 | 9.6 | 0.19 |
| Riverside Labor Force Attachment | 1,188 | 31.3 | 24.5 | 6.7*** | 27.5 | 0.01 |
| Lacked high school diploma or basic skills | 635 | 23.0 | 16.8 | 6.2** | 37.1 | 0.05 |
| Riverside Human Capital Development | 750 | 24.1 | 16.8 | 7.3** | 43.4 | 0.02 |
| Portland | 494 | 47.7 | 44.2 | 3.5 | 7.8 | 0.45 |
Received transitional child care benefits (%) | ||||||
| Atlanta Labor Force Attachment | 1,053 | 19.2 | 9.2 | 10.0*** | 109.0 | 0.00 |
| Atlanta Human Capital Development | 1,120 | 13.0 | 9.2 | 3.9* | 42.1 | 0.05 |
| Grand Rapids Labor Force Attachment | 1,078 | 15.0 | 11.7 | 3.3 | 27.8 | 0.11 |
| Grand Rapids Human Capital Development | 1,089 | 14.7 | 11.7 | 3.0 | 25.2 | 0.15 |
| Riverside Labor Force Attachment | 1,182 | 7.8 | 4.9 | 3.0** | 60.3 | 0.03 |
| Lacked high school diploma or basic skills | 632 | 6.5 | 3.0 | 3.4** | 112.7 | 0.03 |
| Riverside Human Capital Development | 748 | 3.4 | 3.0 | 0.3 | 11.0 | 0.84 |
| Portland | 492 | 34.8 | 31.0 | 3.8 | 12.2 | 0.39 |
Used transitional child care benefits among those who used child care after leaving welfare because of earnings (%) | ||||||
| Atlanta Labor Force Attachment | 53.6 | 37.8 | 15.8 | 41.8 | ||
| Atlanta Human Capital Development | 43.6 | 37.8 | 5.8 | 15.2 | ||
| Grand Rapids Labor Force Attachment | 32.6 | 30.7 | 1.8 | 5.9 | ||
| Grand Rapids Human Capital Development | 35.1 | 30.7 | 4.4 | 14.3 | ||
| Riverside Labor Force Attachment | 25.1 | 20.0 | 5.1 | 25.8 | ||
| Lacked high school diploma or basic skills | 28.1 | 18.1 | 10.0 | 55.1 | ||
| Riverside Human Capital Development | 14.0 | 18.1 | -4.1 | -22.6 | ||
| Portland | 72.9 | 70.0 | 2.9 | 4.1 | ||
Used child care while employed (%) | ||||||
| Atlanta Labor Force Attachment | 1,066 | 33.2 | 33.2 | 0.1 | 0.2 | 0.97 |
| Atlanta Human Capital Development | 1,140 | 32.1 | 33.2 | -1.1 | -3.3 | 0.67 |
| Grand Rapids Labor Force Attachment | 1,093 | 49.2 | 45.3 | 4.0 | 8.8 | 0.12 |
| Grand Rapids Human Capital Development | 1,097 | 43.8 | 45.3 | -1.5 | -3.3 | 0.57 |
| Riverside Labor Force Attachment | 1,212 | 42.9 | 42.4 | 0.5 | 1.2 | 0.84 |
| Lacked high school diploma or basic skills | 653 | 43.9 | 39.3 | 4.6 | 11.8 | 0.19 |
| Riverside Human Capital Development | 774 | 40.6 | 39.3 | 1.3 | 3.3 | 0.72 |
| Portland | 498 | 50.7 | 43.8 | 6.9* | 15.8 | 0.09 |
Paid for child care out-of-pocket (%) | ||||||
| Atlanta Labor Force Attachment | 1,061 | 18.5 | 19.4 | -0.9 | -4.6 | 0.69 |
| Atlanta Human Capital Development | 1,140 | 20.2 | 19.4 | 0.8 | 4.1 | 0.72 |
| Grand Rapids Labor Force Attachment | 1,087 | 29.9 | 29.5 | 0.4 | 1.2 | 0.89 |
| Grand Rapids Human Capital Development | 1,093 | 28.5 | 29.5 | -1.0 | -3.4 | 0.70 |
| Riverside Labor Force Attachment | 1,210 | 26.4 | 26.8 | -0.4 | -1.6 | 0.86 |
| Lacked high school diploma or basic skills | 653 | 26.7 | 22.2 | 4.5 | 20.4 | 0.16 |
| Riverside Human Capital Development | 774 | 24.1 | 22.2 | 1.9 | 8.7 | 0.55 |
| Portland | 496 | 32.3 | 28.1 | 4.3 | 15.2 | 0.32 |
Total monthly out-of-pocket cost of child care ($) | ||||||
| Atlanta Labor Force Attachment | 1,061 | 29.5 | 32.1 | -2.6 | -8.2 | 0.59 |
| Atlanta Human Capital Development | 1,140 | 34.6 | 32.1 | 2.4 | 7.5 | 0.62 |
| Grand Rapids Labor Force Attachment | 1,087 | 70.3 | 68.7 | 1.5 | 2.2 | 0.85 |
| Grand Rapids Human Capital Development | 1,093 | 65.5 | 68.7 | -3.2 | -4.7 | 0.68 |
| Riverside Labor Force Attachment | 1,210 | 62.0 | 61.8 | 0.2 | 0.4 | 0.97 |
| Lacked high school diploma or basic skills | 653 | 59.1 | 48.6 | 10.5 | 21.5 | 0.28 |
| Riverside Human Capital Development | 774 | 61.5 | 48.6 | 12.9 | 26.4 | 0.19 |
| Portland | 496 | 113.5 | 62.1 | 51.4** | 82.8 | 0.02 |
| SOURCE: Child Trends calculations from the Five-Year Client Survey. NOTES: See Appendix A.2. | ||||||
Approximately one-third to one-half of control group members in the client survey sample reported using child care while employed, and one-fifth to one-third of control group members in this sample reported paying for child care out-of-pocket. Thus, of those control group members who used child care while employed, approximately 60 percent paid out-of-pocket for at least some of this child care (the proportion of those who paid out-of-pocket for care divided by the proportion of those who reported using child care while employed).
The wide cross-site variation in take-up of transitional child care benefits may be related to a number of things including whether, or to what extent, families were informed of these benefits and/or encouraged to use them, bureaucratic hurdles in applying for and maintaining benefits, variation in preferences or need for child care, and/or variation in coverage of certain types of care.(20) Yet a majority of control group families who were using child care for purposes of employment had out-of- pocket expenses for care. These findings point to the potential importance of active outreach in informing families of the child care benefits for which they are eligible and suggest that child care expenses may play a role in overall family income for many families, counteracting some of the benefits of earnings.
Impacts on Child Care Use as a Support for Employment
Findings that pertain to the full five-year follow-up show that most of the programs (five of the seven in the client survey sample) increased the likelihood of mothers' child care use after leaving welfare because of earnings. All of the programs except Grand Rapids HCD and Portland increased child care use in this way.(21)
A smaller number of programs significantly increased receipt of transitional child care benefits. Impacts were significant for both Atlanta programs and for the Riverside LFA program for both the full and in-need sample.(22) Atlanta LFA program group members were 11.5 percentage points more likely to have used child care after leaving welfare and were 10 percentage points more likely to have received transitional child care benefits than control group members. In addition, HCD program group members in Atlanta were 5.7 percentage points more likely to have used child care after leaving welfare and 3.9 percentage points more likely to have received transitional child care benefits than control group members. These impacts suggest that much of the increased child care use after leaving welfare because of Atlanta's welfare-to-work program was covered by transitional child care benefits. In contrast, though Riverside HCD program group members were 7.3 percentage points more likely than control group members to use child care after leaving welfare because of earnings, there were no significant differences in receipt of transitional child care benefits.
Across the full follow-up period, then, most programs increased reliance on child care after leaving welfare because of earnings, but fewer than half increased receipt of transitional child care benefits. It has been noted that there was substantial cross-site variation in control group families' receipt of transitional child care benefits. In general, the findings suggest the need for expanding information about subsidies, altering eligibility or increasing funding, and simplifying the administration of subsidies to encourage families to use transitional child care benefits.
Impact findings on child care use in connection with mothers' recent spell of employment are yet more limited. None of the LFA or HCD programs increased child care use while employed in a recent spell (but see the findings presented below for the COS sample on sustained employment for mothers and focal children's time use in both programs in Riverside). Portland program group members, on the other hand, were 6.9 percentage points, or 16 percent, more likely than control group members to use child care while employed in a recent spell. While none of the LFA or HCD programs significantly increased the proportion of families with any out-of-pocket expenses for child care, program group members in Portland had significantly higher total monthly out-of-pocket child care costs than control group members. Portland program group members who used child care while employed paid approximately $82 more in total monthly out-of-pocket costs than control group members who used child care while employed.(23)
In general, the findings for child care as a support for employment are in keeping with the prediction of more extensive impacts for the full follow-up period for which employment impacts were more widespread than for a recent spell of employment when employment impacts were more limited. While there is a fairly widespread pattern of impacts on child care use while transitioning from welfare across the full follow-up, there is very little evidence of impacts on child care use for a recent spell of employment. The impacts that did occur were concentrated in Portland (and in Riverside, as discussed below), where there were impacts on sustained or recent employment. In Portland, there were impacts on full-time employment and employment in the last quarter of year 5 for families with young children.(24)
Child Care as a Context for Development
This section describes the extent and type of child care experienced by young school-age children in the Child Outcomes Study sample, whether used as a support for employment or not. It focuses on child care arrangements for the focal children, on how they spend time and their participation in activities. It should be kept in mind that all of the focal children were aged 8 to 10 and that all measures pertain to a very recent time period, either the past week or a recent weekday.
Control Group Levels of Child Care
Table 10.2 shows that between about 34 and 43 percent of focal children in the Child Outcomes Study control groups were reported by mothers to be in a regular child care arrangement "in the past week," that is, in the week prior to the five-year follow-up. The main arrangement was much more likely to be informal, such as care by a relative or neighbor in the child's home or another home (used by 24 to 36 percent of control group families, depending on site), than formal, such as care provided by a day care center or provided after school by the school (used by 8 to 19 percent). Children who were in a regular arrangement spent 20 to 26 hours per week on average in child care (that is, about 4 to 5 hours per day if care occurred five days a week). Only 6 to 10 percent of families reported relying on multiple arrangements. Only a small proportion of families reported relying on a provider aged 17 or under in the main arrangement for the focal child (about 3 to 5 percent). Use of self-care in the past week was reported for about 9 to 12 percent of control group families, which falls within the range reported for national samples.(25)
|
Site and Program |
Sample size | Program Group | Control Group | Difference (Impact) | Percentage Change (%) |
|---|---|---|---|---|---|
|
Used a regular child care arrangement in the past week (%) |
|||||
| Atlanta Labor Force Attachment | 580 | 41.5 | 39.7 | 1.8 | 4.6 |
| Atlanta Human Capital Development | 650 | 37.8 | 39.4 | -1.6 | -4.1 |
| Grand Rapids Labor Force Attachment | 394 | 41.9 | 43.4 | -1.5 | -3.4 |
| Grand Rapids Human Capital Development | 380 | 36.9 | 41.8 | -4.9 | -11.6 |
| Riverside Labor Force Attachment | 493 | 33.9 | 33.5 | 0.4 | 1.3 |
| Lacked high school diploma or basic skills | 313 | 27.1 | 29.5 | -2.4 | -8.0 |
| Riverside Human Capital Development | 406 | 28.3 | 29.7 | -1.4 | -4.8 |
|
Used a nonregular or sporadic child care arrangement in the past week (%) |
|||||
| Atlanta Labor Force Attachment | 580 | 4.4 | 3.6 | 0.8 | 23.3 |
| Atlanta Human Capital Development | 650 | 3.3 | 3.6 | -0.3 | -7.5 |
| Grand Rapids Labor Force Attachment | 394 | 4.1 | 5.4 | -1.3 | -23.9 |
| Grand Rapids Human Capital Development | 380 | 7.4 | 4.3 | 3.0 | 69.6 |
| Riverside Labor Force Attachment | 493 | 5.6 | 9.4 | -3.8 | -40.2 |
| Lacked high school diploma or basic skills | 313 | 5.9 | 10.4 | -4.5 | -43.4 |
| Riverside Human Capital Development | 406 | 8.9 | 10.2 | -1.3 | -13.1 |
|
Used multiple regular child care arrangements in the past week (%) |
|||||
| Atlanta Labor Force Attachment | 580 | 5.2 | 6.2 | -1.0 | -16.9 |
| Atlanta Human Capital Development | 650 | 4.1 | 6.2 | -2.1 | -33.3 |
| Grand Rapids Labor Force Attachment | 394 | 5.3 | 7.2 | -1.8 | -25.5 |
| Grand Rapids Human Capital Development | 380 | 3.8 | 6.8 | -3.1 | -44.9 |
| Riverside Labor Force Attachment | 493 | 8.0 | 9.6 | -1.6 | -17.0 |
| Lacked high school diploma or basic skills | 313 | 6.6 | 7.0 | -0.4 | -5.5 |
| Riverside Human Capital Development | 406 | 7.9 | 7.1 | 0.8 | 11.2 |
|
Number of hours child spent in all regular care arrangements in the past week |
|||||
| Atlanta Labor Force Attachment | 580 | 9.1 | 10.3 | -1.2 | -11.6 |
| Atlanta Human Capital Development | 650 | 8.7 | 10.1 | -1.3 | -13.3 |
| Grand Rapids Labor Force Attachment | 392 | 11.3 | 10.3 | 1.1 | 10.6 |
| Grand Rapids Human Capital Development | 378 | 7.5 | 10.0 | -2.6 | -25.4 |
| Riverside Labor Force Attachment | 492 | 8.4 | 6.6 | 1.7 | 26.4 |
| Lacked high school diploma or basic skills | 312 | 6.7 | 5.6 | 1.1 | 19.0 |
| Riverside Human Capital Development | 406 | 5.5 | 6.0 | -0.5 | -8.4 |
|
Among those using regular care arrangements in the past week, number of hours used |
|||||
| Atlanta Labor Force Attachment | 232 | 22.1 | 25.6 | ||
| Atlanta Human Capital Development | 249 | 23.2 | 25.2 | ||
| Grand Rapids Labor Force Attachment | 166 | 26.9 | 24.0 | ||
| Grand Rapids Human Capital Development | 147 | 20.1 | 24.5 | ||
| Riverside Labor Force Attachment | 166 | 25.3 | 19.5 | ||
| Lacked high school diploma or basic skills | 89 | 26.4 | 18.2 | ||
| Riverside Human Capital Development | 118 | 19.8 | 20.0 | ||
|
Any self-care in the past week (%) |
|||||
| Atlanta Labor Force Attachment | 578 | 10.1 | 9.8 | 0.3 | 3.1 |
| Atlanta Human Capital Development | 649 | 9.5 | 9.7 | -0.3 | -2.6 |
| Grand Rapids Labor Force Attachment | 392 | 10.8 | 11.6 | -0.8 | -6.9 |
| Grand Rapids Human Capital Development | 375 | 12.8 | 11.5 | 1.3 | 11.6 |
| Riverside Labor Force Attachment | 477 | 9.8 | 8.5 | 1.3 | 15.9 |
| Lacked high school diploma or basic skills | 304 | 8.2 | 4.9 | 3.2 | 65.1 |
| Riverside Human Capital Development | 396 | 5.7 | 5.2 | 0.5 | 10.2 |
|
Used a provider aged 17 or under in the main child care arrangement in the past week (%) |
|||||
| Atlanta Labor Force Attachment | 580 | 3.9 | 3.4 | 0.6 | 16.3 |
| Atlanta Human Capital Development | 650 | 3.9 | 3.7 | 0.2 | 5.7 |
| Grand Rapids Labor Force Attachment | 394 | 4.1 | 5.4 | -1.3 | -24.2 |
| Grand Rapids Human Capital Development | 380 | 2.9 | 5.3 | -2.4 | -45.9 |
| Riverside Labor Force Attachment | 492 | 2.9 | 4.8 | -1.9 | -39.0 |
| Lacked high school diploma or basic skills | 313 | 2.0 | 4.7 | -2.7 | -57.0 |
| Riverside Human Capital Development | 406 | 4.1 | 4.8 | -0.7 | -14.9 |
|
Main arrangement used in the past week was formal (%) |
|||||
| Atlanta Labor Force Attachment | 580 | 18.6 | 19.4 | -0.7 | -3.8 |
| Atlanta Human Capital Development | 650 | 18.2 | 18.9 | -0.6 | -3.4 |
| Grand Rapids Labor Force Attachment | 394 | 8.2 | 10.4 | -2.3 | -21.8 |
| Grand Rapids Human Capital Development | 380 | 6.1 | 10.2 | -4.1 | -40.6 |
| Riverside Labor Force Attachment | 492 | 2.6 | 8.0 | -5.4** | -67.2 |
| Lacked high school diploma or basic skills | 312 | 1.4 | 3.9 | -2.5 | -64.4 |
| Riverside Human Capital Development | 405 | 2.3 | 4.0 | -1.7 | -42.2 |
|
Main arrangement used in the past week was informal (%) |
|||||
| Atlanta Labor Force Attachment | 580 | 26.5 | 23.8 | 2.7 | 11.3 |
| Atlanta Human Capital Development | 650 | 22.2 | 23.9 | -1.8 | -7.3 |
| Grand Rapids Labor Force Attachment | 394 | 35.2 | 36.0 | -0.9 | -2.4 |
| Grand Rapids Human Capital Development | 380 | 31.0 | 34.4 | -3.4 | -9.8 |
| Riverside Labor Force Attachment | 492 | 33.1 | 28.9 | 4.2 | 14.5 |
| Lacked high school diploma or basic skills | 312 | 27.0 | 27.8 | -0.8 | -3.0 |
| Riverside Human Capital Development | 405 | 26.9 | 28.0 | -1.1 | -3.9 |
|
Reported child care as a barrier to school, job training, or work (%) |
|||||
| Atlanta Labor Force Attachment | 575 | 15.4 | 16.3 | -0.9 | -5.3 |
| Atlanta Human Capital Development | 645 | 16.2 | 16.5 | -0.3 | -1.6 |
| Grand Rapids Labor Force Attachment | 392 | 30.0 | 30.8 | -0.8 | -2.6 |
| Grand Rapids Human Capital Development | 377 | 25.2 | 29.2 | -4.1 | -13.9 |
| Riverside Labor Force Attachment | 489 | 27.4 | 21.2 | 6.2 | 29.3 |
| Lacked high school diploma or basic skills | 312 | 21.6 | 18.1 | 3.5 | 19.6 |
| Riverside Human Capital Development | 402 | 20.8 | 18.7 | 2.0 | 10.8 |
| SOURCE: Child Trends calculations from the Five-Year Child Outcomes Study survey (mother reports). NOTES: See Appendix A.2. |
|||||
In sum, findings for control group families indicate use of child care by a minority, but a large one, of families even when the focus is limited to school-age children. When care was used, it involved a substantial portion of the children's day. These findings suggest that attention to the nature of children's child care arrangements should not cease with low-income children's entry into school.
Impacts on Child Care Use
1. Child Care for the Focal Child
There is no evidence on measures of child care use for the focal child in the past week that assignment to one of the LFA or HCD programs affected extent of reliance on child care arrangements at the time of the five-year follow-up. There were no impacts on families' use of a regularly scheduled child care arrangement or on sporadic use of child care in the past week. Total hours in child care across all regular arrangements did not differ in any of the LFA or HCD programs, nor did reliance on multiple arrangements.
There was very little evidence of a shift in type of arrangements relied upon. Notably, there were no program impacts on any use of self-care or on reliance on a child care provider aged 17 or under in the primary arrangement (but see the findings below on time spent with only a young peer or sibling when mothers reported time use on a recent weekday). Thus, there is no indication of an increase in reliance on these potentially problematic arrangements by LFA or HCD program group members when considering child care arrangements in the week prior to the interview.
With a single exception (the Riverside LFA program diminished reliance on formal child care), there were no impacts on use of formal and informal child care as the main arrangement.
2. Supervision and Activities for the Focal Child
Findings will now be presented for the measures of with whom the focal child spent his or her time on a recent weekday afternoon and evening. Because mothers were also asked to report whether the child was in a child care setting for this time period (child care in a center or program, child care by a babysitter, or care in either context on a drop-in basis), it is possible to directly juxtapose impacts on child care and impacts on mothers' reports of whom the child was with. For example, it will be possible to identify whether there were impacts on supervision (whom the child was with) even in the absence of impacts on measures of child care. This set of questions again addresses whether children spent time in self-care, here also asking about time spent with only a young peer or sibling (under age 13). Finally, this section reports on participation in activities or lessons (such as a sport, club, or music lesson) during after-school hours.(26) (See Table 10.3.)
| Site and Program | Sample Size | Program Group | Control Group | Difference (Impact) | Percentage Change (%) |
|---|---|---|---|---|---|
|
Number of half-hour time periods spent with the mother (0-18 periods from 3 pm to midnight) |
|||||
| Atlanta Labor Force Attachment | 578 | 12.9 | 12.7 | 0.2 | 1.6 |
| Atlanta Human Capital Development | 649 | 13.2 | 12.7 | 0.5 | 3.6 |
| Grand Rapids Labor Force Attachment | 392 | 12.5 | 12.2 | 0.3 | 2.5 |
| Grand Rapids Human Capital Development | 376 | 13.0 | 12.2 | 0.8 | 6.8 |
| Riverside Labor Force Attachment | 477 | 13.4 | 14.3 | -0.9* | -6.3 |
| Lacked high school diploma or basic skills | 304 | 13.9 | 14.8 | -0.9* | -6.2 |
| Riverside Human Capital Development | 396 | 14.1 | 14.9 | -0.8* | -5.5 |
|
Number of half-hour time periods spent with an adult other than the mother (0-18 periods from 3 pm to midnight) |
|||||
| Atlanta Labor Force Attachment | 578 | 3.6 | 3.7 | -0.1 | -3.7 |
| Atlanta Human Capital Development | 649 | 3.2 | 3.7 | -0.4 | -12.0 |
| Grand Rapids Labor Force Attachment | 392 | 4.4 | 4.5 | -0.1 | -2.0 |
| Grand Rapids Human Capital Development | 376 | 4.2 | 4.5 | -0.3 | -6.5 |
| Riverside Labor Force Attachment | 477 | 3.5 | 2.5 | 0.9** | 37.4 |
| Lacked high school diploma or basic skills | 304 | 2.8 | 2.1 | 0.6 | 29.8 |
| Riverside Human Capital Development | 396 | 2.9 | 2.1 | 0.9* | 42.0 |
|
Any time spent without an adult present from 3 p.m. to midnight (%) |
|||||
| Atlanta Labor Force Attachment | 578 | 11.5 | 12.7 | -1.1 | -9.0 |
| Atlanta Human Capital Development | 649 | 12.1 | 13.1 | -1.0 | -7.5 |
| Grand Rapids Labor Force Attachment | 392 | 5.4 | 7.2 | -1.8 | -24.5 |
| Grand Rapids Human Capital Development | 376 | 3.5 | 7.3 | -3.8 | -51.7 |
| Riverside Labor Force Attachment | 477 | 11.1 | 10.2 | 0.9 | 8.9 |
| Lacked high school diploma or basic skills | 304 | 15.2 | 9.8 | 5.5 | 55.9 |
| Riverside Human Capital Development | 396 | 10.2 | 9.6 | 0.6 | 6.3 |
|
Number of half-hour time periods spent with only a young peer or sibling (under age 13, 0-18 periods from 3 pm to midnight) |
|||||
| Atlanta Labor Force Attachment | 578 | 0.9 | 0.9 | 0.0 | 2.9 |
| Atlanta Human Capital Development | 649 | 0.9 | 0.9 | 0.1 | 7.4 |
| Grand Rapids Labor Force Attachment | 392 | 0.8 | 0.9 | -0.2 | -17.2 |
| Grand Rapids Human Capital Development | 376 | 0.5 | 0.9 | -0.4* | -39.4 |
| Riverside Labor Force Attachment | 477 | 0.7 | 0.6 | 0.1 | 24.0 |
| Lacked high school diploma or basic skills | 304 | 0.7 | 0.5 | 0.2 | 30.5 |
| Riverside Human Capital Development | 396 | 0.6 | 0.6 | 0.1 | 11.7 |
|
Any time periods in child care (sitter, center or program, or drop-in child care) from 3 p.m. to midnight (%) |
|||||
| Atlanta Labor Force Attachment | 580 | 15.1 | 15.5 | -0.5 | -3.0 |
| Atlanta Human Capital Development | 650 | 10.6 | 15.3 | -4.7* | -30.5 |
| Grand Rapids Labor Force Attachment | 394 | 14.2 | 18.7 | -4.5 | -24.1 |
| Grand Rapids Human Capital Development | 380 | 14.2 | 18.2 | -4.0 | -21.8 |
| Riverside Labor Force Attachment | 493 | 1.4 | 4.4 | -2.9* | -67.0 |
| Lacked high school diploma or basic skills | 313 | 1.6 | 1.5 | 0.1 | 10.0 |
| Riverside Human Capital Development | 406 | 5.8 | 1.7 | 4.0** | 232.0 |
|
Any time periods in an activity or lesson from 3 p.m. to midnight (%) |
|||||
| Atlanta Labor Force Attachment | 580 | 8.2 | 8.4 | -0.2 | -2.3 |
| Atlanta Human Capital Development | 650 | 7.5 | 8.5 | -1.0 | -11.2 |
| Grand Rapids Labor Force Attachment | 394 | 3.9 | 6.1 | -2.3 | -37.0 |
| Grand Rapids Human Capital Development | 380 | 5.8 | 5.5 | 0.3 | 5.3 |
| Riverside Labor Force Attachment | 493 | 2.8 | 6.0 | -3.2 | -53.2 |
| Lacked high school diploma or basic skills | 313 | 1.8 | 5.5 | -3.7 | -67.8 |
| Riverside Human Capital Development | 406 | 2.2 | 4.9 | -2.6 | -54.1 |
| SOURCE: Child Trends calculations from the Five-Year Child Outcomes Study survey (mother reports). NOTES: See Appendix A.2. |
|||||
There were virtually no impacts on any of these variables in the Grand Rapids and Atlanta programs. However, there was a cluster of findings for both programs in Riverside. They indicate a decrease in time with mother and an increase in time with an adult other than the mother in both programs. In the aggregate, none of the programs increased the proportion of children who spent time in after-school activities or who spent time unsupervised by an adult. For the subgroup of families in which the mother lacked basic education at baseline in Riverside, the LFA program increased the proportion of children who spent time without an adult present (though a sizable difference, this was just above the cutoff for significance). These findings are presented in greater detail below.
3. Impacts on the Riverside Programs
The LFA program: all families. For all families assigned to the LFA program in Riverside (irrespective of mothers' educational level at baseline), focal children in the program group spent significantly less time with their mothers and more time with an adult other than the mother. These impacts did not reflect large differences in amounts of time spent with the mother and another adult, however, averaging about a single half-hour less with the mother and about a single half-hour more with another adult.
This increase in time spent with another adult was reported in the absence of an impact on mothers' report of child care use during this period. The impact noted above on type of care for the Riverside LFA program can now be placed in a larger context. In this program, there was apparently a shift in type of care away from reliance on formal child care (detected in the measure of child care use in the past week) and toward reliance on another adult to supervise the child in a situation mothers did not describe as child care.
The LFA program: families in which the mother lacked a high school diploma or GED at baseline. For families in the LFA program in which the mother lacked a high school diploma or GED at baseline, the pattern was similar to that for all LFA families in that program group children spent less time with their mothers (with the difference in time periods again just under a single half-hour) and were less likely to be in a formal child care setting on a regular basis (though the difference in formal child care use was just above the cutoff for statistical significance in this sample). However, unlike those in the full LFA sample, children of mothers in the LFA program who lacked this credential at baseline did not spend more time with another adult. Rather, they were more likely than the control group and the HCD program group to have spent time without an adult present, though this sizable difference, comparable to impacts on other measures, was just above the cutoff for statistical significance. This is the one indication of a finding pointing to a program increasing self-care, and it is of concern that it occurs in a higher-risk subgroup.
The HCD program. Children of HCD program group members spent less time with the mother and more time with another adult than their control group counterparts. These differences also involved only about a single half-hour. However, children of HCD program group members were more likely than their control group counterparts to be in what the mother described as child care (including center or program care, a babysitter, or drop-in care in either setting). Indeed, children of mothers in the HCD program were also significantly more likely than children of mothers with limited education in the LFA program to be in child care on a recent weekday.
The clustering of impacts on measures of supervision and child care in the Riverside site is in accord with impacts on employment in the Child Outcomes Study sample. The measure of mothers' employment that is most closely aligned in terms of time frame is a survey measure of full-time employment at the time of the survey. Impacts on this measure were statistically significant only in the Riverside site and were moderate to large in size.(27) For families in the Riverside LFA program (irrespective of mothers' educational level at baseline), there was a difference of 11 percentage points in reported current full-time employment (40.6 percent of program group mothers and 29.6 percent of control group mothers reported current full-time employment) and the impact for those in the LFA program who lacked a high school diploma or GED was 15.5 percentage points (39.2 of program group members compared to 23.7 of control group members reporting employment). The impact for the Riverside HCD program was 9 percentage points (32.7 percent of program group mothers and 23.7 percent of control group mothers reported current full-time employment). Thus, the child care and activities findings correspond closely to impacts on concurrent employment, specifically full-time employment as reported by the mother, in the Riverside site.
Further impacts on adult outcomes in the Child Outcomes Study sample may also be relevant. At the time of the five-year follow-up, Riverside program group members were more likely than control group members to be cohabiting (significant for the full LFA group; not shown), to have a new baby (significant for the full LFA group and also for those in the LFA group lacking a high school diploma or GED at baseline),(28) and to be living with a partner or spouse as well as children (not shown). In the Riverside LFA group, program group children spent less time with the mother and more time with another adult (though not in the context of what the mother describes as child care). One interpretation of these findings is that program group children more often than control group children were cared for in their mother's absence by the mother's spouse or cohabiting partner.
Conclusions
Impact findings for child care differed according to the time period considered. Across the five-year follow-up, most programs showed impacts on mother's reliance on child care while leaving welfare because of earnings, and several programs showed impacts on use of transitional child care benefits. However, for recent child care use, there were few impacts. Fortunately, this pattern of very few impacts on concurrent measures of child care and supervision includes the measures of self-care and being only with a young peer or sibling. Unfortunately, there is no indication here of the possibly salutary influence of participation in after-school activities and lessons.(29)
For recent child care, it is primarily when going beyond traditional measures of child care that a pattern emerged. When asking about whom the child was with on a recent weekday afternoon, findings indicated a slight shift away from time with the mother and toward time with another adult in both Riverside programs. In the LFA program, this was accompanied by diminished participation in formal child care as the primary arrangement.
In sum, in most of the programs the picture that emerges is of intermittent or weak impacts, captured best by measures across time and considering more than one child in a family. Only in Portland were impacts apparent for a recent spell of employment, and only in Riverside was there evidence for the focal child of impacts occurring on type of care and nature of supervision at the time of the final follow-up.(30)
What do these findings indicate regarding child care as a support for employment in these welfare-to-work programs? The fact that child care impacts mirrored the broad pattern of employment findings, of cumulative effects rather than widespread effects at the time of the final follow-up, suggests that child care use was broadly coordinated with employment, and often functioning to enable employment. Families with an employed mother using child care tended to have out-of-pocket expenses for child care. Yet there were few impacts on the use of transitional child care benefits, and overall use of these benefits was fairly low and quite variable across sites. There are several possible interpretations of this finding, including the possibilities that restrictions on the type of care that could be subsidized with these benefits, limited outreach and information about them provided by caseworkers, or the short period (one year) in which benefits could be used after moving from welfare limited families' abilities to make use of these benefits. Also, it is important to note that across sites there were differences in the types of child care that were encouraged and facilitated. For example, staff members in Atlanta encouraged use of formal child care by reimbursing welfare recipients only for use of licensed or certified care. In contrast, staff in Riverside encouraged informal child care arrangements by emphasizing that welfare recipients should choose child care that they would be able to afford once they were no longer eligible for benefits.
What of child care as a context for children's development? The findings here cannot reflect on the critical issue of child care quality, but do provide a picture of the extent of children's exposure to child care, type of care, use of self-care, and exposure to after-school activities and lessons. Findings generally do not point to elevations in potentially harmful forms of care such as self-care or care by a young caregiver, though findings may suggest the need to continue to monitor this issue. At the same time, they do not point to systematic increases in the possibly beneficial environments of after-school activities and lessons or in use of formal child care arrangements. Rather, children appear to have received an elevated "dose" of child care over time in most programs, but as indicated in the measures of recent child care, this dose does not appear to have been large or sustained in most programs. However, in Riverside (as in Portland) a few findings point to more sustained child care impacts. Some findings in the Riverside LFA program also point to a shift away from formal child care arrangements and toward informal care that the mother may not describe as a child care arrangement. Chapter 12 returns to the possibility that this pattern of child care impacts may help to explain impacts on child outcomes, with a particular focus on findings in the Riverside site, where the impacts extended to current supervision and child care, as well as type of care.
1. Child care benefits may have changed over time, particularly after the 1996 welfare reform and consolidation of child care funding into the Child Care and Development Fund. A thorough review of these changes by site is beyond the scope of this chapter. Moreover, many sample members may not have remained eligible for child care assistance after 1996 since they were off welfare and had been off for some time.
2. Tout et al., 2001.
3. Tout et al. document differences by income level in reasons for child care use by nonemployed mothers, with employment preparation activities playing a larger role for lower-income families.
4. While some previous research suggests that participation in an organized after-school program can have favorable implications for the development of children in low-income families (Bos et al., 1999), these implications are likely to vary also by the quality of after-school care. The quality of child care arrangements has equally important implications for preschool-age children.
5. All impacts on leaving welfare because of earnings, on using child care after leaving welfare because of earnings, and on using transitional child care benefits are experimental. That is, all sample members are included in the analyses.
6. For ease of presentation, throughout this chapter child care during the most recent employment spell will simply be referred to as child care while employed.
7. There are 13,726 children in the Five-Year Client Survey sample: the sample in four sites (Atlanta, Grand Rapids, Riverside, and Portland) for whom there are interviews as well as administrative data (see Chapter 2). Of these children, 6 percent were aged 2 or under at baseline (about 5 to 7 at follow-up), 25 percent were aged 3 to 5 at baseline (about 8 to 10 at follow-up), 22 percent were aged 6 to 9 at baseline (about 11 to 14 at follow-up), and 33 percent were aged 10 to 18 at baseline (about 15 to 23 at follow-up). Approximately 11 percent were born into the families participating in the client survey sample between baseline and the five-year follow-up and another 2 percent did not have their birthdays reported. For more information about the percentage of families containing children born after random assignment, see Chapter 9 of this report. Those who did not have their birthdays reported were included in the all-child analyses but excluded from analyses focusing on age subgroups. The child care measures linking child care and employment focus on children who were under age 13 at the time that the child care was used.
8. See the discussion of control group exposure to new policies in Chapter 1.
9. Tout et al., 2001.
10. For example, findings from the 1997 National Survey of American Families indicate that 55 percent of children aged 6 to 9 and 35 percent of those aged 10 to 12 with employed mothers were in a supervised nonparental child care arrangement (Capizzano, Tout, and Adams, 2000).
11. National Research Council and Institute of Medicine, 2000.
12. National Research Council and Institute of Medicine, 2000; Vandell and Wolfe, 2000
13. NICHD Early Child Care Research Network, 2000; see also U.S. Department of Education, National Center for Education Statistics, 1995. The NICHD findings suggest that extensive participation in formal child care arrangements may also simultaneously increase children's externalizing behavior problems.
14. Zaslow et al., 1998; Zaslow et al., 1999b.
15. In the analyses presented here, formal care included care in a day care center, before- or after-school care sponsored by a school or church, summer camp, a boys'/girls' club, YMCA/YWCA, or a lesson/activity. Informal care included care by the mother's spouse/partner, a relative, or a neighbor.
16. Type of care is more appropriately studied through interview measures than quality of care, which is best studied through direct observation. The interview methodology of the NEWWS Evaluation thus better addresses the issue of type of care than quality of care. The lack of direct observation of child care quality is a clear limitation in this chapter.
17. Capizzano, Tout, and Adams, 2000.
18. Capizzano, Tout, and Adams, 2000.
19. Bos et al., 1999
20. See Chapter 1 for a discussion of these differences.
21. The impact in the Grand Rapids HCD program on using child care after leaving welfare for earnings was significant at the p = .19 level.
22. The impact on receipt of transitional child care benefits is significant at the p = 0.11 level for the Grand Rapids LFA program and at the p = 0.15 level for the HCD program.
23. The total monthly out-of-pocket cost of child care was $225 ($114 divided by 0.507, or the proportion of the sample who used child care while employed) for program group members and $142 for control group members. This is a nonexperimental comparison.
24. Although there were no impacts in the aggregate in Portland, there were increases in employment in the last quarter of year 5 and in full-time employment for families who had preschool-age and young school-age children at baseline.
25. See Capizzano, Tout, and Adams, 2000.
26. Results are summarized in terms of the number of half-hour periods (out of a possible 18) between 3 P.M. and midnight on the chosen weekday that a child was with someone or engaging in a particular care situation or activity. Where distributions were limited, results are summarized instead in terms of the proportion of children for whom this companion or activity was reported at all.
27. See Appendix I.
28. See Appendix I
29. As noted above, while findings of one recent evaluation suggest that participation in after-school child care can be beneficial for children from low-income families, implications for children are likely to vary with the quality of care.
30. The information collected in Atlanta, Grand Rapids, and Riverside on child care and child activities on a recent weekday was not collected in Portland.
Impacts on the Well-Being of All Children
The effects of welfare-to-work programs on child well-being are addressed in two chapters in this report. This chapter captures the breadth of the effects of seven welfare-to-work programs on the well-being of children of all respondents in the Five-Year Client Survey. Program effects are evaluated for children in four age groups,В from toddlers to adolescents at the time of study entry,В and on outcomes in two domains of child developmentВ academic functioning and health and safetyВ as well as on and on selected other selected outcomes. Chapter 12 analyzes, in greater depth, effects on child well-being at the five- year follow-up for a subset of "focal" children, aged 3 to 5 at random assignment, in six welfare-to-work programs in Atlanta, Grand Rapids, and Riverside.(1) Details about the samples examined in these two chapters and how they are derived from the full impact sample in this report are shown in Figure 11.1.
Figure 11.1
Samples and Subsamples Used In Chapter 11 and 12
As discussed in Chapter 1, a central goal of the federal JOBS program, under the 1988 Family Support Act (FSA), was to move single mothers from public assistance to paid employment. This goal was largely implemented by imposing strict participation and work requirements. For 20 years prior to 1988, women receiving welfare who had children under age 6 generally were not subject to these mandates. With the passage of the FSA, women with children as young as age 3 (or as young as age 1, at state option) were newly designated as mandatory participants. Thus, in the early 1990s there was much interest in how welfare-to-work programs might affect children, especially very young children. that differed from the past, particularly for mothers who had children aged less than 6. The well-being of children remained of central concern and produced considerable debate at the passage of this legislation as well as the more recent 1996 welfare reform law (Personal Responsibility and Work Opportunity Reconciliation Act, or PRWORA) that similarly imposed requirements to move welfare recipients into employment and imposed participation mandates on women with children as young as age 1 (or younger, at state option). A two-year follow-up evaluation of the effects of the NEWWS programs on child well-being, which outcomes that included a special study of preschool-age children, was one of the first to inform this debate. The analysis in these two chapters represents the first evaluations of long-term effects of mandatory welfare-to-work programs on the well-being of children.
Earlier chapters in this report primarily focused on the effects of welfare-to-work programs on the outcomes targeted by these programs, such as education, employment, and welfare receipt. As these chapters revealed, during the five-year follow-up period the NEWWS programs, regardless of whether they were program was employment- or education-focused, generally increased participation in employment-related activities, employment and earnings and reduced public assistance payments. for all respondents in the study. The consistency of these findings across programs and sites is not surprising given the stated goals of the programs. Appendix I, summarizes these effects for the Five-Year Client Survey sample and shows that, with some exceptions for the Riverside LFA and HCD programs, and, most important, the Portland program, the pattern of impacts for the respondents in the Five-Year Client Survey sample were quite similar to the impacts for the full impact sample are presented and discussed in Chapters 4-6.
Unlike early childhood intervention programs, the NEWWS programs were not structured to directly affect the well-being of children. It is still possible, however, that welfare-to-work programs could produced either favorable or unfavorable effects on child outcomes. Current theories hypothesize that, by affecting the behavior of mothers' programs could also indirectly affect children's well-being through, for example, changes in mothers' psychological well-being or parenting skills or styles, in child care, and/or in family life and material resources. Some of these changes may bode well for certain child outcomes but prove problematic for others, and thus the effects of a given type of welfare-to-work program on child outcomes may not be uniformly favorable or unfavorable across developmental domains.(2) Also, it is possible that program impacts on child outcomes may vary, depending on whether the effects are enduring, the extent of the exposure to the program, or on the combined effects of all program impacts.
The ways that welfare programs might affect child well-being can be depicted in a simple conceptual framework.(3) A program's features, such as its message, sanctioning rates, and monitoring, can affect the targeted or direct outcomes of the program, such as employment, public assistance receipt, income, and education, and nontargeted outcomes, such as child care and parenting behavior, and ultimately leading to effects on child outcomes. Prior research, though largely based on nonexperimental studies, provides a basis for predicting and understanding how effects on key outcomes such as education, employment, earnings, and income may affect children. For example, mothers' increased educational attainment and employment (depending on the extent and quality of the job), and/or family income may prove beneficial to children in low-income families.(4) Mothers' psychological well-being and parentingВ shown to be related to children's developmental outcomes in the nonexperimental literature(5)В may also be affected by these programs, though the extent and direction of likely change may vary.(6) Child care is another way in which mothers' employment may affect children. Unstable or low-quality child care may produce detrimental effects on children's development.(7)
This chapter reviews general hypotheses and impact findings on child outcomes, organized by child age. It discusses whether or not mothers with children in various age groups may have behaved and responded differently to these welfare-to-work programs.(8),В (9) Also, welfare-to-work programs might have affected children differently at different points in their development. For example, toddlers may be the most vulnerable to possible negative effects of mothers' employment, particularly if they are placed in poor-quality child care. Adolescents, in contrast, may have the most to gain if they are placed in enriching after-school programs. Older children may be left unsupervised or may take on more responsibilities at home as mothers join the workforce, which could lead to unfavorable effects on their development, particularly their social behavior. In reviewing the discussion of impacts throughout this chapter, it is important to remember that, as discussed in Chapter 1, some control group members in Atlanta and Grand Rapids became eligible for program services after the third year of the follow-up period. However, it appears that program impacts on earnings and welfare receipt were only slightly affected by the end of the control group embargo in these sites. The chapter then ends with a discussion, again by child age, of what may have led to that program effects on children.
Table 11.1 summarizes the impacts on the limited set of outcomes examined in this chapter, by child age.
| В | Academic Functioning | Health and Safety | Other | ||||||
|---|---|---|---|---|---|---|---|---|---|
| В | Repeated a Grade | Suspended or Expelled | Attended a Special Class for a Condition | Dropped Out of School | Had a Condition That Required Frequent Medical Attention | Required Emergency Room Visit | Had a Condition That Impeded on Mother's Ability to Go to Work or School | Did Not Live With Mother Because She Could Not Care for Child | Had a Baby as a Teen |
|
Toddlers |
|||||||||
| Grand Rapids LFA | В | F | В | - | F | В | В | В | - |
| Grand Rapids HCD | В | F | В | - | В | В | В | В | - |
| Portland | В | В | В | - | В | В | В | В | - |
|
Preschool-age children |
|||||||||
| Atlanta LFA | В | В | В | - | U | В | U | В | - |
| Atlanta HCD | В | В | В | - | U | В | В | В | - |
| Grand Rapids LFA | В | В | В | - | В | В | В | В | - |
| Grand Rapids HCD | U | В | В | - | В | В | В | В | - |
| Riverside LFA | F | В | В | - | f | В | F | S | - |
| В InВ needa | F | В | В | - | В | В | F | В | - |
| Riverside HCD | В | В | В | - | В | В | В | В | - |
| Portland | В | В | В | - | В | В | В | В | - |
|
Young school-age children |
|||||||||
| Atlanta LFA | В | В | В | В | В | В | В | В | - |
| Atlanta HCD | В | В | В | В | В | В | В | В | - |
| Grand Rapids LFA | u | U | В | В | В | В | В | В | - |
| Grand Rapids HCD | В | В | В | В | F | В | F | В | - |
| Riverside LFA | В | F | В | В | В | В | В | S | - |
| В InВ needa | u | F | В | В | В | В | В | S | - |
| Riverside HCD | В | F | В | В | В | F | В | S | - |
| Portland | В | В | В | В | В | В | В | В | - |
|
Adolescents |
|||||||||
| Atlanta LFA | В | F | В | В | В | В | В | В | f |
|
Atlanta HCD |
В | В | В | В | В | В | В | В | В |
| Grand Rapids LFA | U | В | u | В | u | В | В | В | В |
| Grand Rapids HCD | U | В | u | В | В | В | В | В | В |
| Riverside LFA | U | В | u | В | F | В | В | S | В |
| В InВ needa | u | В | U | В | F | В | В | s | U |
| Riverside HCD | U | В | U | U | В | В | U | В | u |
| Portland | В | В | В | В | В | В | В | В | В |
| NOTES: "F" indicates a statistically significant favorable impact. "U" indicates a statistically significant unfavorable impact. "f" indicates a favorable impact above the cutoff for statistical significance but part of the overall pattern. "u" indicates an unfavorable impact above the cutoff for statistical significance but part of the overall pattern. "S" indicates a statistically significant impact that could not be categorized as favorable or unfavorable. "s" indicates an impact above the cutoff for statistical significance but part of the overall pattern. See Chapter 2 for the definition of a pattern. "-" indicates that a measure was not appropriate for a particular child age group. Blank spaces indicate that there were no impacts. a Sample members lacked a high school diploma or basic skills at random assignment. |
|||||||||
Key Findings
- There is little evidence that these welfare-to-work programs had long-term effects on children in general. Effects that did emerge were clustered by age of the child, with most effects occurring among children who were adolescents at the start of the study (aged 15 to 23 at the five-year follow-up).(10) There is less convincing evidence that the effects of these welfare-to-work programs on children are clustered or vary by domain of child development, program approach, or site.
- There were few differences in the effects on child outcomes by program approach.
- Few program effects were found for children who were toddlers at the start of the study (aged 6 and 7 at the five-year follow-up). It is noteworthy, however, that on the few outcomes measured in the two sites with data for children in this age group more unfavorable effects were not found. In fact, both Grand Rapids programs produced a consistent pattern of favorable effects (though not always statistically significant and thus not always shown in Table 11.1). Perhaps one explanation is that the Grand Rapids programs were more likely to increase part-time employment for the mothers of these toddlers. A similar pattern of effects on part-time employment was not found in Portland or for mothers with older children.
- Few program effects were found for children who were preschool-age or young school-age at the start of the study (aged 8 to 10 and 11 to 14, respectively, at the five-year follow-up). Any effects that did occur for these children were both favorable and unfavorable.
- The effects of the welfare-to-work programs on adolescents who were aged 15 to 23 at the five-year follow-up were generally unfavorable for children with parents in the Grand Rapids and Riverside programs, particularly in the Riverside HCD program. These effects were concentrated in the academic functioning domain and included increased grade repetition, increased attendance in a special class, and, in one program, increased likelihood of dropping out of school. Mothers of adolescents in these two sites experienced the largest increases in employment during the first year of follow-up, decreased average income from earnings and welfare benefits, and, in two programs, were more likely to be married at the five-year follow-up point. Adolescents' academic functioning may have been especially vulnerable to any one or all of these changes.
Measurement Issues
The measures of child well-being analyzed in this section were constructed from mothers' responses to the Five-Year Client Survey. Respondents were asked about their children's academic functioning, health and safety, and other outcomes. (See Table 11.1.) Respondents answered for their own children whether biological children, legally adopted children, or stepchildren. Although they were asked about all children regardless of age, this analysis is limited to children who were aged 18 or under at random assignment. This section provides a discussion about the construction of these measures. (Details about these measures can also be found in Appendix J.)
Respondents were asked a variety of questions about their children, concerning events or situations since a focal date. For most respondents the focal date was the date of the two-year interview. For those who did not answer the two-year survey, the focal date was the date of random assignment. Thus, for most respondents, child outcomes are measured for years 3 to 5 of follow-up (that is, since the two-year interview point). Some outcomes, such as dropping out of school, were not restricted to the period of time since the focal date. Instead these outcomes were measured "ever" in a child's life.
The first group of measures captures children's academic functioning including whether or not they had been suspended or expelled from school, repeated a grade, dropped out of school, or attended a special class or special school for a physical, emotional, or mental condition.
The second group of measures captures children's health and safety, including whether they had a physical, emotional, or mental condition that required frequent medical attention, frequent use of medication, or the use of any special equipment; whether they ever had a physical, emotional, or mental condition that demanded a lot of attention and made it difficult for the respondent to attend work or school; and whether they had an accident, injury, or poisoning that required a visit to a hospital emergency room or clinic. This last outcome is only a rough proxy for child safety. On the one hand, it may measure neglect if children are experiencing more accidents or injuries as mothers increase their work effort and children are left unsupervised, with abusive partners or other adults, or, possibly, placed in unsafe child care arrangements. On the other hand, this outcome may simply reflect a mother's ability to purchase medical care. Emergency room use may be used as a replacement for visits to a doctor or a clinic.
The final group of measures concerns living arrangements (whether a child did not live with their mother because she could not care for them) and teenage parents (whether a child gave birth as a teenager). In this analysis children were considered to have been teenage parents if they had a child at age 18 or under during the five-year follow-up period. Whether children did not live with their mother because she could not care for them may have numerous interpretations: It may capture the consequence or result of a government intervention in the family, for example, being forced to place children in foster care. However, it is also possible that a mother voluntarily elected to place her children in another living arrangement, which may provide a better environment. This outcome may be also interpreted more generally as a measure of household composition or living arrangements rather than a direct measure of child well-being.
Outcomes in this chapter are presented for any child in the family and for each child in the family. Measures presented for any child indicate the percentage of families in which at least one child experienced a certain outcome (for example, "any child ever repeated a grade" indicates that at least one child in the family repeated a grade) and provide a general snapshot of child outcomes at the family level. These measures are similar to those analyzed at the two-year follow-up point. Measures presented for each child in the family indicate the percentage of children who have experienced a certain outcome. Thus, unlike most of the information collected in the survey, these outcomes are specific to a child within a family. Each child in a family who was 18 or under at random assignment is represented in the impact analysis.(11) The 5,342 families in this analysis had a total of 13,726 children.
Four age groups of children are examined:
- Children who were toddlers at random assignment (aged 6 and 7 at the five-year follow-up; 7 percent were age 8);(12)
- Preschool-age children (8 to 10 at the five-year follow-up; 4 percent were age 11);
- Young school-age children (aged 11 to 14 at the five-year follow-up; 3 percent were age 15); and
- Adolescents (aged 15 to 23 at the five-year follow-up; 78 percent were age 15 to 20; 22 percent were 21 to 23, and 1 percent were 24).
The ages of respondent's children varied across sites. As discussed in Chapter 1, mothers with children aged 3 or over were required to participate in most sites. However, in Grand Rapids and Portland, the participation mandate was extended to single mothers with a child as young as age 1. Although this analysis treats children mutually exclusively, the mothers of these children within and across each age group are not mutually exclusive: 41.5 percent of respondents have more than one child within one age group and 49.6 percent have at least one child in more than one age group. The sample size for each of these age categories and the general structure of how the samples were derived for these two chapters are presented in Figure 11.1.
Although the outcomes covered in this chapter provide important information about child well-being, they have some limitations. First, all of them are based on mothers' reports, which may differ from teachers' or children's reports or from direct assessments of the children.(13) Second, the outcomes provide only a snapshot of particular domains of children's development. For example, children's problem behavior (such as their expressions of anxiety, depression, or aggression) and positive behavior (such as their interaction with peers and others) are not captured, and it may be that the NEWWS programs are most likely to affect these behaviors. Measures collected in the Child Outcomes Study described in Chapter 12 address these limitations.
Third, similar measures were collected and constructed for each child regardless of age, yet these measures have different implications by child age. For example, partly because of age requirements for employment, an adolescent who repeats a grade may be much more likely to drop out of school and possibly enter the labor force than a younger school-age child who repeats a grade. Dropping out of school is highly correlated with future labor force participation.(14) In addition, control group levels on these outcomes might differ by age group: Control group levels of suspension or expulsion are naturally higher for adolescents than for early school-age children, making it harder for programs to produce any statistically significant changes.
To provide some basis for evaluating the magnitude of impacts, Tables 11.2-11.6 and Tables 12.1-12.5 report effect sizes in the last column. The accompanying text box describes effect sizes in more detail.
|
Effect Sizes and the Magnitude of Effects on Child Outcomes Evaluating the effects of welfare-to-work programs on child outcomes also requires an assessment of whether the effects are big or small. An impact may be statistically significant, but is it large enough to be deemed important? Evaluating the size of an impact on various measures of adult economic outcomes is relatively straightforward. For example, most can assess whether or not an impact of $200 is a big or small effect on an individual's annual income. It is much more challenging to evaluate whether or not a 10 point change in a scale measuring a child's behavioral problems or a 5 percent change in a scale measuring school progress is big or small. One method of assessing impact size is to standardize it. To do this, impact estimates can be converted into effect sizes. Effect sizes are computed by dividing the impact (the difference in outcomes between the program group and the control group) by the standard deviation of the outcome for the control group. The value of the effect size provides a standardized measure of the program impact that can be used to compare program impacts on outcomes with very different scales. Effect sizes generally range from 0 to 1; a larger absolute value indicates a larger impact on an outcome and a smaller absolute value indicates a smaller impact. Effect sizes rather than percentage changes are reported in the last column of Tables 11.2-11.6 and Tables 12.1-12.5. How large are these effects? Generally effect sizes of 0.1, 0.3, and 0.5 are considered small, medium and large, respectively.a However, these benchmarks are based on nonexperimental studies that cover a broad range of topics. One method is to compare effect sizes on adult economic outcomes and effects sizes on child outcomes. Most welfare and employment programs generate effect sizes of about 0.2 to 0.3 on outcomes such as employment and earnings, and effect sizes on child outcomes are generally half this size. Another method is to compare these effect sizes with effects produced from child-focused interventions such as the Perry Preschool Program and the Abecedarian Project. These child-focused interventions produced effects that generally ranged from 0.2 to 1.0. Finally, it is important to consider that even small effects may have a large impact on the future well-being of a child. Longitudinal studies of children have found that achievement and behavior problems can have important implications for children's well-being as adults.b For example, achievement and problem behavior in early childhood are related to adolescent achievement and behavior. Small effects (for example, 0.1-0.2) may continue to have implications for children over their lives. a Cohen, 1988; Lipsey, 1990. |
The Effect of Welfare-to-Work Approaches on Child Outcomes
Any Child in the Family
Table 11.2 shows impacts on child outcomes for any child in a survey respondent's family. As mentioned above, these analyses are similar to those conducted at the two-year follow-up point. Although some measures are directly comparable with the two-year measures, such as grade repetition and suspensions or expulsions, the estimates are shown for the five-year survey sample only (rather than for the sample that had information collected at both the two-year and five-year points).
Site and Program | Sample Size | Program Group (%) | Control Group (%) | Difference (Impact) | Effect Size |
|---|---|---|---|---|---|
| Any child ever repeated a grade | |||||
| Atlanta Labor Force Attachment | 1,048 | 21.3 | 25.6 | -4.3* | -0.10 |
| Atlanta Human Capital Development | 1,125 | 20.8 | 25.6 | -4.8* | -0.11 |
| Grand Rapids Labor Force Attachment | 1,077 | 20.8 | 16.9 | 3.9 | 0.10 |
| Grand Rapids Human Capital Development | 1,085 | 21.9 | 16.9 | 5.0** | 0.13 |
| Riverside Labor Force Attachment | 1,196 | 11.9 | 11.6 | 0.3 | 0.01 |
| Lacked high school diploma or basic skills | 644 | 11.4 | 12.6 | -1.2 | -0.03 |
| Riverside Human Capital Development | 767 | 14.9 | 12.6 | 2.2 | 0.06 |
| Portland | 493 | 12.1 | 8.2 | 3.9 | 0.14 |
| Any child suspended or expelled | |||||
| Atlanta Labor Force Attachment | 1,049 | 28.6 | 31.9 | -3.3 | -0.07 |
| Atlanta Human Capital Development | 1,125 | 32.4 | 31.9 | 0.5 | 0.01 |
| Grand Rapids Labor Force Attachment | 1,072 | 24.8 | 26.5 | -1.7 | -0.04 |
| Grand Rapids Human Capital Development | 1,081 | 22.0 | 26.5 | -4.4* | -0.10 |
| Riverside Labor Force Attachment | 1,190 | 21.4 | 24.1 | -2.7 | -0.06 |
| Lacked high school diploma or basic skills | 639 | 18.5 | 25.0 | -6.5** | -0.15 |
| Riverside Human Capital Development | 760 | 22.7 | 25.0 | -2.3 | -0.05 |
| Portland | 490 | 29.4 | 29.7 | -0.3 | -0.01 |
| Any child ever dropped out of schoola | |||||
| Atlanta Labor Force Attachment | 1,049 | 17.3 | 18.4 | -1.1 | -0.03 |
| Atlanta Human Capital Development | 1,126 | 19.6 | 18.4 | 1.3 | 0.03 |
| Grand Rapids Labor Force Attachment | 1,077 | 19.3 | 17.8 | 1.5 | 0.04 |
| Grand Rapids Human Capital Development | 1,088 | 18.6 | 17.8 | 0.8 | 0.02 |
| Riverside Labor Force Attachment | 1,198 | 15.5 | 13.7 | 1.8 | 0.05 |
| Lacked high school diploma or basic skills | 643 | 16.3 | 14.2 | 2.2 | 0.06 |
| Riverside Human Capital Development | 768 | 17.9 | 14.2 | 3.8 | 0.11 |
| Portland | 495 | 21.8 | 20.1 | 1.6 | 0.04 |
| Any child attended a special class for a physical, emotional, or mental conditionb | |||||
| Atlanta Labor Force Attachment | 1,050 | 12.6 | 12.0 | 0.6 | 0.02 |
| Atlanta Human Capital Development | 1,128 | 10.5 | 12.0 | -1.5 | -0.05 |
| Grand Rapids Labor Force Attachment | 1,077 | 28.9 | 28.3 | 0.6 | 0.01 |
| Grand Rapids Human Capital Development | 1,084 | 28.5 | 28.3 | 0.2 | 0.00 |
| Riverside Labor Force Attachment | 1,197 | 20.1 | 18.0 | 2.1 | 0.06 |
| Lacked high school diploma or basic skills | 643 | 20.9 | 18.8 | 2.1 | 0.05 |
| Riverside Human Capital Development | 768 | 20.3 | 18.8 | 1.4 | 0.04 |
| Portland | 494 | 29.2 | 24.8 | 4.4 | 0.10 |
| Any child had a physical, emotional, or mental condition that impeded on mother's ability to go to work or schoolb | |||||
| Atlanta Labor Force Attachment | 1,048 | 6.9 | 5.8 | 1.1 | 0.05 |
| Atlanta Human Capital Development | 1,128 | 6.9 | 5.8 | 1.1 | 0.05 |
| Grand Rapids Labor Force Attachment | 1,080 | 13.0 | 14.7 | -1.6 | -0.05 |
| Grand Rapids Human Capital Development | 1,091 | 10.4 | 14.7 | -4.3** | -0.12 |
| Riverside Labor Force Attachment | 1,200 | 10.0 | 12.1 | -2.1 | -0.06 |
| Lacked high school diploma or basic skills | 645 | 7.7 | 9.9 | -2.2 | -0.07 |
| Riverside Human Capital Development | 768 | 13.9 | 9.9 | 4.0* | 0.13 |
| Portland | 493 | 20.6 | 19.0 | 1.6 | 0.04 |
| Any child had a physical, emotional, or mental condition that required frequent medical attentionb | |||||
| Atlanta Labor Force Attachment | 1,049 | 11.7 | 10.6 | 1.1 | 0.04 |
| Atlanta Human Capital Development | 1,127 | 8.9 | 10.6 | -1.7 | -0.06 |
| Grand Rapids Labor Force Attachment | 1,077 | 20.3 | 19.2 | 1.1 | 0.03 |
| Grand Rapids Human Capital Development | 1,085 | 15.5 | 19.2 | -3.7 | -0.10 |
| Riverside Labor Force Attachment | 1,198 | 13.0 | 16.2 | -3.2 | -0.09 |
| Lacked high school diploma or basic skills | 645 | 11.5 | 13.6 | -2.1 | -0.06 |
| Riverside Human Capital Development | 769 | 13.5 | 13.6 | -0.1 | -0.00 |
| Portland | 494 | 23.3 | 20.5 | 2.8 | 0.07 |
| Any child ever had accident, injury, or poisoning that required an emergency room visit | |||||
| Atlanta Labor Force Attachment | 1,042 | 22.4 | 25.4 | -3.1 | -0.07 |
| Atlanta Human Capital Development | 1,118 | 21.4 | 25.4 | -4.0 | -0.09 |
| Grand Rapids Labor Force Attachment | 1,071 | 35.7 | 36.9 | -1.2 | -0.02 |
| Grand Rapids Human Capital Development | 1,085 | 31.5 | 36.9 | -5.4* | -0.11 |
| Riverside Labor Force Attachment | 1,186 | 38.1 | 39.5 | -1.3 | -0.03 |
| Lacked high school diploma or basic skills | 639 | 34.0 | 37.2 | -3.2 | -0.07 |
| Riverside Human Capital Development | 763 | 35.9 | 37.2 | -1.3 | -0.03 |
| Portland | 490 | 42.1 | 41.6 | 0.5 | 0.01 |
| Any child did not live with mother because she could not care for child | |||||
| Atlanta Labor Force Attachment | 1,051 | 4.3 | 5.2 | -1.0 | -0.04 |
| Atlanta Human Capital Development | 1,129 | 4.0 | 5.2 | -1.3 | -0.06 |
| Grand Rapids Labor Force Attachment | 1,082 | 7.3 | 7.2 | 0.1 | 0.01 |
| Grand Rapids Human Capital Development | 1,094 | 7.7 | 7.2 | 0.5 | 0.02 |
| Riverside Labor Force Attachment | 1,203 | 10.9 | 8.6 | 2.3 | 0.08 |
| Lacked high school diploma or basic skills | 647 | 10.3 | 7.0 | 3.3 | 0.12 |
| Riverside Human Capital Development | 770 | 9.0 | 7.0 | 2.0 | 0.07 |
| Portland | 495 | 11.6 | 12.5 | -1.0 | -0.03 |
| Any child ever had a baby as a teenc | |||||
| Atlanta Labor Force Attachment | 1,047 | 14.8 | 19.3 | -4.5** | -0.12 |
| Atlanta Human Capital Development | 1,126 | 18.2 | 19.3 | -1.1 | -0.03 |
| Grand Rapids Labor Force Attachment | 1,081 | 12.5 | 15.1 | -2.6 | -0.07 |
| Grand Rapids Human Capital Development | 1,091 | 15.0 | 15.1 | -0.1 | -0.00 |
| Riverside Labor Force Attachment | 1,199 | 12.0 | 10.0 | 2.0 | 0.07 |
| Lacked high school diploma or basic skills | 645 | 15.5 | 10.6 | 4.9** | 0.16 |
| Riverside Human Capital Development | 768 | 13.3 | 10.6 | 2.7 | 0.09 |
| Portland | 495 | 9.6 | 13.6 | -4.1 | -0.12 |
| SOURCE: MDRC calculations from the Five-Year Client Survey. NOTES: See Appendix A.2. Owing to missing values, sample sizes may vary. a Measures whether the child dropped out of school at any point during the child's lifetime. b Refers to conditions that were current at the time the survey was administered. c Measures whether the child had a baby while a teenager at any point during the five-year follow-up period. | |||||
Although there was no consistent pattern of effects on outcomes for any child in a family, there were more effects than would be expected by chance. Notably, of the effects that occurred, most were produced by the Grand Rapids HCD program. The effects, however, were both favorable and unfavorable within outcome or domain (across program approaches or site). For example, the Atlanta HCD program decreased the proportion of families who had a child retained in grade by approximately 5 percentage points, whereas the Grand Rapids HCD program increased it by 5 percentage points. The Grand Rapids HCD program also decreased the proportion of families with a child who had a physical, emotional, or mental condition that demanded a lot of attention, and the Riverside HCD program increased it. These noted effects also show that effects varied by program approach and site (across outcomes or domains). The conclusion drawn here about the effects of welfare-to-work programs on these outcomes measured for any child in the family are similar to the conclusion drawn at the two-year follow-up point.
As previously discussed, while evaluating effects on any child in a family may be useful in capturing a general snapshot, there are reasons to suspect that this kind of analysis may not be revealing clear patterns of effects on each child. The next sections review impacts presented in Tables 11.3-11.6 and show that some of the impacts that occurred for the analysis of any child in the family (such as effects in Atlanta on grade repetition) also generally occurred, though not always statistically significant, across children in many age groups. As would be expected, other effects, such as on teen childbearing, were concentrated among adolescents.
Toddlers
1. Effects on Mothers' Economic Outcomes
Impacts for survey sample respondents with toddlers (present in Grand Rapids and Portland samples only) are similar to impacts found for the client survey sample. Thus, mothers of these toddlers experienced some increased employment and losses in income from welfare and earnings. Research finds some evidence that mothers' employment during the first few years of life may be particularly detrimental to children's development.(15) The Canadian Self-Sufficiency Project, however, provides some evidence, based on limited outcomes, that welfare and employment programs that increase employment and income do not cause harm to infants and toddlers. Less is known about the sole effect of decreased income.(16)
2. Effects on Child Outcomes
Table 11.3 shows outcomes and impacts for children who were toddlers at study entry in the Grand Rapids and Portland sites. (These children were roughly 1st and 2nd graders at the time of the five-year follow-up.) Approximately 10 percent of these children in the Grand Rapids control group and 4 percent in the Portland control group repeated a grade during the last three years of follow-up. This range is comparable to national figures, which show that 7 percent of all 2nd graders and 10 percent of 2nd graders below poverty were retained in kindergarten or 1st grade in 1996.(17) Incidences of suspensions and expulsions among this age group approximately 6 percent seem quite high and, given that many of these children were aged 3 to 5 three years prior to the five year follow-up, may reflect suspensions or expulsions from child care arrangements rather than school. Possibly more alarming for this age group is that approximately 5 to 6 percent of these children in the control group did not live with their mother at some point during the follow-up period because she could not care for them.
Site and Program | Sample Size | Program Group (%) | Control Group (%) | Difference (Impact) | Effect Size |
|---|---|---|---|---|---|
Ever repeated a grade | |||||
| Grand Rapids Labor Force Attachment | 361 | 8.7 | 10.4 | -1.6 | -0.05 |
| Grand Rapids Human Capital Development | 381 | 8.2 | 10.4 | -2.1 | -0.07 |
| Portland | 217 | 6.2 | 3.6 | 2.6 | 0.19 |
Ever suspended or expelled | |||||
| Grand Rapids Labor Force Attachment | 360 | 0.0 | 6.2 | -6.4*** | -0.27 |
| Grand Rapids Human Capital Development | 380 | 1.6 | 6.2 | -4.6** | -0.19 |
| Portland | 217 | 6.6 | 6.0 | 0.6 | 0.03 |
Attended a special class for physical, emotional, or mental conditiona | |||||
| Grand Rapids Labor Force Attachment | 358 | 11.5 | 15.2 | -3.7 | -0.10 |
| Grand Rapids Human Capital Development | 379 | 14.7 | 15.2 | -0.5 | -0.01 |
| Portland | 215 | 20.0 | 17.2 | 2.8 | 0.08 |
Had a physical, emotional, or mental condition that impeded on mother's ability to go to work or schoola | |||||
| Grand Rapids Labor Force Attachment | 364 | 6.3 | 7.6 | -1.3 | -0.05 |
| Grand Rapids Human Capital Development | 385 | 4.8 | 7.6 | -2.8 | -0.10 |
| Portland | 218 | 13.2 | 12.9 | 0.2 | 0.01 |
Had a physical, emotional, or mental condition that required frequent medical attentiona | |||||
| Grand Rapids Labor Force Attachment | 363 | 7.3 | 14.1 | -6.8** | -0.20 |
| Grand Rapids Human Capital Development | 385 | 10.3 | 14.1 | -3.9 | -0.12 |
| Portland | 218 | 14.4 | 11.7 | 2.7 | 0.09 |
Ever had accident, injury, or poisoning that required an emergency room visit | |||||
| Grand Rapids Labor Force Attachment | 358 | 23.0 | 24.3 | -1.3 | -0.03 |
| Grand Rapids Human Capital Development | 380 | 19.4 | 24.3 | -4.9 | -0.11 |
| Portland | 216 | 20.3 | 27.0 | -6.8 | -0.15 |
Did not live with mother because she could not care for child | |||||
| Grand Rapids Labor Force Attachment | 364 | 6.2 | 4.6 | 1.6 | 0.07 |
| Grand Rapids Human Capital Development | 386 | 5.0 | 4.6 | 0.4 | 0.02 |
| Portland | 218 | 8.4 | 6.0 | 2.4 | 0.10 |
| SOURCE: MDRC calculations from the Five-Year Client Survey. NOTES: See Appendix A.2. Standard errors have been adjusted to account for the presence of multiple siblings within a family. Owing to missing values, sample sizes may vary. a Refers to conditions that were current at the time the survey was administered. | |||||
In general, few program effects were found for this age group. Given that the mothers of these toddlers generally experienced increased employment, it is noteworthy that more unfavorable effects were not found. Some of the unfavorable effects of employment on children may be influenced by the type and hours of employment, whether or not a mother wants to be employed, and the quality of child care. In fact, both Grand Rapids programs produced a consistent (though not always statistically significant) pattern of favorable effects for six of the seven outcome measures examined. The LFA and HCD program effects on suspensions and expulsions and the LFA effect on a condition requiring frequent medical attention were statistically significant and of a modest-to-large size relative to effects on child outcomes observed in comparable experimental studies.(18)
Preschool-Age Children
1. Effects on Mothers' Economic Outcomes
With a few exceptions, impacts on economic outcomes for survey sample respondents with preschool-age children are similar in Atlanta, Grand Rapids, and Portland to impacts noted for the client survey sample.(19) However, in Riverside impacts are larger and more pronounced for survey sample respondents with preschool-age children than for the client survey sample. In particular, the Riverside LFA employment impacts for each of the five years of follow-up and cumulative earnings impacts for survey sample respondents with preschool-age children are nearly one-third to twice as big as impacts for the client survey sample.(20) The effects of mothers' employment for this age group may be either favorable (for example, through role modeling) or unfavorable. Although most of the children in this age group are likely to be in school for a large portion of the day, they still require supervision during off-school hours, and, thus, the effects of mothers' employment may also depend on the quality of child care or out-of-school arrangements. Recent evidence suggests that programs that increased employment and increased income have positive benefits for young school-age children, particularly in their cognitive development.(21) Research on the effects of poverty also finds that the negative effects of poverty are particularly pronounced for this age group of children.(22)
2. Effects on Child Outcomes
Outcomes and impacts for children of preschool-age at study entry (young school-age at the time of the five-year follow-up) are shown in Table 11.4. The table shows that 7 to 12 percent of children in the control group repeated a grade and 5 to 15 percent were ever suspended or expelled during the last three years of the follow-up period. The youngest of these children were likely in 2nd grade. For the oldest of these children, national figures for a roughly comparable age group show that 3.3 percent of 4th to 8th graders were retained in grade in 1995.(23) Roughly 5 to 7 percent of these children did not live with their mother because she could not care for them.
Site and Program | Sample Size | Program Group (%) | Control Group (%) | Difference (Impact) | Effect Size |
|---|---|---|---|---|---|
Ever repeated a grade | |||||
| Atlanta Labor Force Attachment | 766 | 11.9 | 11.9 | -0.0 | -0.00 |
| Atlanta Human Capital Development | 876 | 10.2 | 11.9 | -1.7 | -0.05 |
| Grand Rapids Labor Force Attachment | 562 | 13.5 | 10.4 | 3.1 | 0.10 |
| Grand Rapids Human Capital Development | 534 | 15.7 | 10.4 | 5.2* | 0.17 |
| Riverside Labor Force Attachment | 829 | 4.4 | 9.7 | -5.2*** | -0.18 |
| Lacked high school diploma or basic skills | 501 | 4.5 | 11.4 | -6.9*** | -0.22 |
| Riverside Human Capital Development | 648 | 8.9 | 11.4 | -2.5 | -0.08 |
| Portland | 262 | 6.4 | 7.1 | -0.7 | -0.03 |
Ever suspended or expelled | |||||
| Atlanta Labor Force Attachment | 766 | 8.8 | 8.7 | 0.2 | 0.01 |
| Atlanta Human Capital Development | 876 | 8.8 | 8.7 | 0.1 | 0.00 |
| Grand Rapids Labor Force Attachment | 561 | 6.6 | 8.2 | -1.6 | -0.06 |
| Grand Rapids Human Capital Development | 532 | 7.6 | 8.2 | -0.6 | -0.02 |
| Riverside Labor Force Attachment | 820 | 4.3 | 6.2 | -1.9 | -0.08 |
| Lacked high school diploma or basic skills | 496 | 4.0 | 4.7 | -0.8 | -0.04 |
| Riverside Human Capital Development | 640 | 5.5 | 4.7 | 0.8 | 0.04 |
| Portland | 257 | 9.4 | 14.8 | -5.4 | -0.16 |
Attended a special class for physical, emotional, or mental conditiona | |||||
| Atlanta Labor Force Attachment | 766 | 8.2 | 7.2 | 1.0 | 0.04 |
| Atlanta Human Capital Development | 876 | 6.1 | 7.2 | -1.2 | -0.04 |
| Grand Rapids Labor Force Attachment | 559 | 21.7 | 18.9 | 2.8 | 0.07 |
| Grand Rapids Human Capital Development | 529 | 22.6 | 18.9 | 3.7 | 0.09 |
| Riverside Labor Force Attachment | 828 | 14.5 | 13.7 | 0.9 | 0.03 |
| Lacked high school diploma or basic skills | 498 | 17.2 | 15.6 | 1.6 | 0.04 |
| Riverside Human Capital Development | 646 | 13.0 | 15.6 | -2.7 | -0.07 |
| Portland | 259 | 19.5 | 18.8 | 0.7 | 0.02 |
Had a physical, emotional, or mental condition that impeded on mother's ability to go to work or schoola | |||||
| Atlanta Labor Force Attachment | 767 | 5.3 | 1.8 | 3.5*** | 0.26 |
| Atlanta Human Capital Development | 877 | 3.2 | 1.8 | 1.3 | 0.10 |
| Grand Rapids Labor Force Attachment | 564 | 7.1 | 6.7 | 0.3 | 0.01 |
| Grand Rapids Human Capital Development | 536 | 6.8 | 6.7 | 0.1 | 0.00 |
| Riverside Labor Force Attachment | 832 | 4.9 | 8.5 | -3.6** | -0.13 |
| Lacked high school diploma or basic skills | 501 | 3.4 | 7.0 | -3.6* | -0.14 |
| Riverside Human Capital Development | 648 | 7.3 | 7.0 | 0.3 | 0.01 |
| Portland | 262 | 13.8 | 16.2 | -2.4 | -0.06 |
Had a physical, emotional, or mental condition that required frequent medical attentiona | |||||
| Atlanta Labor Force Attachment | 767 | 7.1 | 3.4 | 3.7** | 0.20 |
| Atlanta Human Capital Development | 877 | 6.8 | 3.4 | 3.4** | 0.18 |
| Grand Rapids Labor Force Attachment | 563 | 13.9 | 11.2 | 2.7 | 0.08 |
| Grand Rapids Human Capital Development | 536 | 13.2 | 11.2 | 2.0 | 0.06 |
| Riverside Labor Force Attachment | 832 | 6.9 | 9.9 | -3.0 | -0.10 |
| Lacked high school diploma or basic skills | 501 | 6.9 | 8.7 | -1.8 | -0.06 |
| Riverside Human Capital Development | 648 | 7.2 | 8.7 | -1.5 | -0.05 |
| Portland | 260 | 16.0 | 11.0 | 5.0 | 0.16 |
Ever had accident, injury, or poisoning that required an emergency room visit | |||||
| Atlanta Labor Force Attachment | 766 | 15.1 | 16.0 | -0.9 | -0.02 |
| Atlanta Human Capital Development | 876 | 14.2 | 16.0 | -1.9 | -0.05 |
| Grand Rapids Labor Force Attachment | 561 | 22.7 | 22.0 | 0.7 | 0.02 |
| Grand Rapids Human Capital Development | 531 | 18.7 | 22.0 | -3.3 | -0.08 |
| Riverside Labor Force Attachment | 819 | 25.0 | 24.5 | 0.6 | 0.01 |
| Lacked high school diploma or basic skills | 498 | 22.9 | 19.3 | 3.6 | 0.09 |
| Riverside Human Capital Development | 640 | 20.2 | 19.3 | 0.9 | 0.02 |
| Portland | 258 | 33.3 | 24.7 | 8.6 | 0.20 |
Did not live with mother because she could not care for child | |||||
| Atlanta Labor Force Attachment | 767 | 3.4 | 4.8 | -1.4 | -0.07 |
| Atlanta Human Capital Development | 877 | 3.0 | 4.8 | -1.8 | -0.09 |
| Grand Rapids Labor Force Attachment | 564 | 6.9 | 4.8 | 2.2 | 0.10 |
| Grand Rapids Human Capital Development | 538 | 5.1 | 4.8 | 0.3 | 0.01 |
| Riverside Labor Force Attachment | 831 | 9.9 | 6.6 | 3.3* | 0.13 |
| Lacked high school diploma or basic skills | 500 | 6.8 | 6.6 | 0.2 | 0.01 |
| Riverside Human Capital Development | 647 | 5.2 | 6.6 | -1.5 | -0.06 |
| Portland | 262 | 9.8 | 7.3 | 2.5 | 0.09 |
| SOURCE: MDRC calculations from the Five-Year Client Survey. NOTES: See Appendix A.2. Standard errors have been adjusted to account for the presence of multiple siblings within a family. Owing to missing values, sample sizes may vary. a Refers to conditions that were current at the time the survey was administered. | |||||
Again, few effects were found for this age group. The effects that were found were concentrated in Atlanta and Riverside, were all unfavorable in Atlanta, and, with one exception, favorable in Riverside. For example, both Atlanta programs increased the likelihood that a child had a condition requiring medical attention and Atlanta LFA increased the likelihood that a child had a condition that demanded a lot of attention. The Riverside LFA program decreased grade repetition by 5.2 percentage points. Attention should be drawn to the fact that the Riverside LFA program increased the likelihood that preschool-age children did not live with their mother because she could not care for them. It is interesting that the Riverside LFA program decreased grade repetition and, at the same time, increased the likelihood of not living with a parent.
Young School-Age Children
1. Effects on Mothers' Economic Outcomes
Impacts for survey sample respondents with young school-age children are similar to impacts noted for the client survey sample. As mentioned above for preschool-age children, young school-age children may also benefit from or be harmed by mothers' employment. Because most of the children in this age group are likely to be in school for a large portion of the day, the effects of mothers' employment depend on the quality of child care or other arrangements for supervision during off-school hours
2. Effects on Child Outcomes
Table 11.5 shows outcomes and impacts for children of young school-age at study entry (aged 11 to 14 at the five-year follow-up). Approximately 4 to 13 percent of children in the control group repeated a grade during the last three years of follow-up and 17 to 25 percent were ever suspended or expelled. Dropout rates were negligible (at about 1 percent). National figures show that 3.3 percent of 4th to 8th graders (as mentioned) and 2.4 percent of 9th to 12th graders were retained in grade in 1995.(24) Even though the age groups are not directly comparable it suggests that NEWWS sample rates of grade retention for this age group are higher than national rates. Rates of suspensions or expulsions, on the other hand, seem comparable to state figures. According to 1999 data from the National Survey of America's Families, 24.5 percent of children aged 12 to17 in families below 200 percent of poverty in California and 23.2 percent in Michigan were suspended or expelled.
|
Site and Program |
Sample Size | Program Group (%) | Control Group (%) | Difference (Impact) | Effect Size |
|---|---|---|---|---|---|
|
Ever repeated a grade |
|||||
| Atlanta Labor Force Attachment | 652 | 11.3 | 13.3 | -2.0 | -0.06 |
| Atlanta Human Capital Development | 697 | 9.7 | 13.3 | -3.6 | -0.10 |
| Grand Rapids Labor Force Attachment | 524 | 14.4 | 9.9 | 4.4 | 0.14 |
| Grand Rapids Human Capital Development | 513 | 12.6 | 9.9 | 2.7 | 0.09 |
| Riverside Labor Force Attachment | 758 | 7.6 | 6.0 | 1.5 | 0.06 |
| Lacked high school diploma or basic skills | 427 | 8.3 | 5.0 | 3.3 | 0.16 |
| Riverside Human Capital Development | 490 | 7.2 | 5.0 | 2.3 | 0.11 |
| Portland | 265 | 4.9 | 4.4 | 0.5 | 0.03 |
|
Ever suspended or expelled |
|||||
| Atlanta Labor Force Attachment | 653 | 25.0 | 25.0 | 0.1 | 0.00 |
| Atlanta Human Capital Development | 694 | 26.0 | 25.0 | 1.1 | 0.02 |
| Grand Rapids Labor Force Attachment | 517 | 25.7 | 18.7 | 7.0* | 0.18 |
| Grand Rapids Human Capital Development | 503 | 16.7 | 18.7 | -1.9 | -0.05 |
| Riverside Labor Force Attachment | 753 | 13.9 | 19.5 | -5.6** | -0.14 |
| Lacked high school diploma or basic skills | 422 | 11.7 | 19.6 | -7.9** | -0.19 |
| Riverside Human Capital Development | 484 | 11.2 | 19.6 | -8.4*** | -0.21 |
| Portland | 262 | 24.6 | 17.1 | 7.5 | 0.20 |
|
Ever dropped out of schoola |
|||||
| Atlanta Labor Force Attachment | 0 | 0.0 | 0.0 | 0.0 | 0.00 |
| Atlanta Human Capital Development | 0 | 0.0 | 0.0 | 0.0 | 0.00 |
| Grand Rapids Labor Force Attachment | 525 | 0.4 | 1.1 | -0.7 | -0.08 |
| Grand Rapids Human Capital Development | 512 | 0.0 | 1.1 | -1.2 | -0.14 |
| Riverside Labor Force Attachment | 760 | 0.7 | 1.5 | -0.8 | -0.06 |
| Lacked high school diploma or basic skills | 427 | 1.3 | 1.1 | 0.2 | 0.02 |
| Riverside Human Capital Development | 490 | 0.6 | 1.1 | -0.5 | -0.05 |
| Portland | 267 | 0.8 | 0.0 | 0.9 | 0.00 |
|
Attended a special class for physical, emotional, or mental conditionb |
|||||
| Atlanta Labor Force Attachment | 654 | 7.9 | 9.1 | -1.2 | -0.04 |
| Atlanta Human Capital Development | 699 | 6.4 | 9.1 | -2.6 | -0.09 |
| Grand Rapids Labor Force Attachment | 522 | 23.1 | 25.8 | -2.7 | -0.06 |
| Grand Rapids Human Capital Development | 507 | 19.6 | 25.8 | -6.2 | -0.14 |
| Riverside Labor Force Attachment | 755 | 13.2 | 13.5 | -0.3 | -0.01 |
| Lacked high school diploma or basic skills | 423 | 12.6 | 14.3 | -1.8 | -0.05 |
| Riverside Human Capital Development | 486 | 14.2 | 14.3 | -0.2 | -0.01 |
| Portland | 264 | 24.1 | 14.4 | 9.7 | 0.27 |
|
Had a physical, emotional, or mental condition that impeded on mother's ability to go to work or schoolb |
|||||
| Atlanta Labor Force Attachment | 655 | 4.4 | 3.3 | 1.0 | 0.06 |
| Atlanta Human Capital Development | 700 | 4.4 | 3.3 | 1.1 | 0.06 |
| Grand Rapids Labor Force Attachment | 529 | 10.9 | 12.7 | -1.9 | -0.06 |
| Grand Rapids Human Capital Development | 518 | 6.3 | 12.7 | -6.4** | -0.19 |
| Riverside Labor Force Attachment | 761 | 7.5 | 7.7 | -0.3 | -0.01 |
| Lacked high school diploma or basic skills | 428 | 5.4 | 7.1 | -1.8 | -0.07 |
| Riverside Human Capital Development | 490 | 6.4 | 7.1 | -0.7 | -0.03 |
| Portland | 267 | 14.0 | 12.9 | 1.1 | 0.03 |
|
Had a physical, emotional, or mental condition that required frequent medical attentionb |
|||||
| Atlanta Labor Force Attachment | 654 | 6.9 | 6.6 | 0.4 | 0.02 |
| Atlanta Human Capital Development | 700 | 4.4 | 6.6 | -2.2 | -0.09 |
| Grand Rapids Labor Force Attachment | 529 | 13.7 | 15.1 | -1.4 | -0.04 |
| Grand Rapids Human Capital Development | 518 | 9.7 | 15.1 | -5.4* | -0.16 |
| Riverside Labor Force Attachment | 761 | 8.9 | 8.0 | 0.9 | 0.03 |
| Lacked high school diploma or basic skills | 428 | 6.0 | 5.7 | 0.3 | 0.01 |
| Riverside Human Capital Development | 490 | 7.2 | 5.7 | 1.5 | 0.06 |
| Portland | 267 | 16.3 | 14.8 | 1.5 | 0.04 |
|
Ever had accident, injury, or poisoning that required an emergency room visit |
|||||
| Atlanta Labor Force Attachment | 652 | 14.2 | 14.2 | 0.0 | 0.00 |
| Atlanta Human Capital Development | 696 | 11.4 | 14.2 | -2.9 | -0.08 |
| Grand Rapids Labor Force Attachment | 524 | 18.4 | 22.4 | -3.9 | -0.09 |
| Grand Rapids Human Capital Development | 514 | 18.2 | 22.4 | -4.2 | -0.10 |
| Riverside Labor Force Attachment | 754 | 22.9 | 21.8 | 1.1 | 0.03 |
| Lacked high school diploma or basic skills | 426 | 23.0 | 18.8 | 4.2 | 0.11 |
| Riverside Human Capital Development | 486 | 12.8 | 18.8 | -6.0* | -0.15 |
| Portland | 262 | 24.3 | 24.1 | 0.2 | 0.00 |
|
Did not live with mother because she could not care for child |
|||||
| Atlanta Labor Force Attachment | 655 | 4.6 | 3.9 | 0.7 | 0.04 |
| Atlanta Human Capital Development | 699 | 3.6 | 3.9 | -0.3 | -0.02 |
| Grand Rapids Labor Force Attachment | 529 | 6.3 | 6.9 | -0.6 | -0.03 |
| Grand Rapids Human Capital Development | 518 | 6.5 | 6.9 | -0.4 | -0.02 |
| Riverside Labor Force Attachment | 759 | 10.8 | 6.8 | 4.0* | 0.15 |
| Lacked high school diploma or basic skills | 428 | 9.5 | 4.2 | 5.3** | 0.25 |
| Riverside Human Capital Development | 490 | 9.4 | 4.2 | 5.2** | 0.24 |
| Portland | 267 | 10.7 | 11.9 | -1.2 | -0.04 |
| SOURCE: MDRC calculations from the Five-Year Client Survey. NOTES: See Appendix A.2. Standard errors have been adjusted to account for the presence of multiple siblings within a family. Owing to missing values, sample sizes may vary. a Measures whether the child dropped out of school at any point during the child's lifetime. b Refers to conditions that were current at the time the survey was administered. |
|||||
There are generally more effects than would be expected by chance for outcomes of young school-age children. Although effects are not consistent within domain, program approach, or site, two interesting general patterns emerge.
First, both Riverside programs increased the likelihood that young school-age children did not live with their mother because she could not care for them. The magnitude of this effect is also relatively large almost doubling the likelihood (and effect sizes of 0.15 to 0.25). Both Riverside programs decreased suspensions and expulsions for young school-age children (though these effects are not supported by other measures that might be expected to be affected by a suspension or expulsion such as grade repetition or dropping out of school). It appears that the Riverside programs are producing favorable effects on one measure of academic functioning and, at the same time, increasing the likelihood of not living with a parent. One hypothesis is that a mother may have decided voluntarily to place her children in an alternative, perhaps temporary, living arrangement and this arrangement may have long-term benefits for the child's development. Or, alternatively, these effects may be capturing two different groups of children: those who experienced a decrease in grade repetition and those who were more likely to not live with their mother because she could not care for them.
Second, a pattern of favorable effects occurred for young school-age children in the Grand Rapids HCD program for seven of the eight outcome measures examined. These children were less likely to be suspended or expelled, less likely to drop out of school (approached statistical significance at p = .14), less likely to have attended a special class (approached statistical significance at p = .14), less likely to have had a condition that demands a lot of attention, and less likely to have had a condition that demands frequent medical attention.(25)
Adolescents
1. Effects on Mothers' Economic Outcomes
Impacts are smaller and more negative in Atlanta and Grand Rapids, similar in Riverside, and more positive in Portland for survey sample respondents of adolescents (aged 10 or over at time of study entry), than for the client survey sample.(26) The effect of mothers' employment may be particularly pronounced for adolescents by, for example, providing a positive role model and encouraging adolescents to prepare for the labor force as young adults. In addition, adolescents in low-income families may take on additional responsibilities at home, such as chores, or may engage in their own employment to help support their family. These activities could have either positive or negative consequences for adolescent outcomes. Finally, adolescents might be harmed by mothers' increased employment because it may translate to lack of supervision during a time when many may initiate risk-taking behaviors. On the other hand, supervised and high-quality out-of-school programs may have particularly beneficial effects for adolescents.(27)
2. Effects on Child Outcomes
Outcomes and impacts for children who were adolescents at study entry (aged 15 to 23 at the five-year follow-up), are shown in Table 11.6. Rates of grade retention during the last three years of follow-up varied considerably across sites among adolescents in the control group, from 4 percent in Portland and Riverside to 17 percent in Atlanta. Dropout rates also varied considerably, from 17 percent in Riverside to 31 percent in Portland. These dropout rates are higher than the dropout rates of 10th to 12th graders for a national sample of 15-to -24-year olds in families at the bottom 20 percent of income levels.(28) Approximately 15 to 23 percent of adolescents in the control group were ever suspended or expelled during the last three years of follow-up. Unsurprisingly, these rates of grade retention, suspensions and expulsions, and dropping out are considerably higher for adolescents than for younger children.
|
Site and Program |
Sample Size | Program Group (%) | Control Group (%) | Difference (Impact) | Effect Size |
|---|---|---|---|---|---|
|
Ever repeated a grade |
|||||
| Atlanta Labor Force Attachment | 834 | 15.2 | 17.2 | -1.9 | -0.05 |
| Atlanta Human Capital Development | 938 | 15.5 | 17.2 | -1.6 | -0.04 |
| Grand Rapids Labor Force Attachment | 890 | 11.7 | 7.5 | 4.2** | 0.16 |
| Grand Rapids Human Capital Development | 919 | 11.8 | 7.5 | 4.3** | 0.16 |
| Riverside Labor Force Attachment | 1126 | 7.1 | 4.2 | 2.9** | 0.14 |
| Lacked high school diploma or basic skills | 638 | 7.0 | 4.3 | 2.8 | 0.13 |
| Riverside Human Capital Development | 657 | 8.2 | 4.3 | 3.9** | 0.18 |
| Portland | 406 | 5.6 | 4.4 | 1.2 | 0.06 |
|
Ever suspended or expelled |
|||||
| Atlanta Labor Force Attachment | 836 | 16.3 | 23.2 | -6.8** | -0.16 |
| Atlanta Human Capital Development | 938 | 21.3 | 23.2 | -1.9 | -0.05 |
| Grand Rapids Labor Force Attachment | 891 | 21.3 | 20.0 | 1.3 | 0.03 |
| Grand Rapids Human Capital Development | 924 | 18.2 | 20.0 | -1.8 | -0.05 |
| Riverside Labor Force Attachment | 1120 | 15.4 | 15.0 | 0.3 | 0.01 |
| Lacked high school diploma or basic skills | 634 | 13.3 | 16.6 | -3.3 | -0.09 |
| Riverside Human Capital Development | 652 | 17.5 | 16.6 | 0.9 | 0.02 |
| Portland | 395 | 16.9 | 18.6 | -1.7 | -0.04 |
|
Ever dropped out of schoola |
|||||
| Atlanta Labor Force Attachment | 836 | 22.8 | 24.1 | -1.3 | -0.03 |
| Atlanta Human Capital Development | 937 | 26.1 | 24.1 | 2.0 | 0.05 |
| Grand Rapids Labor Force Attachment | 899 | 29.5 | 26.0 | 3.5 | 0.08 |
| Grand Rapids Human Capital Development | 934 | 29.1 | 26.0 | 3.1 | 0.07 |
| Riverside Labor Force Attachment | 1122 | 18.5 | 17.9 | 0.6 | 0.01 |
| Lacked high school diploma or basic skills | 635 | 18.2 | 17.3 | 0.9 | 0.02 |
| Riverside Human Capital Development | 657 | 22.8 | 17.3 | 5.4* | 0.14 |
| Portland | 409 | 36.4 | 31.2 | 5.2 | 0.11 |
|
Attended a special class for physical, emotional, or mental conditionb |
|||||
| Atlanta Labor Force Attachment | 835 | 5.0 | 4.4 | 0.6 | 0.04 |
| Atlanta Human Capital Development | 939 | 3.9 | 4.4 | -0.4 | -0.02 |
| Grand Rapids Labor Force Attachment | 894 | 11.5 | 8.5 | 3.0 | 0.10 |
| Grand Rapids Human Capital Development | 928 | 11.0 | 8.5 | 2.5 | 0.09 |
| Riverside Labor Force Attachment | 1128 | 7.3 | 5.0 | 2.2 | 0.10 |
| Lacked high school diploma or basic skills | 640 | 7.5 | 3.9 | 3.6* | 0.18 |
| Riverside Human Capital Development | 660 | 6.9 | 3.9 | 3.0* | 0.15 |
| Portland | 407 | 8.0 | 6.4 | 1.6 | 0.06 |
|
Had a physical, emotional, or mental condition that impeded on mother's ability to go to work or schoolb |
|||||
| Atlanta Labor Force Attachment | 836 | 2.3 | 3.0 | -0.7 | -0.04 |
| Atlanta Human Capital Development | 939 | 2.9 | 3.0 | -0.1 | -0.00 |
| Grand Rapids Labor Force Attachment | 900 | 5.7 | 3.9 | 1.8 | 0.09 |
| Grand Rapids Human Capital Development | 943 | 4.3 | 3.9 | 0.4 | 0.02 |
| Riverside Labor Force Attachment | 1133 | 3.4 | 3.2 | 0.2 | 0.01 |
| Lacked high school diploma or basic skills | 644 | 2.8 | 3.1 | -0.2 | -0.01 |
| Riverside Human Capital Development | 663 | 6.2 | 3.1 | 3.1* | 0.17 |
| Portland | 409 | 4.5 | 2.6 | 1.8 | 0.10 |
|
Had a physical, emotional, or mental condition that required frequent medical attentionb |
|||||
| Atlanta Labor Force Attachment | 835 | 5.3 | 4.6 | 0.6 | 0.03 |
| Atlanta Human Capital Development | 939 | 3.8 | 4.6 | -0.8 | -0.04 |
| Grand Rapids Labor Force Attachment | 899 | 6.6 | 4.3 | 2.3 | 0.11 |
| Grand Rapids Human Capital Development | 941 | 4.7 | 4.3 | 0.4 | 0.02 |
| Riverside Labor Force Attachment | 1134 | 3.4 | 6.3 | -2.8** | -0.12 |
| Lacked high school diploma or basic skills | 644 | 2.8 | 5.4 | -2.6* | -0.11 |
| Riverside Human Capital Development | 664 | 3.8 | 5.4 | -1.6 | -0.07 |
| Portland | 409 | 4.5 | 4.9 | -0.4 | -0.02 |
|
Ever had accident, injury, or poisoning that required an emergency room visit |
|||||
| Atlanta Labor Force Attachment | 822 | 10.8 | 12.0 | -1.2 | -0.04 |
| Atlanta Human Capital Development | 921 | 11.8 | 12.0 | -0.2 | -0.01 |
| Grand Rapids Labor Force Attachment | 873 | 14.1 | 17.4 | -3.2 | -0.09 |
| Grand Rapids Human Capital Development | 910 | 17.3 | 17.4 | -0.1 | -0.00 |
| Riverside Labor Force Attachment | 1104 | 15.1 | 16.9 | -1.8 | -0.05 |
| Lacked high school diploma or basic skills | 628 | 10.8 | 13.3 | -2.5 | -0.07 |
| Riverside Human Capital Development | 648 | 15.9 | 13.3 | 2.6 | 0.08 |
| Portland | 397 | 23.5 | 20.1 | 3.4 | 0.09 |
|
Did not live with mother because she could not care for child |
|||||
| Atlanta Labor Force Attachment | 836 | 4.3 | 3.9 | 0.4 | 0.02 |
| Atlanta Human Capital Development | 937 | 4.1 | 3.9 | 0.2 | 0.01 |
| Grand Rapids Labor Force Attachment | 900 | 3.8 | 4.6 | -0.9 | -0.04 |
| Grand Rapids Human Capital Development | 943 | 6.4 | 4.6 | 1.7 | 0.08 |
| Riverside Labor Force Attachment | 1134 | 8.1 | 5.4 | 2.7* | 0.12 |
| Lacked high school diploma or basic skills | 644 | 7.7 | 4.8 | 2.9 | 0.13 |
| Riverside Human Capital Development | 664 | 3.3 | 4.8 | -1.5 | -0.07 |
| Portland | 409 | 7.9 | 10.1 | -2.2 | -0.08 |
|
Ever had a baby as a teenc |
|||||
| Atlanta Labor Force Attachment | 829 | 16.9 | 21.3 | -4.5 | -0.11 |
| Atlanta Human Capital Development | 936 | 22.3 | 21.3 | 0.9 | 0.02 |
| Grand Rapids Labor Force Attachment | 893 | 19.6 | 19.7 | -0.2 | -0.00 |
| Grand Rapids Human Capital Development | 923 | 20.6 | 19.7 | 0.8 | 0.02 |
| Riverside Labor Force Attachment | 1123 | 15.4 | 13.0 | 2.4 | 0.07 |
| Lacked high school diploma or basic skills | 637 | 19.4 | 13.2 | 6.2** | 0.18 |
| Riverside Human Capital Development | 658 | 16.9 | 13.2 | 3.7 | 0.11 |
| Portland | 399 | 13.8 | 17.2 | -3.4 | -0.10 |
| SOURCE: MDRC calculations from the Five-Year Client Survey. NOTES: See Appendix A.2. Standard errors have been adjusted to account for the presence of multiple siblings within a family. Owing to missing values, sample sizes may vary. a Measures whether the child dropped out of school at any point during the child's lifetime. b Refers to conditions that were current at the time the survey was administered. c Measures whether the child had a baby while a teenager at any point during the five-year follow-up period. |
|||||
One important outcome for adolescents, especially for female adolescents, is teen childbearing, which is correlated with a decreased likelihood of completing schooling and of succeeding in the labor market and an increased likelihood of receiving public assistance. Furthermore, being raised by a teen mother may have negative consequences on children's development.(29) Approximately 13 to 21 percent of adolescents in the NEWWS sample ever had a baby as a teen.(30) These rates are slightly lower when the sample of adolescents is restricted to those aged 10 or over at random assignment but under age 19 at the five-year follow-up (7 to 15 percent; not shown). These are double the national rates, though much of the difference is likely due to the fact that the national figures are not restricted to a low-income or welfare sample.(31) The national birth rate in 1998 for teens aged 15 to 19 was 5 percent.(32) In 1997, teen birth rates for females aged 15 to 19 were 5.7 percent in California, 6.7 percent in Georgia, 4.4 percent in Michigan, and 4.7 percent in Oregon.(33)
The welfare-to-work programs examined in this chapter produced the most effects on outcomes for adolescents. The effects were generally unfavorable in the Grand Rapids and Riverside programs, especially the Riverside HCD program. Both programs in Grand Rapids and both programs in Riverside increased grade repetition by 3 percentage points (Riverside LFA) to 4 percentage points (Grand Rapids HCD). These unfavorable impacts are of concern because children who repeat a grade in high school may be more likely to drop out of school and, as previously noted, completed education is highly correlated with future labor force participation.
In addition to increasing grade repetition, the Riverside HCD program increased the likelihood that an adolescent would drop out of school, increased the percentage of adolescents who attended a special class because of a physical, emotional, or mental condition, and increased the percentage who had a physical, emotional, or mental condition that demanded a lot of the respondents' time. It also increased the percentage of adolescents who were teen parents, an effect that approached statistical significance (p = .18).
Similar unfavorable effects were not found in Atlanta or in Portland. In fact, the Atlanta LFA program decreased suspensions or expulsions by almost 7 percentage points and decreased teenage childbearing by nearly 5 percentage points (with the latter effect approaching statistical significance at p = .11).
Comparisons of NEWWS effects with effects found in other recent studies are discussed in the accompanying text box.
Links Between Effects on Child Outcomes and Program Practices or Particular Effects
The earlier sections began with hypotheses about how targeted outcomes of welfare-to-work programs, such as income and employment, may affect child well-being, and then discussed impacts on child outcomes. This section expands this discussion and hypothesizes ways in which actual program practices or program effects may have led to effects on child outcomes. A complete analysis of all the ways that a program may have affected or generated changes in child outcomes is beyond the scope of this chapter. However, comparing or "lining up" impacts on outcomes presented and discussed in earlier chapters with impacts on child outcomes can reveal potential effects on children. With this in mind, this section additionally discusses impacts on several other outcomes, including characteristics of employment, household composition, and home ownership for mothers of these children (not shown).
Comparing NEWWS Adolescent Findings With Those of Other Recent Studies Recently released studies have documented some unfavorable effects of welfare and employment policy on the outcomes for adolescents. It was found that the Canadian Self-Sufficiency Project (SSP), a program that increased full-time employment and income, had no effect on major delinquency or academic functioning outcomes but did increase minor delinquency and tobacco, alcohol, and drug use among adolescents.a Florida's Family Transition Program (FTP), one of the first that evaluated the effects of a time limit, also found some scattered evidence that adolescents in the program group fared more poorly on a couple of schooling outcomes than adolescents in the control group.b Furthermore, additional analyses of data from New Hope, a program in Milwaukee that offered an earnings supplement and more generous child care and health care benefits for full-time low-income workers, found that New Hope had unfavorable effects on some measures of academic functioning.c The Minnesota Family Investment Program also produced negative effects on academic functioning outcomes of children aged 10 or over at study entry of recent applicants, although this pattern was generally not found for adolescents of long-term welfare recipients.d Three of these studies (FTP, New Hope, and SSP) examined adolescents who were approximately aged 9 to 15 at the time of study entry, and under age 19 at the time of interview. To draw a more precise comparison of the effects of the welfare-to-work programs examined in this report with these former studies, impacts on adolescents were rerun for a similar age cohort. In general, the unfavorable effects found for the full adolescent sample in Grand Rapids and Riverside were also found for the restricted sample of adolescents. However, notably, as was the case for the full sample, no effects were found on these outcomes for the adolescent samples in Atlanta and Portland. Although confidence in these emerging findings could be bolstered by better and broader measures of adolescent development and larger samples, they do provide some consistent evidence that welfare and employment programs may negatively affect some adolescent children. Why these unfavorable effects are occurring in some sites and programs but not in others, in some domains of development but not in others, and whether or not the observed unfavorable effects will result in long-term difficulties as adolescents move into adulthood are especially important issues for further research. a Morris and Michalopoulos, 2000. |
Comparing LFA and HCD Approaches
Fewer differences than would be expected by chance were found in the effects on child outcomes of the LFA program approach compared with the HCD program approach for each of the child age groups.(34) This suggests that children, ranging in age from toddlers to adolescents, were not affected differently by education-focused programs than by employment-focused programs for the outcomes examined in this chapter.
Toddlers
Both Grand Rapids programs increased mothers' employment but decreased average income from earnings and welfare benefits for mothers of toddlers at study entry. The general pattern of impacts on toddlers suggests that the children in these families were faring as well as, if not doing better than, their control group counterparts. A review of impacts on characteristics of employment, income, and child care assistance suggests some ways in which toddlers may have been affected. This review suggests that the reduced hours of work and, possibly, the use of higher-quality child care, and not decreased income, among program group members compared with control group members in Grand Rapids may have contributed to the pattern of effects on outcomes for toddlers.
A closer look at the impacts on hours of employment for these mothers shows that the Grand Rapid's programs increased part-time employment (and decreased full-time employment) during the most recent or current job. In fact, fewer hours worked may be what contributed to lower income for these mothers. They were also significantly more likely to be in jobs with rotating hours and to use child care after leaving welfare because of earnings. In contrast, the Portland program increased both full-time and part-time employment and similar effects on child care use were not found. Field notes also suggest that caseworkers in Grand Rapids had more leeway in administering reimbursement of child care costs (that is, payments were allocated in advance or retroactively), and one adult education center that operated the program's job clubs as well as providing some educational activities used by NEWWS sample members provided on-site child care. These differences may have influenced the quality of care used (for example, if on-site child care was of higher quality). Neither of these circumstances existed in Portland. Finally, there was a general pattern of decreased income in both Grand Rapids programs and the Portland program, suggesting that decreased income, on its own, is not a likely way in which toddlers were affected.
Adolescents
Both programs in Riverside produced larger employment effects for mothers of adolescents than programs in the other sites. As mentioned above, the Riverside programs also produced unfavorable effects on the academic outcomes of adolescents. One possible explanation is that as mothers' employment increased, especially full-time employment, adolescents were less likely to be supervised, giving them more freedom to engage in risk-taking behaviors. Closer inspection of the employment impacts shows that the Grand Rapids and Riverside programs similarly had larger employment effects during the first year of the follow-up than the Atlanta and Portland programs. Unfortunately, actual measures of supervision were not collected in the survey. Nonetheless, unfavorable academic outcomes during the last few years of the follow-up period may be associated with the extent of mothers' employment early in the follow-up period.
A review of effects on other important family outcomes such as income and family structure suggests that there are other possible reasons why adolescents fared poorly on academic outcomes because of these welfare-to-work programs, especially since nearly all of the programs increased employment but did not produce similarly unfavorable effects on adolescent outcomes. Mothers in the Grand Rapids and Riverside programs also experienced decreased income from earnings and welfare benefits during the last three years of the follow-up period. Adolescents may have taken on more responsibility contributing to household resources, increasing their own employment in response to having less income in the household. Employment during adolescence, particularly if it is more than 20 hours per week, is associated with difficulty in school.(35) In addition, and interestingly, the two programs that produced the most unfavorable effects on adolescents the Grand Rapids LFA program and the Riverside HCD program also increased the likelihood that these adolescents' mothers were married and living with a spouse at the five-year follow-up point (not shown). Adolescent children are especially vulnerable to family changes, such as separation, divorce, and marriage.(36)
Conclusions
This chapter examined the effects of these welfare-to-work programs on child outcomes. In general few, if any, effects were found. Most program effects did not vary by program approach. Employment-focused programs and education-focused programs generally generated similar effects. Program effects generally were not concentrated by site, even with the caveat that differences between program and control groups were not as distinct by the final year of follow-up in Atlanta and Grand Rapids. Portland, in particular, produced few effects, though this was not completely surprising since impacts on employment and other economic outcomes were less positive for the client survey sample than for the full impact sample. Atlanta appeared to produce fewer unfavorable effects than the other sites, especially Grand Rapids and Riverside.
The effects found were primarily clustered by age. First, with data on few outcomes in two sites, these welfare-to-work programs produced no unfavorable effects on the outcomes of children who were toddlers at study entry (aged 6 and 7 at the five-year follow-up). The lack of more unfavorable effects for toddlers is somewhat contrary to what has been found in nonexperimental research that suggests mothers' employment during the first few years of a child's life produces unfavorable results. Fewer hours of employment, part-time work versus full-time work, or the quality of child care arrangements may partially explain why. Second, as has been found in the effects of other experimental welfare and employment policies, these welfare-to-work programs produced unfavorable effects on the outcomes of children who were adolescents at study entry (aged 15 to 23 at the five-year follow-up), especially on academic functioning. The unfavorable effects found for adolescents may be associated with lack of supervision, decreased income in the household, or changes in family composition. In any case, it is the well-being of these children that perhaps should be more closely monitored when mothers are required to participate in welfare-to-work programs.
1. This chapter presents program impacts on selected outcomes for all client survey respondents' children aged 3 to 5. See Chapter 12 for more detail about how the Child Outcomes Study (COS) sample differs from this sample.
2. For example, while research generally shows better cognitive and behavioral outcomes for children in formal child care settings (NICHD Early Child Care Research Network, 2000; Zaslow et al., 1998), studies have also linked attendance in larger child care settings with greater ear infections, with detrimental effects on hearing loss and language development more likely to occur in low-quality settings (Vernon-Feagans, Emanuel, and Blood, 1997). Thus, mothers' involvement in welfare-to-work programs that leads to increased use of large, formal settings for their young children may benefit their cognitive and behavioral development but adversely affect their health. In addition, any one program may have different effects on child outcomes, and these effects may reinforce or cancel each other out. For example, any beneficial effect of mothers' increased employment may be attenuated by any declines in income.
3. See McGroder et al., 2000; Hamilton et al., 2000; Zaslow et al., 1998, 1995.
4.For various reviews about the effects of income on children's outcomes see Duncan and Brooks-Gunn, 1997; Harvey, 1999; Hoffman and Youngblade, 1999; Mayer, 1997; Moore and Driscoll, 1997; Vandell and Ramanan, 1992; and Zaslow et al., 1999a. Some argue that Tthe effects of family income may also depend on the source of income, though many studies find no relation between welfare receipt and children's development controlling for demographic and family characteristics (for example, Haveman and Wolfe, 1995; Levine and Zimmerman, 2000; Yoshikawa, 1999). [Is this really true? I thought that this notion had lost a lot of currency lately, i.e., that welfare dollars were not worth as much as other dollars..
5. McGroder, 2000.
6. See, for example, the "stress hypothesis" proposed by Brooks-Gunn and Berlin, 1993. See also Zaslow et al., 1995.
7. Lamb, 1998; Phillips et al., 1994; Zaslow, 1991.
8. Survey respondents may have children who fall into one or more age groups; for example, a respondent may have a preschool-age child and a young school-age child at study entry. For this reason, survey impacts on economic outcomes for the mothers of children in various age groups do not necessarily reflect effects on mutually exclusive groups.
9. Families with children in different age groups may also differ in other ways. For example, respondents with adolescent children are more likely to be older, more likely to have been ever married, less likely to have a high school diploma or a GED, and more likely to be a longer-term recipient of welfare than respondents with younger children. Program impacts may instead reflect other characteristics of these respondents. Conditional subgroup impact analysis, which would test whether or not age of children or other characteristics explain program impacts, is beyond the scope of work for this chapter.
10.In general, there were a greater number of statistically significant findings than expected by chance alone overall and by each of the child age groups. See Chapter 2 for a more detailed description of the standard used to determine the likelihood of chance findings.
11.All standard errors in the impact analysis were adjusted so that the impact estimates account for the presence of two or more children or siblings within a family.
12.Mothers with children under age 3, or as young as age 1 in some sites, were exempt from participation in mandatory services. These mothers were also excluded from being part of NEWWS. However, because the child age groups were created based on information gathered at the five-year survey point, it is possible that some survey respondents provided information about infants (for example, an infant could have joined the household after random assignment through marriage or foster care). Consequently, though the majority of the toddlers group is composed of children aged 1 and 2 at the time of random assignment, 4.7 percent of these children were under age 1.
13.Detailed information about a subset of children aged 3 to 5 in the Child Outcomes Study includes child outcomes as evaluated by the mother, a teacher, and the child. See the discussion in Chapter 12 about ways in which mothers' and children's reports may differ. The New Chance and New Hope evaluations also found that mothers' reports of children's behavior and academic performance differed from teachers' reports of these outcomes (Quint, Bos, and Polit, 1997; Bos et al., 1999).
14.Freeman and Blanchflower, 1999.
15.Baydar and Brooks-Gunn, 1991.
16.Morris and Michalopoulos, 2000.
17.U.S. Department of Education, 2001.
18.Some examples include MFIP (Gennetian and Miller, 2000), SSP (Morris and Michalopoulos, 2000), and FTP (Bloom et al., 2000a).
19.Compared with the client survey sample, the Grand Rapids LFA program decreased earnings (though not significant) in year 5 and the Grand Rapids HCD program had no employment effect during the first year of follow-up.
20.Some, though not all, of this difference is due to lower control group levels in the sample of respondents with a preschool-age child at study entry.
21.Morris et al., 2001.
22.Duncan and Brooks-Gunn, 1997.
23.U.S. Department of Education, 2001.
24.U.S. Department of Education, 2001.
25.A third interesting pattern emerged in Portland. Portland increased suspensions and expulsions and ever attending a special class among young school-age children (though not statistically significant at conventional levels). These differences are of a similar magnitude, small to medium, as has been found in comparable experimental studies (0.2 to 0.3 effect size). The sample size of young school-age children in Portland was relatively small.
26.Compared with the client survey sample, survey sample members with adolescents had a negative pattern of employment effects, small to no significant earnings gains and a negative though not significant effect on cumulative combined income in the Atlanta LFA program; smaller and no significant effect on cumulative earnings in the Atlanta HCD program; smaller cumulated earnings effects and larger decreased cumulative combined income in the Grand Rapids LFA program; significant decreased cumulative combined income in the Grand Rapids HCD program; and slightly larger and more positive employment effects in Portland.
27.Petit et al., 1999; Posner and Vandell, 1994.
28.U.S. Department of Education, 2001.
29.Moore et al., 1993.
30.This measure includes both males and females.
31.Some of the discrepancy could also be due to the inclusion of teen births to males and females in the NEWWS sample.
32.Brown et al., 1999.
33.Child Trends, December 2000.
34.Although the number of significant differences between HCD and LFA outcome levels did not exceed chance for each of the child age groups, two interesting patterns did emerge for young school-age children and adolescents. For young school-age children, the Grand Rapids HCD outcome levels were lower than LFA outcome levels for suspensions and the likelihood of having a physical, emotional, or mental condition that demanded a lot of mothers' time; and the Riverside HCD outcome levels were lower than LFA outcome levels for emergency room visits. For adolescents, the Atlanta LFA outcome levels were lower than HCD outcome levels for suspensions and the likelihood of having a baby while a teen; and the Riverside LFA outcome levels were lower than HCD outcome levels for emergency room visits and the likelihood of having a physical, emotional, or mental condition that demanded a lot of mothers' time. These patterns are interesting in that they suggest that the type of activity that a mother is first required to participate in may have influences on child outcomes that vary by age of the child. For example, an education-focused programs may give mothers more flexibility to better manage their time between participation requirements and their children's need for supervision than an employment-focused program. In contrast, adolescents may benefit more from having a mother engaged in an employment-focused program than an education-focused program because of the role modeling that mothers in full-time employment provide.
35.Mortimer et al., 1996, and Steinberg and Dornbusch, 1991.
36.McLanahan, 1997.
Impacts on Young Children
Chapter 11 presented impacts on a circumscribed set of outcomes for children of all ages in the client survey sample (that is, all children in seven programs in four sites). This chapter examines in greater depth program impacts for a subset of these childrenВ namely, "focal" children in the Atlanta, Grand Rapids, and Riverside sites who were aged 3 to 5 at baseline (aged 8 to 10 at the five-year follow-up).These young children and their families constitute the Child Outcomes Study (COS) sample (see Chapter 2 for details).(1)
Findings reported here should be viewed as extending those presented for "preschool-age children" in Chapter 11, which focused on problem outcomes for children in two developmental domains (academic functioning and health and safety).This chapter examines both problem and positive outcomes for young children, measured in three developmental domains (academic functioning, social skills and behavior, and health and safety). In addition, more numerous outcomes in each domain are examined and include information from mothers, teachers, and the children themselves, as well as from a standardized achievement test.
Because this chapter focuses on impacts for a subset of the children discussed in Chapter 11, it is important to begin with a brief overview of key adult impacts for this sample to set the stage for interpreting impacts on young children. Earlier chapters presented economic impacts for the full NEWWS sample (11 programs in seven sites); detailed information on economic impacts for the COS sample and for the client survey sample can be found in Appendix I. In sum, despite some differences in these samples,(2) the program impacts on economic functioning for COS families were similar to those for the larger samples. With the exception of a decrease in income resulting from the Riverside HCD program in the full sample (compared with a similar size, though non-statistically significant, increase in income in the COS sample), any differences in economic impacts were in the magnitude, rather than the direction, of a given impact. It should be kept in mind that, as discussed in Chapter 1, some control group members in the Atlanta and Grand Rapids sites became eligible for program services prior to the end of the five-year follow-up period. As a result, in these two sites impacts measured as of the end of the follow-up period are probably understated relative to what may have occurred if treatment differences had been maintained in those sites.
This chapter follows a discussion of analysis issues with an examination in depth of program impacts on focal children at the five-year point. (Key patterns of impacts by level of disadvantage and by focal child gender are noted in footnotes.)(3) In an effort to begin to understand the ways in which impacts on young children may have come about, the chapter "maps" impacts found for focal children onto the economic impacts found for their mothers. It concludes with a discussion of findings. (Table 12.1 summarizes impacts on focal child outcomes.)
| Social skills and behavior (19 measures) |
Atlanta | Grand Rapids | Riverside | ||||
|---|---|---|---|---|---|---|---|
| LFA | HCD | LFA | HCD | LFA Full |
LFA In-needa |
HCD | |
| 6 F | 5 F | В | В | В | 2 F | 2 F | |
| В | 2 f | В | В | В | 2 f | 1 f | |
| В | В | 2 U | 6 U | 3 U | В | В | |
| 1 u | В | 1 u | 1 u | 1 u | В | В | |
|
Academic functioning |
|||||||
| Behavioral adjustment to school (5 measures) |
В | 2 F | В | В | 1 F | В | В |
| 1 f | В | В | В | В | В | В | |
| В | В | 1 U | 2 U | 1 U | В | В | |
| Achievement (6 mesures) |
В | В | 1 F | В | В | 1 F | В |
| Academic progress and placement (11 measures) |
1 F | 1 F | В | В | 1 F | 1 F | В |
| В | В | В | В | В | В | 1 f | |
| В | В | В | В | 1 U | В | В | |
| 1b | В | В | В | 1 u | 1 u | В | |
| Attendance (2 measures) |
1 U | 1 U | В | В | В | В | 1 U |
| 1 u | В | В | В | В | В | В | |
| Health and safety (5 measures) |
2 U | 1 U | 1 U | В | В | В | 1 U |
| В | 1 u | В | В | В | В | В | |
| Other (1 measure) | В | В | В | В | В | В | В |
| NOTES: "F" indicates a statistically significant favorable impact. "U" indicates a statistically significant unfavorable impact. "f" indicates a favorable impact above the cutoff for statistical significance but part of the overall pattern. "u" indicates an unfavorable impact above the cutoff for statistical significance but part of the overall pattern. See Chapter 2 for the definition of a pattern. Blank spaces indicate that there were no impacts. a This sample is a subset of the full Riverside LFA sample, containing only those members who lacked a high school diploma or basic skills at random assignment. b The Atlanta LFA program decreased the percentage of children identified by teachers as needing and receiving special services, which would be favorable if the program decreased the percentage of children needing services but unfavorable if the program decreased the percentage of children in need of services who received them. |
|||||||
Key Findings
- Few impacts on young children were found across sites. However, there were more impacts than would be expected by chance. Given that the JOBS program was not aimed at children, it is perhaps not surprising that relatively few impacts were found. It may be that impacts on outcomes important to children such as mothers' employment, family income, and/or the children's immediate environments (home, school, child care) were too few, occurred for too brief a period, or were of an insufficient magnitude to lead to numerous impacts on young children.
- Within specific domains or programs, patterns of impacts were found.
- All six programs affected young children's social skills and behavior, with the direction of impacts differing across sites. Both programs in Atlanta improved children's behavior, whereas both programs in Grand Rapids worsened children's behavior. The Riverside LFA program worsened children's behavior in the full sample, though for children whose mothers lacked a high school diploma or basic skills at study entry, both programs in Riverside improved behavior.
- There were few impacts on young children's math or reading skills or on measures of academic progress and placement (for example, grade repetition, performing above or below grade level in reading or math).
- There were few impacts on measures of health. When impacts were found, however, they were all unfavorable. Specifically, both Atlanta programs increased the likelihood of mothers' reporting that focal children had had a physical, mental, or emotional condition that required frequent medical attention and increased the likelihood of mothers' reports that such a condition impeded on their ability to go to work or school. In addition, mothers in the Grand Rapids LFA program and the Riverside HCD program were less likely than their control group counterparts to rate focal children's overall health as "very good" or "excellent."
- Impacts on young children generally did not vary according to the welfare-to-work strategy employed. The pattern of impacts was similar for the LFA and HCD programs in each site.
Analysis Issues
This section provides an overview of the outcomes examined for young children in the COS and discusses issues relating to interpreting information on child outcome measures from multiple sources. (A more detailed description of measures, including internal consistency reliabilities, can be found in Appendix J.)
Child Outcomes Examined
Child outcomes in three developmental domains were measured: social skills and behavior, academic functioning, and health and safety. Measures tapping both positive and problem outcomes were examined. Outcomes in the social skills and behavior domain consisted of subscales adapted from the Social Skills Ratings System, which was designed to tap "social behaviors that can affect teacher-student relations, peer acceptance, and academic performance."(4) Among the subscales are those that measure children's positive skills and behaviors (such as cooperation and self-control) as well as problem behaviors ("externalizing" behaviors such as fighting and arguing, "internalizing" behaviors such as acting sad or depressed, and hyperactive behaviors such as acting impulsively and being disruptive).
Outcomes in the academic functioning domain are particularly diverse and include (1) measures related to behavioral adjustment to school (for example, school engagement, disciplinary problems), (2) scores on a standardized assessment of focal children's math and reading skills (the Woodcock-Johnson Tests of Achievement Revised),(5) administered by trained interviewers, (3) measures related to academic progress and placement (for example, grade repetition, performance below grade level), and (4) measures of attendance (absenteeism, tardiness).
Outcomes in the health and safety domain include a global rating of focal children's general health; the percentage of mothers rating focal children's health as very good or excellent; the percentage of mothers reporting that focal children had a physical, mental, or emotional condition that required frequent medical attention; the percentage of mothers reporting that focal children had a physical, mental, or emotional condition that impeded on their going to work or school; and the percentage of mothers reporting that focal children had an accident, injury, or poisoning requiring an emergency room visit.
An additional outcome pertains to focal children's living arrangements: the percentage of mothers reporting that focal children had lived away from them at some point since random assignment because they could not care for them. This measure is not a child outcome in the sense that it reflects directly on the developmental status or well-being of children. However, given the potential effect on children of living apart from their mother and a concern that welfare reform not contribute to this outcome, it is reported on here (under the category "other").
The accompanying chart summarizes the outcomes measured and the source of information for each outcome. Mothers reported on focal children's social skills and behavior, behavioral adjustment to school, academic progress and placement, health and safety, and living arrangements. Children reported on their positive social skills and their engagement in school. Teachers reported on focal children's social skills and behavior, behavioral adjustment to school, academic progress and placement, and attendance.(6)
| Assessment | Mother | Child | Teacher | |
|---|---|---|---|---|
| Social Skills and Behavior | ||||
| Externalizing | X | X | ||
| Internalizing | X | X | ||
| Hyperactivity | X | X | ||
| Cooperation | X | X | X | |
| Positive assertion | X | X | X | |
| Self-control | X | X | X | |
| Empathy | X | |||
| Responsibility | X | |||
| Interpersonal skills | X | |||
| Positive approaches to learning | X | |||
Academic Functioning | ||||
Behavioral adjustment to school | ||||
| School engagement | X | X | ||
| Suspended or expelled | X | X | ||
| Disciplinary action | X | |||
Achievement | ||||
| Math skills | X | |||
| Above-average math | X | |||
| Below-average math | X | |||
| Reading skills | X | |||
| Above-average reading | X | |||
| Below-average reading | X | |||
Academic progress and placement | ||||
| Grade repetition | X | X | ||
| Below grade level in math | X | |||
| Above grade level in math | X | |||
| Below grade level in reading | X | |||
| Above grade level in reading | X | |||
| In remedial math group | X | |||
| In remedial reading group | X | |||
| Physical, mental, or emotional condition requiring a special class or school | X | |||
| Needs and receives services* | X | |||
| Needs but does not receive services | X | |||
| Attendance | ||||
| Absent | X | |||
| Tardy | X | |||
Health and Safety | ||||
| General health rating | X | |||
| In very good or excellent health | X | |||
| Physical, mental, or emotional condition requiring frequent medical attention | X | |||
| Physical, mental, or emotional condition impeding mother's work | X | |||
| Accident, injury, or poisoning requiring an emergency room visit | X | |||
Other | ||||
| Lived apart from mother | X | |||
| * See Appendix J for a full description of services examined. | ||||
Multiple Reporters
Obtaining information from multiple sources can provide a more comprehensive picture of children's behavior and development. Parents are an important source of such information, as they are "usually the most knowledgeable about their child's behavior across time and situations."(7) For school-age children, teachers can also provide important information on children's competencies and problems in the school setting. Children's reports can also be informative, for they represent subjective views of their own competencies and shortcomings. Finally, objective assessments of children's academic functioning such as standardized achievement tests provide a measure of performance on a specific "skill," free from subjective biases of informants.(8) Standardized assessments also are useful for comparing children's performance with those of their peers in the sample as well as to age-mates nationwide.
Mothers, teachers, and children provided information on children's social skills and behavior. Survey items in the social skills and behavior domain were worded specifically to capture social skills displayed in the classroom (for the teachers' report measures), social skills displayed in the home (for the mothers' report measures), and social skills more generally, as displayed in the home and/or the school (for the children's report measures). Thus, ratings of children's social skills and behavior by teachers should be seen as reflecting children's classroom behavior in the current school year, ratings of children's social skills and behavior by mothers likely capture the more stable component of children's behavior in the family, and children's ratings of their own social skills and behavior reflect their perceptions of their behavior in both the home and the school settings. Impacts on a social skill as rated by one reporter and not another thus may indicate changes in context-specific behavior. By contrast, a pattern of impacts that occurs on the same measure across reporters suggests a more global impact on children's positive and/or problem behavior.
Ratings of children's academic functioning were provided largely by teachers and, thus, reflect the teachers' views of how the children were performing (academically, socially) in school. While it is reasonable to expect that these reports and any impacts on academic measures would not contradict objective assessments of (and any impacts on) children's math and reading skills, such contradictions may arise if, for example, teachers' evaluations of children's academic capabilities reflect a more global perception of how the children are performing and behaving in school.
Finally, information on focal children's health and safety was obtained solely from mothers. Neither objective ratings of physical health (for example, height or weight) nor diagnoses by medical professionals were obtained. Thus, it is not clear to what extent mean levels and impacts on health and safety measures reflect the focal children's actual health status and to what extent they reflect mothers' perceptions of the children's health.
This section presents impacts of the LFA and HCD programs in Atlanta, Grand Rapids, and Riverside on children's developmental outcomes in the domains of social skills and behavior, academic functioning, health and safety, and the outcome relating to living arrangements.
Impacts on Children
Social Skills and Behavior
Table 12.2 shows impacts on outcomes relating to focal children's social skills and behavior. As described above, outcomes examined in this domain come from reports by mothers, teachers, and children of both positive and problem behaviors.
|
Site and Program |
Sample Size | Program Group | Control Group | Difference (Impact) | Effect Size |
|---|---|---|---|---|---|
|
Externalizing behavior (range of 0 to 18) (mother report) |
|||||
| Atlanta Labor Force Attachment | 542 | 3.8 | 4.2 | -0.5** | -0.20 |
| Atlanta Human Capital Development | 609 | 3.9 | 4.2 | -0.3 | -0.13 |
| Grand Rapids Labor Force Attachment | 376 | 5.5 | 5.6 | -0.1 | -0.02 |
| Grand Rapids Human Capital Development | 358 | 5.6 | 5.7 | -0.1 | -0.02 |
| Riverside Labor Force Attachment | 454 | 4.7 | 4.9 | -0.3 | -0.10 |
| Lacked high school diploma or basic skills | 284 | 4.8 | 5.0 | -0.2 | -0.09 |
| Riverside Human Capital Development | 368 | 4.6 | 5.0 | -0.4 | -0.15 |
|
Externalizing behavior (range of 0 to 15) (teacher report) |
|||||
| Atlanta Labor Force Attachment | 363 | 4.1 | 5.1 | -1.0** | -0.29 |
| Atlanta Human Capital Development | 409 | 4.2 | 5.1 | -0.9** | -0.25 |
| Grand Rapids Labor Force Attachment | 283 | 3.7 | 4.0 | -0.2 | -0.08 |
| Grand Rapids Human Capital Development | 259 | 4.8 | 3.8 | 0.9* | 0.28 |
| Riverside Labor Force Attachment | 323 | 3.9 | 3.1 | 0.8** | 0.28 |
| Lacked high school diploma or basic skills | 205 | 3.9 | 3.4 | 0.5 | 0.18 |
| Riverside Human Capital Development | 272 | 3.0 | 3.5 | -0.4 | -0.16 |
|
Internalizing behavior (range of 0 to 24) (mother report) |
|||||
| Atlanta Labor Force Attachment | 546 | 7.7 | 7.8 | -0.1 | -0.03 |
| Atlanta Human Capital Development | 617 | 7.7 | 7.8 | -0.0 | -0.01 |
| Grand Rapids Labor Force Attachment | 376 | 8.8 | 9.1 | -0.3 | -0.08 |
| Grand Rapids Human Capital Development | 357 | 8.6 | 9.1 | -0.5 | -0.15 |
| Riverside Labor Force Attachment | 456 | 8.6 | 8.2 | 0.4 | 0.14 |
| Lacked high school diploma or basic skills | 286 | 8.5 | 8.4 | 0.1 | 0.03 |
| Riverside Human Capital Development | 370 | 8.4 | 8.3 | 0.0 | 0.02 |
|
Internalizing behavior (range of 0 to 18) (teacher report) |
|||||
| Atlanta Labor Force Attachment | 357 | 4.0 | 5.2 | -1.2** | -0.33 |
| Atlanta Human Capital Development | 404 | 4.2 | 5.1 | -0.9** | -0.25 |
| Grand Rapids Labor Force Attachment | 281 | 3.6 | 4.0 | -0.4 | -0.13 |
| Grand Rapids Human Capital Development | 257 | 4.8 | 3.9 | 0.9* | 0.27 |
| Riverside Labor Force Attachment | 323 | 3.9 | 3.2 | 0.8** | 0.27 |
| Lacked high school diploma or basic skills | 204 | 3.9 | 3.5 | 0.5 | 0.17 |
| Riverside Human Capital Development | 266 | 3.1 | 3.6 | -0.4 | -0.15 |
|
Hyperactivity (range of 0 to 18) (mother report) |
|||||
| Atlanta Labor Force Attachment | 545 | 5.9 | 5.9 | -0.0 | -0.02 |
| Atlanta Human Capital Development | 616 | 5.9 | 5.9 | -0.1 | -0.03 |
| Grand Rapids Labor Force Attachment | 379 | 6.8 | 6.9 | -0.1 | -0.02 |
| Grand Rapids Human Capital Development | 360 | 6.8 | 6.9 | -0.1 | -0.02 |
| Riverside Labor Force Attachment | 459 | 6.6 | 6.1 | 0.5 | 0.17 |
| Lacked high school diploma or basic skills | 290 | 6.4 | 6.1 | 0.2 | 0.11 |
| Riverside Human Capital Development | 373 | 6.2 | 6.1 | 0.1 | 0.03 |
|
Hyperactivity (range of 0 to 6) (teacher report) |
|||||
| Atlanta Labor Force Attachment | 360 | 1.9 | 2.2 | -0.3* | -0.24 |
| Atlanta Human Capital Development | 401 | 1.9 | 2.2 | -0.4** | -0.25 |
| Grand Rapids Labor Force Attachment | 283 | 2.0 | 2.0 | 0.0 | 0.03 |
| Grand Rapids Human Capital Development | 261 | 2.5 | 2.0 | 0.5** | 0.32 |
| Riverside Labor Force Attachment | 321 | 2.3 | 1.9 | 0.4** | 0.29 |
| Lacked high school diploma or basic skills | 204 | 2.2 | 2.0 | 0.2 | 0.16 |
| Riverside Human Capital Development | 270 | 2.1 | 2.1 | -0.0 | -0.01 |
|
Cooperation (range of 0 to 30) (child report) |
|||||
| Atlanta Labor Force Attachment | 558 | 22.5 | 23.2 | -0.7 | -0.17 |
| Atlanta Human Capital Development | 627 | 22.9 | 23.1 | -0.3 | -0.06 |
| Grand Rapids Labor Force Attachment | 378 | 22.7 | 23.4 | -0.6 | -0.14 |
| Grand Rapids Human Capital Development | 362 | 22.3 | 23.4 | -1.0** | -0.23 |
| Riverside Labor Force Attachment | 459 | 21.9 | 22.0 | -0.1 | -0.01 |
| Lacked high school diploma or basic skills | 289 | 22.2 | 21.9 | 0.3 | 0.07 |
| Riverside Human Capital Development | 384 | 22.2 | 21.9 | 0.3 | 0.07 |
|
Cooperation (range of 0 to 39) (mother report) |
|||||
| Atlanta Labor Force Attachment | 549 | 22.4 | 21.9 | 0.5 | 0.10 |
| Atlanta Human Capital Development | 620 | 22.1 | 21.9 | 0.3 | 0.05 |
| Grand Rapids Labor Force Attachment | 382 | 21.3 | 22.1 | -0.9 | -0.13 |
| Grand Rapids Human Capital Development | 363 | 21.4 | 22.1 | -0.7 | -0.11 |
| Riverside Labor Force Attachment | 460 | 22.3 | 22.1 | 0.2 | 0.03 |
| Lacked high school diploma or basic skills | 290 | 22.5 | 22.3 | 0.2 | 0.05 |
| Riverside Human Capital Development | 374 | 23.0 | 22.2 | 0.7 | 0.15 |
|
Cooperation (range of 0 to 27) (teacher report) |
|||||
| Atlanta Labor Force Attachment | 367 | 16.2 | 15.1 | 1.0 | 0.18 |
| Atlanta Human Capital Development | 410 | 16.2 | 15.2 | 1.0 | 0.17 |
| Grand Rapids Labor Force Attachment | 286 | 14.9 | 16.2 | -1.3* | -0.22 |
| Grand Rapids Human Capital Development | 261 | 14.8 | 16.0 | -1.2 | -0.21 |
| Riverside Labor Force Attachment | 325 | 15.1 | 16.2 | -1.1 | -0.19 |
| Lacked high school diploma or basic skills | 206 | 15.4 | 15.2 | 0.2 | 0.03 |
| Riverside Human Capital Development | 275 | 15.1 | 15.2 | -0.2 | -0.03 |
|
Positive assertion (range of 0 to 30) (child report) |
|||||
| Atlanta Labor Force Attachment | 557 | 20.0 | 20.1 | -0.0 | -0.01 |
| Atlanta Human Capital Development | 629 | 19.9 | 20.1 | -0.2 | -0.05 |
| Grand Rapids Labor Force Attachment | 379 | 21.3 | 22.2 | -0.9* | -0.21 |
| Grand Rapids Human Capital Development | 363 | 20.9 | 22.2 | -1.3** | -0.29 |
| Riverside Labor Force Attachment | 458 | 19.9 | 20.0 | -0.2 | -0.03 |
| Lacked high school diploma or basic skills | 289 | 20.0 | 19.5 | 0.6 | 0.12 |
| Riverside Human Capital Development | 381 | 20.1 | 19.6 | 0.5 | 0.12 |
|
Positive assertion (range of 0 to 30) (mother report) |
|||||
| Atlanta Labor Force Attachment | 549 | 20.7 | 20.7 | -0.0 | -0.01 |
| Atlanta Human Capital Development | 619 | 21.0 | 20.7 | 0.3 | 0.06 |
| Grand Rapids Labor Force Attachment | 380 | 21.2 | 21.6 | -0.4 | -0.08 |
| Grand Rapids Human Capital Development | 360 | 21.4 | 21.6 | -0.2 | -0.05 |
| Riverside Labor Force Attachment | 459 | 21.5 | 21.7 | -0.1 | -0.03 |
| Lacked high school diploma or basic skills | 289 | 20.9 | 20.9 | -0.0 | -0.00 |
| Riverside Human Capital Development | 371 | 21.8 | 21.0 | 0.8 | 0.21 |
|
Positive assertion (range of 0 to 18) (teacher report) |
|||||
| Atlanta Labor Force Attachment | 363 | 11.6 | 10.8 | 0.8* | 0.25 |
| Atlanta Human Capital Development | 405 | 11.6 | 10.9 | 0.7 | 0.21 |
| Grand Rapids Labor Force Attachment | 279 | 10.6 | 11.0 | -0.4 | -0.13 |
| Grand Rapids Human Capital Development | 253 | 10.5 | 11.0 | -0.5 | -0.15 |
| Riverside Labor Force Attachment | 318 | 11.8 | 11.8 | 0.1 | 0.02 |
| Lacked high school diploma or basic skills | 201 | 12.2 | 11.1 | 1.2** | 0.39 |
| Riverside Human Capital Development | 266 | 11.2 | 11.3 | -0.0 | -0.01 |
|
Self-control (range of 0 to 30) (child report) |
|||||
| Atlanta Labor Force Attachment | 558 | 19.6 | 19.5 | 0.1 | 0.02 |
| Atlanta Human Capital Development | 629 | 19.3 | 19.5 | -0.3 | -0.06 |
| Grand Rapids Labor Force Attachment | 378 | 19.9 | 20.6 | -0.7 | -0.15 |
| Grand Rapids Human Capital Development | 362 | 19.4 | 20.6 | -1.2** | -0.25 |
| Riverside Labor Force Attachment | 461 | 18.7 | 19.0 | -0.3 | -0.06 |
| Lacked high school diploma or basic skills | 291 | 18.9 | 18.7 | 0.2 | 0.05 |
| Riverside Human Capital Development | 382 | 19.0 | 18.6 | 0.4 | 0.08 |
|
Self-control (range of 0 to 30) (mother report) |
|||||
| Atlanta Labor Force Attachment | 548 | 16.1 | 16.1 | -0.0 | -0.00 |
| Atlanta Human Capital Development | 619 | 15.8 | 16.0 | -0.3 | -0.07 |
| Grand Rapids Labor Force Attachment | 380 | 15.4 | 15.4 | 0.0 | 0.01 |
| Grand Rapids Human Capital Development | 359 | 15.4 | 15.5 | -0.1 | -0.03 |
| Riverside Labor Force Attachment | 461 | 16.4 | 15.8 | 0.6 | 0.14 |
| Lacked high school diploma or basic skills | 291 | 16.2 | 15.2 | 1.0* | 0.26 |
| Riverside Human Capital Development | 373 | 16.5 | 15.3 | 1.2** | 0.29 |
|
Self-control (range of 0 to 27) (teacher report) |
|||||
| Atlanta Labor Force Attachment | 359 | 16.3 | 15.4 | 0.9 | 0.15 |
| Atlanta Human Capital Development | 405 | 16.7 | 15.4 | 1.3* | 0.23 |
| Grand Rapids Labor Force Attachment | 261 | 15.9 | 16.0 | -0.2 | -0.03 |
| Grand Rapids Human Capital Development | 239 | 15.3 | 16.1 | -0.8 | -0.14 |
| Riverside Labor Force Attachment | 307 | 18.0 | 17.7 | 0.3 | 0.05 |
| Lacked high school diploma or basic skills | 198 | 18.4 | 17.0 | 1.5 | 0.28 |
| Riverside Human Capital Development | 260 | 17.3 | 17.0 | 0.2 | 0.05 |
|
Empathy (range of 0 to 30) (child report) |
|||||
| Atlanta Labor Force Attachment | 559 | 21.2 | 20.9 | 0.2 | 0.05 |
| Atlanta Human Capital Development | 627 | 21.4 | 21.0 | 0.4 | 0.08 |
| Grand Rapids Labor Force Attachment | 377 | 22.3 | 22.7 | -0.4 | -0.09 |
| Grand Rapids Human Capital Development | 361 | 22.3 | 22.8 | -0.4 | -0.10 |
| Riverside Labor Force Attachment | 457 | 21.0 | 21.7 | -0.7 | -0.15 |
| Lacked high school diploma or basic skills | 288 | 21.1 | 21.5 | -0.4 | -0.08 |
| Riverside Human Capital Development | 380 | 20.9 | 21.6 | -0.7 | -0.14 |
|
Responsibility (range of 0 to 27) (mother report) |
|||||
| Atlanta Labor Force Attachment | 559 | 17.4 | 17.6 | -0.2 | -0.06 |
| Atlanta Human Capital Development | 624 | 18.1 | 17.5 | 0.5 | 0.14 |
| Grand Rapids Labor Force Attachment | 383 | 17.8 | 17.7 | 0.1 | 0.03 |
| Grand Rapids Human Capital Development | 365 | 17.8 | 17.8 | 0.0 | 0.01 |
| Riverside Labor Force Attachment | 473 | 17.6 | 17.8 | -0.3 | -0.07 |
| Lacked high school diploma or basic skills | 298 | 17.4 | 17.3 | 0.1 | 0.03 |
| Riverside Human Capital Development | 390 | 18.3 | 17.4 | 0.9* | 0.24 |
|
Interpersonal skills (range of 0 to 12) (teacher report) |
|||||
| Atlanta Labor Force Attachment | 364 | 7.7 | 7.1 | 0.6** | 0.26 |
| Atlanta Human Capital Development | 408 | 7.9 | 7.1 | 0.**7 | 0.30 |
| Grand Rapids Labor Force Attachment | 279 | 7.4 | 7.5 | -0.2 | -0.07 |
| Grand Rapids Human Capital Development | 254 | 7.1 | 7.5 | -0.5 | -0.18 |
| Riverside Labor Force Attachment | 322 | 8.2 | 8.3 | -0.1 | -0.04 |
| Lacked high school diploma or basic skills | 205 | 8.4 | 7.9 | 0.5 | 0.26 |
| Riverside Human Capital Development | 272 | 8.0 | 8.0 | -0.1 | -0.03 |
|
Positive approaches to learning (range of 0 to 18)(teacher report) |
|||||
| Atlanta Labor Force Attachment | 366 | 10.9 | 10.4 | 0.5 | 0.16 |
| Atlanta Human Capital Development | 408 | 10.9 | 10.4 | 0.5 | 0.15 |
| Grand Rapids Labor Force Attachment | 279 | 10.1 | 10.7 | -0.6 | -0.16 |
| Grand Rapids Human Capital Development | 254 | 10.0 | 10.7 | -0.7 | -0.19 |
| Riverside Labor Force Attachment | 324 | 10.6 | 10.8 | -0.2 | -0.07 |
| Lacked high school diploma or basic skills | 206 | 10.5 | 10.2 | 0.3 | 0.10 |
| Riverside Human Capital Development | 274 | 10.5 | 10.3 | 0.2 | 0.07 |
| SOURCE: Child Trends calculations from the Five-Year Child Outcomes Study Survey (mother, teacher, and child reports). NOTES: See Appendix A.2. A higher score on each measure indicates that the child demonstrates more of the construct. |
|||||
The largest concentration of impacts across programs, and typically within a given program, occurred in this domain of outcomes. All six programs affected children's social skills and behavior, though the direction of these impacts differed depending on the site (the Atlanta and Grand Rapids programs) or on mothers' level of education at baseline (the Riverside programs). Both Atlanta programs improved social skills and behavior, as did the Riverside HCD program and the Riverside LFA program for children whose mothers lacked a high school diploma or basic skills at baseline. In contrast, both Grand Rapids programs had unfavorable impacts on children's social skills and behavior, as did the Riverside LFA program for the full sample (that is, regardless of mothers' level of education at baseline).
Both Atlanta programs decreased problem behaviors and simultaneously increased positive behaviors.(9) Specifically, both programs decreased teacher-reported levels of externalizing, internalizing, and hyperactive behavior and increased teacher-reported levels of interpersonal skills. Further, both programs resulted in lower levels of mother-rated externalizing behavior and higher levels of teacher-reported positive assertion (a measure of children's positive initiations toward others in social situations), though the differences for the HCD program were just beyond the cutoff for statistical significance. The Atlanta HCD program also increased levels of children's self-control as rated by the teacher.(10)
In contrast to Atlanta, the impacts on social skills and behavior in Grand Rapids were uniformly unfavorable, with both programs decreasing focal children's positive behaviors and the HCD program also increasing problem behaviors. For instance, both programs decreased child-reported levels of assertion and decreased levels of both child- and teacher-reported cooperation, although the differences in child-reported cooperation in the LFA program and teacher-reported cooperation in the HCD program were just beyond the cutoff for statistical significance.(11) The HCD program also decreased children's reports of their own self-control and simultaneously increased their problem behavior, increasing teacher-reported levels of externalizing, internalizing, and hyperactive behavior.
The impacts on social skills and behavior were fewer in Riverside than in Atlanta or in the Grand Rapids HCD program and varied according to mothers' level of education at baseline. The Riverside LFA program increased problem behaviors for the full sample (regardless of mothers' level of education at baseline), while both the LFA and the HCD programs increased positive behaviors for the subsample of children whose mothers had limited education at baseline. Specifically, in the full sample the LFA program increased levels of teacher-reported externalizing, internalizing, and hyperactive behavior.(12) This program also increased mother-reported levels of hyperactive behavior, though this difference was just beyond the cutoff for statistical significance (an effect size of .17). Yet, for the subsample of mothers without a high school diploma or basic skills at baseline, both the LFA and HCD programs improved behavioral outcomes, for example, increasing mother-reported levels of self-control in children. Further, the LFA program increased teacher-rated levels of assertion, self-control, and interpersonal skills for this subgroup.(13) Notably, the favorable impact of the Riverside LFA program on teacher-reported levels of positive assertion for this subgroup was quite large compared with the other impacts in the social skills and behavior domain, with an effect size of .39. The Riverside HCD program also increased mother-reported levels of responsibility and assertion, although the latter impact was just beyond the cutoff for statistical significance.
With few exceptions, the magnitude of the impacts in the social skills and behavior domain across the six programs ranged from .20 to .30 of a standard deviation. These effect sizes fall at the lower end of the effect size range for some of the more successful programs that directly targeted children, such as the Perry Preschool Program and the Abecedarian Project.(14) Given that the programs evaluated in this chapter did not directly intervene with children, the fact that these effect sizes are within the range of those found in the child-focused programs is perhaps surprising and suggests that the welfare-to-work programs examined here affected children's behavior in nontrivial ways.
Further, the impacts described in this section were found on social skills and behavior that are likely to be meaningful for children's future development. Positive social skills may be important for children's success in the school setting, as well as for the development of positive relationships with peers and adults, and therefore may have implications for successful functioning later in life.(15) In addition, research has shown that problem social behaviors that go untreated are related to "poor academic performance, and may result in later social maladjustment problems or serious psychopathology."(16) Early problem behaviors have also been identified as predictors of later delinquency in adolescence.(17)
Academic Functioning
Table 12.3 shows the impacts on child outcomes related to academic functioning. As mentioned, these outcomes were reported by mothers, teachers, and children, as well as obtained from a standardized assessment, and fell into the following subdomains: behavioral adjustment to school, academic achievement, academic progress and placement, and attendance.
Site and Program | Sample Size | Program Group | Control Group | Difference (Impact) | Effect Size |
|---|---|---|---|---|---|
School engagement (range of 0 to 21) (child report) | |||||
| Atlanta Labor Force Attachment | 552 | 16.6 | 16.8 | -0.2 | -0.07 |
| Atlanta Human Capital Development | 617 | 16.7 | 16.8 | -0.1 | -0.05 |
| Grand Rapids Labor Force Attachment | 377 | 16.3 | 17.2 | -0.9** | -0.30 |
| Grand Rapids Human Capital Development | 358 | 16.5 | 17.1 | -0.6* | -0.21 |
| Riverside Labor Force Attachment | 459 | 16.2 | 16.3 | -0.1 | -0.02 |
| Lacked high school diploma or basic skills | 290 | 16.2 | 16.0 | 0.2 | 0.05 |
| Riverside Human Capital Development | 379 | 16.4 | 16.1 | 0.3 | 0.09 |
School engagement (range of 0 to 33) (teacher report) | |||||
| Atlanta Labor Force Attachment | 364 | 19.8 | 19.1 | 0.7 | 0.11 |
| Atlanta Human Capital Development | 414 | 20.6 | 19.2 | 1.4* | 0.21 |
| Grand Rapids Labor Force Attachment | 286 | 18.9 | 19.2 | -0.3 | -0.04 |
| Grand Rapids Human Capital Development | 262 | 18.2 | 19.1 | -0.9 | -0.14 |
| Riverside Labor Force Attachment | 324 | 18.7 | 19.5 | -0.9 | -0.13 |
| Lacked high school diploma or basic skills | 205 | 19.0 | 18.5 | 0.5 | 0.08 |
| Riverside Human Capital Development | 274 | 19.0 | 18.7 | 0.4 | 0.06 |
Suspended or expelled since last interview a (%) (mother report) | |||||
| Atlanta Labor Force Attachment | 599 | 8.5 | 6.5 | 2.0 | 0.10 |
| Atlanta Human Capital Development | 676 | 7.0 | 6.1 | 1.0 | 0.05 |
| Grand Rapids Labor Force Attachment | 428 | 5.7 | 9.0 | -3.3 | -0.13 |
| Grand Rapids Human Capital Development | 410 | 7.0 | 8.3 | -1.3 | -0.05 |
| Riverside Labor Force Attachment | 533 | 2.8 | 6.8 | -4.0* | -0.17 |
| Lacked high school diploma or basic skills | 338 | 3.3 | 4.6 | -1.3 | -0.07 |
| Riverside Human Capital Development | 438 | 6.0 | 4.5 | 1.4 | 0.08 |
Had disciplinary action taken weekly (%) (teacher report) | |||||
| Atlanta Labor Force Attachment | 369 | 30.8 | 37.3 | -6.5 | -0.17 |
| Atlanta Human Capital Development | 412 | 32.1 | 36.6 | -4.5 | -0.12 |
| Grand Rapids Labor Force Attachment | 285 | 34.3 | 31.6 | 2.7 | 0.06 |
| Grand Rapids Human Capital Development | 261 | 37.8 | 31.2 | 6.6 | 0.15 |
| Riverside Labor Force Attachment | 325 | 39.4 | 29.5 | 9.9* | 0.24 |
| Lacked high school diploma or basic skills | 206 | 32.4 | 30.6 | 1.8 | 0.05 |
| Riverside Human Capital Development | 274 | 33.9 | 30.9 | 3.0 | 0.08 |
Had a discipline problem that resulted in parent(s) being notified this school year (%) (teacher report) | |||||
| Atlanta Labor Force Attachment | 365 | 46.4 | 55.3 | -8.9 | -0.22 |
| Atlanta Human Capital Development | 407 | 42.4 | 55.8 | -13.4*** | -0.34 |
| Grand Rapids Labor Force Attachment | 284 | 33.4 | 32.7 | 0.7 | 0.02 |
| Grand Rapids Human Capital Development | 259 | 47.1 | 31.4 | 15.7** | 0.36 |
| Riverside Labor Force Attachment | 325 | 35.7 | 29.2 | 6.4 | 0.15 |
| Lacked high school diploma or basic skills | 205 | 24.6 | 31.9 | -7.3 | -0.19 |
| Riverside Human Capital Development | 274 | 35.0 | 33.2 | 1.8 | 0.05 |
Broad Math Score on the Woodcock-Johnson Tests of Achievement-Revised | |||||
| Atlanta Labor Force Attachment | 557 | 99.7 | 99.5 | 0.2 | 0.01 |
| Atlanta Human Capital Development | 627 | 101.1 | 99.4 | 1.7 | 0.11 |
| Grand Rapids Labor Force Attachment | 377 | 100.0 | 99.2 | 0.8 | 0.05 |
| Grand Rapids Human Capital Development | 363 | 97.8 | 99.0 | -1.2 | -0.08 |
| Riverside Labor Force Attachment | 463 | 98.9 | 97.8 | 1.1 | 0.07 |
| Lacked high school diploma or basic skills | 292 | 97.1 | 94.6 | 2.5 | 0.19 |
| Riverside Human Capital Development | 383 | 95.6 | 95.4 | 0.2 | 0.01 |
Had above-average Broad Math Score on the Woodcock-Johnson Tests of Achievement-Revised (%) | |||||
| Atlanta Labor Force Attachment | 557 | 30.2 | 27.4 | 2.8 | 0.08 |
| Atlanta Human Capital Development | 627 | 31.3 | 27.8 | 3.5 | 0.10 |
| Grand Rapids Labor Force Attachment | 377 | 31.4 | 23.6 | 7.8* | 0.20 |
| Grand Rapids Human Capital Development | 363 | 28.0 | 24.2 | 3.8 | 0.10 |
| Riverside Labor Force Attachment | 463 | 30.8 | 24.2 | 6.6 | 0.16 |
| Lacked high school diploma or basic skills | 292 | 30.7 | 15.9 | 14.8*** | 0.49 |
| Riverside Human Capital Development | 383 | 18.6 | 17.9 | 0.7 | 0.02 |
Had below-average Broad Math Score on the Woodcock-Johnson Tests of Achievement-Revised (%) | |||||
| Atlanta Labor Force Attachment | 557 | 23.5 | 25.8 | -2.3 | -0.06 |
| Atlanta Human Capital Development | 627 | 22.2 | 26.3 | -4.1 | -0.11 |
| Grand Rapids Labor Force Attachment | 377 | 24.5 | 21.4 | 3.1 | 0.08 |
| Grand Rapids Human Capital Development | 363 | 28.5 | 21.8 | 6.8 | 0.18 |
| Riverside Labor Force Attachment | 463 | 25.2 | 27.0 | -1.9 | -0.05 |
| Lacked high school diploma or basic skills | 292 | 27.9 | 31.7 | -3.8 | -0.10 |
| Riverside Human Capital Development | 383 | 34.9 | 30.1 | 4.9 | 0.13 |
Broad Reading Score on the Woodcock-Johnson Tests of Achievement-Revised | |||||
| Atlanta Labor Force Attachment | 557 | 94.7 | 95.0 | -0.2 | -0.02 |
| Atlanta Human Capital Development | 627 | 95.5 | 95.0 | 0.5 | 0.04 |
| Grand Rapids Labor Force Attachment | 377 | 100.2 | 98.8 | 1.4 | 0.09 |
| Grand Rapids Human Capital Development | 363 | 97.7 | 98.7 | -1.0 | -0.06 |
| Riverside Labor Force Attachment | 463 | 95.4 | 95.2 | 0.3 | 0.02 |
| Lacked high school diploma or basic skills | 292 | 93.5 | 92.6 | 0.9 | 0.06 |
| Riverside Human Capital Development | 383 | 93.7 | 93.2 | 0.5 | 0.03 |
Had above-average Broad Reading Score on the Woodcock-Johnson Tests of Achievement-Revised (%) | |||||
| Atlanta Labor Force Attachment | 557 | 15.9 | 17.5 | -1.6 | -0.05 |
| Atlanta Human Capital Development | 627 | 17.5 | 17.6 | -0.1 | -0.00 |
| Grand Rapids Labor Force Attachment | 377 | 31.5 | 25.4 | 6.1 | 0.15 |
| Grand Rapids Human Capital Development | 363 | 27.1 | 26.1 | 1.0 | 0.02 |
| Riverside Labor Force Attachment | 463 | 18.9 | 19.6 | -0.7 | -0.02 |
| Lacked high school diploma or basic skills | 292 | 17.4 | 16.9 | 0.5 | 0.02 |
| Riverside Human Capital Development | 383 | 15.2 | 17.9 | -2.7 | -0.09 |
Had below-average Broad Reading Score on the Woodcock-Johnson Tests of Achievement-Revised (%) | |||||
| Atlanta Labor Force Attachment | 557 | 33.3 | 36.5 | -3.2 | -0.08 |
| Atlanta Human Capital Development | 627 | 32.1 | 36.3 | -4.3 | -0.11 |
| Grand Rapids Labor Force Attachment | 377 | 24.4 | 27.9 | -3.5 | -0.08 |
| Grand Rapids Human Capital Development | 363 | 30.2 | 28.7 | 1.5 | 0.04 |
| Riverside Labor Force Attachment | 463 | 36.0 | 33.7 | 2.2 | 0.05 |
| Lacked high school diploma or basic skills | 292 | 43.2 | 38.7 | 4.5 | 0.11 |
| Riverside Human Capital Development | 383 | 40.1 | 37.1 | 3.0 | 0.08 |
Repeated a grade since last interview a (%) (mother report) | |||||
| Atlanta Labor Force Attachment | 599 | 12.8 | 13.0 | -0.1 | -0.00 |
| Atlanta Human Capital Development | 676 | 10.9 | 12.9 | -2.0 | -0.07 |
| Grand Rapids Labor Force Attachment | 428 | 16.4 | 11.8 | 4.6 | 0.15 |
| Grand Rapids Human Capital Development | 410 | 16.1 | 12.2 | 3.8 | 0.13 |
| Riverside Labor Force Attachment | 533 | 3.2 | 9.4 | -6.1** | -0.23 |
| Lacked high school diploma or basic skills | 338 | 4.8 | 10.2 | -5.4* | -0.22 |
| Riverside Human Capital Development | 438 | 6.1 | 9.9 | -3.8 | -0.16 |
Ever repeated a grade (%) (teacher report) | |||||
| Atlanta Labor Force Attachment | 336 | 15.0 | 11.8 | 3.2 | 0.13 |
| Atlanta Human Capital Development | 381 | 11.8 | 11.9 | -0.1 | -0.00 |
| Grand Rapids Labor Force Attachment | 248 | 8.9 | 13.2 | -4.3 | -0.14 |
| Grand Rapids Human Capital Development | 238 | 15.7 | 12.0 | 3.6 | 0.12 |
| Riverside Labor Force Attachment | 288 | 3.7 | 5.3 | -1.6 | -0.08 |
| Lacked high school diploma or basic skills | 181 | 5.6 | 7.0 | -1.4 | -0.07 |
| Riverside Human Capital Development | 242 | 3.7 | 6.8 | -3.1 | -0.16 |
Below grade level in math (%) (teacher report) | |||||
| Atlanta Labor Force Attachment | 362 | 35.9 | 35.7 | 0.2 | 0.01 |
| Atlanta Human Capital Development | 401 | 32.1 | 35.0 | -2.9 | -0.08 |
| Grand Rapids Labor Force Attachment | 283 | 38.9 | 38.4 | 0.5 | 0.01 |
| Grand Rapids Human Capital Development | 257 | 42.7 | 38.2 | 4.4 | 0.10 |
| Riverside Labor Force Attachment | 321 | 47.4 | 39.7 | 7.8 | 0.18 |
| Lacked high school diploma or basic skills | 204 | 46.6 | 47.1 | -0.5 | -0.01 |
| Riverside Human Capital Development | 269 | 43.0 | 45.0 | -2.0 | -0.05 |
Above grade level in math (%) (teacher report) | |||||
| Atlanta Labor Force Attachment | 362 | 9.8 | 8.0 | 1.8 | 0.08 |
| Atlanta Human Capital Development | 401 | 7.7 | 8.3 | -0.6 | -0.03 |
| Grand Rapids Labor Force Attachment | 283 | 8.1 | 11.7 | -3.6 | -0.12 |
| Grand Rapids Human Capital Development | 257 | 9.2 | 11.9 | -2.7 | -0.09 |
| Riverside Labor Force Attachment | 321 | 6.7 | 8.5 | -1.8 | -0.06 |
| Lacked high school diploma or basic skills | 204 | 4.3 | 4.8 | -0.5 | -0.03 |
| Riverside Human Capital Development | 269 | 5.2 | 5.3 | -0.1 | -0.00 |
Below grade level in reading (%) (teacher report) | |||||
| Atlanta Labor Force Attachment | 369 | 38.8 | 42.7 | -3.9 | -0.10 |
| Atlanta Human Capital Development | 412 | 39.3 | 42.8 | -3.5 | -0.09 |
| Grand Rapids Labor Force Attachment | 285 | 50.3 | 44.7 | 5.6 | 0.12 |
| Grand Rapids Human Capital Development | 259 | 50.9 | 45.6 | 5.4 | 0.12 |
| Riverside Labor Force Attachment | 326 | 49.5 | 42.4 | 7.1 | 0.16 |
| Lacked high school diploma or basic skills | 207 | 56.0 | 54.1 | 2.0 | 0.05 |
| Riverside Human Capital Development | 274 | 50.1 | 53.1 | -3.0 | -0.07 |
Above grade level in reading (%) (teacher report) | |||||
| Atlanta Labor Force Attachment | 369 | 12.6 | 6.9 | 5.6* | 0.27 |
| Atlanta Human Capital Development | 412 | 11.4 | 6.2 | 5.2* | 0.25 |
| Grand Rapids Labor Force Attachment | 285 | 9.6 | 12.9 | -3.2 | -0.11 |
| Grand Rapids Human Capital Development | 259 | 14.8 | 13.1 | 1.7 | 0.05 |
| Riverside Labor Force Attachment | 326 | 9.6 | 15.6 | -6.0 | -0.17 |
| Lacked high school diploma or basic skills | 207 | 7.5 | 9.8 | -2.3 | -0.09 |
| Riverside Human Capital Development | 274 | 9.4 | 10.9 | -1.5 | -0.06 |
In remedial math group (%) (teacher report) | |||||
| Atlanta Labor Force Attachment | 368 | 27.7 | 33.9 | -6.2 | -0.17 |
| Atlanta Human Capital Development | 411 | 31.1 | 32.5 | -1.4 | -0.04 |
| Grand Rapids Labor Force Attachment | 287 | 30.7 | 28.1 | 2.7 | 0.06 |
| Grand Rapids Human Capital Development | 262 | 24.7 | 28.4 | -3.7 | -0.09 |
| Riverside Labor Force Attachment | 326 | 27.5 | 27.5 | -0.0 | -0.00 |
| Lacked high school diploma or basic skills | 207 | 28.7 | 33.5 | -4.7 | -0.13 |
| Riverside Human Capital Development | 272 | 33.2 | 32.0 | 1.2 | 0.03 |
In remedial reading group (%) (teacher report) | |||||
| Atlanta Labor Force Attachment | 368 | 36.7 | 36.7 | -0.0 | -0.00 |
| Atlanta Human Capital Development | 413 | 35.6 | 36.8 | -1.2 | -0.03 |
| Grand Rapids Labor Force Attachment | 285 | 34.1 | 39.3 | -5.3 | -0.12 |
| Grand Rapids Human Capital Development | 262 | 41.8 | 40.6 | 1.2 | 0.03 |
| Riverside Labor Force Attachment | 326 | 44.5 | 34.8 | 9.7* | 0.23 |
| Lacked high school diploma or basic skills | 207 | 54.3 | 42.9 | 11.4 | 0.28 |
| Riverside Human Capital Development | 274 | 42.8 | 43.8 | -0.9 | -0.02 |
Goes to a special class or school, or gets special help in school, for a physical, emotional, or mental condition (%) (mother report) | |||||
| Atlanta Labor Force Attachment | 600 | 8.2 | 6.7 | 1.5 | 0.07 |
| Atlanta Human Capital Development | 678 | 5.9 | 6.7 | -0.8 | -0.04 |
| Grand Rapids Labor Force Attachment | 428 | 21.5 | 20.0 | 1.4 | 0.04 |
| Grand Rapids Human Capital Development | 410 | 23.9 | 19.7 | 4.2 | 0.11 |
| Riverside Labor Force Attachment | 533 | 14.5 | 12.2 | 2.3 | 0.08 |
| Lacked high school diploma or basic skills | 338 | 17.3 | 14.1 | 3.2 | 0.11 |
| Riverside Human Capital Development | 438 | 12.0 | 14.2 | -2.2 | -0.08 |
Identified as needing and receiving special services (%) (teacher report) | |||||
| Atlanta Labor Force Attachment | 377 | 33.5 | 42.9 | -9.4* | -0.24 |
| Atlanta Human Capital Development | 419 | 43.0 | 42.5 | 0.5 | 0.01 |
| Grand Rapids Labor Force Attachment | 288 | 44.4 | 45.5 | -1.1 | -0.02 |
| Grand Rapids Human Capital Development | 264 | 48.4 | 45.8 | 2.5 | 0.05 |
| Riverside Labor Force Attachment | 330 | 46.7 | 44.4 | 2.2 | 0.05 |
| Lacked high school diploma or basic skills | 209 | 52.3 | 54.1 | -1.8 | -0.04 |
| Riverside Human Capital Development | 279 | 50.1 | 54.9 | -4.8 | -0.12 |
Identified as needing and not receiving special services (%) (teacher report) | |||||
| Atlanta Labor Force Attachment | 377 | 40.7 | 38.2 | 2.6 | 0.07 |
| Atlanta Human Capital Development | 419 | 32.9 | 38.3 | -5.4 | -0.14 |
| Grand Rapids Labor Force Attachment | 288 | 33.0 | 29.3 | 3.7 | 0.09 |
| Grand Rapids Human Capital Development | 264 | 31.9 | 28.3 | 3.6 | 0.09 |
| Riverside Labor Force Attachment | 330 | 41.2 | 32.0 | 9.2 | 0.22 |
| Lacked high school diploma or basic skills | 209 | 43.7 | 34.2 | 9.5 | 0.24 |
| Riverside Human Capital Development | 279 | 29.1 | 33.2 | -4.1 | -0.11 |
Days absent during current school year (%) (teacher report) | |||||
| Atlanta Labor Force Attachment | 318 | 5.3 | 4.2 | 1.1 | 0.23 |
| Atlanta Human Capital Development | 359 | 5.9 | 3.8 | 2.1** | 0.43 |
| Grand Rapids Labor Force Attachment | 254 | 7.2 | 6.6 | 0.6 | 0.07 |
| Grand Rapids Human Capital Development | 243 | 6.1 | 6.8 | -0.7 | -0.09 |
| Riverside Labor Force Attachment | 303 | 6.2 | 5.6 | 0.6 | 0.09 |
| Lacked high school diploma or basic skills | 191 | 6.4 | 6.1 | 0.3 | 0.06 |
| Riverside Human Capital Development | 256 | 8.8 | 6.3 | 2.5** | 0.48 |
Days tardy during current school year (%) (teacher report) | |||||
| Atlanta Labor Force Attachment | 316 | 4.0 | 2.3 | 1.7* | 0.44 |
| Atlanta Human Capital Development | 360 | 2.7 | 2.2 | 0.5 | 0.13 |
| Grand Rapids Labor Force Attachment | 253 | 3.0 | 3.9 | -0.9 | -0.13 |
| Grand Rapids Human Capital Development | 242 | 3.9 | 3.9 | -0.0 | -0.00 |
| Riverside Labor Force Attachment | 301 | 4.0 | 3.4 | 0.6 | 0.09 |
| Lacked high school diploma or basic skills | 190 | 3.5 | 3.2 | 0.3 | 0.05 |
| Riverside Human Capital Development | 253 | 3.4 | 3.3 | 0.1 | 0.01 |
SOURCE: Child Trends calculations from the Five-Year Child Outcomes Study survey (mother, teacher, and child reports). | |||||
In general, the impacts on academic functioning were less numerous and their patterns less clear than those found within the social skills and behavior domain. Yet in five of the six programs there were impacts on measures relating to behavioral adjustment to school that, though few, were consistent with the impacts on social skills and behavior noted above. This suggests that impacts on more global ratings of children's social skills and behavior were accompanied by simultaneous changes on measures pertaining specifically to children's behavioral adjustment to school namely, disciplinary problems and engagement in school.
The pattern of impacts on disciplinary problems and school engagement generally follows that found in the social skills and behavior domain. Both Atlanta programs decreased the likelihood that focal children had a discipline problem requiring parental notification, and the Grand Rapids HCD program increased this likelihood. The Riverside LFA program also increased problems in the full sample, increasing the likelihood of having disciplinary action taken at least weekly with the focal child in the few months prior to the survey. These four programs generally altered the likelihood of discipline problems by between 9 and 16 percentage points, decreasing levels from about 56 to 42 percent in both Atlanta programs and increasing levels from 31 to 47 percent in the Grand Rapids HCD program and from 30 to 39 percent in the Riverside LFA program. In addition, following the pattern of social skills and behavior impacts, the Atlanta HCD program also increased teacher-reported levels of children's engagement in school, whereas both Grand Rapids programs decreased child-reported levels of school engagement. The effect sizes of these impacts ranged between .20 and .30. Hence, the general pattern of findings in the social skills and behavior domain was replicated and extended for measures of academic functioning relating to children's behavioral adjustment to school. The singular exception to this pattern is a decrease in suspensions and expulsions (as reported by mothers) in the Riverside LFA program.
Impacts in the other subdomains of academic functioning were less prevalent across the six programs. Impacts on standardized tests of academic achievement were rare, with only two impacts found across the six programs and across the six measures. There were no aggregate impacts on mean age-standardized achievement test scores, with mean scores for program and control group children across the six programs falling between 93 and 100 on the reading assessment (representing the 32nd and 50th percentiles, respectively) and between 95 and 101 on the math assessment (representing the 37th and 51st percentiles, respectively).(18), (19) There were also no impacts on the prevalence of "below-average" scores that is, mean age-standardized scores less than 90. The two impacts that were found were both favorable, increasing the prevalence of "above-average" scores (scores above 110) on the math assessment in the Grand Rapids LFA program and in the Riverside LFA program for the subgroup of children whose mothers had limited education at baseline.
Within the subdomain of academic progress and placement, impacts were also few, with no impacts on five of the eight measures. There were some impacts on measures of whether the focal child had repeated a grade (as reported by the mother), was in a remedial reading group, and performed above grade level in reading. The Riverside LFA program decreased the likelihood of having repeated a grade, dropping to about 4 percent from a control group level of about 10 percent for both the full sample and the subgroup of children whose mothers lacked a high school diploma or GED at baseline. The Riverside HCD program also decreased grade repetition by about 4 percentage points, though this difference was just above the cutoff for statistical significance. Yet the Riverside LFA program simultaneously increased the number of children who were in a remedial reading group by about 10 percentage points for both the full sample and those with limited education (from levels of about 35 and 43 percent, respectively), although the difference for the subgroup with limited education was just beyond the cutoff for statistical significance.(20) The Riverside LFA program also increased the percentage of children needing but not receiving services (an unfavorable finding), a difference just beyond the cutoff for statistical significance.
Finally, both Atlanta programs increased the proportion of children reported by their teachers as being above grade level in reading, increasing the levels from about 6 to 12 percent in both programs. In addition, the Atlanta LFA program increased the proportion of children needing and receiving services (an impact that may be favorable or unfavorable depending on whether it reflects more children needing services or more of those who need services receiving them).
Impacts on measures of attendance were also relatively scarce. Generally, control group children were absent for between 4 and 7 percent of days and were tardy between 2 and 4 percent of days in the current school year. Three of the programs (Atlanta LFA and HCD and Riverside HCD) increased absences, although the difference for the Atlanta LFA program was just beyond the cutoff for statistical significance.(21) The Atlanta LFA program also increased the percentage of days that children were tardy. All four of these differences were in the range of 1 to 3 percentage points. As speculated in the next section, the impacts may be related to unfavorable health impacts that were found in these three programs.
Health and Safety
Impacts on measures of children's health and safety are shown in Table 12.4. The outcomes in this domain are available only from mothers' reports.
| Site and Program | Sample Size | Program Group | Control Group | Difference (Impact) | Effect Size |
|---|---|---|---|---|---|
|
General health rating (range of 1 to 5) (mother report) |
|||||
| Atlanta Labor Force Attachment | 568 | 4.2 | 4.3 | -0.0 | -0.08 |
| Atlanta Human Capital Development | 641 | 4.3 | 4.3 | 0.1 | 0.09 |
| Grand Rapids Labor Force Attachment | 386 | 4.3 | 4.3 | -0.1 | -0.10 |
| Grand Rapids Human Capital Development | 365 | 4.3 | 4.3 | 0.0 | 0.07 |
| Riverside Labor Force Attachment | 473 | 4.2 | 4.3 | -0.1 | -0.11 |
| Lacked high school diploma or basic skills | 299 | 4.2 | 4.2 | 0.0 | 0.07 |
| Riverside Human Capital Development | 395 | 4.1 | 4.2 | -0.1 | -0.16 |
|
In very good or excellent health (%) (mother report) |
|||||
| Atlanta Labor Force Attachment | 568 | 81.8 | 85.2 | -3.4 | -0.12 |
| Atlanta Human Capital Development | 641 | 83.6 | 85.8 | -2.1 | -0.07 |
| Grand Rapids Labor Force Attachment | 386 | 83.4 | 90.4 | -7.0** | -0.26 |
| Grand Rapids Human Capital Development | 365 | 87.0 | 90.1 | -3.2 | -0.12 |
| Riverside Labor Force Attachment | 473 | 85.7 | 87.6 | -2.0 | -0.07 |
| Lacked high school diploma or basic skills | 299 | 85.1 | 82.7 | 2.5 | 0.08 |
| Riverside Human Capital Development | 395 | 75.6 | 83.0 | -7.4* | -0.24 |
|
Has a physical, emotional, or mental condition that requires frequent medical attention, use of medication, or special equipment (%) (mother report) |
|||||
| Atlanta Labor Force Attachment | 600 | 6.6 | 2.7 | 3.9** | 0.30 |
| Atlanta Human Capital Development | 678 | 6.9 | 2.5 | 4.4*** | 0.34 |
| Grand Rapids Labor Force Attachment | 428 | 13.6 | 12.8 | 0.8 | 0.02 |
| Grand Rapids Human Capital Development | 410 | 13.4 | 13.2 | 0.2 | 0.01 |
| Riverside Labor Force Attachment | 533 | 6.7 | 7.3 | -0.6 | -0.02 |
| Lacked high school diploma or basic skills | 338 | 6.3 | 6.5 | -0.2 | -0.01 |
| Riverside Human Capital Development | 438 | 6.5 | 6.8 | -0.4 | -0.02 |
|
Has a physical, emotional, or mental condition that impedes on mother's ability to go to work or school (%) (mother report) |
|||||
| Atlanta Labor Force Attachment | 600 | 4.5 | 1.2 | 3.3** | 0.34 |
| Atlanta Human Capital Development | 678 | 3.1 | 1.3 | 1.8 | 0.18 |
| Grand Rapids Labor Force Attachment | 428 | 6.4 | 8.1 | -1.7 | -0.07 |
| Grand Rapids Human Capital Development | 410 | 6.3 | 8.3 | -2.0 | -0.08 |
| Riverside Labor Force Attachment | 533 | 4.0 | 6.2 | -2.2 | -0.10 |
| Lacked high school diploma or basic skills | 338 | 3.1 | 5.0 | -1.9 | -0.10 |
| Riverside Human Capital Development | 438 | 5.3 | 5.2 | 0.1 | 0.01 |
|
Had an accident or injury requiring emergency medical attention, since the last interview (%) (mother report) |
|||||
| Atlanta Labor Force Attachment | 600 | 15.7 | 16.4 | -0.7 | -0.02 |
| Atlanta Human Capital Development | 678 | 15.3 | 15.9 | -0.6 | -0.02 |
| Grand Rapids Labor Force Attachment | 428 | 26.3 | 23.6 | 2.7 | 0.07 |
| Grand Rapids Human Capital Development | 409 | 22.4 | 24.2 | -1.8 | -0.04 |
| Riverside Labor Force Attachment | 532 | 23.2 | 25.9 | -2.7 | -0.07 |
| Lacked high school diploma or basic skills | 337 | 20.9 | 21.2 | -0.3 | -0.01 |
| Riverside Human Capital Development | 435 | 20.4 | 21.6 | -1.3 | -0.04 |
| SOURCE: Child Trends calculations from the Five-Year Child Outcomes Study survey (mother reports) NOTES: See Appendix A.2. |
|||||
The impacts in this domain of development were relatively few. There were no impacts on three of the six measures of health and safety, including mean levels of children's general health, as reported by mothers. Overall, the children in these sites seemed to be relatively healthy compared with children nationally. The proportion of control group children rated by their mothers as being in very good or excellent health ranged from 85 percent (in Atlanta) to 90 percent (in Grand Rapids). These levels are higher than the 1994 estimates from the National Health Interview Survey, in which 78.5 percent of all children aged 5 to 17 and 61.5 percent of those from families with annual incomes below $10,000 were rated by their parents as being in very good or excellent health.(22)
Yet where impacts were found (in four of the six programs) they were consistently unfavorable.(23) Two programs (Grand Rapids LFA and Riverside HCD) decreased the proportion of children reported as being in very good or excellent health, reducing these levels by about 7 percentage points. It is worth noting, however, that these unfavorable impacts resulted from program mothers being more likely than control group mothers to rate the focal child as being in good not fair or poor health. In addition, both Atlanta programs unfavorably affected two measures of health, increasing the proportion of children whose mothers reported that they had a physical, emotional, or mental condition that required frequent medical attention or the use of medication or special equipment and increasing the proportion with such a condition that impeded mothers' ability to go to work or school (although the difference for the HCD program on the latter measure was just beyond the cutoff for statistical significance). The impacts on conditions requiring frequent medical attention or the use of medication or special equipment were each about 4 percentage points. The impacts on conditions impeding the mothers' work or schooling were generally smaller, about 3 percentage points in the LFA program and about 2 percentage points in the HCD program. These health impacts may shed light on the unfavorable impacts on absenteeism and tardiness that were found in the Atlanta programs and in the Riverside HCD program. It is possible that the decreases in health status caused by these programs led to an increased likelihood of a child's being absent or tardy.
Other Outcome
There were no aggregate impacts on the likelihood that focal children lived away from their mothers because they could not care for them. Between 3 percent (in Atlanta) and 9 percent (in Riverside) of program group mothers reported separation from focal children for this reason, levels that did not differ significantly from those reported by the control groups in any site. (See Table 12.5.)
|
Site and Program |
Sample Size | Program Group (%) | Control Group (%) | Difference (Impact) | Effect Size |
|---|---|---|---|---|---|
|
Lived apart from mother, since last interview,because she could not care for child (mother report) |
|||||
| Atlanta Labor Force Attachment | 600 | 2.5 | 4.6 | -2.1 | -0.12 |
| Atlanta Human Capital Development | 678 | 3.1 | 4.9 | -1.9 | -0.11 |
| Grand Rapids Labor Force Attachment | 428 | 6.0 | 5.9 | 0.1 | 0.00 |
| Grand Rapids Human Capital Development | 410 | 4.6 | 7.0 | -2.4 | -0.11 |
| Riverside Labor Force Attachment | 533 | 8.6 | 6.4 | 2.2 | 0.10 |
| Lacked high school diploma or basic skills | 338 | 6.1 | 6.2 | -0.1 | -0.01 |
| Riverside Human Capital Development | 438 | 5.0 | 6.8 | -1.7 | -0.09 |
| SOURCE: Child Trends calculations from the Five-Year Child Outcomes Study survey (mother reports) NOTES: See Appendix A.2. |
|||||
"Mapping" Child and Adult Impacts Found in the Child Outcomes Study Sample
There were relatively few impacts on young children's developmental outcomes at the five-year point; possible reasons why more numerous impacts on children were not found are addressed in the next section. Nevertheless, there were more statistically significant impacts than would be expected by chance, indicating that welfare-to-work programs can (though modestly) alter children's developmental outcomes though in different ways across programs, sites, and developmental domains.
This section addresses the question: What may have led to the program impacts on children that were found at the five-year point? The sophisticated statistical analyses necessary to answer this question definitively are beyond the scope of the chapter. However, any such "pathways" analyses would need to begin with an examination of impacts on outcomes hypothesized to serve as pathways through which children were affected for example, employment, income, and child care and "map" these impacts onto the impacts found for children. If, for example, the programs producing predominantly favorable impacts on young children were the same programs that also increased mothers' employment, this may suggest that increasing employment is one way to improve child outcomes. If, in addition, the programs that did not produce favorable impacts on children also did not increase employment, then increasing employment would appear to be a necessary condition for improving child outcomes. To address this issue, the following section describes the patterns of key adult impacts that may explain the larger pattern of child impacts across programs.(24)
Educational Attainment
There is not a one-to-one relationship between impacts on mothers' educational attainment and impacts on young children across programs. Given the literature on the importance of mothers' education for children's developmental outcomes, one would expect increases in mothers' education to bode well for children.(25) On the one hand, this appears to be the case: Both the Atlanta and the Riverside HCD programs increased mothers' receipt of an education credential, and impacts of these programs on young children were favorable, whereas the Grand Rapids LFA program decreased receipt of such a credential, and impacts of this program on young children were unfavorable. On the other hand, increases in mothers' education do not appear necessary for improving child outcomes. The Atlanta LFA program did not increase mothers' educational attainment, nor did the Riverside LFA program for the subgroup of mothers lacking basic skills at baseline, yet these programs also improved child outcomes. Similarly, the Grand Rapids HCD program did not alter mothers' educational attainment but, like the LFA program in this site, it led to uniformly unfavorable impacts on children. Thus, impacts on mothers' education may, in part, underlie impacts on children in some programs, but not in other programs.
Employment and Earnings
The pattern of impacts on mothers' employment and earnings also does not appear to explain differences in child impacts across programs. For instance, while both Riverside programs produced relatively large gains in both earnings and employment (with the LFA program having these favorable effects both for the full sample and for the subgroup of mothers lacking a high school diploma at baseline), the pattern of child impacts varied according to mothers' education level, with few, but favorable, child impacts for children of mothers without a diploma and somewhat more numerous and unfavorable impacts for children of mothers in the full sample. Further, there were increases of a similar magnitude in earnings in both Atlanta programs and the Grand Rapids LFA program, yet the Atlanta programs generally improved child outcomes whereas the Grand Rapids programs worsened them. Hence, it is not likely that changes in employment and earnings can account for the different pattern of child impacts across programs.
Cumulative Income and Poverty Status
Some studies have shown that greater income is related to better child outcomes, though other studies have found that increased income is most beneficial when it lifts families out of poverty, since families with income below the poverty line are not likely to have sufficient resources to meet their basic needs.(26) However, there is no apparent relationship in the COS sample between impacts on total combined income across years 1 through 5, or on family poverty status in year 5, and impacts on young children. First of all, there were no impacts on total combined income in the COS sample, so income cannot be the driving force behind either the favorable or the unfavorable impacts on children. With respect to impacts on poverty status at the five-year point, the only program to significantly influence poverty levels, the Riverside HCD program, reduced poverty and improved functioning among focal children. However, the other programs had no significant influences on poverty, providing little support for the hypothesis that changes in poverty status in year 5 of the follow-up are the driving force in impacts on children. It is important to note, however, that this measure of poverty status does not capture program impacts on cumulative poverty over the five-year period. Yet previous research suggests that children's cumulative poverty experiences are more important for their developmental outcomes than is their poverty status in any given year.(27) Hence, although these findings suggest that poverty at the time of the follow-up is not likely to account for the pattern of impacts on children, we can draw few inferences about the role of cumulative poverty in accounting for these impacts.
Child Care
The pattern of impacts in the COS sample on measures of employment-related child care does not appear to explain differences in child impacts across programs. There were increases in the use of child care after leaving welfare (because of earnings) in both programs in Atlanta and Riverside, and in the LFA program in Grand Rapids; however, these programs did not have uniformly favorable or unfavorable impacts on children. Likewise, when impacts on the use of transitional child care benefits emerged in the COS sample, the programs increased the use of this benefit, yet impacts on young children within these programs were, again, both favorable and unfavorable.
Impacts on young children's supervision and child care during nonschool hours occurred almost exclusively in the Riverside site and, therefore, cannot explain the pattern of child impacts found across the programs.
Discussion and Conclusions
The major goal of the Child Outcomes Study was to determine whether mandated participation in a welfare-to-work program could have implications for enrollees' children particularly preschool-age children whose mothers were newly required by the Family Support Act of 1988 to engage in work preparation activities as a condition of receiving welfare. Some policymakers feared that requiring mothers of preschool-age children to secure employment would harm their children; others argued that the economic benefits of employment that would accrue to these families would benefit children. Findings from the COS indicate that there were relatively few long-term impacts of the JOBS programs evaluated in NEWWS on children's developmental outcomes. The lack of pervasive impacts on children is perhaps not surprising, given that the JOBS program was not aimed at children. It may be that impacts on outcomes important to children, such as mothers' employment, family income, and/or the children's immediate environments (home, school, child care), were too few, occurred for too brief a period, or were of an insufficient magnitude to lead to consistent, enduring impacts on young children. It is also possible that impacts at the five-year point are understated in Atlanta and Grand Rapids because some control group members in these sites became eligible for welfare-to-work program services in years 4 or 5 of the follow-up period.
However, the number of impacts found exceeds chance levels, and there was a discernible pattern of impacts on young children.(28) At the five-year follow-up, all six JOBS programs examined affected children's social skills and behavior. Both Atlanta programs increased positive behaviors, and decreased problem behaviors; both Grand Rapids programs decreased positive behaviors, and the HCD program increased problem behaviors; and the Riverside LFA program increased problem behaviors in the full sample.(29), (30) For young children whose mothers lacked a high school diploma or basic skills at study entry, both programs in Riverside increased positive behaviors. Changes in problem behaviors may be especially important; research has shown that a history of antisocial behavior in childhood which is partially measured by the problem behaviors examined in this study is the strongest risk factor for chronic delinquency among adolescents.(31)
Impacts on young children's academic achievement were also relatively rare, but findings relating to behavioral adjustment to school (school engagement, disciplinary problems) were in accord with the patterns described above, with uniformly favorable impacts in Atlanta and uniformly unfavorable impacts in Grand Rapids. Impacts relating to how children were performing academically in school were scarce; however, when impacts were found on these outcomes and others, they were at the lower end of the effect size range for early intervention programs aimed directly at children. These findings suggest that welfare-to-work programs can affect children's performance in school, albeit not as frequently nor as systematically as they appear to affect children's behavior.
Finally, there were also few impacts in the domain of health and safety; however, they were all unfavorable, occurred in four of the six programs,(32) and tended to occur in programs that also increased children's school absenteeism and/or tardiness. In many cases, these unfavorable health impacts were being driven by the unfavorable impacts for boys, with girls' health left largely unaffected by these programs.
A second goal of the COS was to determine whether program impacts on young children of enrollees differed according to the welfare-to-work strategy employed specifically, a Labor Force Attachment or a Human Capital Development strategy. Some believed that increases in mothers' educational attainment resulting from participation in an HCD program would bode particularly well for children particularly for their school success even if such participation did not lead immediately or ultimately to increased employment and earnings. Others argued that the quicker that mothers secured employment (through the LFA approach), the quicker that financial benefits would accrue to children. However, contrary to these initial hypotheses, the welfare-to-work strategy employed did not consistently produce different impacts on children at the five-year follow-up.(33) In direct comparisons of LFA and HCD programs, neither approach emerged as uniformly better or worse for children.
Did five-year impacts on children vary according to initial levels of family risk? The Family Support Act sought explicitly to reduce long-term welfare dependency by providing the services necessary to move long-term recipients into jobs. Many argued that the opportunities that JOBS provided would be more beneficial for the most disadvantaged participants (five-year impact findings for adults discussed in Chapter 7 support this hypothesis), thereby improving outcomes for children. Others feared that higher-risk participants might not be able to mobilize to meet JOBS requirements and would face sanctions, and they and their children would subsequently suffer. Contrary to these expectations, findings from the five-year follow-up do not show a discernible pattern of favorable or unfavorable impacts either across programs or within a single program for children of the "most disadvantaged" mothers.(34) Thus, children aged 8 to 10 in these higher-risk families do not appear to have been consistently helped or hurt by their mothers' enrollment in the LFA and HCD programs. Rather and somewhat unexpectedly there emerged a concentration of unfavorable impacts on children of the least disadvantaged mothers in the Riverside LFA program.(35) Interestingly, the Riverside LFA program unfavorably affected the same problem behavior and discipline outcomes for girls, which begs the question of whether these unfavorable impacts were concentrated among the least disadvantaged girls.(36)
In the Riverside LFA program, the unfavorable impacts on the academic performance, school engagement, and problem behaviors of focal children with the least disadvantaged mothers found at the five-year point are consistent with those found at the two-year point: increased academic problems, emotional problems, and suspensions or expulsions of children in lower-risk families. Other than these impacts, the JOBS programs examined in COS did not appear to have a similar pattern of short- and long-term impacts on children from families varying in initial risk. The only other pattern of findings to emerge by family risk level at the two-year follow-up(37), (38) a pattern of favorable (though relatively small) impacts on children in higher-risk families in all HCD programs and in the Atlanta LFA program did not emerge at the five-year point for children with the most disadvantaged mothers.
What led to the program impacts on children at the five-year point? It is worthwhile to note what did not lead to these impacts. Recent reviews of experimental findings for children indicate that, when welfare-to-work programs increase both maternal employment and family income, children often benefit.(39) However, because none of the six JOBS programs examined here affected the income of COS sample families, income cannot be a pathway through which children were affected. Also, use of child care does not appear to underlie program impacts on children.
There is some evidence from a nonstatistical "mapping" of adult and child impacts at the five-year point that maternal education may have been an important pathway through which some children were affected by their mothers' enrollment in an LFA or HCD program. Three of these programs affected both maternal education and child outcomes in the same direction; in the other three programs, though maternal education was not affected, children were. Thus, changes in mothers' education may not necessarily lead to changes in child outcomes, but when mothers' education is affected, it may have implications for their children. This finding is consistent with results showing that increased participation in educational activities by mothers in these three COS sites predicted greater academic school readiness in their children.(40)
In addition to examining the pattern of impacts across programs, it is important to consider impacts specific to a particular program for identifying possible pathways through which children were affected by that program. For example, the Riverside LFA program reduced time spent in child care activities after school and decreased the use of formal child care as a regular arrangement. Given some emerging evidence on the potential benefits of more formal child care arrangements on children,(41) the reduction in the use of formal care may be related to the pattern of unfavorable impacts of this program on these young children. In addition, research has shown a negative association between family residential moves and children's behavior.(42) This may help to explain the pattern of unfavorable impacts in Grand Rapids, given that both the LFA and the HCD programs in this site increased the likelihood that families had moved since study entry and also had unfavorable impacts on social skills and behavior among children aged 8 to 10. In short, because program impacts on children represent the net effect of all impacts on outcomes important to their development and well-being, and because impacts on these "intervening" outcomes may differ in different programs, future research needs to examine pathways for each program separately and needs to consider, simultaneously, all the intervening outcomes that were affected by each program.(43)
What is clear is that the sites in which these six JOBS programs were implemented varied geographically, economically, demographically (including the racial/ethnic composition of the caseloads), and in the policies, practices, and ethos of the local welfare offices.(44) Differences in program implementation may also help to explain the pattern of impacts. For example, though the Atlanta site successfully implemented distinct LFA and HCD models (in terms of the content and sequence of services), a large proportion of both HCD and LFA case managers in this site believed that an education-oriented welfare-to-work strategy was the best means of leaving welfare.(45), (46) In addition, case managers in the Atlanta site viewed their roles as very customer-oriented; they actively sought out necessary support services for their clients and were less strict about monitoring compliance. It may be that the policies, practices, and ethos of the Atlanta welfare office "fit" well with the needs of its female clients to balance both work and family responsibilities and that this context helps to explain the beneficial results for children in both the LFA and the HCD programs.
Although a national program, the JOBS program was implemented locally in sites that differed in economic conditions, the population served, and in the ethos and practices of the welfare offices. These conditions, though not included in the statistical analyses reported on here, shape the way the JOBS programs were implemented in each site and, thus, can shape the impacts that these programs have on targeted and nontargeted adult outcomes and on child outcomes in as yet unknown ways. Future research on the effectiveness of welfare-to-work programs for both generations may need to focus increasingly on identifying what works for whom, and under what circumstances. Such research would be informative to state and local policymakers and program planners who increasingly bear the responsibility, in this age of devolution, to design and effectively target welfare-to-work programs.
1. Although focal children were aged 8 and 10 at the five-year follow-up, they are characterized as "young" children because they were preschool-age at random assignment.
2. COS families consisted solely of single mothers with preschool-age children at random assignment; the larger samples included married couples and families not necessarily with a preschool-age child but with children of any age. COS mothers were somewhat younger, less likely to have been married, more likely to have a high school diploma or GED, and they had a slightly higher number of children on average than parents in the full evaluation sample. In addition, in Atlanta and Riverside, COS mothers were less likely to have ever worked full time for six months or more for the same employer at baseline. See Hamilton, 2000, p. 11.
3. A more detailed analysis of program impacts on boys and girls at both the two-year and the five-year follow-up is beyond the scope of this report and will be done as part of future analyses.
4. Gresham and Elliot, 1990, p. 1.
5. Woodcock and Johnson, 1989, 1990.
6. As noted in Chapter 2, the sample for the analyses of achievement test scores and of the mother- and child-reported child outcomes comprises 2,332 mothers and their focal children, and the sample for the analyses of teacher-reported child outcomes includes responses from 1,472 teachers of focal children in the final mother and child survey sample.
7. Achenbach, 1991.
8. Admittedly, even objective assessments of childrens' academic skills may contain some cultural biases.
9. Interestingly, the favorable impacts on behavior in the Atlanta programs were largely concentrated among girls, with girls showing fewer problem behaviors and more positive behaviors in both programs and, in the HCD program, fewer disciplinary problems as well.
10. There was a single finding that did not fit with this overall pattern of favorable impacts on social skills in the Atlanta programs: The LFA program decreased child-reported cooperation, though this difference was just beyond the cutoff for statistical significance (p = .11).
11. Although the difference for teacher-reported cooperation in the Grand Rapids HCD program was just beyond the cutoff for statistical significance, it was of a size comparable to similar impacts in other programs that were statistically significant.
12. The impacts on behavior in the Riverside LFA program occurred especially (or, in many cases, only) for the least disadvantaged subgroup and were part of a larger picture of unfavorable impacts for this subgroup that extended past behavior into teachers' reports of academic progress and placement. Further, the increases in problem behavior in this program were concentrated largely among girls and were accompanied by an increase in disciplinary problems among girls.
13. Though the latter two differences were above the cutoff for statistical significance, they were both of a size comparable to impacts in other programs that were statistically significant.
14. Berrueta-Clement et al., 1984; Campbell and Ramey, 1994.
15. Although relatively little is known about predictors and consequences of positive development in children and youth, there are some indications that positive development can lead to more successful future functioning. See Moore and Glei, 1995.
16. Gresham and Elliot, 1990, p. 1. See, for example, Coie and Dodge, 1983; Cowen et al., 1973; Parker and Asher, 1987.
17. Farrington, 1987, as cited in Yoshikawa, 1995.
18. Scores on that fall between 90 and 110, representing the 25th to the 75th percentiles, respectively, are classified as "average" or "normal" (Woodcock and Mather, 1989, 1990).
19. Nevertheless, there were impacts on mean math scores within gender subgroups in both Riverside programs. Both programs increased mean math scores for boys whose mothers lacked a high school diploma or GED at baseline. Additionally, the Riverside HCD program also increased boys' mean reading scores and, interestingly, decreased girls' mean math and mean reading scores.
20. While the impacts of the Riverside LFA program may seem inconsistent (simultaneously decreasing the number of children repeating a grade and increasing the number in a remedial reading group), these impacts occurred on experiences that apply to only a minority of the sample and thus may not pertain to the same children. In other words, the children who were in a remedial reading group may not be the same individuals who repeated a grade. Alternatively, it may be that teachers' evaluations of children's reading and math skills are highly dependent on children's classroom behavior, in which case the impact on being in a remedial reading group may reflect the behavioral impacts of this program more so than its impacts on measures of academic progress or placement. This speculation is supported by a set of findings for the least disadvantaged subgroup, in which there is a general pattern of unfavorable impacts on teachers' reports of behavior. At the same time, however, there is a single finding that appears to diverge from this pattern. Program group children of the least disadvantaged mothers are more likely to score above average on the reading test than controls yet are less likely to be rated as above grade level in reading by their teachers. This suggests that teachers' evaluations of children's performance in reading incorporate more than the children's actual reading skills (which, in fact, increased for program group children). For example, teachers perceiving less school engagement and more disciplinary problems in children (which was more true for the Grand Rapids HCD program group than for the control group) may have difficulty accurately gauging a child's reading skills if the child is having difficulty remaining seated or following instructions during reading class. Likewise, even among children with identical capabilities in reading, a teacher might be less likely to move a child into a higher-level reading group (and/or more likely to move him or her into a lower-level group) if the child tends to be disruptive to other students in the class.
21. The increase in absenteeism in the Atlanta LFA program occurred especially among the least disadvantaged subgroup, a finding that was part of a broader pattern of unfavorable impacts for this subgroup.
22. Adams and Marano, 1995. It may be surprising that mothers in the COS sample reported higher health ratings for their children aged 8 to 10 than were reported for both all children aged 5 to 17 and those from low-income families in a national sample. However, mothers were exempt from participation in JOBS welfare-to-work activities if they were needed at home to care for an ill or incapacitated family member, including a child. Consequently, the COS sample of children is relatively healthy, whereas national samples of children would include some severely and chronically ill children.
23. The pattern of unfavorable health impacts, including some new impacts in subgroups that were masked in the aggregate, was found to be largely concentrated among boys.
24. The adult economic impacts discussed in this section pertain to adults in the COS. See Appendix I for detailed tables of these impacts. The smaller sample sizes of the COS at times led differences that were of a similar magnitude to significant impacts in the larger evaluation samples to be nonsignificant in the COS sample. Hence, this discussion reports on differences in the COS that were either statistically significant or of a similar or greater size than a significant difference in the larger sample of adults within a given program.
25. See Duncan and Magnuson, 2001.
26. Duncan and Brooks-Gunn, 1997.
27. Duncan and Brooks-Gunn, 1997.
28. Impacts on young children two years after enrollment (when focal children were aged approximately 5 to 7) tended to vary by developmental domain, with only favorable aggregate impacts in the cognitive domain and only unfavorable impacts in the health and safety domain. Impacts on young children's behavior and socioemotional functioning were both favorable and unfavorable. See McGroder et al., 2000; and Zaslow, McGroder, and Moore, 2000.
29. These patterns of impacts for young children do not match findings discussed in Chapter 11, as the client survey sample respondents did not provide any information on social skills and behavior for all of their children. Only mothers in the COS provided this information for the focal children, as presented in this chapter, and it is precisely this domain of child outcomes most consistently affected by these programs.
30. As stated previously, the favorable impacts of the Atlanta programs on young children's positive and problem behaviors were especially pronounced for girls. At the same time, the unfavorable impacts of the Riverside LFA program on young children's problem behaviors were also especially pronounced for girls.
31. Farrington, 1987, as cited in Yoshikawa, 1995.
32. Interestingly, as reported in Chapter 11, favorable impacts were found for preschool-age children on whether any child had a condition that impeded on the mother's ability to go to work or school in the Riverside LFA program (for both the full sample and for those in need). Approximately two-thirds of the children in this age group were included in the COS sample presented in this chapter. As shown in Table 12.4, though, there were no favorable health impacts on this measure for the Riverside LFA program in the COS sample. However, the difference in the means between the program and control groups did go in the same direction. Perhaps the larger size of the client survey sample (and, thus, smaller standard errors) accounts for the difference in the presence of impacts.
33. Similarly, impacts on young children at the two-year follow-up did not vary consistently by welfare-to-work strategy employed. See Hamilton, 2000; McGroder et al., 2000; and Zaslow, McGroder, and Moore, 2000.
34. The most disadvantaged mothers did not have a high school diploma as of random assignment, did not work in the year prior to random assignment, and had been on welfare for at least two years as of random assignment. The least disadvantaged mothers had none of these barriers, and the moderately disadvantaged mothers had one or two of these barriers. See Chapter 7.
35. Although samples sizes for the least disadvantaged subgroup in these programs were relatively small, impacts for this subgroup are being reported because they were relatively numerous and were consistent in direction (they were mainly unfavorable). A similar pattern of unfavorable impacts was found for children in lower-risk families (defined by greater employment and low levels of welfare receipt) assigned to Florida's Family Transition Program. See Bloom et al., 2000a.
36. Small sample sizes precluded testing this hypothesis empirically.
37. See McGroder et al., 2000.
38. These comparisons rely on different child outcomes measured at the two-year and five-year follow-ups (with no child or teacher reports in the two-year follow-up) and on different (though similar) characterizations of subgroups.
39. Zaslow et al., forthcoming.
40. Magnuson and McGroder, 2001.
41. NICHD Early Child Care Research Network, 2000; see also U.S. Department of Education, 1995. As discussed in Chapter 10, the NICHD findings also suggest that extensive participation in formal child care arrangements may have the unfavorable result of increasing children's externalizing behavior problems.
42. Zaslow and Eldred, 1998.
43. Findings from the two-year COS suggest that young children may be affected through multiple pathways, that pathways can vary according to the child outcome considered, and that different programs may activate different pathways in affecting children. See McGroder et al., 2000.
44. Hamilton and Brock, 1994.
45. Hamilton et al., 1997.
46. This may be one reason that only the LFA program in Atlanta improved mothers' educational attainment by the two-year follow-up. See McGroder et al., 2000; Zaslow et al., 2000.
Costs and Benefits
Earlier chapters described the implementation of the NEWWS programs and the effects on sample members and their families. In sum, NEWWS programs increased participation in employment-related activities and produced some positive effects for program group members, including higher earnings and less reliance on welfare. Overall, the NEWWS programs had little impactВ positive or negativeВ on the well-being of the children of program group members. This chapter presents an analysis of the costs of providing the array of services that produced these effects and uses the results of the cost analysis to examine the net financial benefits and costs of the NEWWS programs from two perspectives: program group members and government budgets. The benefit-cost analysis includes key financial effects discussed in earlier chapters, such as effects on earnings, cash assistance payments, and Food Stamp payments, and expands the scope to consider effects such as fringe benefits from employment, taxes, and Medicaid coverage.
This chapter extends the analysis of impacts on use of program services (Chapter 3) and impacts on earnings, welfare and Food Stamps, and combined income (Chapters 4-6). It examines the costs and benefits of 10 of the 11 NEWWS programs, those for which five years of follow-up data were available from administrative records. (Owing to data limitations five-year costs and benefits were not estimated for Oklahoma City.)
The chapter presents details on the analysis of the costs of running the NEWWS programs, discusses the financial benefits of NEWWS, and compares the benefits and costs from the two perspectives mentioned above.
Key Findings
- Five-year gross costs for employment-focused programs were similar to those for education-focused programs in the 10 NEWWS programs examined. Although the Grand Rapids LFA and Portland programs were employment-focused, they produced participation rates in post-secondary and vocational training that were similar to or greater than those found in the education-focused programs. The average cost was $7,749 for the employment-focused programs and $7,499 for the education-focused programs.
The costs presented in this chapter consist of all costs associated with providing employment services and related support services to sample members. The gross cost per program group member consists of costs paid by the welfare department and non-welfare agencies while sample members were enrolled in NEWWS programs, as well as for employment and support services after they exited the programs and left the welfare rolls.
- The net costs of the NEWWS programs, over and above the cost of services used by control group members, were higher for education-focused programs ($3,972) than for employment-focused ($3,037).
The net cost per program group member is the gross cost per program group member minus what would have been spent in the absence of the NEWWS program the gross cost per control group member. As discussed in Chapter 3, a sizable number of controls participated on their own in community-provided employment-related activities, usually vocational training or post-secondary education.
- Gross and net costs were higher for the NEWWS programs than for other programs studied by MDRC.
This is not surprising, given the high rates of participation in higher-cost education activities, such as post-secondary education and vocational training. The average cost of the NEWWS programs was comparable to other high-cost programs: the Alameda and Los Angeles Greater Avenues for Independence (GAIN) programs operated in the late 1980s and early 1990s.
- As was the case at the two-year follow-up point, the net costs of the HCD programs continued to be higher than those of the LFA programs by about 40 to 90 percent.
This comparison is based on the three sites where head-to-head LFA-HCD tests were conducted: Atlanta, Grand Rapids, and Riverside. Because the Riverside HCD sample includes only those who did not have a high school diploma or GED when they entered the evaluation, the LFA-HCD comparison in this site is limited to this subgroup.
- Generally, programs spent more on serving those who entered the evaluation with a high school diploma or GED than those who entered without these credentials.
In all programs, gross costs per program group member were higher for the subgroup that entered the evaluation with a high school diploma or GED. As expected, they were more likely to participate in post-secondary education or vocational training activities, whereas those lacking these credentials were more likely to participate in basic education activities, which typically cost less. The differences in net costs between the subgroups were less pronounced, but still higher for those with a high school diploma or GED.
From the perspective of the welfare recipients subject to the programs, the benefit-cost findings show that:
- The three programs that produced the smallest welfare reductions (under $1,000 for both Atlanta programs and the Detroit program) resulted in net gains to sample members, albeit very small ones; sample members effectively broke even in these programs. All programs with larger five-year welfare reductions (ranging from $1,148 in the Columbus Traditional program to $2,868 in the Riverside LFA program) produced net five-year losses for sample members.
- A similar pattern was found for the subgroup of sample members who entered the evaluation with a high school diploma or GED, although the Columbus programs, with similarly small welfare reductions, produced net losses. Those who entered without these education credentials generally suffered net losses. Detroit and Portland were the exceptions: Those without a high school diploma or GED experienced a moderate gain in Portland, and those in Detroit effectively broke even.
From the perspective of government budgets, the findings show that:
- For the full samples, only three programs, all employment-focused, produced gains to government budgets (Grand Rapids LFA, Riverside LFA, and Portland). All programs that produced the smallest welfare reductions, making it difficult for government budgets to break even, resulted in losses. Grand Rapids HCD and Columbus Integrated, with midrange welfare savings, effectively broke even.
- In terms of the return to budget per net dollar invested, on average the results from the government perspective are more positive than those found in the GAIN evaluation. The NEWWS programs averaged a return of $1.29; the GAIN programs averaged $0.76 per dollar invested. Moreover, the two programs with the highest returns, Portland ($2.83) and Grand Rapids LFA ($2.46), compare favorably with the Riverside GAIN program and the Saturation Work Initiative Model (SWIM) program, which returned $2.84 and $2.34, respectively.
- With the exception of the two programs in Atlanta, government budgets recouped their investments in sample members who entered the evaluation without a high school diploma or GED. Among sample members who entered the evaluation with a high school diploma or GED, three of the four employment-focused programs produced gains to government budgets (Atlanta LFA broke even) and all of the education-focused programs produced losses.
Issues in the Cost Analysis
The primary purpose of the cost analysis is to estimate the cost of providing NEWWS services, over and above the cost that would have been incurred in the absence of the program that is, to estimate the average net cost per program group member. The net cost is the difference between the gross cost per program group member and the gross cost per control group member, where the gross costs reflect the cost of all services that sample members used in the NEWWS programs and of the education and training services that they used outside the programs, when they were no longer receiving welfare benefits. In other words, the cost for the control group is the benchmark used to determine the additional costs incurred as a result of the NEWWS programs.
This report updates cost figures based on two years of follow-up data presented in earlier reports.(1) (As noted above, data limitations did not allow inclusion of Oklahoma City in the five-year analysis.) Table 13.1 shows that, at two years, education-focused programs were more costly to operate than employment-focused programs. The average net cost was $1,846 (in 1999 dollars) for the employment-focused programs and $2,523 for the education-focused programs. Similarly, the net costs were 1.5 to 2.6 times greater for the HCD programs than for the LFA programs. The earlier reports presented only program costs because a full cost-benefit accounting with only two years of data was considered premature, in that the total return on program investments would be evident only after several years.
|
Program and Component |
Total Gross Cost per Program Group Member ($) | Total Gross Cost per Control Group Member ($) | Total Net Cost per Program Group Member ($) |
|---|---|---|---|
|
Atlanta Labor Force Attachment |
|||
| Orientation and appraisal | 72 | n/a | 72 |
| Formal assessment | n/a | n/a | n/a |
| Job search | 816 | 8 | 807 |
| Basic education | 507 | 58 | 449 |
| Post-secondary education | 289 | 315 | -26 |
| Vocational training | 751 | 443 | 309 |
| Work experience | 174 | 20 | 152 |
| Subtotal (operating) | 2,609 | 843 | 1,765 |
| Child care | 883 | 291 | 591 |
| Other support services | 193 | 17 | 176 |
| Total | 3,685 | 1,152 | 2,533 |
|
Atlanta Human Capital Development |
|||
| Orientation and appraisal | 72 | n/a | 72 |
| Formal assessment | n/a | n/a | n/a |
| Job search | 194 | 8 | 185 |
| Basic education | 1,263 | 58 | 1,205 |
| Post-secondary education | 300 | 315 | -14 |
| Vocational training | 1,792 | 443 | 1,350 |
| Work experience | 126 | 20 | 105 |
| Subtotal (operating) | 3,746 | 843 | 2,903 |
| Child care | 806 | 291 | 515 |
| Other support services | 413 | 17 | 397 |
| Total | 4,966 | 1,152 | 3,814 |
|
Grand Rapids Labor Force Attachment |
|||
| Orientation and appraisal | 18 | n/a | 18 |
| Formal assessment | 9 | n/a | 9 |
| Job search | 823 | 78 | 745 |
| Basic education | 819 | 793 | 27 |
| Post-secondary education | 1,944 | 1,715 | 229 |
| Vocational training | 730 | 841 | -111 |
| Work experience | 122 | 12 | 110 |
| Subtotal (operating) | 4,465 | 3,438 | 1,026 |
| Child care | 408 | 230 | 177 |
| Other support services | 30 | n/a | 30 |
| Total | 4,903 | 3,669 | 1,233 |
|
Grand Rapids Human Capital Development |
|||
| Orientation and appraisal | 18 | n/a | 18 |
| Formal assessment | 312 | n/a | 312 |
| Job search | 289 | 78 | 210 |
| Basic education | 2,064 | 793 | 1,272 |
| Post-secondary education | 2,123 | 1,715 | 408 |
| Vocational training | 1,205 | 841 | 364 |
| Work experience | 213 | 12 | 201 |
| Subtotal (operating) | 6,224 | 3,438 | 2,786 |
| Child care | 603 | 230 | 372 |
| Other support services | 37 | n/a | 37 |
| Total | 6,865 | 3,669 | 3,195 |
|
Riverside Labor Force Attachment |
|||
| Orientation and appraisal | 111 | n/a | 111 |
| Formal assessment | 6 | n/a | 6 |
| Job search | 877 | 45 | 832 |
| Basic education | 172 | 106 | 67 |
| Post-secondary education | 707 | 421 | 286 |
| Vocational training | 237 | 243 | -6 |
| Work experience | 56 | 65 | -10 |
| Subtotal (operating) | 2,164 | 878 | 1,286 |
| Child care | 98 | 33 | 66 |
| Other support services | 54 | n/a | 54 |
| Total | 2,316 | 911 | 1,405 |
|
Riverside Human Capital Development(without a high school diploma or GED) |
|||
| Orientation and appraisal | 107 | n/a | 107 |
| Formal assessment | 13 | n/a | 13 |
| Job search | 729 | 43 | 685 |
| Basic education | 2,340 | 181 | 2,158 |
| Post-secondary education | 186 | 150 | 36 |
| Vocational training | 236 | 258 | -22 |
| Work experience | 63 | 30 | 33 |
| Subtotal (operating) | 3,674 | 662 | 3,012 |
| Child care | 183 | 16 | 166 |
| Other support services | 82 | n/a | 82 |
| Total | 3,938 | 678 | 3,260 |
|
Columbus Integrated |
|||
| Orientation and appraisal | 17 | n/a | 17 |
| Formal assessment | n/a | n/a | n/a |
| Job search | 79 | 7 | 72 |
| Basic education | 516 | 89 | 427 |
| Post-secondary education | 1,163 | 270 | 892 |
| Vocational training | 693 | 226 | 467 |
| Work experience | 82 | 7 | 76 |
| Subtotal (operating) | 2,550 | 599 | 1,952 |
| Child care | 569 | 346 | 221 |
| Other support services | 221 | 11 | 211 |
| Total | 3,340 | 956 | 2,384 |
|
Columbus Traditional |
|||
| Orientation and appraisal | 9 | n/a | 9 |
| Formal assessment | n/a | n/a | n/a |
| Job search | 100 | 7 | 93 |
| Basic education | 632 | 89 | 543 |
| Post-secondary education | 1,046 | 270 | 775 |
| Vocational training | 313 | 226 | 87 |
| Work experience | 63 | 7 | 57 |
| Subtotal (operating) | 2,163 | 599 | 1,564 |
| Child care | 552 | 346 | 205 |
| Other support services | 148 | 11 | 138 |
| Total | 2,863 | 956 | 1,907 |
|
Detroit |
|||
| Job search | 265 | 47 | 218 |
| Education and training a | 3,541 | 1,890 | 1,651 |
| Work experience | 78 | 10 | 68 |
| Subtotal (operating) | 3,884 | 1,947 | 1,938 |
| Child care | 407 | 337 | 71 |
| Other support services | 61 | 17 | 45 |
| Total | 4,354 | 2,300 | 2,053 |
|
Oklahoma City |
|||
| Orientation and appraisal | n/a | n/a | n/a |
| Formal assessment | n/a | n/a | n/a |
| Job search | 127 | 46 | 80 |
| Basic education | 553 | 276 | 277 |
| Post-secondary education | 814 | 738 | 77 |
| Vocational training | 1,048 | 625 | 423 |
| Work experience | 34 | 14 | 20 |
| Subtotal (operating) | 2,576 | 1,698 | 878 |
| Child care | 593 | 506 | 87 |
| Other support services | 87 | n/a | 87 |
| Total | 3,255 | 2,204 | 1,051 |
|
Portland |
|||
| Orientation and appraisal | 149 | n/a | 149 |
| Formal assessment | n/a | n/a | n/a |
| Job search | 468 | 47 | 422 |
| Basic education | 543 | 337 | 206 |
| Vocational training and post-secondary education | 1,358 | 1,159 | 199 |
| Work experience | 247 | 37 | 210 |
| Subtotal (operating) | 2,766 | 1,580 | 1,186 |
| Child care | 1,582 | 629 | 953 |
| Other support services | 79 | 6 | 73 |
| Total | 4,427 | 2,215 | 2,212 |
| SOURCES: MDRC calculations based on fiscal and participation data from the following: Atlanta - the Fulton County Department of Family and Children Services, the Georgia Department of Human Resources, the Georgia Department of Technical and Adult Education, the Board of Regents University System of Georgia; Grand Rapids - the Michigan Department of Social Services, the Michigan Department of Education Office of Extended Learning Services, the Grand Rapids Community College, the Wyoming Community Education Center; Riverside - the California Department of Social Services, the California Department of Education, the Chancellors Office of California Community Colleges; Columbus - the Franklin County Department of Human Services, the Ohio Department of Education, the Office of Vocational and Adult Education, the Ohio Board of Regents, the National Center for Education Statistics; Detroit - the Michigan Family Independence Agency, the Michigan Department of Education, the Michigan Jobs Commission; Portland - the Oregon Department of Human Resources, Adult and Family Services Division, the Oregon Office of Community College Services; in all sites - information collected on tuition charged at proprietary schools attended by sample members, from MDRC-collected JOBS case file data, the MDRC Two-Year Client Survey, and the MDRC Five-Year Client Survey. MDRC child care and other support service calculations from Fulton County, Kent County, Riverside County, Ohio Department of Human Services, Wayne County, Washington County, and Multnomah County(District 2) payment data, other support service data from country records.
NOTES: Rounding may cause slight discrepancies in calculating sums and differences. |
|||
As described in the earlier reports, costs per sample member are the product of unit costs and behavioral variables. The unit cost of an activity is an estimate of the average cost of serving one person in a specified activity for a specified unit of time (one month or one hour, for example). In general, unit costs were calculated by dividing expenditures for an activity (or service) during a steady-state period by the total number of participant-months in that activity during the same period. The number of participant-months was obtained by counting the number of participants in an activity in each month of the steady-state period and summing across the months.(2) Once the unit cost of an activity was determined, it was multiplied by the average number of months that sample members spent in the activity, called the behavioral variable, to determine the average cost incurred per program group member or control group member during the follow-up period.
The costs presented here were calculated using the same unit costs calculated at the two-year point. However, the behavioral variables used in this analysis cover the five-year period following each sample member's entry into the study. Behavioral variables for years 3 to 5 of follow-up were estimated using Two-Year and Five-Year Client Survey data, as well as administrative records data. The Five-Year Client Survey was not administered in Columbus and Detroit. Therefore, in Columbus, behavioral variables were estimated using participation trends from the two-year and five-year follow-up points in the education-focused sites where longer-term data were available. Because Detroit shifted program focus from education to employment mid-follow-up, it was considered inappropriate to estimate participation for this program using data from the other education-focused programs. Three years of participation data were available from the management information system maintained by the Detroit Work First program. Participation in years 4 and 5 of the follow-up period was estimated based on patterns of participation and welfare receipt in the earlier period.
As discussed in Chapter 2, in several sites service embargoes for control group members were lifted during the follow-up period. As a result, in those sites some control group members were required to participate in employment-related activities in follow-up year 4 or 5. Welfare receipt data were used to determine whether participation should be counted as in-program or out-of-program for both program group members and control group members for whom embargoes were lifted. In addition to the cost of providing the activity, in-program participation incurs an additional cost for case management.
As noted above, costs are estimated for the five-year period following sample members' entrance into the study. Later in the chapter, to assess whether the programs in the NEWWS Evaluation were cost-effective from the perspective of the government's budget, this five-year net cost is compared with the value of any budgetary savings during the same period (for example, from lower welfare or Food Stamp payments) and of any tax revenue increases associated with the additional earnings of program group members.
The costs presented here include the costs of program services as well as the costs of employment-related services that sample members used outside the programs when they were not receiving welfare. The off-welfare costs are important because they represent an additional investment of resources that could have differentially affected program and control group members' future earnings and welfare receipt (effects that are accounted for in the benefit-cost analysis).
All sample members, not just those who participated in mandatory welfare-to-work program services, were included in calculating the net costs because the requirement to participate may have affected some recipients' behavior: Some people may have chosen to avoid the participation mandate by finding a job on their own or by leaving the welfare rolls. In addition, sample members who did not participate in welfare-to-work program services may have taken part in education and training services on their own, and these costs need to be taken into account as well.(3)
Owing to the fact that findings might be expected to differ according to sample members' educational background, all results are presented for the full samples for each program and then separately for those with and without a high school diploma or GED at the time they entered the evaluation. As noted in earlier chapters, all HCDs in Riverside had no high school diploma or GED. Therefore, the Riverside LFA-HCD comparisons include only those without a high school diploma or GED.
Major Components of the Cost Analysis
The costs presented in this chapter consist of all costs associated with providing employment services and related support services to sample members. Figure 13.1 illustrates the cost components. For each group of sample members, costs were calculated for employment-related services that sample members participated in when they were receiving welfare (in-program) and services that they participated in when they were not receiving welfare (out-of-program). The employment-related services are divided into those that were paid for by the welfare department, either directly or indirectly, and those that were paid for by other agencies in the community; this information may be useful to administrators and planners who want to understand the nature of the government's investment.
Figure 13.1
Major Components of Gross and Net Costs for Employment-Related Services
The remainder of the cost portion of the chapter follows the organization of Figure 13.1, beginning with in-program expenditures paid by the welfare departments and ending with the net cost per program group member, which is the sum of the program costs less the costs of services used by control group members.
Expenditures for In-Program Employment-Related Services
Across the programs, costs varied widely for employment-related services for sample members when they were receiving welfare from less than $4,000 in the Riverside LFA and Detroit programs to more than $7,000 in the Grand Rapids and Atlanta HCD programs. The welfare department paid some of these costs, with the remainder picked up by various other agencies in the community. The proportion that was paid for out of welfare department funds varied across programs, from 27 percent in the Grand Rapids LFA program to 63 percent in the Portland program. This section examines these expenditures in more detail. (See Figure 13.1, Boxes 1 and 2.)
Welfare Department Expenditures
Welfare department costs consisted of program operating costs and the costs of support services that sample members received to enable their participation in employment and employment-related activities.
1. Operating costs. The welfare departments paid for the day-to-day operation of the NEWWS programs, including expenditures for employment-related case management services, overhead, program orientation, and other activities. These expenditures cover services provided directly by welfare department staff as well as services provided by other agencies under contract to the welfare departments. Welfare department staff directly provided case management (following up on recipients who failed to attend scheduled appointments, providing employability planning, and referring and monitoring individuals assigned to activities). In some sites, assessments, job search assistance, and job development services were provided by welfare department staff, but in others these activities were provided by contracted outside agencies. Costs incurred by the welfare department to accommodate NEWWS research requirements and requests were excluded from the analysis.
Table 13.2 shows the unit costs (in 1999 dollars) that is, estimates of the average cost of providing specified services to one person for a month used in the cost analysis. Differences in unit measures make it difficult to compare all unit costs across all programs, but it is clear from those with common units that costs varied widely across programs. These differences reflect a number of factors: differences in local markets, the types of employment-related services that participants selected or were steered toward, and the decisions that administrators made in implementing their programs. For example, some programs referred sample members to community providers for basic education services, but did not provide any additional funding to these generally publicly funded programs. However, the welfare departments in Riverside, Detroit, and Portland paid for services provided to sample members through formal contracts with education providers. As a result, welfare department unit costs for basic education (column 1) are higher in these three sites than in the others.(4) Non-welfare agency unit costs for basic education (column 3) were much higher for the Portland program than for the other programs, because community colleges were the largest provider of basic education services in Portland, whereas public adult education systems were the main providers of basic education in the other sites.
| Program and Activity | Program Group | Control Group | |||
|---|---|---|---|---|---|
| Welfare Department Unit Cost | Non-Welfare Agency Unit Cost | Non-Welfare Agency Unit Cost | |||
| Average per Month of Participation ($) | Average per Session ($) | Average per Hour ($) | Average per Month of Participation ($) | Average per Hour ($) | |
Atlanta Labor Force Attachment | |||||
| Orientation and appraisal | n/a | 72 | n/a | n/a | n/a |
| Formal assessment | n/a | n/a | n/a | n/a | n/a |
| Job search | 416 | n/a | n/a | 61 | n/a |
| Basic education | 115 | n/a | 3.18 | n/a | 3.20 |
| Post-secondary education | 73 | n/a | 9.03 | n/a | 8.44 |
| Vocational training | 153 | n/a | 7.31 | n/a | 6.88 |
| Work experience | 167 | n/a | n/a | 185 | n/a |
Atlanta Human Capital Development | |||||
| Orientation and appraisal | n/a | 72 | n/a | n/a | n/a |
| Formal assessment | n/a | n/a | n/a | n/a | n/a |
| Job search | 463 | n/a | n/a | 60.80 | n/a |
| Basic education | 99 | n/a | 3.09 | n/a | 3.20 |
| Post-secondary education | 94 | n/a | 8.07 | n/a | 8.44 |
| Vocational training | 140 | n/a | 7.25 | n/a | 6.88 |
| Work experience | 203 | n/a | n/a | 185 | n/a |
Grand Rapids Labor Force Attachment | |||||
| Orientation and appraisal | n/a | 18 | n/a | n/a | n/a |
| Formal assessment | 395 | n/a | n/a | n/a | n/a |
| Job search a | 259 | n/a | n/a | n/a | n/a |
| Basic education | 132 | n/a | 6.37 | n/a | 6.38 |
| Post-secondary education | 97 | n/a | 9.31 | n/a | 9.37 |
| Vocational training | 110 | n/a | 7.82 | n/a | 7.97 |
| Work experience | 241 | n/a | n/a | 241 | n/a |
Grand Rapids Human Capital Development | |||||
| Orientation and appraisal | n/a | 18 | n/a | n/a | n/a |
| Formal assessment | 395 | n/a | n/a | n/a | n/a |
| Job searcha | 259 | n/a | n/a | n/a | n/a |
| Basic education | 132 | n/a | 6.31 | n/a | 6.38 |
| Post-secondary education | 97 | n/a | 9.15 | n/a | 9.37 |
| Vocational training | 110 | n/a | 7.23 | n/a | 7.97 |
| Work experience | 241 | n/a | n/a | 241 | n/a |
Riverside Labor Force Attachment | |||||
| Orientation and appraisal | n/a | 88 | n/a | n/a | n/a |
| Formal assessment | n/a | 596 | n/a | n/a | n/a |
| Job search | 759 | n/a | n/a | 254 | n/a |
| Basic education | 255 | n/a | 4.45 | n/a | 4.09 |
| Post-secondary education | 122 | n/a | 6.37 | n/a | 6.39 |
| Vocational training | 122 | n/a | 5.52 | n/a | 5.90 |
| Work experience | 572 | n/a | n/a | 572 | n/a |
Riverside Human Capital Development(without a high school diploma or GED) | |||||
| Orientation and appraisal | n/a | 88 | n/a | n/a | n/a |
| Formal assessment | n/a | 596 | n/a | n/a | n/a |
| Job search | 759 | n/a | n/a | 254 | n/a |
| Basic education | 255 | n/a | 4.05 | n/a | 4.09 |
| Post-secondary education | 122 | n/a | 6.29 | n/a | 6.39 |
| Vocational training | 122 | n/a | 5.31 | n/a | 5.90 |
| Work experience | 572 | n/a | n/a | 572 | n/a |
Columbus Integrated | |||||
| Orientation and appraisal | 19 | n/a | n/a | n/a | n/a |
| Formal assessment | n/a | n/a | n/a | n/a | n/a |
| Job search | 220 | n/a | n/a | 100 | n/a |
| Basic education | 86 | n/a | 3.82 | n/a | n/a |
| Post-secondary education | 225 | n/a | 6.16 | n/a | n/a |
| Vocational trainingb | 225 | n/a | n/a | n/a | n/a |
| Work experience | 106 | n/a | n/a | 76 | n/a |
| Columbus Traditional | |||||
| Orientation and appraisal | 14 | n/a | n/a | n/a | n/a |
| Formal assessment | n/a | n/a | n/a | n/a | n/a |
| Job search | 205 | n/a | n/a | 100 | n/a |
| Basic education | 54 | n/a | 3.99 | n/a | n/a |
| Post-secondary education | 55 | n/a | 6.16 | n/a | n/a |
| Vocational trainingb | 55 | n/a | n/a | n/a | n/a |
| Work experience | 46 | n/a | n/a | 76 | n/a |
Detroitc | |||||
| Orientation and appraisal | n/a | n/a | n/a | n/a | n/a |
| Formal assessment | n/a | n/a | n/a | n/a | n/a |
| Job searchd | 752 | n/a | n/a | 597 | n/a |
| Education and traininge | 587 | n/a | 8.44 | n/a | 8.21 |
| Work experience | 557 | n/a | n/a | n/a | n/a |
Portland | |||||
| Orientation and appraisal | 138 | n/a | n/a | n/a | n/a |
| Formal assessment | n/a | n/a | n/a | n/a | n/a |
| Job search | 359 | n/a | n/a | 280 | n/a |
| Basic education | 257 | n/a | 11.75 | n/a | 11.75 |
| Post-secondary education | 131 | n/a | 7.82 | n/a | 8.01 |
| Vocational training | 131 | n/a | 7.82 | n/a | 8.01 |
| Work experience | 400 | n/a | n/a | 322 | n/a |
| SOURCES: MDRC calculations based on fiscal and participation data from the following: Atlanta - the Fulton County Department of Family and Children Services, the Georgia Department of Human Resources, the Georgia Department of Technical and Adult Education, the Board of Regents University System of Georgia; Grand Rapids - the Michigan Department of Social Services, the Michigan Department of Education Office of Extended Learning Services, the Grand Rapids Community College, the Wyoming Community Education Center; Riverside - the California Department of Social Services, the California Department of Education, the Chancellors Office of California Community Colleges; Columbus - the Franklin County Department of Human Services, the Ohio Department of Education, the Office of Vocational and Adult Education, the Ohio Board of Regents, the National Center for Education Statistics; Detroit - the Michigan Family Independence Agency, the Michigan Department of Education, the Michigan Jobs Commission; Portland - the Oregon Department of Human Resources, Adult and Family Services Division, the Oregon Office of Community College Services; in all sites - information collected on tuition charged at proprietary schools attended by sample members, from MDRC-collected JOBS case file data, the MDRC Two-Year Client Survey, and the MDRC Five-Year Client Survey. NOTES: Welfare department unit costs include the cost of providing activities (e.g., classroom instruction, job search facilitation, space rental, and case management expenditures). Work experience unit costs for program and control group members receiving services from non-welfare agencies were assumed to be equal to the welfare department JOBS unit cost. Atlantas unit cost for controls was the average of the LFA and HCD welfare department unit costs. N/a = not applicable. a The estimated unit cost of job search to Grand Rapids non-welfare agencies was $1,025 per participant. b The estimated unit cost of vocational training for non-welfare agencies in Columbus was $5,032 per participant in the Integrated program and $2,771 per participant in the Traditional program. c For the Detroit program, welfare department unit costs were calculated by dividing total activity expenditures by the total number of participant-months for the corresponding activity. At the time of the two-year follow-up, Farrell, 2000, presented a range of unit costs, based on two different methods of calculating participant-months. However, both the two- and five-year gross cost estimates were based only on the upper-bound unit cost estimates, and so only "upper-bound" estimates of unit costs are presented on this table. See Farrell, 2000, for a complete discussion of this issue and the full range of two-year cost and participation estimates for this site. d The estimated unit cost of job search to Detroit non-welfare agencies was $816 per month of participation. In addition, the estimated unit cost to the welfare department for control group participation in job search, education and training, and work experience activities was $271 per month of participation. e Owing to data limitations, in Detroit it was not possible to separate vocational training, post-secondary education, and basic education costs. Thus, the gross costs of all three activities are included in the cost of "education and training." Furthermore, in this site orientation and assessment costs are spread across these three activities. | |||||
The unit cost multiplied by sample members' average number of months of participation in the activity (the behavioral variable) yields the cost per sample member. These costs are shown in Table 13.3 (column 1). In-program welfare department operating costs ranged from $603 (Columbus Traditional) to $2,753 (Riverside HCD) and were not systematically different between employment- and education-focused programs.
| Activity or Service | In-Program Cost | Out-of-Program Cost | Total Gross Cost per Program Group | |||
|---|---|---|---|---|---|---|
| Welfare Department Cost ($) | Non-Welfare Agency Cost ($) | Total Program Cost ($) | Welfare Department Cost ($) | Non-Welfare Agency Cost ($) | ||
Atlanta Labor Force Attachment | ||||||
| Orientation and appraisal | 72 | 0 | 72 | 0 | 0 | 72 |
| Formal assessment | n/a | n/a | n/a | n/a | n/a | n/a |
| Job search | 1,081 | 0 | 1,081 | 0 | 23 | 1,104 |
| Basic education | 210 | 453 | 663 | 0 | 166 | 829 |
| Post-secondary education | 62 | 632 | 694 | 0 | 177 | 871 |
| Vocational training | 265 | 1,166 | 1,431 | 0 | 641 | 2,072 |
| Work experience | 256 | n/a | 256 | 0 | 67 | 323 |
| Subtotal (operating) | 1,946 | 2,251 | 4,198 | 0 | 1,074 | 5,271 |
| Child care | 1,191 | 0 | 1,191 | 301 | 0 | 1,491 |
| Other support services | 317 | 0 | 317 | 0 | 0 | 317 |
| Total | 3,454 | 2,251 | 5,705 | 301 | 1,074 | 7,080 |
Atlanta Human Capital Development | ||||||
| Orientation and appraisal | 72 | 0 | 72 | 0 | 0 | 72 |
| Formal assessment | n/a | n/a | n/a | n/a | n/a | n/a |
| Job search | 503 | 0 | 503 | 0 | 28 | 530 |
| Basic education | 430 | 1,004 | 1,434 | 0 | 161 | 1,596 |
| Post-secondary education | 58 | 425 | 483 | 0 | 415 | 898 |
| Vocational training | 429 | 2,185 | 2,614 | 0 | 605 | 3,219 |
| Work experience | 216 | 0 | 216 | 0 | 45 | 261 |
| Subtotal (operating) | 1,708 | 3,614 | 5,323 | 0 | 1,253 | 6,576 |
| Child care | 1,148 | 0 | 1,148 | 251 | 0 | 1,400 |
| Other support services | 610 | 0 | 610 | 0 | 0 | 610 |
| Total | 3,466 | 3,614 | 7,080 | 251 | 1,253 | 8,585 |
Grand Rapids Labor Force Attachment | ||||||
| Orientation and appraisal | 18 | n/a | 18 | 0 | 0 | 18 |
| Formal assessment | 9 | n/a | 9 | 0 | 0 | 9 |
| Job search | 362 | 908 | 1,270 | 0 | 152 | 1,422 |
| Basic education | 166 | 719 | 884 | 0 | 484 | 1,368 |
| Post-secondary education | 263 | 1,710 | 1,974 | 0 | 1,340 | 3,314 |
| Vocational training | 152 | 1,033 | 1,186 | 0 | 618 | 1,804 |
| Work experience | 130 | 0 | 130 | 0 | 53 | 183 |
| Subtotal (operating) | 1,100 | 4,371 | 5,470 | 0 | 2,648 | 8,118 |
| Child care | 451 | 0 | 451 | 312 | 0 | 762 |
| Other support services | 60 | 0 | 60 | 0 | 0 | 60 |
| Total | 1,610 | 4,371 | 5,981 | 312 | 2,648 | 8,940 |
Grand Rapids Human Capital Development | ||||||
| Orientation and appraisal | 18 | 0 | 18 | 0 | 0 | 18 |
| Formal assessment | 312 | 0 | 312 | 0 | 0 | 312 |
| Job search | 206 | 432 | 638 | 0 | 116 | 754 |
| Basic education | 474 | 1,672 | 2,146 | 0 | 443 | 2,589 |
| Post-secondary education | 302 | 2,285 | 2,586 | 0 | 995 | 3,582 |
| Vocational training | 163 | 900 | 1,062 | 0 | 956 | 2,018 |
| Work experience | 131 | 0 | 131 | 0 | 139 | 271 |
| Subtotal (operating) | 1,607 | 5,288 | 6,895 | 0 | 2,649 | 9,544 |
| Child care | 606 | 0 | 606 | 502 | 0 | 1,108 |
| Other support services | 75 | 0 | 75 | 0 | 0 | 75 |
| Total | 2,288 | 5,288 | 7,576 | 502 | 2,649 | 10,727 |
Riverside Labor Force Attachment | ||||||
| Orientation and appraisal | 111 | 0 | 111 | 0 | 0 | 111 |
| Formal assessment | 6 | 0 | 6 | 0 | 0 | 6 |
| Job search | 1,056 | 0 | 1,056 | 0 | 145 | 1,202 |
| Basic education | 157 | 345 | 502 | 0 | 226 | 728 |
| Post-secondary education | 218 | 988 | 1,205 | 0 | 984 | 2,189 |
| Vocational training | 97 | 392 | 489 | 0 | 586 | 1,075 |
| Work experience | 147 | 0 | 147 | 0 | 134 | 281 |
| Subtotal (operating) | 1,793 | 1,725 | 3,518 | 0 | 2,075 | 5,593 |
| Child care | 252 | 0 | 252 | 29 | 0 | 281 |
| Other support services | 171 | 0 | 171 | 0 | 0 | 171 |
| Total | 2,217 | 1,725 | 3,941 | 29 | 2,075 | 6,045 |
Riverside Human Capital Development | ||||||
| Orientation and appraisal | 107 | 0 | 107 | 0 | 0 | 107 |
| Formal assessment | 13 | 0 | 13 | 0 | 0 | 13 |
| Job search | 1,008 | 0 | 1,008 | 0 | 169 | 1,177 |
| Basic education | 1,188 | 1,420 | 2,608 | 0 | 291 | 2,900 |
| Post-secondary education | 89 | 209 | 298 | 0 | 282 | 580 |
| Vocational training | 179 | 688 | 867 | 0 | 182 | 1,049 |
| Work experience | 169 | 0 | 169 | 0 | 92 | 261 |
| Subtotal (operating) | 2,753 | 2,318 | 5,071 | 0 | 1,015 | 6,086 |
| Child care | 361 | 0 | 361 | 16 | 0 | 376 |
| Other support services | 170 | 0 | 170 | 0 | 0 | 170 |
| Total | 3,284 | 2,318 | 5,601 | 16 | 1,015 | 6,632 |
Columbus Integrated | ||||||
| Orientation and appraisal | 18 | 0 | 18 | 0 | 0 | 18 |
| Formal assessment | n/a | n/a | n/a | n/a | n/a | n/a |
| Job search | 141 | 0 | 141 | 0 | 15 | 156 |
| Basic education | 108 | 259 | 367 | 0 | 334 | 700 |
| Post-secondary education | 732 | 1,241 | 1,973 | 0 | 887 | 2,860 |
| Vocational training | 206 | 542 | 749 | 0 | 534 | 1,283 |
| Work experience | 85 | 0 | 85 | 0 | 170 | 254 |
| Subtotal (operating) | 1,290 | 2,042 | 3,331 | 0 | 1,939 | 5,271 |
| Child care | 571 | 0 | 571 | 558 | 0 | 1,129 |
| Other support services | 334 | 0 | 334 | 0 | 0 | 334 |
| Total | 2,195 | 2,042 | 4,237 | 558 | 1,939 | 6,734 |
| Columbus Traditional | ||||||
| Orientation and appraisal | 9 | 0 | 9 | 0 | 0 | 9 |
| Formal assessment | n/a | n/a | n/a | n/a | n/a | n/a |
| Job search | 192 | 0 | 192 | 0 | 0 | 192 |
| Basic education | 84 | 494 | 577 | 0 | 218 | 796 |
| Post-secondary education | 194 | 1,870 | 2,064 | 0 | 556 | 2,620 |
| Vocational training | 59 | 270 | 329 | 0 | 830 | 1,158 |
| Work experience | 66 | 0 | 66 | 0 | 127 | 194 |
| Subtotal (operating) | 603 | 2,634 | 3,237 | 0 | 1,731 | 4,969 |
| Child care | 584 | 0 | 584 | 547 | 0 | 1,131 |
| Other support services | 224 | 0 | 224 | 0 | 0 | 224 |
| Total | 1,411 | 2,634 | 4,045 | 547 | 1,731 | 6,323 |
Detroit | ||||||
| Job search | 114 | 446 | 560 | 0 | 5 | 565 |
| Education and traininga | 1,198 | 1,403 | 2,602 | 0 | 1,879 | 4,481 |
| Work experience | 77 | 0 | 77 | 0 | 3 | 80 |
| Subtotal (operating) | 1,389 | 1,849 | 3,238 | 0 | 1,887 | 5,126 |
| Child care | 313 | 0 | 313 | 435 | 0 | 748 |
| Other support services | 117 | 0 | 117 | 0 | 0 | 117 |
| Total | 1,819 | 1,849 | 3,668 | 435 | 1,887 | 5,990 |
Portland | ||||||
| Orientation and appraisal | 138 | 11 | 149 | 0 | 0 | 149 |
| Formal assessment | n/a | n/a | n/a | n/a | ||
| Job search | 609 | 14 | 623 | 0 | 131 | 754 |
| Basic education | 361 | 12 | 373 | 0 | 624 | 997 |
| Post-secondary education | 127 | 709 | 836 | 0 | 755 | 1,591 |
| Vocational training | 182 | 928 | 1,110 | 0 | 831 | 1,941 |
| Work experience | 286 | 6 | 292 | 0 | 122 | 414 |
| Subtotal (operating) | 1,702 | 1,680 | 3,382 | 0 | 2,464 | 5,846 |
| Child care | 1,043 | 0 | 1,043 | 1,883 | 0 | 2,926 |
| Other support services | 158 | 0 | 158 | 0 | 0 | 159 |
| Total | 2,903 | 1,680 | 4,583 | 1,883 | 2,464 | 8,930 |
| SOURCES: MDRC calculations based on fiscal and participation data from the following: Atlanta - the Fulton County Department of Family and Children Services, the Georgia Department of Human Resources, the Georgia Department of Technical and Adult Education, the Board of Regents University System of Georgia; Grand Rapids - the Michigan Department of Social Services, the Michigan Department of Education Office of Extended Learning Services, the Grand Rapids Community College, the Wyoming Community Education Center; Riverside - the California Department of Social Services, the California Department of Education, the Chancellors Office of California Community Colleges; Columbus - the Franklin County Department of Human Services, the Ohio Department of Education, the Office of Vocational and Adult Education, the Ohio Board of Regents, the National Center for Education Statistics; Detroit - the Michigan Family Independence Agency, the Michigan Department of Education, the Michigan Jobs Commission; Portland - the Oregon Department of Human Resources, Adult and Family Services Division, the Oregon Office of Community College Services; in all sites - information collected on tuition charged at proprietary schools attended by sample members, from MDRC-collected JOBS case file data, the MDRC Two-Year Client Survey, and the MDRC Five-Year Client Survey. MDRC child care and other support service calculations from Fulton County, Kent County, Riverside County, Ohio Department of Human Services, Wayne County, Washington County, and Multnomah County (District 2) payment data. Other support service data from county records. NOTES: Rounding may cause slight discrepancies in calculating sums and differences. N/a = not applicable. a Owing to data limitations, in Detroit it was not possible to separate vocational training, post-secondary education, and basic education costs. Thus, the gross costs of all three activities are included in the cost of "education and training." Furthermore, in this site orientation and assessment costs are spread across these three activities. | ||||||
2. Support service costs. Programs paid for child care, transportation, and ancillary services (such as uniforms, tools, equipment, and books) to support recipients' participation in employment and employment-related activities. Detailed information on support service payments was available through the end year 2 of follow-up. These data, together with data from the Five-Year Client Survey, were used to estimate payments for the remainder of the follow-up period.
As shown in Table 13.3, the welfare departments spent an average of $252 (Riverside LFA) to more than $1,000 (Atlanta LFA and HCD and Portland) per program group member on in-program child care. The costs of other support services, which included payments to cover transportation costs, class fees, uniforms, and so on, ranged from less than $100 (Grand Rapids LFA and HCD) to $610 (Atlanta HCD).
Non-Welfare Agency Expenditures
Table 13.3 (column 2) shows the non-welfare agency cost of providing employment-related services. These costs ranged from $1,680 (Portland) to $5,288 (Grand Rapids HCD). As discussed above, Portland's low non-welfare agency costs reflect the fact that the welfare department paid for the majority of in-program activities (except for vocational training and college). The costs were high in Grand Rapids, especially for the HCD program, because job search services were provided and paid for by the local community education center and because of the extensive use of education and training activities, which also were not paid for by the welfare department.
Expenditures for Out-of-Program Employment-Related Services
Program and control group members participated in some employment-related activities when they were not receiving welfare benefits. Although these services were not part of the NEWWS programs, if program and control group members participated in such activities and participated at different rates, the off-welfare services have the potential to differentially increase sample members' earnings and reduce their welfare receipt, and thus are included in the cost estimates. These expenditures are examined in more detail below. (See Figure 13.1, Boxes 4 and 5.)
Welfare Department Expenditures
The bulk of the costs incurred when participants were not receiving welfare were paid by non-welfare agencies; welfare agencies provided child care assistance only. Sample members were eligible for one year of transitional child care assistance if they left welfare for work and could also receive income-eligible child care.
As shown in Table 13.3, the average cost of transitional, income-eligible, and other child care ranged from less than $30 per program group member (Riverside LFA and HCD) to $1,883 (Portland). This wide variation in child care payments was seen after two years of follow-up. As explained in the earlier report, Portland's high child care costs may reflect a combination of the following: The program included more families with younger children than most of the other NEWWS programs,(5) the staff believed in the importance of providing child care to everyone who was eligible for it, and integrated case management allowed better administration of child care payments.(6)
Non-Welfare Agency Expenditures
When sample members were not receiving welfare benefits, they most commonly participated in basic education, post-secondary education, and vocational training; some sample members participated in other activities (job search, unpaid work experience, and on-the-job training).(7) These costs ranged from about $1,000 (Riverside HCD) to about $2,500 (Grand Rapids LFA and HCD and Portland). The programs in Grand Rapids and Portland had high rates of self-initiated participation, particularly in education and training activities. As will be discussed later, this was also true for control group members in these programs.
Gross Costs
The gross costs were obtained by adding the cost of in-program services to the out-of-program costs. This total investment must be compared with the total gross cost per control group member to determine the government's net investment per program group member and, in the benefit-cost analysis, the net payoff of that investment. As shown in Table 13.3, the gross costs of the NEWWS programs ranged from $5,990 (Detroit) to $10,727 (Grand Rapids HCD) per program group member over the five-year follow-up period. On average, gross costs were slightly higher for education-focused programs than for employment-focused programs. (See Figure 13.1, Boxes 7 and 10.)
Gross Costs for Program Group Members
Participation rates and duration of participation, as well as whether participation took place while sample members were on or off welfare, determined both overall costs and the proportion of costs paid for by the welfare department. For example, post-secondary education unit costs were similar in the Atlanta and Grand Rapids HCD programs, but participation in both in-program and out-of-program (that is, self-initiated) activities was much greater in Grand Rapids, resulting in a total gross cost for post-secondary education that was nearly four times as high as that for Atlanta. As a result, the Grand Rapids HCD program incurred the highest in-program costs (column 3) and total costs (column 6). However, the proportion of these costs paid by the welfare department was lower in the Grand Rapids programs than in the other programs (column 1 divided by column 6): These programs had more participation in higher-cost activities such as education and training that was leveraged with fewer dollars by the welfare department.
Overall, higher costs were associated with higher rates of participation in education and training activities. Therefore, employment-focused programs might be expected to have lower total gross costs than education-focused programs. However, the gross costs of the Grand Rapids LFA and Portland programs were among the highest in this evaluation because sample members in these programs participated in education and training activities at high rates, much of it self-initiated. As a result, on average, employment-focused programs were slightly more expensive than education-focused ones.
The average gross cost was higher than costs found in other MDRC evaluations of welfare-to-work programs. On average, total gross costs of the NEWWS programs were similar to those found in other high-cost programs, including the Alameda and Los Angeles GAIN programs operated in the late 1980s and early 1990s, which provided extensive education and training services. The average cost per sample member across all NEWWS programs was $7,599, compared with $7,763 in Alameda and $7,123 in Los Angeles (all in 1999 dollars).(8) The higher costs in NEWWS programs (Atlanta and Grand Rapids HCD, Grand Rapids LFA, and Portland) are likely due to higher participation overall, and particularly in high-cost activities such as post-secondary education and vocational training.
Gross Costs for Control Group Members
Control group costs reflect the cost of participation in employment-related services, most of which was self-initiated. However, in the sites where control embargoes were lifted before the end of the five-year follow-up period, a small portion of participation was program-related.(9)
In general, control group costs were high. This was particularly true in Grand Rapids and Portland owing to high rates of self-initiated participation, especially in post-secondary education and vocational training activities. The average control group cost in the GAIN program was $1,638 (in 1999 dollars). The control group costs in Grand Rapids and Portland were about three times higher than the highest control group cost in the GAIN program, which was $2,233 (in 1999 dollars) in San Diego. The control group costs in the other NEWWS programs ranged from 8 percent (Riverside HCD) to 76 percent (Detroit) higher.
Net Costs
The net cost per program group member is the gross cost per program group member minus what would have been spent in the absence of the NEWWS programs, represented by the gross cost per control group member. (See Figure 13.1, Box 11.) As shown in Table 13.4 (column 3), net costs ranged from about $2,000 (Grand Rapids LFA and Detroit) to $5,480 (Atlanta HCD). Depending on the rate of participation, and the types of activities in which control group members enroll on their own, programs with high gross costs can have low net costs. For example, although Grand Rapids HCD had the highest gross costs, owing to high participation rates in education and training activities, similarly high participation rates in these activities among control group members resulted in a net cost that was lower than costs for the other education-focused programs. Likewise, Grand Rapids LFA and Portland had the highest gross costs among the employment-focused programs, but high gross costs for control group members produced low net costs for these two programs. As a result, the average net cost of the NEWWS programs was similar to that of the GAIN program.
Program and Component | Total Gross Cost per Program Group Member ($) | Total Gross Cost per Control Group Member ($) | Total Net Cost per Program Group Member ($) |
|---|---|---|---|
Atlanta Labor Force Attachment | |||
| Orientation and appraisal | 72 | 11 | 61 |
| Formal assessment | n/a | n/a | n/a |
| Job search | 1,104 | 79 | 1,025 |
| Basic education | 829 | 166 | 663 |
| Post-secondary education | 871 | 844 | 27 |
| Vocational training | 2,072 | 1,392 | 680 |
| Work experience | 323 | 126 | 198 |
| Subtotal (operating) | 5,271 | 2,618 | 2,654 |
| Child care | 1,491 | 459 | 1,032 |
| Other support services | 317 | 29 | 288 |
| Total | 7,080 | 3,105 | 3,974 |
Atlanta Human Capital Development | |||
| Orientation and appraisal | 72 | 11 | 61 |
| Formal assessment | n/a | n/a | n/a |
| Job search | 530 | 79 | 452 |
| Basic education | 1,596 | 166 | 1,429 |
| Post-secondary education | 898 | 844 | 54 |
| Vocational training | 3,219 | 1,392 | 1,827 |
| Work experience | 261 | 126 | 135 |
| Subtotal (operating) | 6,576 | 2,618 | 3,958 |
| Child care | 1,400 | 459 | 940 |
| Other support services | 610 | 29 | 581 |
| Total | 8,585 | 3,105 | 5,480 |
Grand Rapids Labor Force Attachment | |||
| Orientation and appraisal | 18 | 1 | 17 |
| Formal assessment | 9 | n/a | 9 |
| Job search | 1,422 | 329 | 1,093 |
| Basic education | 1,368 | 1,170 | 198 |
| Post-secondary education | 3,314 | 2,906 | 408 |
| Vocational training | 1,804 | 1,931 | -127 |
| Work experience | 183 | 54 | 129 |
| Subtotal (operating) | 8,118 | 6,392 | 1,726 |
| Child care | 762 | 561 | 202 |
| Other support services | 60 | n/a | 60 |
| Total | 8,940 | 6,953 | 1,987 |
Grand Rapids Human Capital Development | |||
| Orientation and appraisal | 18 | 1 | 17 |
| Formal assessment | 312 | n/a | 312 |
| Job search | 754 | 329 | 425 |
| Basic education | 2,589 | 1,170 | 1,419 |
| Post-secondary education | 3,582 | 2,906 | 675 |
| Vocational training | 2,018 | 1,931 | 87 |
| Work experience | 271 | 54 | 216 |
| Subtotal (operating) | 9,544 | 6,392 | 3,151 |
| Child care | 1,108 | 561 | 548 |
| Other support services | 75 | n/a | 75 |
| Total | 10,727 | 6,953 | 3,773 |
Riverside Labor Force Attachment | |||
| Orientation and appraisal | 111 | n/a | 111 |
| Formal assessment | 6 | n/a | 6 |
| Job search | 1,202 | 149 | 1,053 |
| Basic education | 728 | 230 | 498 |
| Post-secondary education | 2,189 | 1,301 | 889 |
| Vocational training | 1,075 | 856 | 218 |
| Work experience | 281 | 126 | 155 |
| Subtotal (operating) | 5,593 | 2,661 | 2,931 |
| Child care | 281 | 63 | 218 |
| Other support services | 171 | n/a | 171 |
| Total | 6,045 | 2,724 | 3,320 |
Riverside Human Capital Development | |||
| Orientation and appraisal | 107 | n/a | 107 |
| Formal assessment | 13 | n/a | 13 |
| Job search | 1,177 | 127 | 1,050 |
| Basic education | 2,900 | 409 | 2,491 |
| Post-secondary education | 580 | 606 | -26 |
| Vocational training | 1,049 | 820 | 229 |
| Work experience | 261 | 63 | 198 |
| Subtotal (operating) | 6,086 | 2,025 | 4,061 |
| Child care | 376 | 35 | 342 |
| Other support services | 170 | n/a | 170 |
| Total | 6,632 | 2,059 | 4,573 |
Columbus Integrated | |||
| Orientation and appraisal | 18 | n/a | 18 |
| Formal assessment | n/a | n/a | n/a |
| Job search | 156 | 31 | 125 |
| Basic education | 700 | 180 | 520 |
| Post-secondary education | 2,860 | 606 | 2,254 |
| Vocational training | 1,283 | 593 | 690 |
| Work experience | 254 | 25 | 229 |
| Subtotal (operating) | 5,271 | 1,434 | 3,837 |
| Child care | 1,129 | 1,084 | 45 |
| Other support services | 334 | 33 | 301 |
| Total | 6,734 | 2,551 | 4,183 |
Columbus Traditional | |||
| Orientation and appraisal | 9 | n/a | 9 |
| Formal assessment | n/a | n/a | n/a |
| Job search | 192 | 31 | 161 |
| Basic education | 796 | 180 | 616 |
| Post-secondary education | 2,620 | 606 | 2,014 |
| Vocational training | 1,158 | 593 | 566 |
| Work experience | 194 | 25 | 169 |
| Subtotal (operating) | 4,969 | 1,434 | 3,534 |
| Child care | 1,131 | 1,084 | 47 |
| Other support services | 224 | 33 | 190 |
| Total | 6,323 | 2,551 | 3,772 |
Detroit | |||
| Job search | 565 | 328 | 237 |
| Education and traininga | 4,481 | 2,830 | 1,651 |
| Work experience | 80 | 12 | 67 |
| Subtotal (operating) | 5,126 | 3,170 | 1,956 |
| Child care | 748 | 734 | 14 |
| Other support services | 117 | 34 | 83 |
| Total | 5,990 | 3,937 | 2,053 |
Portland | |||
| Orientation and appraisal | 149 | n/a | 149 |
| Formal assessment | n/a | n/a | n/a |
| Job search | 754 | 273 | 481 |
| Basic education | 997 | 593 | 404 |
| Post-secondary education | 1,591 | 2,353 | -763 |
| Vocational training | 1,941 | 1,049 | 892 |
| Work experience | 414 | 240 | 174 |
| Subtotal (operating) | 5,846 | 4,508 | 1,337 |
| Child care | 2,926 | 1,542 | 1,384 |
| Other support services | 159 | 15 | 144 |
| Total | 8,930 | 6,065 | 2,865 |
| SOURCES: See Table 13.3. NOTES: Rounding may cause slight discrepancies in calculating sums and differences. N/a = not applicable. a Owing to data limitations, in Detroit it was not possible to separate vocational training, post-secondary education, and basic education costs. Thus, the gross costs of all three activities are included in the cost of "education and training." Furthermore, in this site orientation and assessment costs are spread across these three activities. | |||
Net costs were generally lower for employment-focused programs than for education-focused ones, largely due to the particularly low net costs in Grand Rapids LFA and Portland. As the two-year findings suggested, the Grand Rapids LFA program may have diverted program group members into job search who would have enrolled on their own in education or training programs. The same may have been true of the Portland program.
Results by Educational Attainment Subgroups
Table 13.5 shows gross and net costs for those with and without a high school diploma or GED when they entered the evaluation. For the most part, gross and net costs were higher for graduates. This is not surprising, given that those with high school credentials were more likely to participate in higher-cost activities, such as post-secondary education or vocational training, whereas those who lacked these credentials were more likely to participate in typically lower-cost activities, such as basic education.
| Program and Component | Total Gross Cost per Program Group Member ($) | Total Gross Cost per Control Group Member ($) | Total Net Cost per Program Group Member ($) |
|---|---|---|---|
| Without a high school diploma or GED | |||
| Atlanta Labor Force Attachment | |||
| Operating costs | 3,761 | 1,118 | 2,642 |
| Support services | 1,469 | 370 | 1,099 |
| Total | 5,230 | 1,489 | 3,741 |
| Atlanta Human Capital Development | |||
| Operating costs | 5,646 | 1,118 | 4,528 |
| Support services | 1,982 | 370 | 1,612 |
| Total | 7,628 | 1,489 | 6,140 |
| Grand Rapids Labor Force Attachment | |||
| Operating costs | 6,244 | 5,699 | 544 |
| Support services | 493 | 445 | 48 |
| Total | 6,736 | 6,144 | 592 |
| Grand Rapids Human Capital Development | |||
| Operating costs | 8,789 | 5,699 | 3,090 |
| Support services | 1,001 | 445 | 556 |
| Total | 9,790 | 6,144 | 3,645 |
| Riverside Labor Force Attachment | |||
| Operating costs | 5,306 | 2,025 | 3,282 |
| Support services | 309 | 35 | 274 |
| Total | 5,615 | 2,060 | 3,555 |
| Riverside Human Capital Development | |||
| Operating costs | 6,086 | 2,025 | 4,061 |
| Support services | 546 | 35 | 511 |
| Total | 6,632 | 2,060 | 4,572 |
| Columbus Integrated | |||
| Operating costs | 4,564 | 914 | 3,650 |
| Support services | 1,357 | 624 | 733 |
| Total | 5,921 | 1,538 | 4,383 |
| Columbus Traditional | |||
| Operating costs | 3,500 | 914 | 2,585 |
| Support services | 1,078 | 624 | 454 |
| Total | 4,577 | 1,538 | 3,039 |
| Detroit | |||
| Operating costs | 4,560 | 3,033 | 1,527 |
| Support services | 781 | 902 | -121 |
| Total | 5,341 | 3,935 | 1,406 |
| Portland | |||
| Operating costs | 4,690 | 2,927 | 1,763 |
| Support services | 2,874 | 1,346 | 1,528 |
| Total | 7,563 | 4,273 | 3,290 |
| With a high school diploma or GED | |||
| Atlanta Labor Force Attachment | |||
| Operating costs | 6,259 | 3,596 | 2,663 |
| Support services | 2,105 | 559 | 1,546 |
| Total | 8,365 | 4,155 | 4,209 |
| Atlanta Human Capital Development | |||
| Operating costs | 7,207 | 3,596 | 3,611 |
| Support services | 2,021 | 559 | 1,462 |
| Total | 9,228 | 4,155 | 5,073 |
| Grand Rapids Labor Force Attachment | |||
| Operating costs | 8,961 | 7,112 | 1,849 |
| Support services | 958 | 630 | 328 |
| Total | 9,918 | 7,741 | 2,177 |
| Grand Rapids Human Capital Development | |||
| Operating costs | 10,273 | 7,112 | 3,162 |
| Support services | 1,440 | 630 | 810 |
| Total | 11,713 | 7,741 | 3,972 |
| Riverside Labor Force Attachment | |||
| Operating costs | 5,877 | 3,625 | 2,252 |
| Support services | 581 | 58 | 523 |
| Total | 6,457 | 3,682 | 2,775 |
| Columbus Integrated | |||
| Operating costs | 5,938 | 1,790 | 4,149 |
| Support services | 1,524 | 1,368 | 157 |
| Total | 7,463 | 3,157 | 4,305 |
| Columbus Traditional | |||
| Operating costs | 5,759 | 1,790 | 3,969 |
| Support services | 1,495 | 1,368 | 127 |
| Total | 7,253 | 3,157 | 4,096 |
| Detroit | |||
| Operating costs | 5,510 | 3,292 | 2,218 |
| Support services | 859 | 589 | 270 |
| Total | 6,370 | 3,882 | 2,488 |
| Portland | |||
| Operating costs | 6,568 | 5,438 | 1,131 |
| Support services | 3,358 | 1,540 | 1,818 |
| Total | 9,926 | 6,978 | 2,948 |
| SOURCES: See Table 13.3. NOTES: Rounding may cause slight discrepancies in calculating sums and differences. | |||
LFA-HCD Program Differences
As noted above, net costs were generally higher for education-focused programs than for employment-focused programs. Similarly, HCD programs had higher net costs than LFA programs. Again, this was not a surprising finding. As discussed in Chapter 1, the HCD programs were designed to make larger upfront investments in building skills, mainly through education and training activities, which tend to be longer in duration and more expensive to operate than job search activities. It was anticipated that the programs would incur higher costs, but with the hope that this greater investment in human capital would produce greater returns (earnings increases and welfare savings) over time.
After two years of follow-up, HCD programs were 1.5 to 2.6 times more expensive than LFA programs. Although the net cost gap at the five-year point narrowed relative to net costs at the two-year point, HCD programs continued to be more expensive than LFA programs from 1.3 to 1.9 times more expensive.(10)
Analytical Approach for the Benefit-Cost Analysis
Accounting Methods
The analytical approach used in this benefit-cost analysis is similar to that used in previous evaluations conducted by MDRC.(11) The general approach is to place dollar values on the program's effects and its use of resources whenever possible, either by directly measuring them or by estimating them.
Program effects on earnings, welfare, and Food Stamp payments were measured directly. Effects on earnings were based on quarterly earnings reported by employers to states' unemployment insurance (UI) systems, and effects on welfare and Food Stamp payments were measured using computerized administrative records (the same data sources were used in the impact analysis presented in Chapters 4 and 5). Effects on Medicaid, fringe benefits, federal income taxes, state income and sales taxes, and the costs of administering transfer programs could not be measured directly, but were estimated or imputed using various data sources (details are provided below).
All of these effects were considered along with the estimated net costs of the NEWWS programs, presented above, to ascertain the net gains and losses to program group members and to government budgets.
The benefit-cost estimates cover a five-year period starting with the quarter after random assignment (quarter 2) for each sample member. This time frame is similar to that used in most previous MDRC benefit-cost analyses of welfare-to-work programs.Projecting benefits and costs beyond five years would be problematic because it is difficult to predict future behavior. However, given the extent to which impacts on earnings and welfare had decayed in most programs by the end of the five-year period, it seems unlikely that there would be many future net returns to the programs.
The benefit-cost estimates in this analysis are expressed in terms of net present values per program group member. The "net" means that, like impacts, the estimated amounts represent differences between estimates for program and control group members. The estimates are in "present value" terms because the accounting method of discounting was used to express the dollar worth at the end of the first year of follow-up of program effects that occurred later in the follow-up period.(12) As in the cost analysis, all estimated amounts in the benefit-cost analysis are expressed in 1999 dollars, eliminating the effects of inflation on the values.
Analytical Perspectives
An important issue in benefit-cost analyses of government programs is determining who bears any benefits or costs of the program. A program effect can generate gains from one perspective while generating losses from another. For example, a decrease in welfare payments is a financial loss from the perspective of the program group but a financial gain from the perspective of the government. This makes it important to consider the perspectives of all the directly affected groups when assessing each main program effect.
This analysis presents the net benefits and costs of the NEWWS programs from the perspective of program participants and government budgets. In the following table, the main financial effects are shown as an expected gain or benefit (+), a loss or cost (-), or neither a benefit nor a cost (0), according to a priori expectations regarding their value. (The tables in the following sections show the actual gains and losses in dollars.)
Expected Main Financial Effects of NEWWS Programs | Accounting Perspective | |
|---|---|---|
| Welfare Sample | Government Budget | |
| Increased earnings and fringe benefits | + | 0 |
| Increased tax payments | - | + |
| Reduced use of transfer programs | - | + |
| NEWWS operating costs | 0 | - |
| Increased use of support services | 0 | - |
The welfare sample perspective identifies net gains or losses for program group members, indicating how they fared as a result of the program. As illustrated, it is expected that earnings impacts represent gains for participants, whereas reductions in welfare payments and higher tax payments (resulting from earnings gains) represent losses. The program may be considered a net gain from the standpoint of program group members if the gains from earnings exceed losses from reduced transfer payments and higher taxes. The net cost of providing eligibility and employment-related services to participants has no direct effect on their income.
The government budget perspective identifies net gains and losses incurred by a combination of federal, state, and local government budgets. Net gains to the government budget occur through savings in transfer payments and their related administrative costs and through higher taxes paid by program group members compared with control group members. The government budget comes out ahead if tax increases and savings in transfer payments and administrative costs exceed the net cost of providing NEWWS program services. Program group members' earnings gains do not directly affect the government budget's net gains or losses.
Limitations of the Analysis
Some limits on the comprehensiveness of the benefit-cost analysis should be recognized. Some program effects, whose costs and benefits are difficult to quantify or to express in dollars, are not estimated. For example, this analysis does not include estimates of out-of-pocket expenditures (for child care, transportation, clothing, and so on) that sample members incurred when they went to work.(13) In addition, the estimates in this chapter reflect the direct effects of programs and do not consider secondary effects. These secondary effects include the possible displacement of other workers by the increased employment of program group members; these displaced workers may have become unemployed or employed in lower-paying jobs. In addition, the analysis does not consider the sample members' forgone personal and family activities that resulted from increased work or the intrinsic benefits of education that are not reflected in earnings. The analysis does not place a dollar value on family or child well-being or the clear but difficult-to-monetize benefits associated with society's (or sample members') preference for work over welfare.
Effects for Sample Members
This section presents estimates of the financial benefits of the NEWWS programs per program group member during the five-year follow-up period.(14) It presents an account of the main benefit components: earnings and fringe benefits, personal taxes, and transfer payments and benefits.
Earnings and Fringe Benefits
As reported in Chapter 4, the NEWWS programs produced increases in employment and earnings for program group members (compared with control group members) over the five-year follow-up period. Table 13.6 shows that the impact on earnings ranged from about $900 (Grand Rapids HCD) to more than $5,000 (Portland) per program group member.
| Component of Analysis | Atlanta LFA ($) | Atlanta HCD ($) | Grand Rapids LFA ($) | Grand Rapids HCD ($) | Riverside LFA ($) | Riverside HCD ($) | Columbus Integrated ($) | Columbus Traditional ($) | Detroit ($) | Portland ($) |
|---|---|---|---|---|---|---|---|---|---|---|
| Earnings | 2,575 | 2,089 | 1,678 | 906 | 2,697 | 1,398 | 2,086 | 1,456 | 1,490 | 5,171 |
| Fringe benefitsa | 384 | 311 | 265 | 143 | 369 | 192 | 330 | 230 | 235 | 708 |
| Total | 2,959 | 2,400 | 1,943 | 1,050 | 3,066 | 1,590 | 2,416 | 1,687 | 1,726 | 5,879 |
|
Personal taxes |
||||||||||
| Social Security taxb | 174 | 141 | 113 | 61 | 182 | 94 | 141 | 98 | 101 | 349 |
| Federal income tax | 9 | 62 | -321 | -99 | -278 | -340 | 0 | -53 | 59 | -258 |
| State income tax | 77 | 73 | 46 | 33 | 5 | 2 | 27 | 15 | 64 | 175 |
| State sales and excise tax | 25 | 20 | -21 | -19 | -5 | -44 | 10 | 6 | 19 | 0 |
| Total taxes | 285 | 296 | -182 | -24 | -95 | -287 | 178 | 67 | 242 | 266 |
| Sample size | 2,938 | 2,992 | 3,012 | 2,997 | 6,726 | 3,135 | 4,672 | 4,729 | 4,459 | 4,028 |
| SOURCES: MDRC calculations from state unemployment insurance (UI) earnings records and published data on fringe benefits and tax rates. NOTES: Estimates reflect discounting and adjustment for inflation. Estimates were regression-adjusted using ordinary least squares, controlling for pre-random assignment characteristics of sample members. Rounding may cause slight discrepancies in calculating sums and differences. Tests of statistical significance were not performed. a These include employer-paid health and life insurance, pension contributions, and workers' compensation. Paid leave is captured directly by the earnings estimate. Employee-paid Social Security and Medicare taxes are included as tax payments. b Employee portion only. |
||||||||||
Fringe benefits employer-provided health and life insurance, pension contributions, and worker's compensation were part of sample members' total compensation from working and thus were included in the benefit-cost analysis.(15) They ranged by region from 13.7 percent (the West) to 15.8 percent (the Midwest) of earnings.(16) Increases from earnings and fringe benefits yielded an average increase in total work-related compensation that ranged from $1,050 per program group member in Grand Rapids HCD to $5,879 per program group member in Portland.(17)
Personal Taxes
Since all of the programs increased earnings, one might expect them to also increase income taxes, payroll taxes, and sales and excise taxes (there is no sales tax in Oregon). Federal tax payments, along with the Earned Income Tax Credit (EITC),(18) were imputed from the relevant earnings base using tax rates and rules for 1995.(19) As shown in Table 13.6, in fact, about half of the programs decreased income taxes by $53 (Columbus Traditional) to $340 (Riverside HCD) per sample member. In these cases, the expected increase in tax payments was outweighed by an increase in federal EITC payments to sample members. There was an increase in federal income taxes in the remaining sites, ranging from less than $1 (Columbus Integrated) to $62 (Atlanta HCD).
Transfer Payments and Benefits
As discussed in Chapter 5, all of the programs in the evaluation decreased welfare and Food Stamp payments to program group members over the follow-up period. As shown in Table 13.7, the impacts on welfare payments ranged from -$571 in the Detroit program to -$3,058 in the Riverside HCD program. During the same period, reductions in Food Stamp payments ranged from $160 in the Atlanta HCD program to over $1,000 in the Riverside HCD and Columbus Integrated programs. In sum, program group members received from about $900 (Detroit and Atlanta) to over $4,000 (Riverside HCD) less in cash payments over the five-year period than their control group counterparts.
| Component of Analysis | Atlanta LFA ($) | Atlanta HCD ($) | Grand Rapids LFA ($) | Grand Rapids HCD ($) | Riverside LFA ($) | Riverside HCD ($) | Columbus Integrated ($) | Columbus Traditional ($) | Detroit ($) | Portland ($) |
|---|---|---|---|---|---|---|---|---|---|---|
|
Transfer payments |
||||||||||
| Welfare | -919 | -747 | -2,711 | -1,860 | -2,868 | -3,058 | -1,575 | -1,148 | -571 | -2,783 |
| Food Stamps | -434 | -160 | -643 | -396 | -928 | -1,040 | -1,052 | -673 | -341 | -848 |
| Medicaid | -1,159 | -727 | -1,026 | -775 | -510 | -766 | -1,102 | -875 | -309 | -2,598 |
| Total | -2,512 | -1,633 | -4,380 | -3,031 | -4,306 | -4,864 | -3,728 | -2,696 | -1,222 | -6,228 |
|
Administrative costs |
||||||||||
| Welfare | -119 | -97 | -434 | -298 | -258 | -275 | -173 | -126 | -91 | -918 |
| Food Stamps | -67 | -25 | -99 | -61 | -143 | -160 | -162 | -104 | -53 | -131 |
| Medicaid | -46 | -29 | -51 | -39 | -71 | -107 | -44 | -35 | -15 | -208 |
| Total | -233 | -151 | -584 | -397 | -472 | -543 | -379 | -265 | -159 | -1,257 |
| Sample size | 2,938 | 2,992 | 3,012 | 2,997 | 6,726 | 3,135 | 4,672 | 4,729 | 4,459 | 4,028 |
| SOURCES: MDRC calculations from state welfare and Food Stamp payment records, unemployment insurance (UI) earnings records, and published data on administrative costs. NOTES: Estimates reflect discounting and adjustment for inflation. Estimates were regression-adjusted using ordinary least squares, controlling for pre-random assignment characteristics of sample members. Rounding may cause slight discrepancies in calculating sums and differences. Tests of statistical significance were not performed. |
||||||||||
In addition to cash benefits, sample members were also eligible for Medicaid benefits. In low-grant states especially, Medicaid benefits can have a larger cash value than welfare payments. Sample members and their children were categorically eligible for Medicaid while receiving welfare. In addition, those who left welfare because of employment were eligible for Transitional Medicaid for one year.(20) Eligibility for these programs was estimated using welfare administrative data, Transitional Medicaid take-up rates calculated from the Five-Year Client Survey,(21) and published data on Medicaid expenditures.(22)
As noted in Chapter 8, the programs in the NEWWS Evaluation did not result in significant differences in health care coverage between program and control group members at the end of the follow-up period. It is important to note that the earlier chapter looked at differences in health coverage at a single point in time, that is, in the month before the five-year survey. As shown in Figure 5.1, in most programs the smallest differences in rate of welfare receipt were observed at the end of the follow-up period. However, as shown in Table 5.1, the differences in months of welfare receipt (and, therefore, Medicaid coverage) over the entire five-year period were substantial, ranging from 1.6 months (Detroit) to 5.6 months (Portland). Although program group members were more likely than their control group counterparts to receive Transitional Medicaid, this did not make up for the loss in Medicaid benefits due to their shorter stay on welfare. As shown in Table 13.7, average Medicaid losses over the five-year period ranged from $309 (Detroit) to $2,598 (Portland).
The costs of administering transfer programs were calculated as the cost per benefit paid. As a result, the NEWWS programs also decreased transfer program administration costs.(23) These changes, shown in Table 13.7, yielded decreases of between about $150 (Atlanta HCD and Detroit) and about $1,250 in Portland.(24)
Net Gains and Losses by Accounting Perspective
Welfare Sample Members
Table 13.8 summarizes the main financial effects from the perspective of welfare sample members. Program-control group differences were defined as gains, indicated by positive values, and losses, indicated by negative values. These results were then summed to attain an estimate of the overall net gain or loss of each program.
| Component of Analysis | Atlanta LFA ($) | Atlanta HCD ($) | Grand Rapids LFA ($) | Grand Rapids HCD ($) | Riverside LFA ($) | Columbus Integrated ($) | Columbus Traditional ($) | Detroit ($) | Portland ($) |
|---|---|---|---|---|---|---|---|---|---|
Gains | |||||||||
| Earnings | 2,575 | 2,089 | 1,678 | 906 | 2,697 | 2,086 | 1,456 | 1,490 | 5,171 |
| Fringe benefitsa | 384 | 311 | 265 | 143 | 369 | 330 | 230 | 235 | 708 |
| Total | 2,959 | 2,400 | 1,943 | 1,050 | 3,066 | 2,416 | 1,687 | 1,726 | 5,879 |
Losses | |||||||||
| Tax payments | -285 | -296 | 182 | 24 | 95 | -178 | -67 | -242 | -266 |
| Welfare payments | -919 | -747 | -2,711 | -1,860 | -2,868 | -1,575 | -1,148 | -571 | -2,783 |
| Food Stamps | -434 | -160 | -643 | -396 | -928 | -1,052 | -673 | -341 | -848 |
| Medicaid | -1,159 | -727 | -1,026 | -775 | -510 | -1,102 | -875 | -309 | -2,598 |
| Total | -2,797 | -1,930 | -4,197 | -3,007 | -4,211 | -3,907 | -2,763 | -1,464 | -6,494 |
| Net gain or loss(net present value) | 162 | 471 | -2,254 | -1,957 | -1,145 | -1,491 | -1,076 | 262 | -615 |
| SOURCES: MDRC calculations from state welfare and Food Stamp payment records, unemployment insurance (UI) earnings records, and published data on fringe benefits, taxes, and administrative costs. NOTES: Estimates reflect discounting and adjustment for inflation. Estimates were regression-adjusted using ordinary least squares, controlling for pre-random assignment characteristics of sample members. Rounding may cause slight discrepancies in calculating sums and differences. Tests of statistical significance were not performed. a These include employer-paid health and life insurance, pension contributions, and worker's compensation. Paid leave is captured directly by the earnings estimate. Employee-paid Social Security and Medicare taxes are included as tax payments. | |||||||||
The overall financial effect from the welfare sample perspective was estimated by subtracting the value of transfer payment losses from the value of gains in earnings, fringe benefits, and taxes. Clearly, programs where sample members experience large losses in transfer payments are less likely to produce net gains to sample members, because they have greater losses to offset with earned income. The three programs with the smallest welfare reductions (under $1,000 for both Atlanta programs and the Detroit program) produced very small net gains for sample members. However, these gains were very small; sample members effectively broke even. All of the programs with larger welfare reductions (ranging from about $1,148 in the Columbus Traditional program to $2,868 in the Riverside LFA program) produced net losses for sample members.
As noted earlier, this analysis does not take into account losses of time for personal and family activities (leisure) and out-of-pocket expenditures associated with increases in employment. Given that the three programs with gains did no more than break even, even small values attached to these effects would have resulted in losses for all programs.
Table 13.9 shows the net present values from the perspective of welfare sample members who did not have a high school diploma or GED at the time they entered the evaluation. Only two programs (Detroit and Portland) produced gains for these welfare sample members. However, these gains were small (sample members in Detroit effectively broke even). For the subgroup of sample members with a high school diploma or GED, three programs produced gains: both programs in Atlanta and the Detroit program. Again, gains were small. In fact, welfare sample members in the Atlanta LFA and Detroit programs did little more than break even. Generally, those with a high school diploma did better (that is, bigger gains or smaller losses accrued to this subgroup). However, for the Grand Rapids LFA and Portland programs, this trend was reversed. Although welfare reductions in Portland were as large for nongraduates as for graduates, nongraduates experienced smaller losses in Food Stamps and Medicaid, resulting in smaller reductions in total benefits from transfer programs.
Component of Analysis | Atlanta LFA ($) | Atlanta HCD ($) | Grand Rapids LFA ($) | Grand Rapids HCD ($) | Riverside LFA ($) | Riverside HCD ($) | Columbus Integrated ($) | Columbus Traditional ($) | Detroit ($) | Portland ($) |
|---|---|---|---|---|---|---|---|---|---|---|
Without a high school diploma or GED | ||||||||||
Gains | ||||||||||
| Earnings | 1,143 | 0 | 3,241 | 1,143 | 2,378 | 1,398 | 3,066 | 1,352 | 1,533 | 5,067 |
| Fringe benefitsa | 170 | 0 | 512 | 181 | 326 | 192 | 484 | 214 | 242 | 694 |
| Total | 1,313 | 0 | 3,753 | 1,323 | 2,704 | 1,590 | 3,551 | 1,565 | 1,775 | 5,761 |
Losses | ||||||||||
| Tax payments | 195 | -133 | 17 | -101 | 452 | 324 | -467 | -225 | -196 | 81 |
| Welfare payments | -774 | -249 | -3,218 | -2,240 | -3,121 | -3,058 | -2,207 | -1,393 | -518 | -2,628 |
| Food Stamps | 14 | 227 | -849 | -776 | -967 | -1,040 | -1,489 | -809 | -444 | -273 |
| Medicaid | -964 | -276 | -1,183 | -1,019 | -533 | -767 | -1,674 | -1,014 | -411 | -1,858 |
| Total | -1,528 | -432 | -5,233 | -4,135 | -4,168 | -4,541 | -5,837 | -3,441 | -1,568 | -4,677 |
Net gain or loss(net present value) | -215 | -432 | -1,479 | -2,812 | -1,465 | -2,951 | -2,286 | -1,876 | 207 | 1,084 |
With a high school diploma or GED | ||||||||||
Gains | ||||||||||
| Earnings | 3,452 | 3,358 | 535 | 763 | 3,117 | 1,252 | 1,467 | 1,318 | 5,075 | |
| Fringe benefitsa | 514 | 500 | 85 | 121 | 427 | 198 | 232 | 208 | 695 | |
| Total | 3,966 | 3,858 | 619 | 883 | 3,545 | 1,449 | 1,699 | 1,527 | 5,770 | |
Losses | ||||||||||
| Tax payments | -571 | -370 | 299 | 87 | -292 | 57 | 5 | -198 | -384 | |
| Welfare payments | -1,014 | -1,034 | -2,359 | -1,572 | -2,539 | -1,124 | -969 | -576 | -2,939 | |
| Food Stamps | -732 | -378 | -513 | -159 | -878 | -759 | -587 | -238 | -1,132 | |
| Medicaid | -1,290 | -964 | -906 | -598 | -481 | -713 | -840 | -211 | -3,066 | |
| Total | -3,608 | -2,746 | -3,478 | -2,242 | -4,190 | -2,540 | -2,391 | -1,224 | -7,521 | |
Net gain or loss(net present value) | 358 | 1,112 | -2,859 | -1,359 | -646 | -1,090 | -693 | 303 | -1,751 | |
| SOURCES: See Table 13.8. NOTES: Estimates reflect discounting and adjustment for inflation. Estimates were regression-adjusted using ordinary least squares, controlling for pre-random assignment characteristics of sample members. Rounding may cause slight discrepancies in calculating sums and differences. Tests of statistical significance were not performed. a These include employer-paid health and life insurance, pension contributions, and worker's compensation. Paid leave is captured directly by the earnings estimate. Employee-paid Social Security and Medicare taxes are included as tax payments. | ||||||||||
Government Budgets
From the perspective of government budgets, programs are considered "budget neutral" if they can generate savings that equal the expenditures for services. It follows, then, that programs that produce large welfare savings have the best chance of paying for themselves and programs with small welfare savings will have a harder time breaking even. As shown in Table 13.10 the three programs (Grand Rapids LFA, Riverside LFA, and Portland) that resulted in gains to government budgets had the largest welfare savings. The programs with the smallest welfare reductions, ranging from $571 (Detroit) to $1,148 (Columbus Traditional), resulted in net losses. Programs with welfare reductions in the midrange (Columbus Integrated and Grand Rapids HCD) essentially broke even.
Component of Analysis | Atlanta LFA ($) | Atlanta HCD ($) | Grand Rapids LFA ($) | Grand Rapids HCD ($) | Riverside LFA ($) | Columbus Integrated ($) | Columbus Traditional ($) | Detroit ($) | Portland ($) |
|---|---|---|---|---|---|---|---|---|---|
Gains | |||||||||
| Payroll taxes a | 348 | 282 | 227 | 122 | 364 | 282 | 197 | 201 | 698 |
| Income and sales tax | 111 | 155 | -295 | -86 | -277 | 37 | -31 | 141 | -83 |
| Welfare payments | 919 | 747 | 2,711 | 1,860 | 2,868 | 1,575 | 1,148 | 571 | 2,783 |
| Food Stamps | 434 | 160 | 643 | 396 | 928 | 1,052 | 673 | 341 | 848 |
| Medicaid | 1,159 | 727 | 1,026 | 775 | 510 | 1,102 | 875 | 309 | 2,598 |
| Transfer administration | 233 | 151 | 584 | 397 | 472 | 379 | 265 | 159 | 1,257 |
| Total | 3,204 | 2,221 | 4,895 | 3,465 | 4,865 | 4,427 | 3,126 | 1,724 | 8,100 |
| Losses | |||||||||
| Net cost of program and nonprogram activities and services | -3,974 | -5,480 | -1,987 | -3,773 | -3,320 | -4,183 | -3,772 | -2,053 | -2,865 |
| Net gain or loss (net present value) | -770 | -3,259 | 2,908 | -308 | 1,545 | 244 | -646 | -329 | 5,235 |
| Return to budget per net dollar invested in program and nonprogram activities and services b | 0.81 | 0.41 | 2.46 | 0.92 | 1.47 | 1.06 | 0.83 | 0.84 | 2.83 |
| SOURCES: See Table 13.8. NOTES: Estimates reflect discounting and adjustment for inflation. Estimates were regression-adjusted using ordinary least squares, controlling for pre-random assignment characteristics of sample members. Rounding may cause slight discrepancies in calculating sums and differences. Tests of statistical significance were not performed. a Payroll taxes include employer- and employee-paid Social Security and Medicare taxes. b The return to budget per net dollar invested is computed by dividing total savings and tax increases by the net cost of activities and services. | |||||||||
To provide some basis for interpreting these results and to facilitate cross-program comparisons, Table 13.10 also presents an additional measure of the cost-effectiveness of the NEWWS programs from the government budget perspective. This measure is called the return to budget per net dollar invested, and it is calculated by dividing the gains (taxes and savings in transfer payments and associated administrative costs) by the total net costs of services. Using this metric, government budgets come out ahead if programs produce more than a dollar's worth of additional revenues and savings for each dollar spent on employment-related services to program group members (compared with controls).
Both the Portland and the Grand Rapids LFA programs produced over $2.00 in increased revenue and savings for every additional dollar spent on program group members. The Riverside LFA program also produced a considerable return, $1.47 per dollar invested. As noted above, the Grand Rapids HCD and Columbus Integrated programs essentially caused the government to break even ($0.92 to $1.06, respectively). The Atlanta, Columbus Traditional, and Detroit programs were not as successful, returning considerably less than one dollar for each dollar invested, ranging from $0.41 in the Atlanta HCD program to just over $0.80 in the other programs. The average return across all of the programs was $1.29 (not shown in any table).
On average, these results are more positive than those found in the GAIN evaluation, in which returns to the government budget ranged from $0.17 per dollar invested (Tulare) to $2.84 (Riverside). The average across all counties in that study was $0.76. In the NEWWS Evaluation, the return on investments in Portland ($2.83) and Grand Rapids LFA ($2.46) compare favorably with the Riverside GAIN program and the SWIM program, which returned $2.84 and $2.34, respectively.
From the perspective of government budgets, the results were mostly positive for those without a high school diploma or GED. As shown in Table 13.11, both Atlanta programs produced losses. Gains ranged from $437 in Detroit to $5,564 in Grand Rapids LFA. For those with a high school diploma or GED, only the employment-focused programs paid for themselves (Atlanta LFA broke even). This is consistent with other findings: The welfare savings were larger for employment-focused programs; and those entering education-focused programs with a high school diploma or GED were more likely to participate in higher-cost activities, such as post-secondary education or vocational training, than those entering employment-focused (particularly LFA). However, this is not to say that programs with high rates of participation in higher-cost activities cannot pay for themselves: The mixed-approach Portland program had the highest returns to the government budget.
Component of Analysis | Atlanta LFA ($) | Atlanta HCD ($) | Grand Rapids LFA ($) | Grand Rapids HCD ($) | Riverside LFA ($) | Riverside HCD ($) | Columbus Integrated ($) | Columbus Traditional ($) | Detroit ($) | Portland ($) |
|---|---|---|---|---|---|---|---|---|---|---|
Without a high school diploma or GED | ||||||||||
Gains | ||||||||||
| Payroll taxesa | 154 | -0 | 438 | 154 | 321 | 189 | 414 | 182 | 207 | 684 |
| Income and sales tax | -272 | 133 | -236 | 24 | -613 | -418 | 260 | 133 | 92 | -423 |
| Welfare payments | 774 | 249 | 3,218 | 2,240 | 3,121 | 3,058 | 2,207 | 1,393 | 518 | 2,628 |
| Food Stamps | -14 | -227 | 849 | 776 | 967 | 1,040 | 1,489 | 809 | 444 | 273 |
| Medicaid | 964 | 276 | 1,183 | 1,019 | 533 | 767 | 1,674 | 1,014 | 411 | 1,858 |
| Transfer administration | 137 | 9 | 705 | 529 | 504 | 543 | 539 | 318 | 172 | 1,058 |
| Total | 1,742 | 440 | 6,156 | 4,741 | 4,833 | 5,178 | 6,583 | 3,850 | 1,843 | 6,077 |
Losses | ||||||||||
| Net cost of program and nonprogram activities and services | -3,741 | -6,140 | -592 | -3,645 | -3,555 | -4,572 | -4,383 | -3,039 | -1,406 | -3,290 |
| Net gain or loss(net present value) | -1,999 | -5,700 | 5,564 | 1,096 | 1,278 | 606 | 2,200 | 811 | 437 | 2,787 |
| Return to budget per net dollar invested in program and nonprogram activities and servicesb | 0.47 | 0.07 | 10.40 | 1.30 | 1.36 | 1.13 | 1.50 | 1.27 | 1.31 | 1.85 |
With a high school diploma or GED | ||||||||||
Gains | ||||||||||
| Payroll taxesa | 466 | 453 | 72 | 103 | 421 | 169 | 198 | 178 | 685 | |
| Income and sales tax | 338 | 143 | -335 | -138 | 82 | -141 | -104 | 109 | 42 | |
| Welfare payments | 1,014 | 1,034 | 2,359 | 1,572 | 2,539 | 1,124 | 969 | 576 | 2,939 | |
| Food Stamps | 732 | 378 | 513 | 159 | 878 | 759 | 587 | 238 | 1,132 | |
| Medicaid | 1,290 | 964 | 906 | 598 | 481 | 713 | 840 | 211 | 3,066 | |
| Transfer administration | 296 | 231 | 502 | 306 | 431 | 269 | 231 | 139 | 1,390 | |
| Total | 4,137 | 3,204 | 4,016 | 2,599 | 4,832 | 2,893 | 2,721 | 1,452 | 9,253 | |
Losses | ||||||||||
| Net cost of program and nonprogram activities and services | -4,209 | -5,073 | -2,177 | -3,972 | -2,775 | -4,305 | -4,096 | -2,488 | -2,948 | |
| Net gain or loss (net present value) | -72 | -1,869 | 1,839 | -1,373 | 2,057 | -1,412 | -1,375 | -1,036 | 6,305 | |
| Return to budget per net dollar invested in program and nonprogram activities and servicesb | 0.98 | 0.63 | 1.84 | 0.65 | 1.74 | 0.67 | 0.66 | 0.58 | 3.14 | |
| SOURCES: See Table 13.8. NOTES: Estimates reflect discounting and adjustment for inflation. Estimates were regression-adjusted using ordinary least squares, controlling for pre-random assignment characteristics of sample members. Rounding may cause slight discrepancies in calculating sums and differences. Tests of statistical significance were not performed. a Payroll taxes include employer- and employee-paid Social Security and Medicare taxes. b The return to budget per net dollar invested is computed by dividing total savings and tax increases by the net cost of activities and services. | ||||||||||
Sensitivity of the Results
As is common in cost-benefit analyses, assumptions were made in producing the estimates in this analysis. Owing to data limitations, several major assumptions had to be made about the levels of participation in follow-up years 3 to 5, as well as the proportion of participation that was considered in-program or out-of-program. The Five-Year Client Survey asked about participation at any point during the follow-up period, as well as specifically about participation in the last year of the follow-up period. This information was put together with data from the Two-Year Client Survey, which provided more detailed information about participation during the two-year period following program entry, to classify the likelihood of having participated during follow-up years 3 and 4. For the cost-benefit analysis, sample members were considered to have participated during years 3 and 4 if there was a good chance that they could have participated during this period.
In addition, the available data could not provide details on whether participation took place as part of the program or as a self-initiated activity. Therefore, in the cost-benefit analysis, it was assumed that the proportion of in-program participation increased with the number of months of welfare receipt during the period in question. This was the best approximation of the participation patterns seen in the detailed two-year data used in earlier reports.
To measure the effect of these two assumptions, a sensitivity test was conducted (see the following table). In this analysis, the alternative assumptions were that only those with the highest likelihood of having participated in years 3 and 4 did so, and all participation was evenly divided between "in-program" and "out-of-program" for those who received welfare for a portion of the period.(25) The resulting gross and net costs, as well as the net present value from the perspective of government budgets, are presented below. (The welfare perspective is unaffected by program costs.)
Site and Program | Gross Cost | Net Cost ($) | From the Government Budget Perspective | ||
|---|---|---|---|---|---|
| Program Group ($) | Control Group ($) | Net Gain or Loss (NPV) ($) | Return to Budget per Net Dollar Invested in Program and Nonprogram Activities and Services ($) | ||
| Atlanta LFA | 6,432 | 3,006 | 3,426 | -222 | 0.94 |
| Atlanta HCD | 7,843 | 3,006 | 4,837 | -2,616 | 0.46 |
| Grand Rapids LFA | 8,311 | 6,513 | 1,798 | 3,097 | 2.72 |
| Grand Rapids HCD | 9,993 | 6,513 | 3,480 | -15 | 1.00 |
| Riverside LFA | 5,280 | 2,538 | 2,742 | 2,123 | 1.77 |
| Columbus Integrated | 6,158 | 2,433 | 3,724 | 703 | 1.19 |
| Columbus Traditional | 5,782 | 2,433 | 3,348 | -222 | 0.93 |
| Detroit | 5,478 | 3,751 | 1,727 | -3 | 1.00 |
| Portland | 8,456 | 5,884 | 2,572 | 5,528 | 3.15 |
Using these alternative assumptions, gross costs were lower by 5 percent (Portland) to 13 percent (Riverside LFA) for program group members and by 3 percent (Atlanta LFA and HCD and Portland) to 7 percent (Riverside LFA) for control group members. The alternative assumptions resulted in reductions in net costs ranging from 8 percent (Grand Rapids HCD) to 17 percent (Riverside LFA).
As these results show, the point estimates of gross and net costs are indeed sensitive to the assumptions made. However, using these lower cost estimates, the same conclusions were drawn with regard to the ranking of the various sites overall and between program approaches in the NEWWS Evaluation. In addition, the comparisons with the GAIN programs also remain largely unaffected. In the benefit-cost analysis, the lower cost estimates result in very few changes to the bottom line for government budgets: Although there were changes in the magnitude of gains and losses, none of the estimates changed direction. In the sites where there were small losses to government budgets using the original assumptions, the losses are smaller yet, and these programs could be considered to have reached the "break-even" point.
Conclusions
The costs of services were relatively high everywhere. This was true for controls as well. Gross costs for employment-focused programs were similar to those for education-focused programs. However, net costs were lower for employment-focused programs. Similarly, LFA programs continued to have lower costs than HCD programs.
The NEWWS programs mostly left sample members worse off. Where there were gains to the welfare sample, they were very small. In half of the programs, the average earnings gains over the five-year period were smaller than reductions in welfare and Food Stamps. In Atlanta, which had the lowest welfare benefit levels, program group members had less to lose. As a result, both programs in this site produced earnings gains that were greater than the loss in combined welfare, Food Stamp, and Medicaid benefits. Detroit was the only other program to produce earnings gains greater than its reductions in transfer benefits. However, it bears repeating that the gains were very small and could easily have become losses had leisure and out-of-pocket work expenditures been included.
Except for Portland and Grand Rapids LFA, sample members without a high school diploma or GED at random assignment fared worse than those with these credentials regardless of the program approach. However, even among those who entered the evaluation with a high school diploma or GED, programs mostly left sample members worse off.
Although earnings increases were too small to cover benefit losses for program group members, in most programs the savings to government budgets were not large enough to cover their investments. The three programs that produced gains to government budgets were all employment-focused and had among the lowest net costs. Similar results were seen for sample members who entered the programs with a high school diploma or GED. That is, within this subgroup, the four employment-focused programs broke even or produced gains to government budgets, and all of the education-focused programs produced losses. For the subgroup without these credentials, all programs, except for the two in Atlanta, produced gains to government budgets. In general, the greater investment of education-focused programs did not pay off for government budgets or for program group members.
For the full samples, no program produced gains to both the welfare sample and government budgets. This was true for the educational attainment subgroups as well, except for nongraduates in Portland: Small gains ($1,084 over five years) accrued to nongraduates in that program, and the government budget experienced a moderate gain of $2,787. (Detroit produced a small gain to the government budget but broke even from the perspective of the welfare sample.)
1. See Hamilton et al., 1997; Scrivener et al., 1998; Farrell, 2000; Storto et al., 2000; and Scrivener and Walter, 2001.
2. See the two-year reports mentioned in footnote 1 for more detailed information on the calculation of unit costs.
3. As noted in Chapter 2, unlike those in other programs in this analysis, Columbus sample members were randomly assigned to the two programs prior to an orientation. These differences in sample composition are reflected in the cost and benefit outcomes.
4. A unit cost for basic education in Detroit is not available. Table 13.2 presents an "education and training" unit cost for Detroit, which combines vocational training, post-secondary education, and basic education. (See footnote "e" on Table 13.2.)
5. The Portland, Grand Rapids, and Detroit programs required welfare recipients with children as young as age 1 to participate in the NEWWS program, whereas the requirement extended only to recipients with children as young as age 3 in the other programs. The Grand Rapids and Detroit programs did not have particularly high child care costs.
6. A comparison of child care costs for the Columbus Integrated and Traditional programs does not support the idea that the case management approach alone was responsible for this result.
7. The Two-Year and Five-Year Client Survey data, along with welfare payment records, were used to estimate participation in employment-related activities that sample members took part in when they were not receiving welfare.
8. In Riccio, Friedlander, and Freedman, 1994, the GAIN costs were presented in 1993 dollars and were as follows: Alameda, $6,977; and Los Angeles, $6,402.
9. A portion of job search participation that occurred after the embargoes were lifted was considered program-related. The proportion of participation that was considered in-program was estimated based on the number of months that sample members received welfare in the post-embargo period. All participation in education and training activities was considered to be self-initiated.
10. The Riverside comparison includes only HCD and LFA program group members without a high school diploma or GED at random assignment.
11. Many of the techniques were originally developed for the benefit-cost analysis conducted as part of the MDRC Demonstration of State Work/Welfare Initiatives (for additional information, see Long and Knox, 1985). In this report the description of the analytical approach was adapted from previous MDRC reports (Riccio, Friedlander, and Freedman, 1994; Kemple, Friedlander, and Fellerath, 1995; Miller et al., 2000; and Bloom et al., 2000a).
12. In programs such as these, particularly education-focused programs, many costs are incurred early, when welfare receipt is heaviest; however, many benefits, such as earnings gains, continue to be realized in later years. Simply comparing the nominal dollar value of program costs with benefits over multiple years would be problematic because a dollar's value is greater in the present than in the future: A dollar available today, to either sample members or the government, can be invested and may produce income over time, making it worth more than a dollar available in the future. In order to make a fair comparison between benefits and costs over multiple years, it is essential to determine their value at a common point in time for example, the present. In benefit-cost analyses, this is often accomplished by discounting, a method for reducing the value of benefits and costs accrued in later years relative to benefits and costs accrued in early years. In this analysis, the end of the first year following random assignment was used as the comparison point for the investment period. In other words, gains that were accrued after that point were discounted to reflect their value at the end of year 1. In calculating these discounted values, it was assumed that a dollar invested at the end of year 1 would earn a real rate of return of 5 percent annually (this assumption has been used in other MDRC benefit-cost analyses).
13. Some work-related expenditures were paid by the welfare agencies and are reflected as a cost to government budgets in the support services line.
14. The presentation of benefit-cost results in this report was adapted from previous MDRC reports (Miller et al., 2000; Bloom et al., 2000a).
15. Data limitations did not allow for an accounting of out-of-pocket costs paid by sample members for health care coverage.
16. These rates were based on published information on employers' compensation costs for 1995 from the U.S. Department of Labor, Bureau of Labor Statistics.
17. This analysis does not account for out-of-pocket child care costs. With the exception of the Portland program, no out-of-pocket cost differences were found between program and control group members in the programs for which Five-Year Client Survey data were available.
18. The federal Earned Income Tax Credit is a credit against federal income taxes for taxpayers with annual earnings below a certain level. For 1995, taxpayers with earnings up to $26,673 were eligible for the EITC. Not all eligible taxpayers receive the EITC. As has been the practice in earlier benefit-cost analyses performed by MDRC, the EITC "take-up" rate was estimated at 80 percent.
19. Income from earnings was used in calculating federal and state income taxes. Income from earnings and welfare benefits was used in calculating sales and excise taxes. Sales and excise tax rates were based on information from state tax boards.
20. This analysis does not include an accounting of public health care coverage for children through programs for low-income families, such as the Child Health Insurance Program.
21. Take-up rates for programs without Five-Year Client Survey data were estimated from those with these data.
22. Average statewide Medicaid costs per eligible month were calculated using 1995 data from the Health Care Financing Administration Web site (www.hcfa.gov).
23. The costs of administering welfare, Food Stamp, and Medicaid benefits were estimated using statewide administrative cost data.
24. One of the hypotheses of the case management experiment in Columbus was that integrated case management would allow more efficient delivery of services. As discussed in Scrivener and Walter, 2001, integrated case managers reported seeing such benefits. This analysis assumes that the costs of administering benefits were the same for both programs. Because the five-year administration net costs were relatively low ($379 for the Integrated program and $265 for the Traditional program), even if the Integrated program had realized some savings in this area, the overall cost of the program would still have been slightly higher than the cost for the Traditional program. That is, the combined net operating and transfer administration costs for the Traditional program were $4,037, about $150 less than the net operating cost of the Integrated program.
25. Both scenarios counted all participation as "in-program" for those who received welfare for the full period and all participation as "out-of-program" for those who did not receive any welfare during the entire period.
Appendices
Appendix A: Notes for Tables and Figures
Notes for Tables and Figures Displaying Impacts
Calculated with Administrative Records Data
- Estimates were regression-adjusted using ordinary least squares, controlling for pre-random assignment characteristics of sample members.
- "Percentage change" equals 100 times "difference" divided by "control group."
- Rounding may cause slight discrepancies in calculating sums and differences.
- A two-tailed t-test was applied to differences between outcomes for the program and control groups. Statistical significance levels are indicated as: * = 10 percent; ** = 5 percent; and *** = 1 percent.
- Riverside limited enrollment in its HCD program to individuals determined by program regulations to need basic education because they lacked a high school diploma or GED certificate, attained low scores on a reading or math test administered at program entry, or had limited proficiency in English. As a result, control group means differ for the Riverside LFA and HCD programs.
- If outcomes are shown in italics, differences between program group members and control group members are not true experimental comparisons; statistical tests were not performed.
- The quarter of random assignment refers to the calendar quarter in which random assignment occurred. Because quarter 1, the quarter of random assignment, may contain some earnings from the period prior to random assignment, it is excluded from follow-up measures. Thus, "year 1" is quarters 2 through 5, "year 2" is quarters 6 through 9, "year 3" is quarters 10 through 13, "year 4" is quarters 14 through 17, and "year 5" is quarters 18 through 21.
- N/A = not applicable.
Appendix A.2
Notes for Tables and Figures Displaying Impacts
Calculated with Responses to the Two-Year Client Survey, Five-Year Client Survey, Child Outcomes Study, and Teacher Survey
- Measures for program and control group members represent weighted averages. In all sites, certain subgroups were overrepresented (for research purposes) among those chosen to be surveyed. Members of the survey samples are weighted to replicate the proportion of program and control group members in the full impact sample.
- The Five-Year Client Survey sample includes 434 respondents who were not interviewed for the Two-Year Client Survey. Measures calculated from responses to both surveys exclude these sample members.
- Estimates were regression-adjusted using ordinary least squares, controlling for pre-random assignment characteristics of sample members.
- "Percentage change" equals 100 times "difference" divided by "control group."
- Rounding may cause slight discrepancies in calculating sums and differences.
- A two-tailed t-test was applied to differences between outcomes for the program and control groups. Statistical significance levels are indicated as: * = 10 percent; ** = 5 percent; and *** = 1 percent.
- Riverside limited enrollment in its HCD program to individuals determined by program regulations to need basic education because they lacked a high school diploma or GED certificate, attained low scores on a reading or math test administered at program entry, or had limited proficiency in English. As a result, control group means differ for the Riverside LFA and HCD programs.
- If outcomes are shown in italics, differences between program group members and control group members are not true experimental comparisons; statistical tests were not performed.
- In Chapters 11 and 12, "effect size" equals "difference" divided by the standard deviation (not shown in tables) in the "control group." If measures are presented as percentages, "effect size" equals "difference" divided by 100, then divided by the standard deviation in the "control group."
Appendix B: Supplementary Tables to Chapter 3
| Site and Program | Sample Size | Program Group (%) | Control Group (%) | Difference (Impact) | Percentage Change (%) |
|---|---|---|---|---|---|
| Basic education | |||||
| Atlanta Labor Force Attachment | 1,071 | 20.4 | 11.4 | 9.1 *** | 79.8 |
| Atlanta Human Capital Development | 1,146 | 29.9 | 11.4 | 18.5 *** | 163.0 |
| Grand Rapids Labor Force Attachment | 1,097 | 20.0 | 18.9 | 1.2 | 6.3 |
| Grand Rapids Human Capital Development | 1,109 | 32.1 | 18.9 | 13.2 *** | 70.2 |
| Riverside Labor Force Attachment | 1,219 | 17.1 | 17.9 | -0.8 | -4.5 |
| Lacked high school diploma or basic skills | 657 | 24.9 | 28.0 | -3.2 | -11.3 |
| Riverside Human Capital Development | 778 | 59.0 | 28.0 | 31.0 *** | 110.6 |
| Portland | 504 | 27.2 | 20.9 | 6.3 * | 30.2 |
| Post-secondary education | |||||
| Atlanta Labor Force Attachment | 1,071 | 7.5 | 8.7 | -1.2 | -13.5 |
| Atlanta Human Capital Development | 1,146 | 11.5 | 8.7 | 2.8 | 31.7 |
| Grand Rapids Labor Force Attachment | 1,097 | 27.0 | 26.2 | 0.7 | 2.8 |
| Grand Rapids Human Capital Development | 1,109 | 29.6 | 26.2 | 3.3 | 12.7 |
| Riverside Labor Force Attachment | 1,219 | 24.5 | 25.0 | -0.5 | -1.9 |
| Lacked high school diploma or basic skills | 657 | 12.6 | 15.5 | -2.9 | -18.7 |
| Riverside Human Capital Development | 778 | 13.1 | 15.5 | -2.4 | -15.2 |
| Portland | 504 | 42.4 | 26.0 | 16.4 *** | 63.0 |
| Vocational training | |||||
| Atlanta Labor Force Attachment | 1,071 | 23.8 | 23.2 | 0.6 | 2.7 |
| Atlanta Human Capital Development | 1,146 | 33.0 | 23.2 | 9.7 *** | 42.0 |
| Grand Rapids Labor Force Attachment | 1,097 | 23.3 | 24.2 | -0.8 | -3.5 |
| Grand Rapids Human Capital Development | 1,109 | 28.5 | 24.2 | 4.3 | 17.7 |
| Riverside Labor Force Attachment | 1,219 | 23.7 | 24.5 | -0.8 | -3.4 |
| Lacked high school diploma or basic skills | 657 | 21.5 | 20.9 | 0.6 | 2.8 |
| Riverside Human Capital Development | 778 | 21.2 | 20.9 | 0.3 | 1.4 |
| Portland | 504 | 25.3 | 28.7 | -3.4 | -11.8 |
| Work experience | |||||
| Atlanta Labor Force Attachment | 1,071 | 17.2 | 7.6 | 9.6 *** | 126.3 |
| Atlanta Human Capital Development | 1,146 | 12.2 | 7.6 | 4.6 ** | 61.2 |
| Grand Rapids Labor Force Attachment | 1,097 | 10.5 | 5.2 | 5.3 *** | 101.1 |
| Grand Rapids Human Capital Development | 1,109 | 7.6 | 5.2 | 2.4 | 45.3 |
| Riverside Labor Force Attachment | 1,219 | 7.8 | 3.2 | 4.6 *** | 147.1 |
| Lacked high school diploma or basic skills | 657 | 7.3 | 2.2 | 5.2 *** | 235.9 |
| Riverside Human Capital Development | 778 | 6.3 | 2.2 | 4.2 ** | 190.0 |
| Portland | 504 | 14.7 | 13.7 | 1.0 | 7.1 |
| SOURCE: MDRC calculations from the Five-Year Client Survey. NOTES: See Appendix A.2. |
|||||
|
Site and Program |
Sample Size | Program Group (%) | Control Group (%) | Difference (Impact) | Percentage Change (%) |
|---|---|---|---|---|---|
|
Basic education |
|||||
| Atlanta Labor Force Attachment | 407 | 34.5 | 18.5 | 16.0 *** | 86.6 |
| Atlanta Human Capital Development | 437 | 59.8 | 18.5 | 41.3 *** | 223.1 |
| Grand Rapids Labor Force Attachment | 434 | 44.8 | 41.8 | 3.0 | 7.2 |
| Grand Rapids Human Capital Development | 468 | 68.8 | 41.8 | 27.0 *** | 64.8 |
| Riverside Labor Force Attachment | 657 | 24.9 | 28.0 | -3.2 | -11.3 |
| Riverside Human Capital Development | 778 | 59.0 | 28.0 | 31.0 *** | 110.6 |
| Portland | 163 | 59.3 | 49.5 | 9.8 | 19.7 |
|
Post-secondary education |
|||||
| Atlanta Labor Force Attachment | 407 | 1.2 | 0.8 | 0.3 | 37.1 |
| Atlanta Human Capital Development | 437 | 2.5 | 0.8 | 1.6 | 193.9 |
| Grand Rapids Labor Force Attachment | 434 | 9.3 | 6.4 | 2.9 | 45.8 |
| Grand Rapids Human Capital Development | 468 | 11.1 | 6.4 | 4.8 * | 74.5 |
| Riverside Labor Force Attachment | 657 | 12.6 | 15.5 | -2.9 | -18.7 |
| Riverside Human Capital Development | 778 | 13.1 | 15.5 | -2.4 | -15.2 |
| Portland | 163 | 21.6 | 13.8 | 7.8 | 56.9 |
|
Vocational training |
|||||
| Atlanta Labor Force Attachment | 407 | 14.0 | 16.3 | -2.3 | -14.3 |
| Atlanta Human Capital Development | 437 | 15.2 | 16.3 | -1.1 | -6.8 |
| Grand Rapids Labor Force Attachment | 434 | 19.9 | 18.9 | 1.0 | 5.2 |
| Grand Rapids Human Capital Development | 468 | 20.6 | 18.9 | 1.7 | 9.0 |
| Riverside Labor Force Attachment | 657 | 21.5 | 20.9 | 0.6 | 2.8 |
| Riverside Human Capital Development | 778 | 21.2 | 20.9 | 0.3 | 1.4 |
| Portland | 163 | 20.4 | 29.6 | -9.2 | -31.2 |
|
Work experience |
|||||
| Atlanta Labor Force Attachment | 407 | 14.5 | 7.2 | 7.3 ** | 101.2 |
| Atlanta Human Capital Development | 437 | 8.4 | 7.2 | 1.2 | 16.1 |
| Grand Rapids Labor Force Attachment | 434 | 9.7 | 5.5 | 4.1 * | 74.9 |
| Grand Rapids Human Capital Development | 468 | 4.7 | 5.5 | -0.8 | -15.3 |
| Riverside Labor Force Attachment | 657 | 7.3 | 2.2 | 5.2 *** | 235.9 |
| Riverside Human Capital Development | 778 | 6.3 | 2.2 | 4.2 ** | 190.0 |
| Portland | 163 | 17.8 | 7.9 | 9.9 | 125.3 |
| SOURCE: MDRC calculations from the Five-Year Client Survey. NOTES: See Appendix A.2. |
|||||
| Site and Program | Sample Size | Program Group (%) | Control Group (%) | Difference (Impact) | Percentage Change (%) |
|---|---|---|---|---|---|
| Basic education | |||||
| Atlanta Labor Force Attachment | 664 | 11.4 | 6.9 | 4.5 ** | 65.1 |
| Atlanta Human Capital Development | 709 | 11.1 | 6.9 | 4.1 * | 59.8 |
| Grand Rapids Labor Force Attachment | 663 | 6.3 | 6.6 | -0.2 | -3.4 |
| Grand Rapids Human Capital Development | 641 | 12.9 | 6.6 | 6.4 *** | 97.3 |
| Riverside Labor Force Attachment | 562 | 6.7 | 4.7 | 2.0 | 42.7 |
| Portland | 334 | 12.2 | 6.8 | 5.4 | 79.7 |
| Post-secondary education | |||||
| Atlanta Labor Force Attachment | 664 | 11.5 | 13.7 | -2.2 | -16.1 |
| Atlanta Human Capital Development | 709 | 16.8 | 13.7 | 3.1 | 22.7 |
| Grand Rapids Labor Force Attachment | 663 | 36.7 | 37.1 | -0.4 | -1.1 |
| Grand Rapids Human Capital Development | 641 | 39.4 | 37.1 | 2.3 | 6.1 |
| Riverside Labor Force Attachment | 562 | 40.5 | 37.7 | 2.8 | 7.6 |
| Portland | 334 | 52.6 | 31.7 | 20.9 *** | 66.2 |
| Vocational training | |||||
| Atlanta Labor Force Attachment | 664 | 30.7 | 27.1 | 3.7 | 13.6 |
| Atlanta Human Capital Development | 709 | 43.7 | 27.1 | 16.6 *** | 61.5 |
| Grand Rapids Labor Force Attachment | 663 | 25.4 | 27.1 | -1.6 | -6.1 |
| Grand Rapids Human Capital Development | 641 | 32.5 | 27.1 | 5.4 | 20.0 |
| Riverside Labor Force Attachment | 562 | 26.6 | 29.2 | -2.6 | -8.8 |
| Portland | 334 | 27.7 | 29.1 | -1.4 | -4.8 |
| Work experience | |||||
| Atlanta Labor Force Attachment | 664 | 18.7 | 7.8 | 10.9 *** | 140.7 |
| Atlanta Human Capital Development | 709 | 14.8 | 7.8 | 7.0 *** | 90.2 |
| Grand Rapids Labor Force Attachment | 663 | 11.0 | 4.9 | 6.1 *** | 124.3 |
| Grand Rapids Human Capital Development | 641 | 9.2 | 4.9 | 4.3 ** | 88.2 |
| Riverside Labor Force Attachment | 562 | 8.4 | 4.4 | 4.0 * | 91.0 |
| Portland | 334 | 12.9 | 15.7 | -2.8 0.0 | -17.7 |
| SOURCE: MDRC calculations from the Five-Year Client Survey. NOTES: See Appendix A.2. |
|||||
Appendix C: Supplementary Tables to Chapter 4
| Site and Program | Program Group (%) | Control Group (%) | Difference (Impact) | Percentage Change (%) |
|---|---|---|---|---|
| Atlanta Labor Force Attachment | ||||
| Year 1 | 49.1 | 44.4 | 4.7 *** | 10.6 |
| Year 2 | 55.3 | 49.8 | 5.5 *** | 11.0 |
| Year 3 | 58.4 | 54.1 | 4.4 ** | 8.1 |
| Year 4 | 61.5 | 58.5 | 3.0 * | 5.2 |
| Year 5 | 65.1 | 63.0 | 2.1 | 3.3 |
| Sample size | 1441.0 | 1497.0 | ||
| Atlanta Human Capital Development | ||||
| Year 1 | 45.5 | 44.4 | 1.1 | 2.5 |
| Year 2 | 54.3 | 49.8 | 4.5 *** | 9.0 |
| Year 3 | 58.2 | 54.1 | 4.2 ** | 7.7 |
| Year 4 | 60.3 | 58.5 | 1.8 | 3.1 |
| Year 5 | 63.6 | 63.0 | 0.6 | 1.0 |
| Sample size | 1495.0 | 1497.0 | ||
| Grand Rapids Labor Force Attachment | ||||
| Year 1 | 63.6 | 52.9 | 10.7 *** | 20.2 |
| Year 2 | 67.2 | 60.6 | 6.6 *** | 10.8 |
| Year 3 | 69.8 | 65.0 | 4.7 *** | 7.3 |
| Year 4 | 71.3 | 70.0 | 1.3 | 1.9 |
| Year 5 | 70.0 | 73.0 | -2.9 * | -4.0 |
| Sample size | 1557.0 | 1455.0 | ||
| Grand Rapids Human Capital Development | ||||
| Year 1 | 58.1 | 52.9 | 5.3 *** | 9.9 |
| Year 2 | 65.5 | 60.6 | 4.9 *** | 8.0 |
| Year 3 | 67.2 | 65.0 | 2.2 | 3.4 |
| Year 4 | 70.0 | 70.0 | 0.0 | 0.0 |
| Year 5 | 70.3 | 73.0 | -2.7 * | -3.7 |
| Sample size | 1542.0 | 1455.0 | ||
| Riverside Labor Force Attachment | ||||
| Year 1 | 50.6 | 34.1 | 16.5 *** | 48.5 |
| Year 2 | 45.4 | 37.0 | 8.4 *** | 22.6 |
| Year 3 | 44.5 | 39.8 | 4.7 *** | 11.9 |
| Year 4 | 45.8 | 41.9 | 3.9 *** | 9.2 |
| Year 5 | 48.7 | 44.5 | 4.2 *** | 9.5 |
| Sample size | 3384.0 | 3342.0 | ||
| Riverside Labor Force Attachment Lacked high school diploma or basic skills | ||||
| Year 1 | 45.8 | 29.1 | 16.7 *** | 57.4 |
| Year 2 | 40.9 | 31.9 | 9.0 *** | 28.3 |
| Year 3 | 40.8 | 34.8 | 6.1 *** | 17.5 |
| Year 4 | 41.5 | 37.1 | 4.4 *** | 11.9 |
| Year 5 | 45.8 | 39.9 | 5.9 *** | 14.8 |
| Sample size | 1586.0 | 1539.0 | ||
| Riverside Human Capital Development | ||||
| Year 1 | 35.7 | 29.1 | 6.6 *** | 22.9 |
| Year 2 | 37.7 | 31.9 | 5.8 *** | 18.2 |
| Year 3 | 39.3 | 34.8 | 4.6 *** | 13.2 |
| Year 4 | 40.9 | 37.1 | 3.9 ** | 10.4 |
| Year 5 | 44.9 | 39.9 | 5.0 *** | 12.5 |
| Sample size | 1596.0 | 1539.0 | ||
| Columbus Integrated | ||||
| Year 1 | 60.0 | 60.1 | -0.1 | -0.2 |
| Year 2 | 65.2 | 62.9 | 2.3 * | 3.7 |
| Year 3 | 68.9 | 65.3 | 3.6 *** | 5.5 |
| Year 4 | 69.9 | 67.7 | 2.2 * | 3.2 |
| Year 5 | 69.1 | 68.8 | 0.3 | 0.5 |
| Sample size | 2513.0 | 2159.0 | ||
| Columbus Traditional | ||||
| Year 1 | 59.9 | 60.1 | -0.1 | -0.2 |
| Year 2 | 64.5 | 62.9 | 1.6 | 2.6 |
| Year 3 | 67.9 | 65.3 | 2.6 ** | 3.9 |
| Year 4 | 68.9 | 67.7 | 1.2 | 1.8 |
| Year 5 | 69.3 | 68.8 | 0.5 | 0.7 |
| Sample size | 2570.0 | 2159.0 | ||
| Detroit | ||||
| Year 1 | 41.2 | 40.6 | 0.6 | 1.6 |
| Year 2 | 54.2 | 51.5 | 2.6 * | 5.1 |
| Year 3 | 62.3 | 59.0 | 3.3 ** | 5.5 |
| Year 4 | 66.4 | 67.5 | -1.1 | -1.7 |
| Year 5 | 68.8 | 68.8 | -0.0 | -0.1 |
| Sample size | 2226.0 | 2233.0 | ||
| Oklahoma City | ||||
| Year 1 | 50.9 | 51.6 | -0.7 | -1.4 |
| Year 2 | 51.0 | 51.6 | -0.6 | -1.2 |
| Year 3 | 51.4 | 52.1 | -0.7 | -1.4 |
| Year 4 | 52.9 | 52.7 | 0.3 | 0.5 |
| Year 5 | 53.2 | 54.2 | -1.0 | -1.8 |
| Sample size | 4309.0 | 4368.0 | ||
| Portland | ||||
| Year 1 | 58.5 | 49.8 | 8.6 *** | 17.3 |
| Year 2 | 62.3 | 51.4 | 10.9 *** | 21.2 |
| Year 3 | 64.1 | 52.0 | 12.1 *** | 23.2 |
| Year 4 | 64.2 | 55.8 | 8.5 *** | 15.2 |
| Year 5 | 62.4 | 58.6 | 3.8 * | 6.4 |
| Sample size | 3529.0 | 499.0 | ||
| SOURCE: MDRC calculations from state and county administrative records. NOTES: See Appendix A.1. |
||||
| Site and Program | Program Group (%) | Control Group (%) | Difference (Impact) | Percentage Change (%) |
|---|---|---|---|---|
| Atlanta Labor Force Attachment | ||||
| Year 1 | 2,043 | 1,647 | 396*** | 24.0 |
| Year 2 | 3,228 | 2,578 | 650*** | 25.2 |
| Year 3 | 3,930 | 3,313 | 618*** | 18.7 |
| Year 4 | 4,815 | 4,255 | 560** | 13.2 |
| Year 5 | 5,821 | 5,586 | 235 | 4.2 |
| Sample size | 1,441 | 1,497 | ||
| Atlanta Human Capital Development | ||||
| Year 1 | 1,846 | 1,647 | 199* | 12.1 |
| Year 2 | 3,059 | 2,578 | 480*** | 18.6 |
| Year 3 | 3,912 | 3,313 | 599*** | 18.1 |
| Year 4 | 4,910 | 4,255 | 655*** | 15.4 |
| Year 5 | 5,671 | 5,586 | 84 | 1.5 |
| Sample size | 1,495 | 1,497 | ||
| Grand Rapids Labor Force Attachment | ||||
| Year 1 | 2,289 | 1,759 | 530*** | 30.2 |
| Year 2 | 3,385 | 2,874 | 511*** | 17.8 |
| Year 3 | 4,658 | 4,211 | 447** | 10.6 |
| Year 4 | 5,615 | 5,480 | 135 | 2.5 |
| Year 5 | 6,376 | 6,447 | -71 | -1.1 |
| Sample size | 1,557 | 1,455 | ||
| Grand Rapids Human Capital Development | ||||
| Year 1 | 1,869 | 1,759 | 110 | 6.3 |
| Year 2 | 3,354 | 2,874 | 479*** | 16.7 |
| Year 3 | 4,514 | 4,211 | 303 | 7.2 |
| Year 4 | 5,535 | 5,480 | 56 | 1.0 |
| Year 5 | 6,345 | 6,447 | -102 | -1.6 |
| Sample size | 1,542 | 1,455 | ||
| Riverside Labor Force Attachment | ||||
| Year 1 | 2,460 | 1,741 | 719*** | 41.3 |
| Year 2 | 3,039 | 2,482 | 557*** | 22.4 |
| Year 3 | 3,389 | 2,957 | 432*** | 14.6 |
| Year 4 | 3,867 | 3,554 | 313* | 8.8 |
| Year 5 | 4,683 | 4,155 | 528*** | 12.7 |
| Sample size | 3,384 | 3,342 | ||
| Riverside Labor Force Attachment | ||||
| Lacked high school diploma or basic skills | ||||
| Year 1 | 1,866 | 1,250 | 616*** | 49.3 |
| Year 2 | 2,270 | 1,886 | 384** | 20.4 |
| Year 3 | 2,541 | 2,150 | 391** | 18.2 |
| Year 4 | 2,882 | 2,601 | 281 | 10.8 |
| Year 5 | 3,634 | 3,025 | 608*** | 20.1 |
| Sample size | 1,586 | 1,539 | ||
| Riverside Human Capital Development | ||||
| Year 1 | 1,446 | 1,250 | 196* | 15.7 |
| Year 2 | 2,025 | 1,886 | 139 | 7.4 |
| Year 3 | 2,561 | 2,150 | 411** | 19.1 |
| Year 4 | 2,857 | 2,601 | 255 | 9.8 |
| Year 5 | 3,384 | 3,025 | 359 | 11.9 |
| Sample size | 1,596 | 1,539 | ||
| Columbus Integrated | ||||
| Year 1 | 2,994 | 2,914 | 80 | 2.8 |
| Year 2 | 4,578 | 3,982 | 595*** | 15.0 |
| Year 3 | 5,644 | 5,134 | 510*** | 9.9 |
| Year 4 | 6,722 | 6,260 | 463** | 7.4 |
| Year 5 | 7,682 | 7,276 | 406 | 5.6 |
| Sample size | 2,513 | 2,159 | ||
| Columbus Traditional | ||||
| Year 1 | 3,099 | 2,914 | 185 | 6.4 |
| Year 2 | 4,472 | 3,982 | 490*** | 12.3 |
| Year 3 | 5,459 | 5,134 | 325* | 6.3 |
| Year 4 | 6,557 | 6,260 | 298 | 4.8 |
| Year 5 | 7,389 | 7,276 | 113 | 1.6 |
| Sample size | 2,570 | 2,159 | ||
| Detroit | ||||
| Year 1 | 1,398 | 1,341 | 57 | 4.2 |
| Year 2 | 2,976 | 2,663 | 313** | 11.8 |
| Year 3 | 4,593 | 4,112 | 481** | 11.7 |
| Year 4 | 5,920 | 5,570 | 351 | 6.3 |
| Year 5 | 7,081 | 6,823 | 258 | 3.8 |
| Sample size | 2,226 | 2,233 | ||
| Oklahoma City | ||||
| Year 1 | 1,404 | 1,387 | 16 | 1.2 |
| Year 2 | 2,121 | 2,127 | -5 | -0.3 |
| Year 3 | 2,632 | 2,672 | -40 | -1.5 |
| Year 4 | 3,114 | 3,068 | 45 | 1.5 |
| Year 5 | 3,597 | 3,498 | 100 | 2.8 |
| Sample size | 4,309 | 4,368 | ||
| Portland | ||||
| Year 1 | 2,828 | 2,484 | 343* | 13.8 |
| Year 2 | 4,421 | 3,150 | 1,271*** | 40.4 |
| Year 3 | 5,491 | 4,050 | 1,441*** | 35.6 |
| Year 4 | 6,319 | 5,112 | 1,207*** | 23.6 |
| Year 5 | 6,982 | 6,095 | 887** | 14.6 |
| Sample size | 3,529 | 499 | ||
| SOURCE: MDRC calculations from state and county administrative records. NOTES: See Appendix A.1. |
||||
| Site and Program | Sample Size | Program Group | Control Group | Difference (Impact) | Percentage Change (%) |
|---|---|---|---|---|---|
| Ever employed (%) | |||||
| Atlanta Labor Force Attachment | 2,938 | 72.0 | 68.2 | 3.8** | 5.6 |
| Atlanta Human Capital Development | 2,992 | 71.3 | 68.2 | 3.0* | 4.4 |
| Grand Rapids Labor Force Attachment | 3,012 | 84.0 | 77.6 | 6.4*** | 8.2 |
| Grand Rapids Human Capital Development | 2,997 | 82.0 | 77.6 | 4.4*** | 5.6 |
| Riverside Labor Force Attachment | 6,726 | 66.1 | 53.8 | 12.3*** | 22.8 |
| Lacked high school diploma or basic skills | 3,125 | 61.5 | 47.1 | 14.3*** | 30.4 |
| Riverside Human Capital Development | 3,135 | 55.9 | 47.1 | 8.8*** | 18.7 |
| Columbus Integrated | 4,672 | 81.1 | 78.5 | 2.6** | 3.3 |
| Columbus Traditional | 4,729 | 80.7 | 78.5 | 2.2** | 2.8 |
| Detroit | 4,459 | 74.0 | 70.4 | 3.6 | 5.1 |
| Oklahoma City | 8,677 | 71.5 | 71.8 | -0.4 | -0.5 |
| Portland | 4,028 | 80.3 | 71.5 | 8.8*** | 12.2 |
| Average number of quarters employed (%) | |||||
| Atlanta Labor Force Attachment | 2,938 | 4.6 | 4.0 | 0.6*** | 14.0 |
| Atlanta Human Capital Development | 2,992 | 4.4 | 4.0 | 0.4*** | 9.7 |
| Grand Rapids Labor Force Attachment | 3,012 | 5.4 | 4.7 | 0.7*** | 15.7 |
| Grand Rapids Human Capital Development | 2,997 | 5.1 | 4.7 | 0.4*** | 8.5 |
| Riverside Labor Force Attachment | 6,726 | 3.9 | 3.0 | 0.9*** | 28.9 |
| Lacked high school diploma or basic skills | 3,125 | 3.4 | 2.5 | 0.9*** | 36.2 |
| Riverside Human Capital Development | 3,135 | 2.9 | 2.5 | 0.5*** | 18.2 |
| Columbus Integrated | 4,672 | 5.8 | 5.5 | 0.3*** | 5.3 |
| Columbus Traditional | 4,729 | 5.7 | 5.5 | 0.2** | 4.1 |
| Detroit | 4,459 | 4.1 | 3.9 | 0.2** | 6.1 |
| Oklahoma City | 8,677 | 3.7 | 3.8 | -0.1 | -2.9 |
| Portland | 4,028 | 5.4 | 4.2 | 1.1*** | 26.3 |
| Average total earnings ($) | |||||
| Atlanta Labor Force Attachment | 2,938 | 9,202 | 7,538 | 1,664*** | 22.1 |
| Atlanta Human Capital Development | 2,992 | 8,816 | 7,538 | 1,278*** | 17.0 |
| Grand Rapids Labor Force Attachment | 3,012 | 10,332 | 8,844 | 1,488*** | 16.8 |
| Grand Rapids Human Capital Development | 2,997 | 9,736 | 8,844 | 892*** | 10.1 |
| Riverside Labor Force Attachment | 6,726 | 8,888 | 7,180 | 1,708*** | 23.8 |
| Lacked high school diploma or basic skills | 3,125 | 6,678 | 5,286 | 1,392 | 26.3 |
| Riverside Human Capital Development | 3,135 | 6,032 | 5,286 | 746** | 14.1 |
| Columbus Integrated | 4,672 | 13,216 | 12,030 | 1,186** | 9.9 |
| Columbus Traditional | 4,729 | 13,030 | 12,030 | 1,000*** | 8.3 |
| Detroit | 4,459 | 8,967 | 8,115 | 851** | 10.5 |
| Oklahoma City | 8,677 | 6,157 | 6,186 | -29 | -0.5 |
| Portland | 4,028 | 12,740 | 9,684 | 3,055*** | 31.5 |
| SOURCE: MDRC calculations from state and county administrative records. NOTES: See Appendix A.1. |
|||||
Impacts on Employment and Earnings in the Last Quarter of Year 5
| Site and Program | Sample Size | Program Group | Control Group | Difference (Impact) | Percentage Change (%) |
|---|---|---|---|---|---|
| Ever employed (%) | |||||
| Atlanta Labor Force Attachment | 2,938 | 50.9 | 51.5 | -0.5 | -1.1 |
| Atlanta Human Capital Development | 2,992 | 51.3 | 51.5 | -0.2 | -0.4 |
| Grand Rapids Labor Force Attachment | 3,012 | 54.9 | 59.0 | -4.1** | -6.9 |
| Grand Rapids Human Capital Development | 2,997 | 56.7 | 59.0 | -2.3 | -4.0 |
| Riverside Labor Force Attachment | 6,726 | 38.4 | 35.5 | 2.9** | 8.1 |
| Lacked high school diploma or basic skills | 3,125 | 35.3 | 31.1 | 4.2** | 13.5 |
| Riverside Human Capital Development | 3,135 | 33.8 | 31.1 | 2.7* | 8.8 |
| Columbus Integrated | 4,672 | 54.4 | 55.0 | -0.6 | -1.2 |
| Columbus Traditional | 4,729 | 54.2 | 55.0 | -0.8 | -1.5 |
| Detroit | 4,459 | 55.0 | 55.2 | -0.2 | -0.4 |
| Oklahoma City | 8,677 | 37.3 | 38.2 | -0.9 | -2.4 |
| Portland | 4,028 | 50.1 | 44.7 | 5.4** | 12.2 |
| Total earnings ($) | |||||
| Atlanta Labor Force Attachment | 2,938 | 1,544 | 1,496 | 48 | 3.2 |
| Atlanta Human Capital Development | 2,992 | 1,469 | 1,496 | -26 | -1.7 |
| Grand Rapids Labor Force Attachment | 3,012 | 1,659 | 1,746 | -87 | -5.0 |
| Grand Rapids Human Capital Development | 2,997 | 1,675 | 1,746 | -71 | -4.1 |
| Riverside Labor Force Attachment | 6,726 | 1,236 | 1,113 | 123** | 11.0 |
| Lacked high school diploma or basic skills | 3,125 | 986 | 829 | 156** | 18.9 |
| Riverside Human Capital Development | 3,135 | 885 | 829 | 56 | 6.8 |
| Columbus Integrated | 4,672 | 2,008 | 1,914 | 94 | 4.9 |
| Columbus Traditional | 4,729 | 1,913 | 1,914 | -1 | -0.0 |
| Detroit | 4,459 | 1,847 | 1,849 | -2 | -0.1 |
| Oklahoma City | 8,677 | 934 | 939 | -5 | -0.5 |
| Portland | 4,028 | 1,793 | 1,555 | 238** | 15.3 |
| SOURCE: MDRC calculations from state and county administrative records. NOTES: See Appendix A.1. |
|||||
| Site and Program | Sample Size | Program Group (%) | Control Group (%) | Difference (Impact) | Percentage Change (%) |
|---|---|---|---|---|---|
| Ever employed in years 1 to 5 | |||||
| Atlanta Labor Force Attachment | 1,071 | 83.3 | 82.9 | 0.4 | 0.4 |
| Atlanta Human Capital Development | 1,146 | 81.2 | 82.9 | -1.7 | -2.0 |
| Grand Rapids Labor Force Attachment | 1,097 | 94.7 | 91.9 | 2.98* | 3.1 |
| Grand Rapids Human Capital Development | 1,109 | 93.6 | 91.9 | 1.7 | 1.9 |
| Riverside Labor Force Attachment | 1,219 | 85.7 | 80.8 | 4.9** | 6.0 |
| Lacked high school diploma or basic skills | 657 | 81.8 | 74.2 | 7.6** | 10.2 |
| Riverside Human Capital Development | 778 | 82.8 | 74.2 | 8.6*** | 11.6 |
| Portland | 504 | 93.5 | 93.3 | 0.2 | 0.2 |
| Employed at interview | |||||
| Atlanta Labor Force Attachment | 1,071 | 57.2 | 54.3 | 2.9 | 5.4 |
| Atlanta Human Capital Development | 1,146 | 52.9 | 54.3 | -1.4 | -2.5 |
| Grand Rapids Labor Force Attachment | 1,097 | 65.1 | 67.7 | -2.6 | -3.8 |
| Grand Rapids Human Capital Development | 1,109 | 66.2 | 67.7 | -1.5 | -2.2 |
| Riverside Labor Force Attachment | 1,219 | 55.0 | 48.9 | 6.1** | 12.4 |
| Lacked high school diploma or basic skills | 657 | 48.9 | 43.0 | 5.9 | 13.6 |
| Riverside Human Capital Development | 778 | 51.1 | 43.0 | 8.0** | 18.7 |
| Portland | 504 | 61.7 | 58.3 | 3.4 | 5.9 |
| Employed full time at interview | |||||
| Atlanta Labor Force Attachment | 1,071 | 47.2 | 45.5 | 1.7 | 3.8 |
| Atlanta Human Capital Development | 1,146 | 44.1 | 45.5 | -1.4 | -3.2 |
| Grand Rapids Labor Force Attachment | 1,097 | 53.5 | 54.3 | -0.8 | -1.5 |
| Grand Rapids Human Capital Development | 1,109 | 52.9 | 54.3 | -1.3 | -2.5 |
| Riverside Labor Force Attachment | 1,219 | 44.1 | 35.9 | 8.2*** | 22.7 |
| Lacked high school diploma or basic skills | 657 | 40.2 | 31.8 | 8.4** | 26.4 |
| Riverside Human Capital Development | 778 | 36.8 | 31.8 | 4.9 | 15.4 |
| Portland | 504 | 50.3 | 46.9 | 3.4 | 7.2 |
| Employed with employer-provided medical insurance at interview | |||||
| Atlanta Labor Force Attachment | 1,071 | 22.2 | 19.0 | 3.2 | 16.6 |
| Atlanta Human Capital Development | 1,146 | 21.2 | 19.0 | 2.2 | 11.5 |
| Grand Rapids Labor Force Attachment | 1,097 | 30.5 | 30.0 | 0.5 | 1.7 |
| Grand Rapids Human Capital Development | 1,109 | 30.4 | 30.0 | 0.4 | 1.5 |
| Riverside Labor Force Attachment | 1,219 | 17.5 | 15.0 | 2.5 | 16.6 |
| Lacked high school diploma or basic skills | 657 | 12.0 | 10.9 | 1.1 | 9.8 |
| Riverside Human Capital Development | 778 | 14.8 | 10.9 | 3.9 | 35.5 |
| Portland | 504 | 31.6 | 27.3 | 4.4 | 16.0 |
| SOURCE: MDRC calculations from state and county administrative records. NOTES: See Appendix A.1. |
|||||
| Site and Program | Sample Size | Program Group (%) | Control Group (%) | Difference (Impact) | Percentage Change (%) |
|---|---|---|---|---|---|
| Employed in 75 percent or more of quarters in years 3 to 5 | |||||
| Atlanta Labor Force Attachment | 2,938 | 37.0 | 32.5 | 4.5*** | 14.0 |
| Atlanta Human Capital Development | 2,992 | 35.6 | 32.5 | 3.1* | 9.5 |
| Grand Rapids Labor Force Attachment | 3,012 | 40.8 | 38.0 | 2.9* | 7.6 |
| Grand Rapids Human Capital Development | 2,997 | 39.8 | 38.0 | 1.8 | 4.8 |
| Riverside Labor Force Attachment | 6,726 | 23.7 | 20.6 | 3.2*** | 15.5 |
| Lacked high school diploma or basic skills | 3,125 | 20.0 | 16.2 | 3.8*** | 23.4 |
| Riverside Human Capital Development | 3,135 | 20.1 | 16.2 | 3.9*** | 24.0 |
| Columbus Integrated | 4,672 | 43.7 | 42.4 | 1.3 | 3.1 |
| Columbus Traditional | 4,729 | 43.4 | 42.4 | 1.0 | 2.2 |
| Detroit | 4,459 | 35.9 | 34.3 | 1.7 | 4.9 |
| Oklahoma City | 8,677 | 22.1 | 22.8 | -0.6 | -2.7 |
| Portland | 4,028 | 38.6 | 31.2 | 7.5*** | 23.9 |
| Increased average earnings from first to last measured quarters in years 1 to 5 | |||||
| Atlanta Labor Force Attachment | 2,938 | 40.4 | 38.6 | 1.8 | 4.7 |
| Atlanta Human Capital Development | 2,992 | 40.8 | 38.6 | 2.3 | 5.9 |
| Grand Rapids Labor Force Attachment | 3,012 | 46.8 | 46.0 | 0.8 | 1.6 |
| Grand Rapids Human Capital Development | 2,997 | 46.3 | 46.0 | 0.3 | 0.7 |
| Riverside Labor Force Attachment | 6,726 | 31.9 | 27.4 | 4.4*** | 16.2 |
| Lacked high school diploma or basic skills | 3,125 | 28.2 | 21.8 | 6.4*** | 29.4 |
| Riverside Human Capital Development | 3,135 | 26.9 | 21.8 | 5.1*** | 23.2 |
| Columbus Integrated | 4,672 | 48.9 | 46.6 | 2.3 | 4.9 |
| Columbus Traditional | 4,729 | 48.2 | 46.6 | 1.6 | 3.5 |
| Detroit | 4,459 | 42.0 | 42.3 | -0.3 | -0.8 |
| Oklahoma City | 8,677 | 30.4 | 30.0 | 0.4 | 1.2 |
| Portland | 4,028 | 45.2 | 38.3 | 6.9*** | 17.9 |
| SOURCE: MDRC calculations from state and county administrative records. NOTES: See Appendix A.1. |
|||||
Appendix D: Supplementary Tables to Chapter 5
| Site and Program | Sample Size | Program Group | Control Group | Difference (Impact) | Percentage Change (%) |
|---|---|---|---|---|---|
| Average number of months receiving welfare in years 1 to 3 | |||||
| Atlanta Labor Force Attachment | 2,938 | 24.6 | 26.3 | -1.8*** | -6.7 |
| Atlanta Human Capital Development | 2,992 | 25.1 | 26.3 | -1.2*** | -4.6 |
| Grand Rapids Labor Force Attachment | 3,012 | 20.1 | 23.2 | -3.1*** | -13.4 |
| Grand Rapids Human Capital Development | 2,997 | 21.3 | 23.2 | -1.9*** | -8.3 |
| Riverside Labor Force Attachment | 6,726 | 19.7 | 21.9 | -2.2*** | -9.9 |
| Lacked high school diploma or basic skills | 3,125 | 21.0 | 23.0 | -2.1*** | -8.9 |
| Riverside Human Capital Development | 3,135 | 21.3 | 23.0 | -1.7*** | -7.5 |
| Columbus Integrated | 4,672 | 18.9 | 21.5 | -2.6*** | -12.2 |
| Columbus Traditional | 4,729 | 19.8 | 21.5 | -1.7*** | -8.0 |
| Detroit | 4,459 | 26.2 | 27.3 | -1.1*** | -3.9 |
| Oklahoma City | 6,896 | 14.7 | 15.6 | -0.9*** | -5.8 |
| Portland | 4,028 | 16.3 | 20.1 | -3.8*** | -18.9 |
| Average total welfare payments received in years 1 to 3 ($) | |||||
| Atlanta Labor Force Attachment | 2,938 | 6,504 | 7,058 | -554*** | -7.9 |
| Atlanta Human Capital Development | 2,992 | 6,578 | 7,058 | -480*** | -6.8 |
| Grand Rapids Labor Force Attachment | 3,012 | 7,853 | 9,793 | -1,940*** | -19.8 |
| Grand Rapids Human Capital Development | 2,997 | 8,545 | 9,793 | -1,247*** | -12.7 |
| Riverside Labor Force Attachment | 6,726 | 11,175 | 13,068 | -1,893*** | -14.5 |
| Lacked high school diploma or basic skills | 3,125 | 12,122 | 14,164 | -2,042*** | -14.4 |
| Riverside Human Capital Development | 3,135 | 12,432 | 14,164 | -1,732*** | -12.2 |
| Columbus Integrated | 4,672 | 6,072 | 7,151 | -1,079*** | -15.1 |
| Columbus Traditional | 4,729 | 6,335 | 7,151 | -816*** | -11.4 |
| Detroit | 4,459 | 11,538 | 11,921 | -383** | -3.2 |
| Oklahoma City | 6,896 | 4,532 | 4,822 | -290*** | -6.0 |
| Portland | 4,028 | 7,270 | 9,179 | -1,910*** | -20.8 |
| SOURCE: MDRC calculations from state and county administrative records. NOTES: See Appendix A.1. |
|||||
|
Site and Program |
Program Group (%) | Control Group (%) | Difference (Impact) | Percentage Change (%) |
|---|---|---|---|---|
|
Atlanta Labor Force Attachment |
||||
| Year 1 | 78.2 | 82.7 | -4.5*** | -5.5 |
| Year 2 | 64.7 | 70.9 | -6.2*** | -8.7 |
| Year 3 | 56.1 | 61.2 | -5.1*** | -8.3 |
| Year 4 | 44.5 | 50.1 | -5.7*** | -11.3 |
| Year 5 | 32.6 | 36.5 | -3.9** | -10.8 |
| Sample size | 1441.0 | 1497.0 | ||
|
Atlanta Human Capital Development |
||||
| Year 1 | 80.8 | 82.7 | -2.0 | -2.4 |
| Year 2 | 66.2 | 70.9 | -4.8*** | -6.7 |
| Year 3 | 57.4 | 61.2 | -3.8** | -6.2 |
| Year 4 | 46.9 | 50.1 | -3.2* | -6.4 |
| Year 5 | 34.5 | 36.5 | -2.1 | -5.6 |
| Sample size | 1495.0 | 1497.0 | ||
|
Grand Rapids Labor Force Attachment |
||||
| Year 1 | 68.8 | 77.8 | -9.1*** | -11.6 |
| Year 2 | 53.5 | 60.8 | -7.3*** | -12.1 |
| Year 3 | 41.2 | 48.0 | -6.8*** | -14.2 |
| Year 4 | 32.5 | 38.0 | -5.6*** | -14.6 |
| Year 5 | 24.3 | 27.8 | -3.5** | -12.5 |
| Sample size | 1557.0 | 1455.0 | ||
|
Grand Rapids Human Capital Development |
||||
| Year 1 | 73.7 | 77.8 | -4.2*** | -5.4 |
| Year 2 | 54.3 | 60.8 | -6.4*** | -10.6 |
| Year 3 | 41.7 | 48.0 | -6.3*** | -13.1 |
| Year 4 | 32.5 | 38.0 | -5.6*** | -14.6 |
| Year 5 | 25.8 | 27.8 | -2.0 | -7.1 |
| Sample size | 1542.0 | 1455.0 | ||
|
Riverside Labor Force Attachment |
||||
| Year 1 | 62.9 | 69.6 | -6.7*** | -9.6 |
| Year 2 | 50.1 | 56.4 | -6.4*** | -11.3 |
| Year 3 | 42.7 | 47.6 | -4.8*** | -10.2 |
| Year 4 | 37.0 | 41.1 | -4.2*** | -10.1 |
| Year 5 | 30.2 | 34.6 | -4.4*** | -12.8 |
| Sample size | 3384.0 | 3342.0 | ||
|
Riverside Labor Force Attachment |
||||
| Year 1 | 66.4 | 72.3 | -6.0*** | -8.3 |
| Year 2 | 54.2 | 60.0 | -5.8*** | -9.7 |
| Year 3 | 47.1 | 52.1 | -5.0*** | -9.6 |
| Year 4 | 41.5 | 46.3 | -4.7*** | -10.2 |
| Year 5 | 34.5 | 39.0 | -4.5*** | -11.6 |
| Sample size | 1586.0 | 1539.0 | ||
|
Riverside Human Capital Development |
||||
| Year 1 | 68.2 | 72.3 | -4.1*** | -5.7 |
| Year 2 | 55.9 | 60.0 | -4.1** | -6.8 |
| Year 3 | 46.2 | 52.1 | -5.9*** | -11.4 |
| Year 4 | 38.7 | 46.3 | -7.5*** | -16.3 |
| Year 5 | 33.3 | 39.0 | -5.7*** | -14.7 |
| Sample size | 1596.0 | 1539.0 | ||
|
Columbus Integrated |
||||
| Year 1 | 68.0 | 72.5 | -4.5*** | -6.2 |
| Year 2 | 47.1 | 53.8 | -6.8*** | -12.5 |
| Year 3 | 33.2 | 40.3 | -7.1*** | -17.6 |
| Year 4 | 22.3 | 27.6 | -5.3*** | -19.3 |
| Year 5 | 12.7 | 16.5 | -3.8*** | -23.1 |
| Sample size | 2513.0 | 2159.0 | ||
|
Columbus Traditional |
||||
| Year 1 | 68.9 | 72.5 | -3.6*** | -5.0 |
| Year 2 | 49.3 | 53.8 | -4.6*** | -8.5 |
| Year 3 | 34.9 | 40.3 | -5.5*** | -13.5 |
| Year 4 | 24.7 | 27.6 | -3.0** | -10.7 |
| Year 5 | 14.1 | 16.5 | -2.4** | -14.3 |
| Sample size | 2570.0 | 2159.0 | ||
|
Detroit |
||||
| Year 1 | 85.4 | 86.5 | -1.1 | -1.2 |
| Year 2 | 70.1 | 73.7 | -3.6*** | -4.8 |
| Year 3 | 56.6 | 61.4 | -4.8*** | -7.9 |
| Year 4 | 45.1 | 48.1 | -3.0** | -6.3 |
| Year 5 | 35.0 | 35.2 | -0.2 | -0.5 |
| Sample size | 2226.0 | 2233.0 | ||
|
Oklahoma City |
||||
| Year 1 | 50.0 | 53.1 | -3.1*** | -5.9 |
| Year 2 | 39.1 | 41.0 | -1.9 | -4.6 |
| Year 3 | 30.3 | 32.7 | -2.4** | -7.3 |
| Year 4 | n/a | n/a | n/a | n/a |
| Year 5 | n/a | n/a | n/a | n/a |
| Sample size | 3430.0 | 3466.0 | ||
|
Portland |
||||
| Year 1 | 59.8 | 68.0 | -8.3*** | -12.2 |
| Year 2 | 40.6 | 53.3 | -12.7*** | -23.9 |
| Year 3 | 25.8 | 38.6 | -12.8*** | -33.2 |
| Year 4 | 16.5 | 22.5 | -6.0*** | -26.7 |
| Year 5 | 12.3 | 17.1 | -4.8*** | -28.2 |
| Sample size | 3529.0 | 499.0 | ||
| SOURCE: MDRC calculations from state and county administrative records. NOTES: See Appendix A.1. |
||||
| Site and Program | Sample Size | Program Group | Control Group | Difference (Impact) | Percentage Change (%) |
|---|---|---|---|---|---|
| Average number of months receiving Food Stamps in years 1 to 3 | |||||
| Atlanta Labor Force Attachment | 2,938 | 28.4 | 29.0 | -0.7* | -2.3 |
| Atlanta Human Capital Development | 2,992 | 28.4 | 29.0 | -0.6* | -2.2 |
| Grand Rapids Labor Force Attachment | 3,012 | 22.6 | 25.1 | -2.5*** | -10.0 |
| Grand Rapids Human Capital Development | 2,997 | 23.5 | 25.1 | -1.5*** | -6.1 |
| Riverside Labor Force Attachment | 6,726 | 17.6 | 20.0 | -2.5*** | -12.3 |
| Lacked high school diploma or basic skills | 3,125 | 18.9 | 21.2 | -2.3*** | -10.9 |
| Riverside Human Capital Development | 3,135 | 19.0 | 21.2 | -2.2*** | -10.6 |
| Columbus Integrated | 4,672 | 21.7 | 23.7 | -2.0*** | -8.5 |
| Columbus Traditional | 4,729 | 22.3 | 23.7 | -1.4*** | -5.9 |
| Detroit | 4,459 | 28.3 | 29.4 | -1.0*** | -3.5 |
| Oklahoma City | 6,896 | 19.6 | 20.0 | -0.4 | -1.9 |
| Portland | 4,028 | 21.4 | 23.5 | -2.0*** | -8.7 |
| Average total Food Stamps received in years 1 to 3 ($) | |||||
| Atlanta Labor Force Attachment | 2,938 | 7,165 | 7,371 | -207** | -2.8 |
| Atlanta Human Capital Development | 2,992 | 7,302 | 7,371 | -69 | -0.9 |
| Grand Rapids Labor Force Attachment | 3,012 | 4,680 | 5,092 | -412*** | -8.1 |
| Grand Rapids Human Capital Development | 2,997 | 4,858 | 5,092 | -234** | -4.6 |
| Riverside Labor Force Attachment | 6,726 | 3,322 | 3,875 | -553*** | -14.3 |
| Lacked high school diploma or basic skills | 3,125 | 3,652 | 4,218 | -566*** | -13.4 |
| Riverside Human Capital Development | 3,135 | 3,683 | 4,218 | -536*** | -12.7 |
| Columbus Integrated | 4,672 | 5,618 | 6,312 | -694*** | -11.0 |
| Columbus Traditional | 4,729 | 5,830 | 6,312 | -482*** | -7.6 |
| Detroit | 4,459 | 6,646 | 6,888 | -241*** | -3.5 |
| Oklahoma City | 6,896 | 4,887 | 4,988 | -100 | -2.0 |
| Portland | 4,028 | 5,182 | 5,822 | -640*** | -11.0 |
| SOURCE: MDRC calculations from state and county administrative records. NOTES: See Appendix A.1. |
|||||
|
Site and Program |
Sample Size | Program Group (%) | Control Group (%) | Difference (Impact) | Percentage Change (%) |
|---|---|---|---|---|---|
|
Ever received Food Stamps in last quarter of year 5 |
|||||
| Atlanta Labor Force Attachment | 2,938 | 51.6 | 54.7 | -3.1* | -5.7 |
| Atlanta Human Capital Development | 2,992 | 52.6 | 54.7 | -2.2 | -3.9 |
| Grand Rapids Labor Force Attachment | 3,012 | 27.4 | 29.8 | -2.4 | -8.0 |
| Grand Rapids Human Capital Development | 2,997 | 29.1 | 29.8 | -0.6 | -2.1 |
| Riverside Labor Force Attachment | 6,726 | 30.4 | 34.4 | -4.0*** | -11.6 |
| Lacked high school diploma or basic skills | 3,125 | 34.8 | 38.6 | -3.8** | -9.8 |
| Riverside Human Capital Development | 3,135 | 33.3 | 38.6 | -5.3*** | -13.8 |
| Columbus Integrated | 4,672 | 21.5 | 26.1 | -4.5*** | -17.4 |
| Columbus Traditional | 4,729 | 24.7 | 26.1 | -1.4 | -5.2 |
| Detroit | 4,459 | 35.6 | 36.1 | -0.5 | -1.4 |
| Oklahoma City | n/a | n/a | n/a | n/a | n/a |
| Portland | 4,028 | 30.9 | 34.0 | -3.1 | -9.1 |
|
Received Food Stamps but not welfare in last quarter of year 5 |
|||||
| Atlanta Labor Force Attachment | 1,929 | 22.3 | 22.7 | -0.4 | -1.8 |
| Atlanta Human Capital Development | 1,953 | 23.6 | 22.7 | 0.9 | 4.1 |
| Grand Rapids Labor Force Attachment | 1,087 | 8.8 | 7.3 | 1.5 | 20.7 |
| Grand Rapids Human Capital Development | 1,084 | 7.8 | 7.3 | 0.5 | 6.4 |
| Riverside Labor Force Attachment | 6,726 | 3.4 | 3.2 | 0.2 | 5.2 |
| Lacked high school diploma or basic skills | 3,125 | 3.8 | 3.1 | 0.8 | 25.0 |
| Riverside Human Capital Development | 3,135 | 3.8 | 3.1 | 0.8 | 24.7 |
| Columbus Integrated | 3,095 | 9.5 | 10.3 | -0.7 | -7.1 |
| Columbus Traditional | 3,100 | 11.1 | 10.3 | 0.8 | 8.0 |
| Detroit | 4,459 | 7.9 | 8.2 | -0.3 | -4.2 |
| Oklahoma City | n/a | n/a | n/a | n/a | n/a |
| Portland | 4,028 | 20.4 | 19.0 | 1.4 | 7.3 |
| SOURCE: MDRC calculations from state and county administrative records. NOTES: See Appendix A.1. |
|||||
Appendix E: Supplementary Tables to Chapter 6
| Site and Program | Program Group ($) | Control Group ($) | Difference (Impact) | Percentage Change (%) |
|---|---|---|---|---|
|
Atlanta Labor Force Attachment |
||||
| Year 1 | 7,409 | 7,188 | 221** | 3.1 |
| Year 2 | 7,888 | 7,531 | 357** | 4.7 |
| Year 3 | 8,271 | 7,928 | 343* | 4.3 |
| Year 4 | 8,587 | 8,364 | 223 | 2.7 |
| Year 5 | 8,983 | 8,976 | 7 | 0.1 |
| Sample size | 1,441 | 1,497 | ||
|
Atlanta Human Capital Development |
||||
| Year 1 | 7,264 | 7,188 | 76 | 1.1 |
| Year 2 | 7,834 | 7,531 | 303** | 4.0 |
| Year 3 | 8,283 | 7,928 | 355** | 4.5 |
| Year 4 | 8,805 | 8,364 | 441** | 5.3 |
| Year 5 | 8,933 | 8,976 | -42 | -0.5 |
| Sample size | 1,495 | 1,497 | ||
|
Grand Rapids Labor Force Attachment |
||||
| Year 1 | 7,743 | 8,037 | -294** | -3.7 |
| Year 2 | 7,723 | 7,987 | -264* | -3.3 |
| Year 3 | 8,220 | 8,391 | -171 | -2.0 |
| Year 4 | 8,485 | 8,807 | -323 | -3.7 |
| Year 5 | 8,569 | 8,950 | -381 | -4.3 |
| Sample size | 1,557 | 1,455 | ||
|
Grand Rapids Human Capital Development |
||||
| Year 1 | 7,777 | 8,037 | -260** | -3.2 |
| Year 2 | 7,905 | 7,987 | -82 | -1.0 |
| Year 3 | 8,165 | 8,391 | -227 | -2.7 |
| Year 4 | 8,413 | 8,807 | -395* | -4.5 |
| Year 5 | 8,666 | 8,950 | -284 | -3.2 |
| Sample size | 1,542 | 1,455 | ||
|
Riverside Labor Force Attachment |
||||
| Year 1 | 8,722 | 8,706 | 16 | 0.2 |
| Year 2 | 7,663 | 7,983 | -321** | -4.0 |
| Year 3 | 7,420 | 7,735 | -315** | -4.1 |
| Year 4 | 7,469 | 7,730 | -261 | -3.4 |
| Year 5 | 7,655 | 7,650 | 6 | 0.1 |
| Sample size | 3,384 | 3,342 | ||
|
Riverside Labor Force Attachment |
||||
| Year 1 | 8,510 | 8,635 | -125 | -1.4 |
| Year 2 | 7,342 | 7,865 | -523*** | -6.6 |
| Year 3 | 7,064 | 7,454 | -390* | -5.2 |
| Year 4 | 7,017 | 7,332 | -314 | -4.3 |
| Year 5 | 7,096 | 7,025 | 71 | 1.0 |
| Sample size | 1,586 | 1,539 | ||
|
Riverside Human Capital Development |
||||
| Year 1 | 8,249 | 8,635 | -386*** | -4.5 |
| Year 2 | 7,303 | 7,865 | -562*** | -7.2 |
| Year 3 | 6,997 | 7,454 | -457** | -6.1 |
| Year 4 | 6,768 | 7,332 | -564** | -7.7 |
| Year 5 | 6,607 | 7,025 | -418* | -6.0 |
| Sample size | 1,596 | 1,539 | ||
|
Columbus Integrated |
||||
| Year 1 | 8,543 | 8,932 | -389*** | -4.4 |
| Year 2 | 8,663 | 8,654 | 8 | 0.1 |
| Year 3 | 8,686 | 8,815 | -129 | -1.5 |
| Year 4 | 8,934 | 8,964 | -29 | -0.3 |
| Year 5 | 9,211 | 9,112 | 98 | 1.1 |
| Sample size | 2,513 | 2,159 | ||
|
Columbus Traditional |
||||
| Year 1 | 8,766 | 8,932 | -166 | -1.9 |
| Year 2 | 8,709 | 8,654 | 55 | 0.6 |
| Year 3 | 8,695 | 8,815 | -120 | -1.4 |
| Year 4 | 8,966 | 8,964 | 3 | 0.0 |
| Year 5 | 9,096 | 9,112 | -17 | -0.2 |
| Sample size | 2,570 | 2,159 | ||
|
Detroit |
||||
| Year 1 | 8,694 | 8,673 | 21 | 0.2 |
| Year 2 | 9,266 | 9,121 | 145 | 1.6 |
| Year 3 | 9,907 | 9,779 | 128 | 1.3 |
| Year 4 | 10,146 | 10,056 | 90 | 0.9 |
| Year 5 | 10,243 | 10,056 | 187 | 1.9 |
| Sample size | 2,226 | 2,233 | ||
|
Oklahoma City |
||||
| Year 1 | 5,363 | 5,478 | -115 | -2.1 |
| Year 2 | 5,269 | 5,349 | -80 | -1.5 |
| Year 3 | 5,368 | 5,541 | -173 | -3.1 |
| Year 4 | n/a | n/a | n/a | n/a |
| Year 5 | n/a | n/a | n/a | n/a |
| Sample size | 3,430 | 3,466 | ||
|
Portland |
||||
| Year 1 | 8,836 | 8,964 | -128 | -1.4 |
| Year 2 | 8,700 | 8,306 | 394 | 4.7 |
| Year 3 | 8,723 | 8,200 | 522 | 6.4 |
| Year 4 | 8,666 | 8,054 | 612* | 7.6 |
| Year 5 | 8,752 | 8,282 | 470 | 5.7 |
| Sample size | 3,529 | 499 | ||
| SOURCE: MDRC calculations from state and county administrative records. NOTES: See Appendix A.1. |
||||
|
Site and Program |
Sample Size | Program Group ($) | Control Group ($) | Difference (Impact) | Percentage Change (%) |
|---|---|---|---|---|---|
|
Average combined income in years 1 to 3 |
|||||
| Atlanta Labor Force Attachment | 2,938 | 22,870 | 21,967 | 903** | 4.1 |
| Atlanta Human Capital Development | 2,992 | 22,696 | 21,967 | 729** | 3.3 |
| Grand Rapids Labor Force Attachment | 3,012 | 22,865 | 23,729 | -865** | -3.6 |
| Grand Rapids Human Capital Development | 2,997 | 23,140 | 23,729 | -589 | -2.5 |
| Riverside Labor Force Attachment | 6,726 | 23,385 | 24,123 | -738** | -3.1 |
| Lacked high school diploma or basic skills | 3,125 | 22,451 | 23,668 | -1,217*** | -5.1 |
| Riverside Human Capital Development | 3,135 | 22,147 | 23,668 | -1,521*** | -6.4 |
| Columbus Integrated | 4,672 | 24,906 | 25,493 | -588 | -2.3 |
| Columbus Traditional | 4,729 | 25,195 | 25,493 | -298 | -1.2 |
| Detroit | 4,459 | 27,150 | 26,924 | 227 | 0.8 |
| Oklahoma City | 8,677 | n/a | n/a | n/a | n/a |
| Portland | 4,028 | 25,191 | 24,686 | 506 | 2.0 |
|
Average combined income in last quarter of year 5 |
|||||
| Atlanta Labor Force Attachment | 2,938 | 2,155 | 2,156 | -1 | -0.0 |
| Atlanta Human Capital Development | 2,992 | 2,117 | 2,156 | -39 | -1.8 |
| Grand Rapids Labor Force Attachment | 3,012 | 2,041 | 2,206 | -165** | -7.5 |
| Grand Rapids Human Capital Development | 2,997 | 2,092 | 2,206 | -113 | -5.1 |
| Riverside Labor Force Attachment | 6,726 | 1,857 | 1,855 | 2 | 0.1 |
| Lacked high school diploma or basic skills | 3,125 | 1,703 | 1,687 | 16 | 0.9 |
| Riverside Human Capital Development | 3,135 | 1,556 | 1,687 | -131** | -7.8 |
| Columbus Integrated | 4,672 | 2,255 | 2,217 | 38 | 1.7 |
| Columbus Traditional | 4,729 | 2,182 | 2,217 | -35 | -1.6 |
| Detroit | 4,459 | 2,456 | 2,453 | 3 | 0.1 |
| Oklahoma City | 8,677 | n/a | n/a | n/a | n/a |
| Portland | 4,028 | 2,125 | 1,977 | 148 | 7.5 |
| SOURCE: MDRC calculations from state and county administrative records. NOTES: See Appendix A.1. |
|||||
| Site and Program | Sample Size | Program Group (%) | Control Group (%) | Difference (Impact) | Percentage Change (%) |
|---|---|---|---|---|---|
|
Employed and not on welfare |
|||||
| Atlanta Labor Force Attachment | 2,938 | 40.5 | 39.9 | 0.6 | 1.4 |
| Atlanta Human Capital Development | 2,992 | 40.0 | 39.9 | 0.1 | 0.3 |
| Grand Rapids Labor Force Attachment | 3,012 | 43.5 | 46.5 | -3.0* | -6.4 |
| Grand Rapids Human Capital Development | 2,997 | 45.3 | 46.5 | -1.2 | -2.5 |
| Riverside Labor Force Attachment | 6,726 | 28.3 | 26.0 | 2.3** | 8.8 |
| Lacked high school diploma or basic skills | 3,125 | 24.5 | 21.3 | 3.3** | 15.4 |
| Riverside Human Capital Development | 3,135 | 22.7 | 21.3 | 1.4 | 6.6 |
| Columbus Integrated | 4,672 | 50.0 | 48.0 | 1.9 | 4.0 |
| Columbus Traditional | 4,729 | 48.5 | 48.0 | 0.5 | 1.0 |
| Detroit | 4,459 | 39.4 | 40.5 | -1.1 | -2.7 |
| Oklahoma City | 8,677 | n/a | n/a | n/a | n/a |
| Portland | 4,028 | 46.2 | 40.4 | 5.8** | 14.3 |
|
Employed and on welfare |
|||||
| Atlanta Labor Force Attachment | 2,938 | 10.4 | 11.6 | -1.1 | -9.7 |
| Atlanta Human Capital Development | 2,992 | 11.3 | 11.6 | -0.3 | -2.6 |
| Grand Rapids Labor Force Attachment | 3,012 | 11.5 | 12.6 | -1.1 | -8.5 |
| Grand Rapids Human Capital Development | 2,997 | 11.4 | 12.6 | -1.2 | -9.3 |
| Riverside Labor Force Attachment | 6,726 | 10.1 | 9.5 | 0.6 | 6.3 |
| Lacked high school diploma or basic skills | 3,125 | 10.8 | 9.8 | 0.9 | 9.3 |
| Riverside Human Capital Development | 3,135 | 11.2 | 9.8 | 1.3 | 13.5 |
| Columbus Integrated | 4,672 | 4.4 | 7.0 | -2.6*** | -36.9 |
| Columbus Traditional | 4,729 | 5.6 | 7.0 | -1.3** | -18.9 |
| Detroit | 4,459 | 15.7 | 14.8 | 0.9 | 6.0 |
| Oklahoma City | 8,677 | n/a | n/a | n/a | n/a |
| Portland | 4,028 | 3.9 | 4.3 | -0.4 | -8.2 |
|
Not employed and on welfare |
|||||
| Atlanta Labor Force Attachment | 2,938 | 22.1 | 24.9 | -2.8* | -11.3 |
| Atlanta Human Capital Development | 2,992 | 23.2 | 24.9 | -1.8 | -7.0 |
| Grand Rapids Labor Force Attachment | 3,012 | 12.8 | 15.2 | -2.4* | -15.8 |
| Grand Rapids Human Capital Development | 2,997 | 14.4 | 15.2 | -0.8 | -5.3 |
| Riverside Labor Force Attachment | 6,726 | 20.1 | 25.1 | -5.0*** | -20.0 |
| Lacked high school diploma or basic skills | 3,125 | 23.8 | 29.2 | -5.4*** | -18.6 |
| Riverside Human Capital Development | 3,135 | 22.1 | 29.2 | -7.1*** | -24.2 |
| Columbus Integrated | 4,672 | 8.3 | 9.5 | -1.2 | -13.1 |
| Columbus Traditional | 4,729 | 8.5 | 9.5 | -1.0 | -11.0 |
| Detroit | 4,459 | 19.3 | 20.4 | -1.1 | -5.3 |
| Oklahoma City | 8,677 | n/a | n/a | n/a | n/a |
| Portland | 4,028 | 8.4 | 12.9 | -4.5*** | -34.8 |
|
Not employed and not on welfare |
|||||
| Atlanta Labor Force Attachment | 2,938 | 27.0 | 23.6 | 3.4** | 14.3 |
| Atlanta Human Capital Development | 2,992 | 25.6 | 23.6 | 2.0 | 8.3 |
| Grand Rapids Labor Force Attachment | 3,012 | 32.2 | 25.8 | 6.5*** | 25.1 |
| Grand Rapids Human Capital Development | 2,997 | 28.9 | 25.8 | 3.1* | 12.2 |
| Riverside Labor Force Attachment | 6,726 | 41.5 | 39.4 | 2.1* | 5.4 |
| Lacked high school diploma or basic skills | 3,125 | 40.9 | 39.7 | 1.2 | 3.1 |
| Riverside Human Capital Development | 3,135 | 44.0 | 39.7 | 4.3** | 10.9 |
| Columbus Integrated | 4,672 | 37.3 | 35.4 | 1.9 | 5.3 |
| Columbus Traditional | 4,729 | 37.3 | 35.4 | 1.9 | 5.3 |
| Detroit | 4,459 | 25.6 | 24.3 | 1.3 | 5.2 |
| Oklahoma City | 8,677 | n/a | n/a | n/a | n/a |
| Portland | 4,028 | 41.5 | 42.5 | -1.0 | -2.3 |
| SOURCE: MDRC calculations from state and county administrative records. NOTES: See Appendix A.1. |
|||||
|
Site, Program, and Income Received |
Program Group (%) | Control Group (%) | Difference (Impact) | Percentage Change (%) |
|---|---|---|---|---|
|
Atlanta LFA |
||||
|
Any household member |
98.1 | 98.2 | -0.1 | -0.1 |
| Earnings | 68.7 | 65.5 | 3.3 | 5.0 |
| AFDC/TANF | 31.4 | 38.6 | -7.2*** | -18.7 |
| Food Stamps | 54.0 | 62.4 | -8.4*** | -13.4 |
| Child support | 15.3 | 14.5 | 0.8 | 5.5 |
| SSI/disability | 14.8 | 16.6 | -1.8 | -10.6 |
| Social Security/pension | 6.8 | 7.6 | -0.8 | -10.5 |
| Other income | 6.4 | 4.6 | 1.8 | 38.1 |
|
Respondent |
93.7 | 95.2 | -1.5 | -1.5 |
| Earnings | 59.7 | 55.0 | 4.7 | 8.6 |
| AFDC/TANF | 29.2 | 36.3 | -7.1*** | -19.5 |
| Food Stamps | 51.5 | 59.4 | -7.8*** | -13.2 |
| Child support | 14.2 | 13.7 | 0.5 | 3.7 |
| SSI/disability | 7.8 | 8.1 | -0.3 | -3.8 |
| Social Security/pension | 2.5 | 2.2 | 0.3 | 13.1 |
| Other income | 5.2 | 4.2 | 1.0 | 24.2 |
|
Other household member |
42.0 | 40.7 | 1.3 | 3.2 |
| Earnings | 29.1 | 27.6 | 1.5 | 5.4 |
| AFDC/TANF | 3.3 | 4.0 | -0.7 | -17.4 |
| Food Stamps | 4.2 | 5.2 | -1.0 | -18.9 |
| Child support | 1.1 | 0.8 | 0.3 | 34.5 |
| SSI/disability | 7.9 | 9.6 | -1.7 | -17.7 |
| Social Security/pension | 4.9 | 5.8 | -0.9 | -14.9 |
| Other income | 1.1 | 0.4 | 0.7 | 191.0 |
| Sample size | 519 | 552 | ||
|
Atlanta HCD |
||||
|
Any household member |
97.9 | 98.2 | -0.3 | -0.3 |
| Earnings | 67.2 | 65.5 | 1.7 | 2.6 |
| AFDC/TANF | 36.7 | 38.6 | -1.9 | -4.9 |
| Food Stamps | 55.2 | 62.4 | -7.2** | -11.6 |
| Child support | 18.2 | 14.5 | 3.6* | 24.9 |
| SSI/disability | 16.0 | 16.6 | -0.6 | -3.5 |
| Social Security/pension | 9.1 | 7.6 | 1.4 | 18.9 |
| Other income | 4.6 | 4.6 | -0.0 | -0.8 |
|
Respondent |
93.2 | 95.2 | -2.1 | -2.2 |
| Earnings | 54.5 | 55.0 | -0.5 | -0.9 |
| AFDC/TANF | 34.0 | 36.3 | -2.3 | -6.5 |
| Food Stamps | 52.1 | 59.4 | -7.3** | -12.3 |
| Child support | 17.9 | 13.7 | 4.2* | 30.6 |
| SSI/disability | 8.1 | 8.1 | 0.0 | 0.4 |
| Social Security/pension | 2.5 | 2.2 | 0.3 | 14.1 |
| Other income | 3.7 | 4.2 | -0.5 | -12.8 |
|
Other household member |
43.2 | 40.7 | 2.5 | 6.1 |
| Earnings | 30.2 | 27.6 | 2.6 | 9.4 |
| AFDC/TANF | 5.4 | 4.0 | 1.4 | 35.1 |
| Food Stamps | 4.9 | 5.2 | -0.3 | -5.8 |
| Child support | 0.3 | 0.8 | -0.6 | -69.5 |
| SSI/disability | 9.3 | 9.6 | -0.3 | -2.6 |
| Social Security/pension | 6.6 | 5.8 | 0.9 | 15.4 |
| Other income | 0.9 | 0.4 | 0.5 | 131.3 |
| Sample size | 594 | 552 | ||
|
Grand Rapids LFA |
||||
| Any household member | 98.2 | 99.2 | -1.0 | -1.1 |
| Earnings | 82.1 | 80.3 | 1.8 | 2.2 |
| AFDC/TANF | 20.4 | 22.2 | -1.8 | -7.9 |
| Food Stamps | 33.6 | 36.4 | -2.8 | -7.8 |
| Child support | 25.7 | 29.0 | -3.4 | -11.6 |
| SSI/disability | 25.3 | 21.7 | 3.6 | 16.6 |
| Social Security/pension | 5.1 | 7.2 | -2.1 | -28.9 |
| Other income | 8.2 | 7.6 | 0.6 | 7.5 |
|
Respondent |
92.8 | 93.5 | -0.8 | -0.8 |
| Earnings | 68.4 | 68.7 | -0.3 | -0.4 |
| AFDC/TANF | 19.7 | 21.2 | -1.5 | -7.0 |
| Food Stamps | 32.2 | 35.0 | -2.8 | -8.0 |
| Child support | 25.7 | 27.5 | -1.8 | -6.6 |
| SSI/disability | 12.2 | 10.4 | 1.8 | 17.7 |
| Social Security/pension | 1.2 | 1.6 | -0.4 | -22.9 |
| Other income | 5.9 | 6.0 | -0.2 | -2.8 |
|
Other household member |
59.3 | 55.8 | 3.5 | 6.3 |
| Earnings | 46.8 | 43.0 | 3.8 | 8.9 |
| AFDC/TANF | 1.2 | 1.9 | -0.7 | -36.8 |
| Food Stamps | 2.4 | 2.4 | 0.0 | 1.0 |
| Child support | 0.6 | 1.6 | -0.9 | -60.1 |
| SSI/disability | 16.5 | 13.1 | 3.4 | 25.5 |
| Social Security/pension | 3.8 | 6.0 | -2.2* | -36.3 |
| Other income | 2.8 | 2.1 | 0.8 | 38.7 |
| Sample size | 535 | 562 | ||
|
Grand Rapids HCD |
||||
| Any household member | 98.1 | 99.2 | -1.1 | -1.1 |
| Earnings | 78.8 | 80.3 | -1.5 | -1.9 |
| AFDC/TANF | 22.7 | 22.2 | 0.5 | 2.3 |
| Food Stamps | 35.0 | 36.4 | -1.5 | -4.0 |
| Child support | 30.3 | 29.0 | 1.2 | 4.3 |
| SSI/disability | 27.7 | 21.7 | 6.0** | 27.6 |
| Social Security/pension | 3.6 | 7.2 | -3.6*** | -49.7 |
| Other income | 6.7 | 7.6 | -0.9 | -12.2 |
|
Respondent |
92.9 | 93.5 | -0.6 | -0.7 |
| Earnings | 67.6 | 68.7 | -1.1 | -1.7 |
| AFDC/TANF | 20.5 | 21.2 | -0.8 | -3.6 |
| Food Stamps | 31.9 | 35.0 | -3.0 | -8.7 |
| Child support | 29.0 | 27.5 | 1.5 | 5.4 |
| SSI/disability | 14.0 | 10.4 | 3.6* | 35.1 |
| Social Security/pension | 0.7 | 1.6 | -0.9 | -54.9 |
| Other income | 5.5 | 6.0 | -0.6 | -9.4 |
|
Other household member |
57.0 | 55.8 | 1.3 | 2.3 |
| Earnings | 40.1 | 43.0 | -2.9 | -6.7 |
| AFDC/TANF | 2.9 | 1.9 | 1.0 | 53.7 |
| Food Stamps | 3.9 | 2.4 | 1.5 | 64.3 |
| Child support | 1.8 | 1.6 | 0.2 | 13.2 |
| SSI/disability | 16.3 | 13.1 | 3.2 | 24.0 |
| Social Security/pension | 3.0 | 6.0 | -3.0** | -49.8 |
| Other income | 1.2 | 2.1 | -0.8 | -40.1 |
| Sample size | 547 | 562 | ||
|
Riverside LFA |
||||
| Any household member | 99.4 | 98.8 | 0.6 | 0.6 |
| Earnings | 73.7 | 69.8 | 3.8 | 5.5 |
| AFDC/TANF | 41.6 | 49.3 | -7.8*** | -15.8 |
| Food Stamps | 43.6 | 50.9 | -7.3*** | -14.4 |
| Child support | 11.7 | 12.5 | -0.8 | -6.1 |
| SSI/disability | 15.3 | 13.9 | 1.4 | 10.0 |
| Social Security/pension | 8.8 | 8.2 | 0.7 | 8.3 |
| Other income | 10.0 | 11.0 | -1.0 | -9.5 |
|
Respondent |
92.0 | 90.3 | 1.7 | 1.9 |
| Earnings | 57.9 | 51.7 | 6.2** | 12.0 |
| AFDC/TANF | 38.3 | 46.5 | -8.2*** | -17.6 |
| Food Stamps | 39.7 | 48.7 | -8.9*** | -18.4 |
| Child support | 10.9 | 11.3 | -0.4 | -3.3 |
| SSI/disability | 8.6 | 4.7 | 3.8*** | 81.2 |
| Social Security/pension | 3.0 | 2.2 | 0.8 | 34.7 |
| Other income | 7.0 | 8.8 | -1.9 | -21.1 |
|
Other household member |
52.1 | 56.3 | -4.2 | -7.4 |
| Earnings | 37.6 | 42.2 | -4.7* | -11.1 |
| AFDC/TANF | 4.9 | 5.1 | -0.2 | -4.8 |
| Food Stamps | 5.2 | 3.7 | 1.6 | 43.4 |
| Child support | 0.8 | 1.2 | -0.4 | -31.9 |
| SSI/disability | 7.5 | 10.0 | -2.5 | -24.6 |
| Social Security/pension | 6.8 | 6.9 | -0.1 | -0.7 |
| Other income | 3.6 | 2.5 | 1.1 | 44.7 |
| Sample size | 499 | 720 | ||
|
Riverside LFA-lacked high school diploma or basic skills |
||||
| Any household member | 99.4 | 98.8 | 0.6 | 0.6 |
| Earnings | 69.6 | 64.0 | 5.6 | 8.8 |
| AFDC/TANF | 50.6 | 59.4 | -8.8** | -14.9 |
| Food Stamps | 49.1 | 59.6 | -10.5*** | -17.6 |
| Child support | 7.0 | 11.0 | -4.0* | -36.4 |
| SSI/disability | 14.0 | 14.6 | -0.6 | -4.2 |
| Social Security/pension | 8.3 | 7.3 | 1.0 | 13.5 |
| Other income | 6.9 | 11.5 | -4.5** | -39.7 |
|
Respondent |
91.3 | 91.2 | 0.2 | 0.2 |
| Earnings | 52.7 | 45.1 | 7.6** | 16.7 |
| AFDC/TANF | 47.1 | 56.4 | -9.3** | -16.5 |
| Food Stamps | 45.1 | 56.4 | -11.3*** | -20.0 |
| Child support | 6.5 | 9.8 | -3.2 | -33.3 |
| SSI/disability | 10.0 | 4.8 | 5.1*** | 106.5 |
| Social Security/pension | 3.7 | 2.2 | 1.6 | 72.2 |
| Other income | 3.8 | 9.0 | -5.3*** | -58.5 |
| Other household member | 47.3 | 53.7 | -6.4* | -11.8 |
| Earnings | 32.6 | 39.7 | -7.1** | -17.9 |
| AFDC/TANF | 5.4 | 5.6 | -0.2 | -3.1 |
| Food Stamps | 5.5 | 4.6 | 0.9 | 19.2 |
| Child support | 0.5 | 1.2 | -0.8 | -60.8 |
| SSI/disability | 4.9 | 10.4 | -5.5*** | -52.7 |
| Social Security/pension | 6.1 | 5.4 | 0.6 | 11.7 |
| Other income | 3.5 | 2.7 | 0.8 | 30.6 |
| Sample size | 255 | 402 | ||
|
Riverside HCD |
||||
|
Any household member |
98.6 | 98.8 | -0.3 | -0.3 |
| Earnings | 71.7 | 64.0 | 7.7** | 12.0 |
| AFDC/TANF | 51.2 | 59.4 | -8.2** | -13.8 |
| Food Stamps | 52.2 | 59.6 | -7.4** | -12.4 |
| Child support | 13.5 | 11.0 | 2.5 | 22.7 |
| SSI/disability | 14.8 | 14.6 | 0.2 | 1.2 |
| Social Security/pension | 5.3 | 7.3 | -2.0 | -27.2 |
| Other income | 7.6 | 11.5 | -3.9* | -33.7 |
|
Respondent |
88.6 | 91.2 | -2.6 | -2.9 |
| Earnings | 54.3 | 45.1 | 9.1** | 20.2 |
| AFDC/TANF | 49.5 | 56.4 | -6.9* | -12.2 |
| Food Stamps | 50.2 | 56.4 | -6.1 | -10.9 |
| Child support | 12.5 | 9.8 | 2.7 | 27.9 |
| SSI/disability | 6.5 | 4.8 | 1.6 | 34.0 |
| Social Security/pension | 1.1 | 2.2 | -1.1 | -51.2 |
| Other income | 4.5 | 9.0 | -4.5** | -50.0 |
|
Other household member |
54.0 | 53.7 | 0.3 | 0.5 |
| Earnings | 39.6 | 39.7 | -0.1 | -0.2 |
| AFDC/TANF | 4.4 | 5.6 | -1.2 | -21.0 |
| Food Stamps | 3.9 | 4.6 | -0.8 | -17.0 |
| Child support | 1.0 | 1.2 | -0.2 | -17.6 |
| SSI/disability | 9.2 | 10.4 | -1.2 | -11.2 |
| Social Security/pension | 4.7 | 5.4 | -0.8 | -14.0 |
| Other income | 3.4 | 2.7 | 0.7 | 27.6 |
| Sample size | 376 | 402 | ||
|
Portland |
||||
|
Any household member |
97.7 | 100.1 | -2.4** | -2.4 |
| Earnings | 81.0 | 79.4 | 1.6 | 2.0 |
| AFDC/TANF | 19.1 | 17.6 | 1.5 | 8.3 |
| Food Stamps | 33.8 | 42.3 | -8.4* | -20.0 |
| Child support | 20.2 | 21.7 | -1.5 | -6.9 |
| SSI/disability | 16.3 | 10.2 | 6.1* | 60.0 |
| Social Security/pension | 12.7 | 8.7 | 4.0 | 45.8 |
| Other income | 7.3 | 9.2 | -1.9 | -20.7 |
|
Respondent |
90.1 | 89.7 | 0.4 | 0.4 |
| Earnings | 62.5 | 60.6 | 1.9 | 3.1 |
| AFDC/TANF | 17.1 | 15.3 | 1.8 | 11.6 |
| Food Stamps | 32.0 | 37.6 | -5.7 | -15.1 |
| Child support | 17.7 | 21.8 | -4.1 | -19.0 |
| SSI/disability | 8.2 | 6.2 | 2.0 | 32.0 |
| Social Security/pension | 0.8 | 2.9 | -2.0* | -71.1 |
| Other income | 6.5 | 6.8 | -0.3 | -4.3 |
|
Other household member |
60.1 | 54.2 | 6.0 | 11.0 |
| Earnings | 45.0 | 42.5 | 2.5 | 5.9 |
| AFDC/TANF | 1.9 | 2.6 | -0.7 | -25.5 |
| Food Stamps | 2.8 | 6.8 | -4.1** | -59.6 |
| Child support | 2.5 | -0.1 | 2.6** | -3487.8 |
| SSI/disability | 9.9 | 4.6 | 5.3** | 116.0 |
| Social Security/pension | 11.9 | 5.8 | 6.0** | 103.1 |
| Other income | 1.5 | 2.9 | -1.4 | -48.3 |
| Sample size | 281 | 223 | ||
| SOURCE: MDRC calculations from the Five-year Client Survey. NOTES: See Appendix A.2. |
||||
| Site and Program | Sample Size | Program Group (%) | Control Group (%) | Difference (Impact) | Percentage Change (%) |
|---|---|---|---|---|---|
|
Only respondent employed |
|||||
| Atlanta Labor Force Attachment | 1,071 | 39.6 | 37.9 | 1.8 | 4.7 |
| Atlanta Human Capital Development | 1,146 | 37.0 | 37.9 | -0.9 | -2.4 |
| Grand Rapids Labor Force Attachment | 1,097 | 35.3 | 37.3 | -2.0 | -5.4 |
| Grand Rapids Human Capital Development | 1,109 | 38.7 | 37.3 | 1.4 | 3.7 |
| Riverside Labor Force Attachment | 1,219 | 36.1 | 27.6 | 8.5*** | 30.9 |
| Lacked high school diploma or basic skills | 657 | 37.1 | 24.3 | 12.7*** | 52.4 |
| Riverside Human Capital Development | 778 | 32.1 | 24.3 | 7.8** | 31.9 |
| Portland | 504 | 36.0 | 37.0 | -0.9 | -2.6 |
|
Respondent and other household member employed |
|||||
| Atlanta Labor Force Attachment | 1,071 | 20.1 | 17.2 | 2.9 | 17.1 |
| Atlanta Human Capital Development | 1,146 | 17.6 | 17.2 | 0.4 | 2.3 |
| Grand Rapids Labor Force Attachment | 1,097 | 33.1 | 31.4 | 1.7 | 5.5 |
| Grand Rapids Human Capital Development | 1,109 | 28.9 | 31.4 | -2.5 | -8.0 |
| Riverside Labor Force Attachment | 1,219 | 21.8 | 24.1 | -2.3 | -9.6 |
| Lacked high school diploma or basic skills | 657 | 15.7 | 20.8 | -5.2* | -24.9 |
| Riverside Human Capital Development | 778 | 22.2 | 20.8 | 1.3 | 6.5 |
| Portland | 504 | 26.5 | 23.7 | 2.8 | 11.8 |
|
Only other household member employed |
|||||
| Atlanta Labor Force Attachment | 1,071 | 9.0 | 10.4 | -1.5 | -14.0 |
| Atlanta Human Capital Development | 1,146 | 12.6 | 10.4 | 2.2 | 21.2 |
| Grand Rapids Labor Force Attachment | 1,097 | 13.7 | 11.6 | 2.1 | 18.0 |
| Grand Rapids Human Capital Development | 1,109 | 11.3 | 11.6 | -0.4 | -3.2 |
| Riverside Labor Force Attachment | 1,219 | 15.8 | 18.1 | -2.4 | -13.2 |
| Lacked high school diploma or basic skills | 657 | 16.9 | 18.8 | -1.9 | -10.2 |
| Riverside Human Capital Development | 778 | 17.4 | 18.8 | -1.4 | -7.5 |
| Portland | 504 | 18.5 | 18.8 | -0.3 | -1.5 |
|
Neither respondent nor other household member employed |
|||||
| Atlanta Labor Force Attachment | 1,071 | 31.3 | 34.5 | -3.3 | -9.4 |
| Atlanta Human Capital Development | 1,146 | 32.8 | 34.5 | -1.7 | -4.9 |
| Grand Rapids Labor Force Attachment | 1,097 | 17.9 | 19.7 | -1.8 | -9.2 |
| Grand Rapids Human Capital Development | 1,109 | 21.2 | 19.7 | 1.5 | 7.7 |
| Riverside Labor Force Attachment | 1,219 | 26.3 | 30.2 | -3.8 | -12.7 |
| Lacked high school diploma or basic skills | 657 | 30.4 | 36.0 | -5.6 | -15.7 |
| Riverside Human Capital Development | 778 | 28.3 | 36.0 | -7.7** | -21.4 |
| Portland | 504 | 19.0 | 20.6 | -1.6 | -7.7 |
| SOURCE: MDRC calculations from the Five-year Client Survey. NOTES: See Appendix A.2. |
|||||
Appendix F: Supplementary Tables to Chapter 7
| Site and Program | Sample Size | Average Total Earnings in Years 1 to 5 ($) | Average Welfare Payments in Years 1 to 5 ($) | Combined Income in Years 1 to 5 ($) |
|---|---|---|---|---|
|
On welfare for two years or more |
||||
| Atlanta Labor Force Attachment | 2,063 | 14,067 | 11,202 | 39,133 |
| Atlanta Human Capital Development | 2,100 | 14,067 | 11,202 | 39,133 |
| Grand Rapids Labor Force Attachment | 1,791 | 18,507 | 14,551 | 42,370 |
| Grand Rapids Human Capital Development | 1,775 | 18,507 | 14,551 | 42,370 |
| Riverside Labor Force Attachment | 3,510 | 10,827 | 21,539 | 39,971 |
| Lacked high school diploma or basic skills | 1,831 | 7,677 | 22,800 | 38,504 |
| Riverside Human Capital Development | 1,841 | 7,677 | 22,800 | 38,504 |
| Columbus Integrated | 3,392 | 22,855 | 10,095 | 43,925 |
| Columbus Traditional | 3,415 | 22,855 | 10,095 | 43,925 |
| Detroit | 3,313 | 19,404 | 17,040 | 48,018 |
| Oklahoma City | 2,076 | 11,771 | n/a | n/a |
| Portland | 2,443 | 16,885 | 13,370 | 40,755 |
|
On welfare for less than two years |
||||
| Atlanta Labor Force Attachment | 840 | 25,528 | 6,969 | 42,277 |
| Atlanta Human Capital Development | 847 | 25,528 | 6,969 | 42,277 |
| Grand Rapids Labor Force Attachment | 1,219 | 23,985 | 10,670 | 41,847 |
| Grand Rapids Human Capital Development | 1,215 | 23,985 | 10,670 | 41,847 |
| Riverside Labor Force Attachment | 3,101 | 19,393 | 14,624 | 39,513 |
| Lacked high school diploma or basic skills | 1,248 | 15,370 | 16,303 | 37,829 |
| Riverside Human Capital Development | 1,238 | 15,370 | 16,303 | 37,829 |
| Columbus Integrated | 806 | 33,403 | 6,508 | 47,620 |
| Columbus Traditional | 793 | 33,403 | 6,508 | 47,620 |
| Detroit | 1,015 | 24,142 | 13,688 | 46,658 |
| Oklahoma City | 2,683 | 13,787 | n/a | n/a |
| Portland | 1,494 | 26,955 | 9,104 | 43,050 |
| SOURCES: MDRC calculations from unemployment insurance (UI) earnings records and AFDC records. NOTE: N/a = not applicable. |
||||
|
Site and Program |
Sample Size | Average Total Earnings in Years 1 to 5 ($) | Average Welfare Payments in Years 1 to 5 ($) | Combined Income in Years1 to 5 ($) |
|---|---|---|---|---|
|
Did not work in year prior to random assignment |
||||
| Atlanta Labor Force Attachment | 1,869 | 11,756 | 11,024 | 36,191 |
| Atlanta Human Capital Development | 1,937 | 11,756 | 11,024 | 36,191 |
| Grand Rapids Labor Force Attachment | 1,527 | 15,109 | 14,680 | 38,649 |
| Grand Rapids Human Capital Development | 1,489 | 15,109 | 14,680 | 38,649 |
| Riverside Labor Force Attachment | 4,010 | 9,143 | 20,198 | 36,357 |
| Lacked high school diploma or basic skills | 2,074 | 6,649 | 21,772 | 35,917 |
| Riverside Human Capital Development | 2,065 | 6,649 | 21,772 | 35,917 |
| Columbus Integrated | 2,143 | 15,916 | 9,984 | 36,132 |
| Columbus Traditional | 2,160 | 15,916 | 9,984 | 36,132 |
| Detroit | 2,978 | 16,065 | 17,395 | 44,839 |
| Oklahoma City | 3,910 | 8,128 | n/a | n/a |
| Portland | 2,317 | 15,927 | 13,662 | 39,323 |
|
Worked in year prior to random assignment |
||||
| Atlanta Labor Force Attachment | 1,069 | 27,313 | 8,038 | 46,689 |
| Atlanta Human Capital Development | 1,055 | 27,313 | 8,038 | 46,689 |
| Grand Rapids Labor Force Attachment | 1,485 | 26,380 | 11,266 | 45,667 |
| Grand Rapids Human Capital Development | 1,508 | 26,380 | 11,266 | 45,667 |
| Riverside Labor Force Attachment | 2,716 | 23,764 | 15,333 | 45,088 |
| Lacked high school diploma or basic skills | 1,051 | 19,101 | 16,888 | 42,797 |
| Riverside Human Capital Development | 1,070 | 19,101 | 16,888 | 42,797 |
| Columbus Integrated | 2,529 | 33,597 | 8,176 | 51,434 |
| Columbus Traditional | 2,569 | 33,597 | 8,176 | 51,434 |
| Detroit | 1,481 | 29,425 | 13,953 | 53,424 |
| Oklahoma City | 4,767 | 16,544 | n/a | n/a |
| Portland | 1,711 | 27,710 | 9,056 | 45,381 |
| SOURCES: MDRC calculations from unemployment insurance (UI) earnings records and AFDC records. NOTE: N/a = not applicable. |
||||
|
Site and Program |
Sample Size | Average Total Earnings in Years 1 to 5 ($) | Average Welfare Payments in Years 1 to 5 ($) | Combined Income in Years 1 to 5 ($) |
|---|---|---|---|---|
|
Most disadvantaged |
||||
| Atlanta Labor Force Attachment | 698 | 8,486 | 12,244 | 35,381 |
| Atlanta Human Capital Development | 734 | 8,486 | 12,244 | 35,381 |
| Grand Rapids Labor Force Attachment | 458 | 8,622 | 17,943 | 36,697 |
| Grand Rapids Human Capital Development | 453 | 8,622 | 17,943 | 36,697 |
| Riverside Labor Force Attachment | 1,362 | 5,044 | 23,895 | 37,108 |
| Riverside Human Capital Development | 1,362 | 5,044 | 23,895 | 37,108 |
| Columbus Integrated | 911 | 11,383 | 11,557 | 34,421 |
| Columbus Traditional | 901 | 11,383 | 11,557 | 34,421 |
| Detroit | 1,119 | 11,532 | 20,072 | 44,519 |
| Oklahoma City | 429 | 3,888 | n/a | n/a |
| Portland | 617 | 10,715 | 16,575 | 38,513 |
|
Moderately disadvantaged |
||||
| Atlanta Labor Force Attachment | 1,887 | 17,542 | 9,829 | 40,004 |
| Atlanta Human Capital Development | 1,911 | 17,542 | 9,829 | 40,004 |
| Grand Rapids Labor Force Attachment | 2,123 | 20,668 | 12,850 | 42,108 |
| Grand Rapids Human Capital Development | 2,078 | 20,668 | 12,850 | 42,108 |
| Riverside Labor Force Attachment | 4,298 | 15,916 | 17,185 | 39,417 |
| Riverside Human Capital Development | 3,049 | 15,916 | 17,185 | 39,417 |
| Columbus Integrated | 3,155 | 26,251 | 8,916 | 45,276 |
| Columbus Traditional | 3,236 | 26,251 | 8,916 | 45,276 |
| Detroit | 3,018 | 21,404 | 15,413 | 47,440 |
| Oklahoma City | 6,170 | 10,914 | n/a | n/a |
| Portland | 2,803 | 20,143 | 11,896 | 41,358 |
|
Least disadvantaged |
||||
| Atlanta Labor Force Attachment | 353 | 35,469 | 5,762 | 49,746 |
| Atlanta Human Capital Development | 347 | 35,469 | 5,762 | 49,746 |
| Grand Rapids Labor Force Attachment | 431 | 33,891 | 8,139 | 48,040 |
| Grand Rapids Human Capital Development | 466 | 33,891 | 8,139 | 48,040 |
| Riverside Labor Force Attachment | 1,066 | 29,903 | 12,281 | 46,887 |
| Columbus Integrated | 606 | 44,779 | 5,585 | 56,893 |
| Columbus Traditional | 592 | 44,779 | 5,585 | 56,893 |
| Detroit | 322 | 41,675 | 10,898 | 59,547 |
| Oklahoma City | 2,078 | 20,057 | n/a | n/a |
| Portland | 608 | 35,591 | 5,875 | 48,183 |
| SOURCES: MDRC calculations from unemployment insurance (UI) earnings records and AFDC records. NOTE: N/a = not applicable. |
||||
|
Site and Program |
Sample Size | Average Total Earnings in Years 1 to 5 ($) | Average Welfare Payments in Years 1 to 5 ($) | Combined Income in Years 1 to 5 ($) |
|---|---|---|---|---|
|
White, non-Hispanic |
||||
| Atlanta Labor Force Attachment | n/a | n/a | n/a | n/a |
| Atlanta Human Capital Development | n/a | n/a | n/a | n/a |
| Grand Rapids Labor Force Attachment | 1,470 | 24,279 | 11,278 | 43,288 |
| Grand Rapids Human Capital Development | 1,515 | 24,279 | 11,278 | 43,288 |
| Riverside Labor Force Attachment | 3,464 | 15,056 | 16,394 | 37,123 |
| Lacked high school diploma or basic skills | 1,245 | 10,232 | 18,157 | 34,423 |
| Riverside Human Capital Development | 1,208 | 10,232 | 18,157 | 34,423 |
| Columbus Integrated | 2,161 | 24,551 | 7,981 | 41,388 |
| Columbus Traditional | 2,204 | 24,551 | 7,981 | 41,388 |
| Detroit | 481 | 18,179 | 14,535 | 42,377 |
| Oklahoma City | 5,109 | 11,997 | n/a | n/a |
| Portland | 2,754 | 19,783 | 11,176 | 39,941 |
|
Black, non-Hispanic |
||||
| Atlanta Labor Force Attachment | 2,791 | 17,584 | 9,992 | 40,319 |
| Atlanta Human Capital Development | 2,838 | 17,584 | 9,992 | 40,319 |
| Grand Rapids Labor Force Attachment | 1,214 | 17,841 | 14,938 | 42,055 |
| Grand Rapids Human Capital Development | 1,158 | 17,841 | 14,938 | 42,055 |
| Riverside Labor Force Attachment | 1,121 | 14,411 | 19,593 | 41,157 |
| Lacked high school diploma or basic skills | 501 | 10,551 | 21,281 | 39,362 |
| Riverside Human Capital Development | 510 | 10,551 | 21,281 | 39,362 |
| Columbus Integrated | 2,414 | 26,572 | 9,868 | 47,350 |
| Columbus Traditional | 2,431 | 26,572 | 9,868 | 47,350 |
| Detroit | 3,836 | 20,842 | 16,482 | 48,429 |
| Oklahoma City | 2,484 | 14,484 | n/a | n/a |
| Portland | 798 | 24,242 | 13,200 | 47,414 |
|
Hispanic |
||||
| Atlanta Labor Force Attachment | n/a | n/a | n/a | n/a |
| Atlanta Human Capital Development | n/a | n/a | n/a | n/a |
| Grand Rapids Labor Force Attachment | 244 | 15,515 | 14,019 | 38,617 |
| Grand Rapids Human Capital Development | 249 | 15,515 | 14,019 | 38,617 |
| Riverside Labor Force Attachment | 1,858 | 14,847 | 20,855 | 43,658 |
| Lacked high school diploma or basic skills | 1,210 | 11,590 | 21,919 | 41,991 |
| Riverside Human Capital Development | 1,240 | 11,590 | 21,919 | 41,991 |
| Columbus Integrated | n/a | n/a | n/a | n/a |
| Columbus Traditional | n/a | n/a | n/a | n/a |
| Detroit | n/a | n/a | n/a | n/a |
| Oklahoma City | 392 | 11,549 | n/a | n/a |
| Portland | n/a | n/a | n/a | n/a |
| SOURCES: MDRC calculations from unemployment insurance (UI) earnings records and AFDC records. NOTE: N/a = not applicable. |
||||
|
Site and Program |
Sample Size | Average Total Earnings in Years 1 to 5 ($) | Average Welfare Payments in Years 1 to 5 ($) | Combined Income in Years 1 to 5 ($) |
|---|---|---|---|---|
|
On welfare for two years or more |
||||
| Atlanta Labor Force Attachment | 2,063 | 832 | 223 | 746 |
| Atlanta Human Capital Development | 2,100 | 824 | 221 | 739 |
| Grand Rapids Labor Force Attachment | 1,791 | 1,012 | 360 | 948 |
| Grand Rapids Human Capital Development | 1,775 | 1,017 | 361 | 953 |
| Riverside Labor Force Attachment | 3,510 | 714 | 443 | 804 |
| Lacked high school diploma or basic skills | 1,831 | 789 | 610 | 1,010 |
| Riverside Human Capital Development | 1,841 | 788 | 610 | 1,009 |
| Columbus Integrated | 3,392 | 848 | 211 | 800 |
| Columbus Traditional | 3,415 | 847 | 211 | 799 |
| Detroit | 3,313 | 891 | 293 | 791 |
| Oklahoma City | 2,076 | 634 | n/a | n/a |
| Portland | 2,443 | 1,566 | 535 | 1,541 |
|
On welfare for less than two years |
||||
| Atlanta Labor Force Attachment | 840 | 1,911 | 344 | 1,713 |
| Atlanta Human Capital Development | 847 | 1,903 | 342 | 1,707 |
| Grand Rapids Labor Force Attachment | 1,219 | 1,416 | 400 | 1,303 |
| Grand Rapids Human Capital Development | 1,215 | 1,423 | 402 | 1,310 |
| Riverside Labor Force Attachment | 3,101 | 996 | 436 | 998 |
| Lacked high school diploma or basic skills | 1,248 | 1,271 | 692 | 1,388 |
| Riverside Human Capital Development | 1,238 | 1,274 | 694 | 1,391 |
| Columbus Integrated | 806 | 2,214 | 362 | 2,070 |
| Columbus Traditional | 793 | 2,236 | 365 | 2,091 |
| Detroit | 1,015 | 1,828 | 511 | 1,608 |
| Oklahoma City | 2,683 | 645 | n/a | n/a |
| Portland | 1,494 | 2,493 | 516 | 2,315 |
| SOURCES: MDRC calculations from unemployment insurance (UI) earnings records and AFDC records. NOTE: N/a = not applicable. |
||||
| Site and Program | Sample Size | Average Total Earnings in Years 1 to 5 ($) | Average Welfare Payments in Years 1 to 5 ($) | Combined Income in Years 1 to 5 ($) |
|---|---|---|---|---|
|
Did not work in year prior to random assignment |
||||
| Atlanta Labor Force Attachment | 1,869 | 950 | 236 | 857 |
| Atlanta Human Capital Development | 1,937 | 931 | 231 | 840 |
| Grand Rapids Labor Force Attachment | 1,527 | 1,019 | 399 | 992 |
| Grand Rapids Human Capital Development | 1,489 | 1,033 | 404 | 1,005 |
| Riverside Labor Force Attachment | 4,010 | 645 | 411 | 750 |
| Lacked high school diploma or basic skills | 2,074 | 711 | 574 | 953 |
| Riverside Human Capital Development | 2,065 | 713 | 576 | 956 |
| Columbus Integrated | 2,143 | 1,003 | 268 | 1,002 |
| Columbus Traditional | 2,160 | 1,001 | 267 | 1,000 |
| Detroit | 2,978 | 898 | 318 | 822 |
| Oklahoma City | 3,910 | 471 | n/a | n/a |
| Portland | 2,317 | 1,703 | 528 | 1,664 |
|
Worked in year prior to random assignment |
||||
| Atlanta Labor Force Attachment | 1,069 | 1,477 | 308 | 1,309 |
| Atlanta Human Capital Development | 1,055 | 1,491 | 311 | 1,321 |
| Grand Rapids Labor Force Attachment | 1,485 | 1,322 | 360 | 1,193 |
| Grand Rapids Human Capital Development | 1,508 | 1,316 | 358 | 1,187 |
| Riverside Labor Force Attachment | 2,716 | 1,142 | 465 | 1,099 |
| Lacked high school diploma or basic skills | 1,051 | 1,523 | 738 | 1,547 |
| Riverside Human Capital Development | 1,070 | 1,507 | 730 | 1,530 |
| Columbus Integrated | 2,529 | 1,184 | 222 | 1,079 |
| Columbus Traditional | 2,569 | 1,177 | 221 | 1,072 |
| Detroit | 1,481 | 1,566 | 420 | 1,335 |
| Oklahoma City | 4,767 | 517 | n/a | n/a |
| Portland | 1,711 | 2,159 | 529 | 2,014 |
| SOURCES: MDRC calculations from unemployment insurance (UI) earnings records and AFDC records. NOTE: N/a = not applicable. |
||||
|
Site and Program |
Sample Size | Average Total Earnings in Years 1 to 5 ($) | Average Welfare Payments in Years 1 to 5 ($) | Combined Income in Years 1 to 5 ($) |
|---|---|---|---|---|
|
Most disadvantaged |
||||
| Atlanta Labor Force Attachment | 698 | 1,095 | 377 | 1,056 |
| Atlanta Human Capital Development | 734 | 1,063 | 366 | 1,025 |
| Grand Rapids Labor Force Attachment | 458 | 1,402 | 728 | 1,542 |
| Grand Rapids Human Capital Development | 453 | 1,416 | 735 | 1,558 |
| Riverside Labor Force Attachment | 1,362 | 800 | 715 | 1,140 |
| Riverside Human Capital Development | 1,362 | 802 | 717 | 1,143 |
| Columbus Integrated | 911 | 1,143 | 429 | 1,267 |
| Columbus Traditional | 901 | 1,150 | 432 | 1,274 |
| Detroit | 1,119 | 1,186 | 522 | 1,148 |
| Oklahoma City | 429 | 924 | n/a | n/a |
| Portland | 617 | 2,352 | 1,232 | 2,730 |
|
Moderately disadvantaged |
||||
| Atlanta Labor Force Attachment | 1,887 | 1,099 | 239 | 977 |
| Atlanta Human Capital Development | 1,911 | 1,092 | 238 | 970 |
| Grand Rapids Labor Force Attachment | 2,123 | 1,069 | 324 | 984 |
| Grand Rapids Human Capital Development | 2,078 | 1,082 | 328 | 997 |
| Riverside Labor Force Attachment | 4,298 | 799 | 376 | 810 |
| Riverside Human Capital Development | 3,049 | 1,052 | 495 | 1,067 |
| Columbus Integrated | 3,155 | 1,044 | 212 | 967 |
| Columbus Traditional | 3,236 | 1,034 | 210 | 958 |
| Detroit | 3,018 | 1,021 | 312 | 891 |
| Oklahoma City | 6,170 | 395 | n/a | n/a |
| Portland | 2,803 | 1,653 | 446 | 1,564 |
|
Least disadvantaged |
||||
| Atlanta Labor Force Attachment | 353 | 3,121 | 495 | 2,755 |
| Atlanta Human Capital Development | 347 | 3,143 | 498 | 2,774 |
| Grand Rapids Labor Force Attachment | 431 | 2,947 | 624 | 2,616 |
| Grand Rapids Human Capital Development | 466 | 2,846 | 603 | 2,526 |
| Riverside Labor Force Attachment | 1,066 | 288 | 688 | 1,928 |
| Columbus Integrated | 606 | 2,958 | 365 | 2,687 |
| Columbus Traditional | 592 | 2,990 | 369 | 2,715 |
| Detroit | 322 | 4,344 | 833 | 3,661 |
| Oklahoma City | 2,078 | 936 | n/a | n/a |
| Portland | 608 | 4,204 | 724 | 3,845 |
| SOURCES: MDRC calculations from unemployment insurance (UI) earnings records and AFDC records. NOTE: N/a = not applicable. |
||||
|
Site and Program |
Sample Size | Average Total Earnings in Years 1 to 5 ($) | Average Welfare Payments in Years 1 to 5 ($) | Combined Income in Years 1 to 5 ($) |
|---|---|---|---|---|
|
White, non-Hispanic |
||||
| Atlanta Labor Force Attachment | n/a | n/a | n/a | n/a |
| Atlanta Human Capital Development | n/a | n/a | n/a | n/a |
| Grand Rapids Labor Force Attachment | 1,470 | 1,276 | 373 | 1,177 |
| Grand Rapids Human Capital Development | 1,515 | 1,263 | 370 | 1,165 |
| Riverside Labor Force Attachment | 3,464 | 847 | 406 | 888 |
| Lacked high school diploma or basic skills | 1,245 | 1,139 | 675 | 1,303 |
| Riverside Human Capital Development | 1,208 | 1,157 | 686 | 1,323 |
| Columbus Integrated | 2,161 | 1,154 | 243 | 1,099 |
| Columbus Traditional | 2,204 | 1,142 | 240 | 1,088 |
| Detroit | 481 | 2,328 | 768 | 2,181 |
| Oklahoma City | 5,109 | 459 | n/a | n/a |
| Portland | 2,754 | 1,627 | 434 | 1,552 |
|
Black, non-Hispanic |
||||
| Atlanta Labor Force Attachment | 2,791 | 826 | 193 | 738 |
| Atlanta Human Capital Development | 2,838 | 819 | 191 | 732 |
| Grand Rapids Labor Force Attachment | 1,214 | 1,196 | 430 | 1,114 |
| Grand Rapids Human Capital Development | 1,158 | 1,224 | 440 | 1,140 |
| Riverside Labor Force Attachment | 1,121 | 1,475 | 778 | 1,572 |
| Lacked high school diploma or basic skills | 501 | 1,682 | 1,163 | 2,122 |
| Riverside Human Capital Development | 510 | 1,671 | 1,155 | 2,108 |
| Columbus Integrated | 2,414 | 1,102 | 246 | 1,024 |
| Columbus Traditional | 2,431 | 1,099 | 245 | 1,022 |
| Detroit | 3,836 | 858 | 273 | 755 |
| Oklahoma City | 2,484 | 682 | n/a | n/a |
| Portland | 798 | 2,783 | 975 | 2,729 |
|
Hispanic |
||||
| Atlanta Labor Force Attachment | n/a | n/a | n/a | n/a |
| Atlanta Human Capital Development | n/a | n/a | n/a | n/a |
| Grand Rapids Labor Force Attachment | 244 | 2,771 | 1,027 | 2,636 |
| Grand Rapids Human Capital Development | 249 | 2,787 | 1,033 | 2,651 |
| Riverside Labor Force Attachment | 1,858 | 1,098 | 624 | 1,143 |
| Lacked high school diploma or basic skills | 1,210 | 1,099 | 761 | 1,287 |
| Riverside Human Capital Development | 1,240 | 1,085 | 751 | 1,271 |
| Columbus Integrated | n/a | n/a | n/a | n/a |
| Columbus Traditional | n/a | n/a | n/a | n/a |
| Detroit | n/a | n/a | n/a | n/a |
| Oklahoma City | 392 | 1,607 | n/a | n/a |
| Portland | n/a | n/a | n/a | n/a |
| SOURCES: MDRC calculations from unemployment insurance (UI) earnings records and AFDC records. NOTE: N/a = not applicable. |
||||
Appendix G: Survey Response Analysis
Data on participation, degree receipt, job quality, income, transitional benefits, health care coverage, child care, child outcomes, and several other measures used in this report come from responses to the Five-Year Client Survey, Child Outcomes Study (COS) survey, and Teacher Survey. This appendix assesses the reliability of impact results for the three survey samples. It also considers whether impacts estimated for sample members who responded to the survey may be generalized to the "eligible" sample that is, to sample members who were randomly assigned during the months when the sample was chosen and who met the criteria for selection.
The analysis addresses the following questions for each sample:
- Are the response rates (sample members interviewed as a percentage of sample members chosen to be interviewed) high enough to satisfy the usual standards of impact analysis?
- Are differences in response rates across research groups small enough to indicate that comparisons between those groups will yield unbiased impact estimates?
- Are impact estimates based on unemployment insurance earnings records and welfare payment records similar for respondents and the eligible samples?
For respondents in the Client Survey and COS survey samples, the findings from these multiple tests were generally, but not entirely, positive. For the LFA and HCD programs in Atlanta and Grand Rapids, impact results for respondents appear reliable and representative of effects for members of the eligible sample. Somewhat greater caution is required in generalizing about the impacts for Riverside LFA survey respondents, especially for the COS sample because (1) COS response rates in Riverside were relatively low and (2) relatively large differences were found when comparing five-year impacts on welfare payments and (for the COS) total earnings for the respondent and eligible samples. More extreme disparity was found in comparing impacts for the eligible and respondent samples for Portland (Client Survey) and for Riverside HCD (both surveys). These results suggest that impacts on survey outcomes for Riverside HCD and Portland are not representative of effects for the eligible sample. It is also uncertain whether results from the Teacher Survey represent program effects for the eligible sample because of low response rates in all sites and, for several programs, inconsistency across samples in estimated earnings and welfare impacts.
The response analysis involves comparing background characteristics and impact results for the following samples drawn from the full research sample:
The survey-eligible sample ("eligibles"). Sample members in the full research sample who were randomly assigned during months in which the survey sample was selected and who met the criteria for inclusion.
The fielded sample ("fieldeds"). Members of the eligible sample who were chosen to be interviewed.
The respondent sample ("respondents"). Members of the eligible sample, chosen to be interviewed (that is, fieldeds), who were interviewed.
The nonrespondent sample ("nonrespondents"). Members of the eligible sample, chosen to be interviewed (that is, fieldeds), who were not interviewed. They could not be located or declined to be interviewed.
II. Survey Selection and Sampling Ratios
As discussed in Chapter 2, in Atlanta, Grand Rapids, and Riverside, the samples for the Five-Year Client Survey, COS survey, and Teacher Survey were nested: COS sample members make up a part of the larger Client Survey sample, and teacher surveys were collected for a portion of the COS sample.
With very few exceptions, the eligible samples originally chosen for the Two-Year Client Survey and COS survey were the same for the later surveys.(1) Sample members were randomly assigned during some, but not all, months of sample intake. (See Chapter 2, Table 2.2.) Limiting the eligible sample in this way can introduce "cohort effects": impact estimates that are especially large or small for sample members randomly assigned during particular months. A cohort effect may occur because members of the survey-eligible sample differ in measured or unmeasured background characteristics from persons randomly assigned in other months. Changes in area labor markets or in program implementation that occur at some point after the start-up of random assignment may also introduce cohort effects for example, by increasing or decreasing a program's relative success in moving welfare recipients from welfare to work. In addition, the research strategy in Atlanta, Grand Rapids, and Riverside required exclusion of sample members with certain background characteristics: teen parents, parents with children under age 3 (in Atlanta and Riverside), men with children aged 3 to 5, people who did not speak either English or Spanish, and people who did not provide information on their education status and children's ages prior to random assignment. Survey "eligibles" in Atlanta, Grand Rapids, and Riverside who were also eligible for selection to the COS sample included women with at least one child aged 3 to 5 at the time of random assignment.
Differences of moderate size that is, not enough to change the overall findings about a program were found for the Atlanta LFA and HCD, Grand Rapids LFA, and Portland programs, when impacts for the survey-eligible and full samples were compared. For instance, five-year earnings impacts for both Atlanta programs were about $1,000 larger when calculated for the eligible sample rather than the full sample. Earnings gains averaged about $700 more for LFA-eligibles in Grand Rapids. In contrast, impacts for program group members in the Portland survey-eligible sample fell about $900 below the increase for the full sample. Differences in impacts were smaller for Riverside LFA and HCD and Grand Rapids HCD.
The fielded samples for the Five-Year Client Survey and COS survey were selected from among the sample members originally fielded for the two-year surveys. Sample selection for these earlier surveys occurred in the following way: The eligible samples in Atlanta, Grand Rapids, Riverside were divided into strata according to sample members' research group, date of random assignment, age of youngest child, and pre-random assignment educational attainment. In Portland, the strata were defined by sample members' research group and date of random assignment. For research purposes, different sampling ratios, ranging from 16 to 100 percent, were used in selecting members of the fielded sample from within each stratum. (The sampling ratio is the percentage of eligible sample members selected.) Sample members were chosen at random within each stratum. Although corrected for, as discussed below, differences in sampling ratios may affect survey impact estimates. For instance, unless the total sample size is large, different sampling ratios increase the likelihood that persons chosen in one research group differ (perhaps in unmeasured characteristics) from persons chosen in another research group.
In Portland and Grand Rapids all sample members fielded at two years were again fielded at five years, whereas in Atlanta and Riverside funding limitations made it necessary to select a subsample from the original fielded sample. For research purposes, MDRC gave priority to members of the Two-Year COS survey fielded sample in selecting the Five-Year Client Survey sample.
Members of the five-year fielded samples included respondents and nonrespondents to the earlier survey interviews. In addition, the boy or girl chosen as the "focal child" for the two-year COS survey continued as the focal child for the five-year study.
By the strictest definition, the fielded sample for the Five-Year COS survey also constituted the fielded sample for the Five-Year Teacher Survey, because researchers were prepared to interview teachers of every COS focal child. However, surveys were attempted only for COS focal children whose mother was interviewed in person at five years and who then gave her written permission to contact her child's school. Thus, an alternative definition of the fielded sample for the Teacher Survey would limit the sample to focal children of COS survey respondents, or even to children whose mothers signed a permission form during their interview.
Survey respondents are members of the fielded sample who completed an interview (or a sufficient portion to be usable for research). The concept is straightforward for the Five-Year Client Survey, but not for the COS survey. Sample members fielded only for the Client Survey could be interviewed by phone or in person. However, sample members fielded for the COS survey were supposed to answer the Client Survey and additional COS survey questions during an in-person session that also included observations of interactions between the COS mother and focal child, an interview with the focal child, and administration of a standard assessment of the focal child's intellectual development. As noted in Chapter 2, 203 mothers in the COS fielded sample had moved too far away to be visited by interviewers or could not participate a COS in-person interview for other reasons. They did, however, answer the Client Survey by phone, including the questions on child care and child outcomes asked of all respondents to the Client Survey. These sample members are counted as respondents to the Client Survey, but not as respondents to the COS survey. Thus, the COS respondent sample is limited to mothers who participated in the in-person interviews, observations, and assessments. Finally, for this analysis, the "respondent" sample for the Teacher Survey includes COS survey respondents with a completed Teacher Survey about the focal child.
For this report, weights were applied to the survey respondent sample to correct for differences in sampling ratios between the strata. In the unweighted fielded survey sample in these sites, strata (that is, sample members who share background characteristics and have the same sampling ratio) with high sampling ratios are overrepresented and strata with low sampling ratios are underrepresented. To make the fielded sample more closely replicate the background characteristics of survey eligibles, weights for each stratum were set to equal the inverse of the sampling ratio for that stratum. For example, a stratum in which 1 eligible person in 4 was chosen would receive a weight of 4 (or 4/1), whereas a stratum in which every eligible person was chosen would receive a weight of 1 (or 1/1). The same weights are used for the respondent sample.
It should be noted that under some conditions impacts for a weighted respondent sample may still be different from those for the eligible sample. For example, this result could occur if very different proportions of program and control group fieldeds answered the survey, or if members of a subgroup within one research group were more likely to be interviewed than their counterparts in a different research group. These issues are addressed in the next section.
Table G.1 shows the response rate, the percentage of the fielded sample who responded, by survey sample, program, and research group. The goal of each survey effort was to obtain responses from at least 70 percent of the fielded sample in every research group. For the Five-Year Client Survey, response rates exceed 70 percent for all programs and research groups (and 80 percent in Atlanta and Grand Rapids) and are high enough to suggest that the survey probably represents the eligible sample. These results inspire confidence in the impacts for respondents.
|
Site and Program |
Number of Fielded Members | Response Rate (%) | Number of Fielded Members | Response Rate (%) | Number of Fielded Members | Response Rate (%) |
|---|---|---|---|---|---|---|
| Five-Year Client Survey | Five-Year Child Outcomes Study Survey | Five-Year Teacher Survey | ||||
| Atlanta Labor Force Attachment | 597 | 86.9 | 349 | 82.8 | 349 | 52.7 |
| Atlanta Human Capital Development | 717 | 82.8 | 473 | 77.6 | 473 | 47.8 |
| Atlanta Control | 649 | 85.1 | 389 | 79.9 | 389 | 49.6 |
| Grand Rapids Labor Force Attachment | 605 | 88.4 | 252 | 84.5 | 252 | 57.1 |
| Grand Rapids Human Capital Development | 619 | 88.4 | 244 | 80.3 | 244 | 49.2 |
| Grand Rapids Control | 606 | 92.7 | 249 | 85.9 | 249 | 57.8 |
| Riverside Labor Force Attachment | 680 | 73.4 | 294 | 62.9 | 294 | 36.7 |
| Riverside Human Capital Development | 509 | 73.9 | 309 | 67.3 | 309 | 42.4 |
| Riverside Control | 933 | 77.2 | 536 | 64.9 | 536 | 41.4 |
| Portland Program | 364 | 77.2 | ||||
| Portland Control | 289 | 77.2 | ||||
| SOURCES: MDRC calculations from the Five-Year Client Survey, Child Outcomes Study survey, and Teacher Survey. NOTES: See Appendix A.2. |
||||||
Response rates also exceed the 70 percent threshold for the COS sample fielded in Atlanta and Grand Rapids. In Riverside, however, COS response rates fell below this standard, ranging from about 63 percent (LFA) to 67 percent (HCD).(2) Therefore, greater caution is required when interpreting results for the COS sample in Riverside.
Response rates were much lower for the Teacher Survey, using the strictest definition of the fielded sample: about 40 percent for Riverside, 50 percent for Atlanta and Grand Rapids HCD, and 58 percent for Grand Rapids LFA and control group members. In each site, response rates increase about 10 percentage points when COS survey respondents constitute the fielded sample. (Results not shown.) These data should be considered the least reliable indicators of program effects on children.
V. Research Group Differences in Response Rates
Different response rates among research groups can be a potential source of bias in research group comparisons. Such differences suggest that research groups may differ by unobservable characteristics that cannot be controlled for and may affect impact estimates. The results for the Five-Year Client Survey fielded samples indicate that within each site response rates for each research group differ by 5 percentage points or less, which should be interpreted as a good result. (See Table G.1.) Variation in response rates was only slighter greater for the smaller COS survey and Teacher Survey fielded samples.
VI. Research Group Differences in Background Characteristics
An additional concern when estimating impacts from survey responses is that research groups may differ in background characteristics that affect future employment, welfare receipt, and other outcomes. Differences in these observable characteristics can be corrected for in the regression impact model and do not pose a large problem. These differences, however, may indicate variation in unobservable characteristics that as noted above, cannot be controlled for in the impact analysis. The following results show that background characteristics differ by research group in three programs.
To determine whether there are any observable program-control differences within the survey respondent sample, the 0/1 dummy variable indicating membership in the program group was regressed on pre-random assignment demographic information for the fielded and the respondent samples. A statistically significant p-value of the R-square of the regression described above indicates that research groups have different background characteristics and that greater caution is required in interpreting impact results. The results show that differences in demographic characteristics are evident in Atlanta HCD, Riverside LFA, and Portland among respondents to the Client Survey and in Riverside LFA among respondents to the COS survey. However, this problem was not severe. Even in programs for which program-control group differences were found, these differences rarely exceeded 5 percentage points.(3) (Results not shown.) No statistically significant differences were found for respondents to the Teacher Survey. (Results not shown.)
VII. A Comparison of Impacts for Survey Respondents and the Eligible Samples
Impacts on five-year earnings and welfare payments based on administrative records were estimated for the eligible and respondent samples for the Client Survey, COS survey, and Teacher Survey. The results are summarized in Figures G.1-G.3. In these figures, impacts for the eligible sample (weighting not required) are compared with the weighted impacts for the respondent sample. Programs that fall near the 45-degree line that is drawn on these figures have similar impacts for the respondent sample and the eligible sample, whereas programs that fall well above or below the 45- degree line show large variation in impacts. Similarity in results suggests that impacts estimated with survey data for respondents represent the effects that would be found for the eligible sample especially on measures that are often affected by employment and welfare levels, such as child care use, health care coverage, and child outcomes. Differences in impact estimates suggest the opposite, though some variation in impacts should be expected because of differences in sample sizes.
For these comparisons a problematic result is considered to have occurred when the impact for respondents exceeded or fell below the impact for the eligible sample by an amount sufficient to change the findings. For example, results would be problematic if a program led to an unusually large impact on total earnings or welfare payments for the eligible sample, based on the range of impacts found in previous evaluations of welfare-to-work programs, but to an unusually small impact when calculated for the respondent sample (or vice versa).(4) Similarly, within a site, results would be problematic if the relative effects of LFA and HCD differed substantially when calculated for the two samples. Under these circumstances, findings on other outcomes may not represent the likely impact for the larger sample.
A. Comparing Impacts for the Client Survey Eligible and Respondent Samples
For the Client Survey eligible and respondent samples, five-year earnings impacts were nearly identical for the Riverside LFA program and relatively close (within $450 to $550) for the Atlanta and Grand Rapids HCD programs (Figure G.1). Impact estimates for the two samples varied by a somewhat larger amount for Atlanta and Grand Rapids LFA ($800 to $900). For both programs, impacts were smaller for the respondent sample, but not by enough to change the overall conclusions that the two programs led to moderately large impacts on earnings.(5)
Figure G.1
Five-Year Earnings and Welfare Impacts for Five-Year Client Survey Respondents and Eligibles
SOURCE: MDRC calculations from the Five-Year Client Survey.
NOTES: See Appendix A.2.
Earnings impacts differed dramatically and are problematic for Riverside HCD and for Portland. For the eligible sample, Portland led to a substantial impact on cumulative earnings (averaging about $850 per year) for the full impact sample, but only a small effect (of about $225 per year) for the respondent sample.(6) The opposite result occurred for Riverside HCD. Five-year earnings impacts for the respondent sample (of about $700 per year) greatly exceed impacts estimated for the eligible sample (about $180 per year). Furthermore, impacts for HCD respondents in Riverside also exceed effects for LFAs, a very different result from what was found for the full sample.(7) For both of these programs, these differences in impacts change the conclusion about program effects and create considerable doubt as to whether other effects estimated with survey data may be generalized to the eligible sample.(8)
Comparisons of impacts on total welfare payments show greater consistency between the eligible and respondent samples. The Riverside LFA and Portland programs led to noticeably larger reductions in average welfare payments over five years when estimated for the respondent sample. However, reductions for the eligible samples in the programs averaged more than 15 percent below control group levels. Therefore, these differences in impacts do not change the overall finding (discussed in Chapter 5) that Riverside LFA and Portland programs led to large decreases in welfare payments.(9)
B. Comparing Impacts for the COS Survey Eligible and Respondent Samples
1. COS Respondents
Five-year earnings impacts were very consistent for Atlanta and Grand Rapids HCD when calculated for the COS eligible and respondent samples. Estimated impacts for Grand Rapids LFA differed to a greater extent (about $1,500 higher for respondents) but still support a conclusion that these programs led to a moderate impact on earnings. Impacts varied more dramatically for the two programs in Riverside, especially for Riverside HCD. For COS survey respondents in Riverside, each program led to unusually large gains in earnings above control group levels that averaged more than $1,200 per year. For Riverside LFA, this change in impacts is less of a problem, because LFAs in the eligible sample averaged close to $800 per year more than control group members, an impact that few programs have achieved for parents of young children. For Riverside HCDs, however, estimated earnings gains more than tripled when calculated for COS survey respondents and lead to a conclusion that the program was much more successful than suggested by impacts for the eligible sample.(10)
Less variation was found in comparing five-year impacts on welfare payments for the COS survey eligible and respondent samples (Figure G.2, lower panel). Reductions in welfare dollars were noticeably different for the respondent sample only for Riverside LFA. However, the impact for the eligible sample was already large for Riverside LFA.
Figure G.2
Five-Year Earnings and Welfare Impacts for Five-Year Child Outcomes Study Respondents and Eligibles
SOURCE: MDRC calculations from the Five-Year Client Survey.
NOTES: See Appendix A.2.
2. Teacher Survey Respondents
As noted above, the "respondent" sample for the Teacher Survey is made up of the mothers in the COS survey respondent sample whose focal child was the subject of a completed Teacher Survey. For most sites and programs, the comparison of five-year impacts on total earnings and welfare payments yielded similar results as the comparison for COS eligibles and all COS respondents discussed above (Figure G.3). The main exception is the Grand Rapids LFA program, which led to earnings impacts that averaged nearly $1,000 per year for Teacher Survey respondents, but only $350 per year for COS survey eligibles. In addition, the Grand Rapids HCD program resulted in a large decrease in welfare payments for COS survey eligibles, but only a small reduction for Teacher Survey respondents.(11)
Figure G.3
Five-Year Earnings and Welfare Impacts for Five-Year Teacher Survey Respondents and Eligibles
SOURCE: MDRC calculations from the Five-Year Client Survey.
NOTES: See Appendix A.2.
1. See Freedman et al., 2000a, Appendix E, for a response analysis of the Two-Year Client Survey. A small number of sample members who were chosen to be surveyed were subsequently discovered to have background characteristics (such as nonproficiency in English or Spanish) that made them ineligible. These sample members were dropped from the eligible samples for the survey at five years. In addition, it was decided to drop single fathers from the Five-Year Client Survey eligible sample, although some were interviewed at two years.
2. In Riverside, response rates would exceed 75 percent for all research groups if the COS mothers who answered only the Client Survey were counted as respondents.
3. An important exception occurred in Portland, where whites made up about 63 percent of program group respondents but 71 percent of control group respondents. As noted in Chapter 7, impacts on earnings were much larger for whites than for African-Americans among members of the full sample.
4. More specifically, programs are considered to have led to a "large" impact on earnings if program group members' average earnings exceeded the control group average by $900 or more per year. Earnings impacts of between $300 and $900 per year are considered "moderate"; and impacts of between $100 and $300 are considered "small." Programs that lead to earnings impacts of less than $100 per year are considered to have resulted in no impact. Thus, in comparing impacts for the respondent and eligible samples, a difference of impacts in earnings of $600 or more per year, or $3,000 or more over five years, could change a finding of "small" impacts to "large" impacts, or vice versa. This difference in impacts would be considered problematic. In contrast, a variation in impacts of $1,000 or less over five years would be unlikely to change the overall findings about a program's relative success in increasing earnings. Differences in impacts of between $1,000 and $3,000 may also cause concern, particularly if one sample shows no effects on earnings and the other sample shows a moderate gain or loss compared with the control group.
Reductions in welfare expenditures of 10 percent or more below the control group average may be considered "large," whereas reductions of between 2 and 5 percent may be considered "small."
5. By standards described in footnote 4, earnings impacts would be characterized as moderate for Grand Rapids LFAs in the eligible sample, but as small for LFAs in the respondents sample. However, the change in impacts is not that large: from about $450 per year for the eligible sample and about $290 per year for the respondent sample.
6. Earnings impacts were only slightly smaller for the eligible sample in Portland than for the full sample. (Results not shown.)
7. Earnings impacts for Riverside LFA and HCD eligible samples are much closer to impacts estimated for the full samples. (Results not shown.)
8. In Portland, two-year impact estimates were much closer for the eligible and respondent samples. See Freedman et al., 2000, Appendix E.
9. For Portland, there is also a large discrepancy in impacts on combined income for the eligible sample (+$1,600) and respondent sample (-$3,900). This result shows again that findings for respondents on survey outcomes are likely not representative of impacts for the eligible (or full) samples. The difference in impacts on combined income was also substantial for Riverside HCD (-$3,000 eligibles; -$500 respondents) and Riverside LFA (-$1,300 eligibles; -$3,300 respondents). (Results not shown.)
10.The same degree of variation in earnings impacts was also seen for Riverside LFAs in need of basic education. (Results not shown.)
11.The discrepancy in earnings impacts is less extreme for Grand Rapids LFA when COS respondents are considered as the eligible sample: $1,000 per year above the control group compared with about $650 per year. For Grand Rapids HCD, however, impact estimates were only slightly closer when COS respondents were compared with Teacher Survey respondents.
Appendix H: A Comparison of Impacts Estimated from Survey and UI Earnings Data
Employment and earnings impacts in this report are estimated from statewide-automated unemployment insurance (UI) earnings records and from responses to the Two-Year Client Survey and Five-Year Client Survey. This appendix compares employment impacts from these sources over five years and during year 5 and investigates why they differ in some programs. The results demonstrate that surveys sometimes recorded jobs that were missed by statewide UI earnings reporting systems and other times underreported employment. Further, in some sites program and control groups varied in the degree to which employment was underreported on the survey or with UI records.
I. Possible Reasons for Differences Between Survey and UI Earnings Data
Survey data are self-reported. They include jobs that are not covered or not reported to the state UI system, such as self-employment, some domestic work, federal government or military jobs, informal employment, or out-of-state jobs. UI earnings data, however, may include jobs that respondents fail to recall or are reluctant to report on the survey. Survey respondents may also have had problems recalling start and end dates of some jobs, particularly those that started early in the follow-up and lasted for a short period of time. On the other hand, some employers may have delayed reporting employment to the UI system until after the files were created for this report.(1)
Furthermore, survey and UI earnings data presented in this report cover somewhat different time periods. UI earnings data were available for each quarter of follow-up, whereas the two survey interviews recorded employment in a less complete way. The two-year survey collected information on up to six jobs during years 1 and 2, but the five-year survey recorded only information about respondents' current or most recent job.(2) In addition, the indicator of "ever employed during years 1 to 5" used in the report covers somewhat different follow-up periods, when measured with UI earnings or survey data. UI data cover quarters 2-21 after random assignment, whereas the follow-up period for survey data extended from the random assignment month to the interview date, which for some respondents occurred several months after the end of year 5.(3) Moreover, year 5 survey impacts cover months 48-60, starting and ending slightly earlier than the follow-up for UI earnings for most sample members.(4)
II. Reporting Discrepancies for Sample Members with Both Survey and UI Earnings Data
One potential source of differences in impact estimates from survey and UI earnings data is discrepant reporting. To see if this were a problem, for each sample member in the survey respondent sample, employment reported any time after random assignment from the survey data was directly compared with UI earnings recorded during quarters 2-21 (or years 1 to 5).(5) A second comparison was made for year 5. The results are shown in Table H.1.
|
Site and Program |
Five-Year Survey Only (%) | UI Earnings Data Only (%) |
|---|---|---|
|
Years 1 to 5 |
||
| Atlanta | 5.9 | 6.2 |
| Grand Rapids | 5.9 | 2.4 |
| Portland | 12.9 | 1.5 |
| Riverside | 15.3 | 2.1 |
|
Year 5 |
||
| Atlanta | 8.5 | 10.0 |
| Grand Rapids | 11.1 | 8.7 |
| Portland | 17.4 | 9.5 |
| Riverside | 18.5 | 5.7 |
|
SOURCE: MDRC calculations from the Five-Year Client Survey and unemployment insurance(UI) records. |
||
For each comparison, a match occurred if both sources recorded earnings during at least one quarter of follow-up or if neither source recorded earnings during any quarter. Match rates over five years ranged from 82.6 percent in Riverside to 91.7 percent in Grand Rapids when program and control group members were considered together. As shown in Table H.1, patterns of discrepancies differed by site. In Atlanta and Grand Rapids, relatively few mismatches occurred. In Atlanta, neither source recorded a higher incidence of employment, and in Grand Rapids the employment level was only slightly higher when recorded with survey data. Discrepancies in reported employment occurred most often in Portland and Riverside, and nearly all mismatches resulted from employment that was recorded only on the survey.
Results for year 5 show a similar pattern, except that the overall match rates were lower for each site, ranging from 73.1 percent in Portland to 81.5 percent in Atlanta. As with the five-year employment measure, in Atlanta and Grand Rapids about the same percentage of respondents had information on employment recorded only on UI earnings records as had information recorded only from their survey responses. However, in Portland and, especially, in Riverside survey responses provided considerably higher rates of employment than UI records.
III. Observed Patterns of Differences Between Survey and UI Earnings Impacts
Table H.2 compares the incidence of employment in years 1 to 5 for program and control group survey respondents in each program, as well as program impacts, estimated from UI earnings (row 1: records impact) and survey responses (row 2: survey impact). A comparison of these two rows highlights the difference in estimates from survey and UI earnings data.
|
Site and Program |
Program Group (%) | Control Group (%) | Difference (Impact) | Percentage Change (%) |
|---|---|---|---|---|
|
Ever employed in years 1 to 5 |
||||
|
Atlanta Labor Force Attachment |
||||
| Records impact: survey sample | 84.3 | 84.6 | -0.3 | -0.4 |
| Survey impact: survey sample | 83.3 | 82.9 | 0.4 | 0.4 |
|
Atlanta Human Capital Development |
||||
| Records impact: survey sample | 83.4 | 84.6 | -1.2 | -1.4 |
| Survey impact: survey sample | 81.2 | 82.9 | -1.7 | -2.0 |
|
Grand Rapids Labor Force Attachment |
||||
| Records impact: survey sample | 90.9 | 88.4 | 2.5 | 2.8 |
| Survey impact: survey sample | 94.7 | 91.9 | 2.9* | 3.1 |
|
Grand Rapids Human Capital Development |
||||
| Records impact: survey sample | 90.3 | 88.4 | 1.9 | 2.1 |
| Survey impact: survey sample | 93.6 | 91.9 | 1.7 | 1.9 |
|
Riverside Labor Force Attachment |
||||
| Records impact: survey sample | 74.3 | 65.0 | 9.3*** | 14.3 |
| Survey impact: survey sample | 85.7 | 80.8 | 4.9** | 6.0 |
|
Riverside Human Capital Development |
||||
| Records impact: survey sample | 69.3 | 60.0 | 9.2*** | 15.4 |
| Survey impact: survey sample | 82.8 | 74.2 | 8.6*** | 11.6 |
|
Portland |
||||
| Records impact: survey sample | 84.6 | 84.0 | 0.5 | 0.6 |
| Survey impact: survey sample | 93.5 | 93.3 | 0.2 | 0.2 |
|
SOURCE: MDRC calculations from state and county administrative records. |
||||
From the standpoint of consistency, the preferred result would be (by definition) for both sources to record the same information for each person. This result occurred in Atlanta, where survey responses and UI records captured the same employment levels for LFAs, HCDs, and control group members, as well as similar impacts. Results for Grand Rapids were nearly as good. For all three research groups, survey responses display employment levels that were between 3 and 4 percentage points higher than were recorded with UI earnings data, but impacts for LFAs and HCDs were similar when calculated with either data source. The next best result occurs when both program and control group members have similar rates of discrepant reporting, because impacts estimated from UI earnings and survey data will be similar. This situation is demonstrated by results for Riverside HCD and Portland. As shown in Table H.2, the survey records higher employment levels than do UI earnings (especially in Riverside); but differences are consistent for program and control groups, leaving impact levels nearly unchanged.
Variation in rates of discrepant reporting by research group is more problematic, because it affects impact results. As shown in Table H.2, this result occurred only for Riverside LFA. In Riverside, the proportion of control group members who ever worked for pay was nearly 16 percentage points higher when recorded from survey responses than UI earnings data. LFAs also reported a higher incidence of employment on the survey, but the discrepancy was not as great (11 percentage points). As a result, survey impacts on employment were more than 4 percentage points smaller than UI earnings impacts for the survey respondent sample. Nonetheless, both sources record a statistically significant impact on employment for Riverside LFA.(6)
The results for year 5 are more problematic, in that most programs show at least a small positive difference in employment when calculated with survey data, but not when calculated with UI earnings (Table H.3). The reasons for this discrepancy differ by site and program. In general, program group members were more likely than control group members to have higher levels of employment recorded on the survey and, correspondingly, less likely to have higher levels of employment recorded on UI earnings. The difference is most extreme in Portland, where the program-control group difference changes from a loss in employment when calculated with UI records to a gain when calculated with survey data, although neither difference was statistically significant.(7)
|
Site and Program |
Program Group (%) | Control Group (%) | Difference (Impact) | Percentage Change (%) |
|---|---|---|---|---|
|
Ever employed in year 5 |
||||
|
Atlanta Labor Force Attachment |
||||
| Records impact: survey sample | 68.1 | 69.5 | -1.3 | -1.9 |
| Survey impact: survey sample | 69.0 | 65.2 | 3.7 | 5.7 |
|
Atlanta Human Capital Development |
||||
| Records impact: survey sample | 69.2 | 69.5 | -0.3 | -0.5 |
| Survey impact: survey sample | 64.8 | 65.2 | -0.4 | -0.6 |
|
Grand Rapids Labor Force Attachment |
||||
| Records impact: survey sample | 74.7 | 74.6 | 0.1 | 0.1 |
| Survey impact: survey sample | 79.5 | 75.7 | 3.8 | 5.0 |
|
Grand Rapids Human Capital Development |
||||
| Records impact: survey sample | 72.0 | 74.6 | -2.6 | -3.5 |
| Survey impact: survey sample | 77.7 | 75.7 | 2.1 | 2.7 |
|
Riverside Labor Force Attachment |
||||
| Records impact: survey sample | 49.1 | 45.7 | 3.4 | 7.5 |
| Survey impact: survey sample | 63.8 | 57.6 | 6.2** | 10.8 |
|
Riverside Human Capital Development |
||||
| Records impact: survey sample | 47.8 | 41.7 | 6.1* | 14.7 |
| Survey impact: survey sample | 58.0 | 49.2 | 8.8** | 17.8 |
|
Portland |
||||
| Records impact: survey sample | 57.6 | 64.2 | -6.7 | -10.4 |
| Survey impact: survey sample | 60.9 | 56.0 | 4.9 | 8.7 |
| SOURCE: MDRC calculations from state and county administrative records. NOTES: See Appendix A. |
||||
1. In addition, sample members' Social Security number is needed to match to UI earnings. Some Social Security numbers are reported or recorded incorrectly at random assignment. Sometimes the incorrect number does not match to any UI earnings records; other times it matches to the records for another person.
2. In addition, 434 respondents to the Five-Year Client Survey were not interviewed after two years.
3. As noted in the report, UI data are recorded quarterly. Quarter 1 may include earnings from before random assignment and is therefore excluded from the analysis of program impacts. Quarters 2-21 correspond to months 2 to 61, 3 to 62, or 4 to 63 after random assignment, depending on whether sample members were randomly assigned during the first, second, or third month of a calendar quarter.
4. Year 5 as measured with UI earnings records includes quarters 18-21, which correspond to months 50 to 61, 51 to 62, or 52 to 63, depending on the respondent's month of random assignment.
5. Some sample members were interviewed after the follow-up period for UI earnings and were excluded from this comparison.
6. The survey results for Riverside show that joblessness was not as pervasive as indicated by UI data. Interestingly, in all four sites respondents' most recent jobs that were reported only on the survey provided fewer hours of work per week on average and lower hourly pay and were much less likely to provide medical coverage than jobs that were reported on both sources. This nonexperimental finding suggests that sample members in Riverside relied more on self-employment, household employment, and service jobs for small employers than those in the other sites in the evaluation. This result is still consistent with the overall characterization of Riverside as having a weaker labor market than the other sites in the evaluation.
7. The p-value of the UI earnings impact was .13.
Appendix I: Impacts on Employment, Welfare, Income, and Related Outcomes for the Client Survey, Child Outcomes Study Survey, and Teacher Survey Samples
This appendix presents a brief summary of program impacts on employment, earnings, welfare payments, combined income, and other outcomes calculated with administrative records and survey responses for the three survey samples. It then compares program impacts for each sample with impacts calculated for the larger samples from which its members were selected, including (for administrative data) the full impact sample. These findings provide context for interpreting program impacts on other outcomes measured for each survey sample, including child care use and cost and child outcomes.
Table I.1 (for administrative records data) and Table I.2 (for survey data) show the control means and impacts for the three survey samples and (for Table I.1) for the full impact sample. There are several reasons why impacts for a particular outcome may vary by sample. Samples may be chosen from different months of random assignment or selected from different subgroups. In addition, response rates may vary for different samples (or may differ by research group within a sample), and the samples may differ in the degree to which certain subgroups may be over- or underrepresented among respondents. This appendix does not attempt to explain which source of variation accounts for differences in impacts across samples.(1)
|
Site |
Full Sample | Client Survey Sample | COS Sample | Teacher Survey Sample | ||||
|---|---|---|---|---|---|---|---|---|
| Control Group | Impact | Control Group | Impact | Control Group | Impact | Control Group | Impact | |
|
Employed in years 1 to 5 (%) |
||||||||
| Atlanta Labor Force Attachment | 80.3 | 2.8** | 84.6 | -0.3 | 88.6 | -1.0 | 90.0 | 1.9 |
| Atlanta Human Capital Development | 80.3 | 1.0 | 84.6 | -1.2 | 88.6 | 0.0 | 90.0 | -1.1 |
| Grand Rapids Labor Force Attachment | 88.3 | 1.5 | 88.4 | 2.5 | 91.7 | 4.7** | 91.0 | 6.2** |
| Grand Rapids Human Capital Development | 88.3 | 1.2 | 88.4 | 1.9 | 91.7 | 2.6 | 91.0 | 2.0 |
| Riverside Labor Force Attachment | 66.1 | 8.4*** | 65.0 | 9.3*** | 63.5 | 14.8*** | 68.5 | 10.9** |
| Lacked high school diploma or basic skills | 61.1 | 9.4*** | 60.0 | 11.5*** | 59.1 | 17.4*** | 63.6 | 11.4* |
| Riverside Human Capital Development | 61.1 | 5.8*** | 60.0 | 9.2*** | 59.1 | 13.8*** | 63.6 | 13.7** |
| Portland | 81.7 | 4.1** | 84.0 | 0.5 | ||||
|
Employed in all four quarters of year 5 (%) |
||||||||
| Atlanta Labor Force Attachment | 36.6 | 0.3 | 39.5 | 2.2 | 45.1 | 1.0 | 43.6 | 9.4* |
| Atlanta Human Capital Development | 36.6 | -0.1 | 39.5 | 0.8 | 45.1 | -0.5 | 43.6 | 1.7 |
| Grand Rapids Labor Force Attachment | 38.7 | 0.7 | 41.7 | 3.5 | 43.3 | 7.9* | 43.5 | 15.7*** |
| Grand Rapids Human Capital Development | 38.7 | -0.4 | 41.7 | 0.8 | 43.3 | 2.4 | 43.5 | 6.1 |
| Riverside Labor Force Attachment | 23.2 | 3.1*** | 22.7 | 3.1 | 20.4 | 7.6* | 25.1 | 7.3 |
| Lacked high school diploma or basic skills | 18.8 | 4.3*** | 17.5 | 5.3 | 14.2 | 11.4** | 17.2 | 11.4** |
| Riverside Human Capital Development | 18.8 | 2.7* | 17.5 | 7.1** | 14.2 | 13.0*** | 17.2 | 13.0** |
| Portland | 33.6 | 4.6** | 34.7 | 2.8 | ||||
|
Earnings in years 1 to 5 ($) |
||||||||
| Atlanta Labor Force Attachment | 17,380 | 2,459*** | 18,801 | 2,531* | 20,516 | 2,547 | 20,709 | 4,443* |
| Atlanta Human Capital Development | 17,380 | 2,017** | 18,801 | 2,484* | 20,516 | 2,444 | 20,709 | 2,769 |
| Grand Rapids Labor Force Attachment | 20,770 | 1,552* | 23,532 | 1,446 | 23,340 | 3,285 | 22,450 | 4,898* |
| Grand Rapids Human Capital Development | 20,770 | 846 | 23,532 | 505 | 23,340 | 635 | 22,450 | 1,466 |
| Riverside Labor Force Attachment | 14,889 | 2,549*** | 13,652 | 2,505* | 10,805 | 6,537*** | 13,810 | 6,727*** |
| Lacked high school diploma or basic skills | 10,912 | 2,281*** | 9,412 | 2,697* | 7,279 | 6,932*** | 8,268 | 7,080*** |
| Riverside Human Capital Development | 10,912 | 1,361* | 9,412 | 3,503*** | 7,279 | 6,265*** | 8,268 | 6,343*** |
| Portland | 20,891 | 5,150*** | 21,893 | 1,119 | ||||
|
Total months on welfare in years 1 to 5 (%) |
||||||||
| Atlanta Labor Force Attachment | 37.2 | -2.9*** | 38.5 | -1.8* | 39.1 | -1.6 | 39.3 | -2.8 |
| Atlanta Human Capital Development | 37.2 | -1.9*** | 38.5 | -1.0 | 39.1 | -0.7 | 39.3 | -0.7 |
| Grand Rapids Labor Force Attachment | 31.1 | -4.2*** | 30.0 | -5.1*** | 31.6 | -4.5*** | 31.0 | -4.5** |
| Grand Rapids Human Capital Development | 31.1 | -2.9*** | 30.0 | -2.6*** | 31.6 | -1.5 | 31.0 | 1.1 |
| Riverside Labor Force Attachment | 31.0 | -3.2*** | 34.9 | -6.7*** | 41.7 | -8.0*** | 41.1 | -7.4*** |
| Lacked high school diploma or basic skills | 33.3 | -3.2*** | 37.9 | -7.5*** | 44.0 | -9.2*** | 45.2 | -9.4*** |
| Riverside Human Capital Development | 33.3 | -3.3*** | 37.9 | -3.7*** | 44.0 | -3.1* | 45.2 | -3.5 |
| Portland | 25.3 | -5.6*** | 26.1 | -5.9*** | ||||
|
Total welfare payments in years 1 to 5 ($) |
||||||||
| Atlanta Labor Force Attachment | 9,946 | -881*** | 10,383 | -595* | 10,892 | -663 | 10,916 | -982* |
| Atlanta Human Capital Development | 9,946 | -710*** | 10,383 | -340 | 10,892 | -218 | 10,916 | -120 |
| Grand Rapids Labor Force Attachment | 12,966 | -2,552*** | 12,602 | -2,892*** | 14,027 | -2,987*** | 13,958 | -3,282*** |
| Grand Rapids Human Capital Development | 12,966 | -1,767*** | 12,602 | -1,623*** | 14,027 | -1,249 | 13,958 | -471*** |
| Riverside Labor Force Attachment | 18,294 | -2,710*** | 20,315 | -4,613*** | 25,450 | -5,707*** | 25,060 | -5,548 |
| Lacked high school diploma or basic skills | 20,126 | -2,955*** | 22,634 | -5,513*** | 27,355 | -6,817*** | 28,342 | -7,521*** |
| Riverside Human Capital Development | 20,126 | -2,949*** | 22,634 | -3,080*** | 27,355 | -3,134*** | 28,342 | -3,353** |
| Portland | 11,686 | -2,746*** | 12,240 | -3,225*** | ||||
|
Total combined income in years 1 to 5 ($) |
||||||||
| Atlanta Labor Force Attachment | 39,987 | 1,152 | 42,756 | 1,753 | 46,069 | 1,909 | 46,397 | 3,414* |
| Atlanta Human Capital Development | 39,987 | 1,133 | 42,756 | 1,854 | 46,069 | 2,019 | 46,397 | 2,307 |
| Grand Rapids Labor Force Attachment | 42,172 | -1,433* | 44,750 | -1,911 | 46,927 | 52 | 46,167 | 1,074 |
| Grand Rapids Human Capital Development | 42,172 | -1,247 | 44,750 | -1,398 | 46,927 | -302 | 46,167 | 1,556 |
| Riverside Labor Force Attachment | 39,804 | -875 | 41,446 | -3,334** | 45,432 | -646 | 48,211 | -310 |
| Lacked high school diploma or basic skills | 38,311 | -1,280 | 40,334 | -4,212** | 44,323 | -1,442 | 47,093 | -2,271 |
| Riverside Human Capital Development | 38,311 | -2,387*** | 40,334 | -525 | 44,323 | 2,300 | 47,093 | 1,521 |
| Portland | 41,807 | 1,870 | 44,691 | -3,875* | ||||
| SOURCE: MDRC calculations from state and county administrative records. NOTES: See Appendix A. |
||||||||
| Site | Client Survey Sample | COS Sample | Teacher Survey Sample | |||
|---|---|---|---|---|---|---|
| Control Group (%) | Impact | Control Group (%) | Impact | Control Group (%) | Impact | |
| Received education credential since random assignment | ||||||
| Atlanta Labor Force Attachment | 10.7 | 4.8** | 12.8 | 0.8 | 10.9 | 5.4 |
| Atlanta Human Capital Development | 10.7 | 10.9*** | 12.8 | 14.6*** | 10.9 | 12.9*** |
| Grand Rapids Labor Force Attachment | 26.1 | -7.4*** | 29.1 | -11.3*** | 27.4 | -11.1** |
| Grand Rapids Human Capital Development | 26.1 | 2.2 | 29.1 | 2.3 | 27.4 | 4.2 |
| Riverside Labor Force Attachment | 19.1 | -0.4 | 19.5 | 3.2 | 19.6 | 3.0 |
| Lacked high school diploma or basic skills | 14.5 | 3.0 | 14.8 | 5.5 | 15.9 | 7.5 |
| Riverside Human Capital Development | 14.5 | 10.6*** | 14.8 | 14.7*** | 15.9 | 11.9** |
| Employed at interview | ||||||
| Atlanta Labor Force Attachment | 54.3 | 2.9 | 59.6 | -0.8 | 63.7 | -1.6 |
| Atlanta Human Capital Development | 54.3 | -1.4 | 59.6 | 0.3 | 63.7 | -3.0 |
| Grand Rapids Labor Force Attachment | 67.7 | -2.6 | 67.2 | 3.9 | 64.9 | 8.8 |
| Grand Rapids Human Capital Development | 67.7 | -1.5 | 67.2 | -1.4 | 64.9 | 7.2 |
| Riverside Labor Force Attachment | 48.9 | 6.1** | 44.7 | 6.2 | 50.0 | 6.8 |
| Lacked high school diploma or basic skills | 43.0 | 5.9 | 37.3 | 9.7* | 44.9 | 9.2 |
| Riverside Human Capital Development | 43.0 | 8.0** | 37.3 | 13.5** | 44.9 | 8.5 |
| Employed full time at interview | ||||||
| Atlanta Labor Force Attachment | 45.5 | 1.7 | 51.6 | 0.3 | 55.0 | -2.2 |
| Atlanta Human Capital Development | 45.5 | -1.4 | 51.6 | -0.3 | 55.0 | -3.3 |
| Grand Rapids Labor Force Attachment | 54.3 | -0.8 | 55.2 | 0.7 | 51.7 | 6.8 |
| Grand Rapids Human Capital Development | 54.3 | -1.3 | 55.2 | -3.1 | 51.7 | 7.9 |
| Riverside Labor Force Attachment | 35.9 | 8.2*** | 29.6 | 11.0*** | 32.3 | 13.8*** |
| Lacked high school diploma or basic skills | 31.8 | 8.4** | 23.7 | 15.5*** | 28.6 | 19.3*** |
| Riverside Human Capital Development | 31.8 | 4.9 | 23.7 | 9.0* | 28.6 | 6.5 |
| Employed and using child care at interview | ||||||
| Atlanta Labor Force Attachment | 21.5 | 0.8 | 38.9 | 1.9 | 42.4 | 0.5 |
| Atlanta Human Capital Development | 21.5 | -1.1 | 38.9 | -2.0 | 42.4 | -3.7 |
| Grand Rapids Labor Force Attachment | 30.7 | 4.3* | 49.5 | 2.4 | 47.4 | 6.6 |
| Grand Rapids Human Capital Development | 30.7 | 1.4 | 49.5 | -0.7 | 47.4 | 6.7 |
| Riverside Labor Force Attachment | 26.6 | 0.8 | 35.1 | 3.3 | 41.1 | 2.5 |
| Lacked high school diploma or basic skills | 25.2 | 0.3 | 31.5 | 3.3 | 40.7 | -1.3 |
| Riverside Human Capital Development | 25.2 | -0.2 | 31.5 | 2.1 | 40.7 | -8.5 |
| Respondent and all children had medical coverage in month before interview | ||||||
| Atlanta Labor Force Attachment | 71.1 | -1.0 | 74.3 | -1.5 | 77.8 | -7.5* |
| Atlanta Human Capital Development | 71.1 | 0.2 | 74.3 | 0.2 | 77.8 | 0.0 |
| Grand Rapids Labor Force Attachment | 70.9 | -1.7 | 76.4 | -1.7 | 74.7 | -0.3 |
| Grand Rapids Human Capital Development | 70.9 | -1.2 | 76.4 | -4.3 | 74.7 | 0.9 |
| Riverside Labor Force Attachment | 77.6 | -1.9 | 83.7 | -3.1 | 85.5 | -3.0 |
| Lacked high school diploma or basic skills | 77.6 | 0.2 | 84.6 | 0.4 | 85.7 | 3.1 |
| Riverside Human Capital Development | 77.6 | 0.8 | 84.6 | -2.9 | 85.7 | -3.8 |
| Married and living with spouse | ||||||
| Atlanta Labor Force Attachment | 8.4 | 1.3 | 7.7 | 2.3 | 7.3 | 2.7 |
| Atlanta Human Capital Development | 8.4 | -1.5 | 7.7 | 0.9 | 7.3 | 2.0 |
| Grand Rapids Labor Force Attachment | 20.5 | 2.3 | 22.8 | 2.6 | 24.0 | 3.4 |
| Grand Rapids Human Capital Development | 20.5 | -0.2 | 22.8 | 1.1 | 24.0 | -5.0 |
| Riverside Labor Force Attachment | 22.0 | -1.4 | 21.2 | 2.4 | 25.6 | -0.8 |
| Lacked high school diploma or basic skills | 18.1 | 0.5 | 18.4 | 2.7 | 21.6 | -1.5 |
| Riverside Human Capital Development | 18.1 | 3.7 | 18.4 | 0.8 | 21.6 | 0.4 |
| Gave birth to another child since random assignment | ||||||
| Atlanta Labor Force Attachment | 12.4 | -0.8 | 20.5 | -1.4 | 24.3 | -3.7 |
| Atlanta Human Capital Development | 12.4 | 0.1 | 20.5 | 0.1 | 24.3 | -0.5 |
| Grand Rapids Labor Force Attachment | 21.7 | 0.9 | 29.4 | -2.2 | 32.6 | -9.2* |
| Grand Rapids Human Capital Development | 21.7 | 0.5 | 29.4 | 0.4 | 32.6 | -2.8 |
| Riverside Labor Force Attachment | 22.1 | 3.4 | 31.0 | 10.4*** | 32.4 | 6.6 |
| Lacked high school diploma or basic skills | 23.1 | 5.1* | 34.0 | 9.6** | 37.5 | 1.8 |
| Riverside Human Capital Development | 23.1 | 1.0 | 34.0 | -0.7 | 37.5 | -5.7 |
| Living with another wage earner in month before interview | ||||||
| Atlanta Labor Force Attachment | 27.6 | 1.5 | 24.2 | 3.6 | 29.3 | -2.2 |
| Atlanta Human Capital Development | 27.6 | 2.6 | 24.2 | 4.2 | 29.3 | 1.9 |
| Grand Rapids Labor Force Attachment | 43.0 | 3.8 | 41.4 | 4.7 | 42.8 | 5.8 |
| Grand Rapids Human Capital Development | 43.0 | -2.9 | 41.4 | 2.3 | 42.8 | -1.3 |
| Riverside Labor Force Attachment | 42.2 | -4.7* | 38.2 | -2.6 | 39.6 | -4.7 |
| Lacked high school diploma or basic skills | 39.7 | -7.1** | 37.2 | -6.0 | 37.1 | -5.3 |
| Riverside Human Capital Development | 39.7 | -0.1 | 37.2 | -0.9 | 37.1 | -2.6 |
| At least one household member received $1,000 or more in earnings | ||||||
| Atlanta Labor Force Attachment | 9.8 | 0.4 | 11.8 | 0.9 | 15.2 | -3.0 |
| Atlanta Human Capital Development | 9.8 | -0.8 | 11.8 | 3.5 | 15.2 | 2.7 |
| Grand Rapids Labor Force Attachment | 21.9 | 0.7 | 26.4 | 1.7 | 27.4 | 0.4 |
| Grand Rapids Human Capital Development | 21.9 | -0.7 | 26.4 | -0.8 | 27.4 | -1.6 |
| Riverside Labor Force Attachment | 21.1 | -0.7 | 20.5 | 1.4 | 22.6 | -2.1 |
| Lacked high school diploma or basic skills | 17.5 | -2.5 | 17.3 | -2.8 | 17.2 | -1.6 |
| Riverside Human Capital Development | 17.5 | 2.6 | 17.3 | 0.9 | 17.2 | 0.1 |
| SOURCE: MDRC calculations from the Five-Year Client Survey, Child Outcomes study, and Teacher Survey. NOTES; See Appendix A.2 |
||||||
As discussed below, for Riverside HCDs impacts are larger and more positive in each survey sample than for the full impact sample. In addition, effects were generally more positive for the COS sample and, especially, the Teacher Survey sample than for the full impact sample. In contrast, impact results were less positive for Portland's Client Survey respondents than for the full impact sample.
I. Impacts for the Five-Year Client Survey Sample
A. Administrative Records Outcomes
For the Client Survey sample, most programs did not increase job finding (percentage ever employed) over five years, but both programs in Atlanta and Riverside led to impacts on earnings of between $2,500 and $3,500. The other programs led to positive differences in total earnings that were small and not statistically significant. All programs also reduced months of welfare receipt and total payments over five years, though the Atlanta HCD-control group differences on these measures were small and not statistically significant. Impacts on combined income over five years showed greater variation by program. Both Atlanta programs led to small increases in income, whereas both Grand Rapids programs reduced income by a small amount. (None of these differences were statistically significant.) Relatively large losses in income (between $3,000 and $4,000) over five years were found for Riverside LFA and Portland. Riverside HCD had almost no effect on income.
The pattern of impacts for the Client Survey sample resembled impacts for the full impact sample for both programs in Atlanta and Grand Rapids, but varied substantially for Riverside LFA and HCD and for Portland. For Riverside LFA, employment and earnings impacts were similar for the two samples, but welfare reductions were considerably larger for the Client Survey sample. As a result, Riverside LFAs in the survey sample show a much larger reduction in combined income over five years than those in the full impact sample. A large disparity in impacts was found for Riverside HCD as well, but in the opposite direction. For the Client Survey sample, the program led to substantial gains in total earnings over five years ($3,503) and in stable employment in year 5 (7.1 percentage points); these effects were not seen for the full impact sample. Reductions in welfare were similar for the two samples, however. Therefore, the effect on combined income is more positive for Riverside HCDs in the Client Survey sample. Finally, the Portland program had little effect on total earnings over five years for the Client Survey sample, but unusually large increases for the full impact sample. Moreover, the Portland program led to a relatively large decrease in combined income for the Client Survey sample, but a small (and not statistically significant) increase for the full impact sample. Differences for these three programs should be kept in mind when interpreting impacts on other outcomes.
B. Survey Outcomes
Table I.2 shows program impacts on a selection of outcomes for parents and families that may indirectly affect the well-being of children. As discussed in the text, programs led to relatively few impacts on these measures for the Client Survey sample. Of note, the Atlanta and Riverside HCD programs led to relatively large increases in attainment of an education credential after random assignment and both Riverside programs increased employment at the end of five years. Grand Rapids LFA did not increase employment at the end of year 5, but led to a small increase in the percentage of Client Survey respondents who used child care for employment.
II. Impacts for the Five-Year Child Outcomes Study sample
A. Administrative Records Outcomes
Most programs led to positive impacts on employment or on earnings for COS sample although some differences were not statistically significant. Both Riverside programs raised average earnings by more than $6,000 per program group member, an unusually large amount. These two programs and the Grand Rapids LFA program also led to impacts on stable employment in year 5. All programs reduced months on welfare and total payments over five years, but effects were small and not statistically significant for both Atlanta programs and for Grand Rapids HCD. Impacts on combined income were mixed. The Riverside HCD program and both Atlanta programs led to small and not statistically significant increases in income, whereas both Grand Rapids programs and the Riverside LFA program had almost no effect.
To some extent, the pattern of impact is more positive for the COS sample than for the Client Survey and full impact samples, particularly for the measure of combined income. The biggest differences in impact estimates are for the two Riverside programs, especially Riverside HCD. That program's five-year earnings gains for the COS sample exceeded impacts for the full impact sample by nearly $5,000 per sample member. The difference in impacts on combined income was nearly as large. For Grand Rapids LFAs, results were also more positive for the COS sample than for the Client Survey or full impact sample.(2)
B. Survey Outcomes
For the COS sample, the pattern of impacts on survey outcomes looks similar to impacts for all respondents to the Client Survey. The two Riverside programs led to the most positive results. They increased employment and employment with full-time hours at the end of five years. Riverside HCD also increased degree attainment by nearly 15 percentage points above the control group level.
Less positively, Riverside LFA increased the proportion of COS sample members who gave birth to another child after random assignment by 10 percentage points above the control group average. Only a small difference was found when all Client Survey respondents in Riverside were included in the calculation.
III. Impacts for the Teacher Survey Sample
A. Administrative Records Outcomes
In general, COS families who participated in the Teacher Survey experienced the most positive impacts of the four research samples. All programs increased five-year earnings above control group levels, although some differences were small and not statistically significant. Moreover, impacts on earnings averaged more than $800 per year for the three LFA programs and for Riverside HCD. Most programs also increased stable employment in year 5. All programs reduced total welfare payments over five years, but effects were small and not statistically significant for Atlanta and Grand Rapids HCD. Over five years, all programs except Riverside LFA increased combined income, although effects were small and not statistically significant in most programs.
The disparity of impact results across samples was greatest for Atlanta and Grand Rapids LFA. These programs led to much more positive effects on earnings, on stable employment, and on combined income for the Teacher Survey sample, than for the other samples. Furthermore, as for the COS sample, Riverside HCD led to much larger impacts on stable employment and earnings and more positive effects on combined income for the Teacher Survey sample than for the full impact and Client Survey samples.
B. Survey Outcomes
For survey outcomes, more program-control group differences of at least moderate size (+/- 5 percentage points) were found for the Teacher Survey sample than for the other survey samples. However, most of these differences were not statistically significant. As for the other samples, both Riverside programs increased employment at the end of year 5 above control group levels. In addition, a higher percentage of LFAs gave birth to a child during the follow-up, an effect also seen for all LFAs in the Riverside COS sample. Unlike its effect on the COS and Client Survey samples, Riverside HCD decreased below control group levels the proportion of Teacher Survey sample members who were working and using child care at the end of year 5 and decreased the incidence of childbearing after random assignment.(3)
The Grand Rapids LFA and HCD programs also led to larger effects for the Teacher Survey sample than for the COS samples, although not all differences were statistically significant. Each program increased employment and employment with full-time hours at the end of year 5. Grand Rapids LFAs were also more likely than control group members to be living with another person who was working for pay, more likely to be using child care for employment at the end of year 5, and less likely to have given birth to a child after random assignment. The Grand Rapids HCD program also increased child care use for employment at the end of year 5 and decreased the incidence of being married and living with spouse.
Finally, programs had as little effect on medical coverage at the end of year 5 for the Teacher Survey sample as they did for the other survey samples. However, Atlanta LFA led to a statistically significant decrease in coverage, a result not seen for the other samples.
1. See Appendix G for a discussion of these issues.
2. It should be remembered that both the full impact and Client Survey samples include parents whose youngest child was aged 6 or over at random assignment, but the COS sample does not.
3. For Riverside LFA, only the impact on full-time employment was statistically significant.
Appendix J: Child Outcome Measures
|
Measure |
Source | For Whom Available | Description |
|---|---|---|---|
|
Social Skills and Behavior a |
|||
| Externalizing behavior | Mother | Focal child | Includes such behaviors as fighting and arguing with others. Scores could range from 0 to 18. Cronbach's alpha of .78. |
| Teacher | Focal child | Includes behaviors such as fighting and arguing. Scores could range from 0 to 15. Cronbach's alpha of .89. | |
| Internalizing behavior | Mother | Focal child | Includes such behaviors as acting sad and being anxious about being with others. Scores could range from 0 to 24. Cronbach's alpha of .66. |
| Teacher | Focal child | Includes such behaviors as acting sad and being anxious about being with others. Scores could range from 0 to 18. Cronbach's alpha of .75. | |
| Hyperactivity | Mother | Focal child | Includes such behaviors as disrupting others and acting without thinking. Scores could range from 0 to 18. Cronbach's alpha of .76. |
| Teacher | Focal child | Includes such behaviors as disturbing others and acting without thinking. Scores could range from 0 to 6. Cronbach's alpha of .73. | |
| Cooperation | Child | Focal child | Includes such behaviors as listening to adults and finishing schoolwork and homework on time. Scores could range from 0 to 30. Cronbach's alpha of .77. |
| Mother | Focal child | Includes items addressing neatness, helpfulness at home, and giving of compliments. Scores could range from 0 to 39. Cronbach's alpha of .85. | |
| Teacher | Focal child | Includes such school-related behaviors as staying focused and completing work. Scores could range from 0 to 27. Cronbach's alpha of .92. | |
| Positive assertion | Child | Focal child | Includes ratings by children themselves on the frequency with which they demonstrate behaviors such as being active with other children and properly handling children who are behaving inappropriately. Scores could range from 0 to 30. Cronbach's alpha of .71. |
| Mother | Focal child | Includes frequency with which children demonstrate such skills as initiating interaction with other children and making friends. Scores could range from 0 to 30. Cronbach's alpha of .83. | |
| Teacher | Focal child | Includes frequency with which children show such skills as interacting well with other children and making friends. Scores could range from 0 to 18. Cronbach's alpha of .84. | |
| Self-control | Child | Focal child | Includes ratings by children themselves on the frequency with which they demonstrate behaviors such as controlling anger and properly handling children who are behaving inappropriately. Scores could range from 0 to 30. Cronbach's alpha of .69. |
| Mother | Focal child | Includes such behaviors as speaking politely and responding appropriately to conflict. Scores could range from 0 to 30. Cronbach's alpha of .78. | |
| Teacher | Focal child | Includes such behaviors as responding appropriately to conflict and interacting well with peers. Scores range from 0 to 27. Cronbach's alpha of .92. | |
| Empathy | Child | Focal child | Includes such behaviors as listening to others and trying to understand others' feelings. Scores could range from 0 to 30. Cronbach's alpha of .79. |
| Responsibility | Mother | Focal child | Includes such behaviors as speaking politely to others and listening to others. Scores could range from 0 to 27. Cronbach's alpha of .80. |
| Interpersonal skills | Teacher | Focal child | Includes such skills being sensitive to others and making friendships. Scores could range from 0 to 12. Cronbach's alpha of .77. |
| Positive approaches to learning | Teacher | Focal child | Includes such habits as working carefully and being creative. Scores could range from 0 to 18. Cronbach's alpha of .85. |
| Academic Functioning | |||
| School engagement | Childb | Focal child | From 0 to 3 on seven items measuring school engagement, where 0 = "never true"; 1 = "sometimes true"; 2 = "often true"; 3 = "very often true." Scores therefore range from 0 to 21, with appropriate items reverse-coded so that higher scores indicate more engagement in schools. The items are:
"When I'm in school, I feel happy." "I work very hard in school." "When I'm in school, I feel bored." "When I'm in school, I pay attention in class." "I try to learn as much as I can about my school subjects." "When I'm in class, I just pretend to work." "When I'm in class, I try very hard." Cronbach's alpha of .75. |
| Teacher | Focal child | Where 1 = "very true"; 2 = "sort of true"; 3 = "not very true"; 4 = "not at all true":
"This student . . . Comes to class unprepared." Pays attention." Works as hard as s/he can." Does more than is required." Is interested in schoolwork outside of the classroom." Thinks about other things in class." "In my class, this student seems . . .A million miles away." Interested." Tuned in." Sleepy." Angry." Responses to the items were recoded and given values from 0 to 3. Scores therefore range from 0 to 33. Cronbach's alpha of .92. |
|
| Suspended or expelled since last interview (%) | Mother | Each Child | "Since the last interview, (has this child/have any of these children) ever been suspended, excluded, or expelled from school?" |
| Had disciplinary action taken weekly (%) | Teacher | Focal child | "In the last month or two, how often have you taken any disciplinary action regarding this student? (1) Several times a week; (2) About once a week; (3) One to three times a month; (4) Once in awhile, less than once a month; (5) Never." Categories (1) and (2) were collapsed and the responses were dichotomized to determine whether teachers reported that disciplinary action had taken place weekly. |
| Had a discipline problem that resulted in parent(s) being notified this school year (%) | Teacher | Focal child | "So far this school year, has this student had any behavior or discipline problem at school that resulted in his/her parent(s) or guardian(s) being sent a note or being asked to come in to meet with you or the principal? (1) No, never; (2) Yes, once; (3) Yes, more than once." Categories (2) and (3) were collapsed and the responses were dichotomized to determine whether focal children had a discipline problem that resulted in their parents being notified this school year. |
| Dropped out of school (%) | Mother | Each Child | "(Has this child/Have any of these children) ever dropped out of school?" Analyzed for children aged 10 to 24 at the five-year follow-up. |
| Broad Math Score on the Woodcock-Johnson Tests of Achievement-Revisedc | Assessment | Focal child | Administered the Calculation and Applied Problems tests. Composite score is age-standardized, with a mean of 100 and a standard deviation of 15. |
| Above-average Broad Math Score (%) | Assessment | Focal child | Focal children with Broad Math scores higher than 110 were considered to have above average scores. |
| Below-average Broad Math Score (%) | Assessment | Focal child | Focal children with Broad Math scores lower than 90 were considered to have below average scores. |
| Broad Reading Score on the Woodcock-Johnson Tests of Achievement-Revisedc | Assessment | Focal child | Administered the Passage Comprehension and Letter-Word Identification tests. Composite score is age-standardized, with a mean of 100 and a standard deviation of 15. |
| Above-average Broad Reading Score (%) | Assessment | Focal child | Focal children with Broad Reading scores higher than 110 were considered to have above average scores. |
| Below-average Broad Reading Score (%) | Assessment | Focal child | Focal children with Broad Reading scores lower than 90 were considered to have below average scores. |
| Repeated a grade (%) | Mother | Each Child | "Since the last interview, (has this child/have any of these children) repeated a grade for any reason?" |
| Teacher | Focal child | "Not counting kindergarten, has this student repeated any grades or been retained in grade for any reason?" | |
| Below grade level, math (%) | Teacher | Focal child | "Is this student currently performing above grade level, below grade level, or right around grade level in reading and math?" Responses were coded so as to obtain the measure of whether the student was below grade level in math. |
| Above grade level, math (%) | Teacher | Focal child | "Is this student currently performing above grade level, below grade level, or right around grade level in reading and math?" Responses were coded so as to obtain the measure of whether the student was above grade level in math. |
| Below grade level, reading (%) | Teacher | Focal child | "Is this student currently performing above grade level, below grade level, or right around grade level in reading and math?" Responses were coded so as to obtain the measure of whether the student was below grade level in reading. |
| Above grade level, reading (%) | Teacher | Focal child | "Is this student currently performing above grade level, below grade level, or right around grade level in reading and math?" Responses were coded to obtain the measure of whether the student was above grade level in reading. |
| In remedial math group (%) | Teacher | Focal child | "What level-of-ability group is this student in for mathematics? (1) High-ability group, (2) Middle-ability group, (3) Low-ability (or remedial) group, (4) No grouping by ability, or heterogeneous grouping." Responses were dichotomized to obtain the measure of whether the student was in a remedial group for math. |
| In remedial reading group (%) | Teacher | Focal child | "What level-of-ability group is this student in for reading? (1) High-ability group, (2) Middle-ability group, (3) Low-ability (or remedial) group, (4) No grouping by ability, or heterogeneous grouping." Responses were dichotomized to obtain the measure of whether the student was in a remedial group for reading. |
| Attended a special class for physical, emotional, or mental condition (%) | Mother | Each child | "(Does your child/Do any of your children) go to a special class or special school, or get special help in school for any physical, emotional, or mental condition?" |
| Children identified as needing and receiving special services (%) | Teacher | Focal child | Services included Title I services, English as a Second Language (ESL) or bilingual instruction, remedial instruction, speech therapy, or a resource for the gifted. Responses were dichotomized such that focal children were classified as needing and receiving special services if teachers reported they needed and received any of the services. |
| Children identified as needing and not receiving special services (%) | Teacher | Focal child | Services included Title I services, English as a Second Language (ESL) or bilingual instruction, remedial instruction, speech therapy, or a resource for the gifted. Responses were dichotomized such that focal children were classified as needing and not receiving special services if teachers reported they needed but did not receive any of the services listed. The measures of needing and receiving and needing and not receiving are not mutually exclusive; children could have been identified as needing and receiving some services while needing and not receiving other services. |
| Days child had been absent during current school year (%) | Teacher | Focal child | "So far this school year, about what percentage of the days that this student has been enrolled in your classroom has he/she been absent for any reason including sickness?" |
| Days child had been tardy during current school year (%) | Teacher | Focal child | "So far this school year, about what percentage of the days that this student has been enrolled in your classroom has he/she been tardy for any reason including sickness?" |
| Health and Safety | |||
| General health rating | Mother | Focal child | "Would you say that your child's health in general is: (1) Excellent, (2) Very good, (3) Good, (4) Fair, (5) Poor?" |
| In very good or excellent health (%) | Mother | Focal child | Responses to the general health rating item were dichotomized so as to obtain the measure of whether the focal child was in very good or excellent health. |
| Had a physical, emotional, or mental condition that required frequent medical attention (%) | Mother | Each child | "(Does your child/Do any of your children) have a physical, emotional, or mental condition that requires frequent medical attention, frequent use of medication, or the use of any special equipment such as a wheelchair or a breathing mask?" It was specified that this does not include eyeglasses. |
| Had a physical, emotional, or mental condition that impeded on mother's ability to go to work/school (%) | Mother | Each child | "(Does your child/Do any of your children) have a physical, emotional, or mental condition that demands a lot of your attention and makes it hard for you to go to school or work?" |
| Accident, injury, or poisoning that required an emergency room visit (%) | Mother | Each child | "Since the last interview, (has your child/have any of your children) had an accident, injury, or poisoning requiring a visit to a hospital emergency room or clinic?" |
| Other Outcomes | |||
| Did not live with parent, since last interview, because parent could not care for child (%) | Mother | Each child | "Since the last interview, (has your child/have any of your children) not lived with you because you couldn't care for (him/her/them)?" |
| Ever had a baby as a teen (%) | Mother | Each child | "(Has your child/Have any of your children) had a baby?" Analyzed for children aged 14 to 24 at the five-year follow-up, though analysis was limited to those having had a baby as a teenager. |
| NOTES: Some items were asked of each child in the family while others were asked of only the focal child. The "for whom available" column notes for which group of children each item was asked. The items asked of each child are reported in Chapters 11 and 12. The items asked of the focal child only are reported only in Chapter 12. Cronbach's alphas are reported on the mother/child Child Outcomes Study sample (N = 2,332) for the mother- and child-report items and on the teacher sample (N = 1,472) for the teacher-report items. a Items adapted from Gresham and Elliot, 1990. In the social skills domain, focal children responded to items with 0 = "Never"; 1 = "Sometimes"; 2 = "Often"; or 3 = "Very often." Mothers also responded to items with 0 = "Never" to 3 = "Very often.". Teachers responded to items with 1 = "Never"; 2 = "Sometimes"; 3 = "Often"; or 4 = "Very often"; responses were recoded from 0 to 3. bThe child school engagement scale is by Connell, 1990. c Woodcock and Johnson, 1989, 1990. |
|||
References
Achenbach, Thomas M. 1991. Manual for the Child Behavior Checklist/4-18 and 1991 Profile. Burlington: University of Vermont Department of Psychiatry.
Adams, Patricia F., and Marie A. Marano. 1995. "Current Estimates from the National Health Interview Survey, 1994." National Center for Health Statistics. Vital Health Statistics 10(193): 111-112.
Allard, Mary Ann, Randy Albelda, Mary Ellen Colten, and Carol Cosenza. 1997. In Harm's Way? Domestic Violence, AFDC Receipt, and Welfare Reform in Massachusetts. Boston: Center for Social Policy Research, University of Massachusetts.
Bane, Mary Jo, and David T. Ellwood. 1983. The Dynamics of Dependence: The Routes to Self Sufficiency. Washington, DC: U.S. Department of Health and Human Services, Office of the Assistant Secretary for Planning and Evaluation.
Bane, Mary Jo, and David T. Ellwood. 1994. Welfare Realities: From Rhetoric to Reform. Cambridge, MA: Harvard University Press.
Bardach, Eugene. 1993. Improving the Productivity of JOBS Programs. New York: Manpower Demonstration Research Corporation.
Baydar, Nazli, and Jeanne Brooks-Gunn. 1991. "Effects of Maternal Employment and Child-Care Arrangements in Infancy on Preschoolers' Cognitive and Behavioral Outcomes: Evidence from the Children of the NLSY." Developmental Psychology 27(6): 932-945.
Berlin, Gordon L. 2000. Encouraging Work, Reducing Poverty: The Impact of Work Incentive Programs. New York: Manpower Demonstration Research Corporation.
Berrueta-Clement, John R., Lawrence J. Schweinhart, W. Steven Barnett, Ann S. Epstein, and David P. Weikart. 1984. Changed Lives: The Effects of the Perry Preschool Program on Youths Through Age 19. Ypsilanti, MI: High/Scope.
Bloom, Dan. 1997. After AFDC: Welfare-to-Work Choices and Challenges for States. New York: Manpower Demonstration Research Corporation.
Bloom, Dan, James Kemple, Pamela Morris, Susan Scrivener, Nandita Verma, and Richard Hendra. 2000a. The Family Transition Program: Final Report on Florida's Initial Time-Limited Welfare. New York: Manpower Demonstration Research Corporation.
Bloom, Dan, Laura Melton, Charles Michalopoulos, Susan Scrivener, and Johanna Walter. 2000b. Jobs First: Implementation and Early Impacts of Connecticut's Welfare Reform Initiative. New York: Manpower Demonstration Research Corporation.
Bloom, Dan, and Charles Michalopoulos. 2001. How Welfare and Work Policies Affect Employment and Income: A Synthesis of Research. New York: Manpower Demonstration Research Corporation.
Bos, Johannes M., Aletha C. Huston, Robert C. Granger, Greg J. Duncan, Thomas W. Brock, and Vonnie C. McLoyd. 1999. New Hope for People with Low Incomes: Two-Year Results of a Program to Reduce Poverty and Reform Welfare. New York: Manpower Demonstration Research Corporation.
Bos, Johannes M., Susan Scrivener, Jason Snipes, and Gayle Hamilton. 2001. Improving Basic Skills: The Effects of Adult Education in Welfare-to-Work Programs. Washington, DC: U.S. Department of Education, Office of the Under Secretary and Office of Vocational and Adult Education; and U.S. Department of Health and Human Services, Administration for Children and Families and Office of the Assistant Secretary for Planning and Evaluation.
Bos, Johannes M., and Wanda G. Vargas. 2001. Maternal Employment and Changes in Adolescent Outcomes: Evidence from Two Evaluations of Programs to Promote Work. Paper presented at the Society for Research in Child Development biennial meeting, Minneapolis.
Brock, Thomas, and Kristen Harknett. 1998. Welfare-to-Work Case Management: A Comparison of Two Models. New York: Manpower Demonstration Research Corporation.
Brooks-Gunn, Jeanne, and Lisa Berlin. 1993. A Consideration of Self-Sufficiency and Parenting in the Context of the Teenage Parent Demonstration Program. Paper presented at the Second National Head Start Conference, symposium "Effects of a Two-Generation Program on Mothers and Children: The Teenage Parent Demonstration." Washington, DC
Brown, Brett, Sharon Vandivere, Laura Duberstein Lindberg, Scott Boggess, Laura Porter, and Sean Williams. 1999. Trends in the Well-Being of America's Children and Youth, 1999. Washington, DC: U.S. Department of Health and Human Services, Office of the Assistant Secretary for Planning and Evaluation. Stock no. 017-022-01484-0. From web site: <http://aspe.hhs.gov/hsp/99trends/intro.pdf>
Campbell, Frances A., and Craig T. Ramey. 1994. "Effects of Early Intervention on Intellectual and Academic Achievement: A Follow-Up Study of Children from Low-Income Families." Child Development 65(2): 684-698.
Capizzano, Jeffrey, Kathryn Tout, and Gina Adams. 2000. Child Care Patterns of School-Age Children with Employed Mothers. Occasional paper no. 41. Washington, DC: Urban Institute Press.
Caspi, Avshalom, Bradley R. Wright, Terrie E. Moffit, and Paul A. Silva. 1998. "Early Failure in the Labor Market: Childhood and Adolescent Predictors of Unemployment in the Transition to Adulthood." American Sociological Review 63(June): 424-451.
Center for Law and Social Policy (CLASP). 1995. The CLASP Guide to Welfare Waivers: 1992-1995. Washington, DC: Center for Law and Social Policy.
Center on Social Welfare Policy and Law. 1994. Living at the Bottom: An Analysis of 1994 AFDC Benefit Levels. Washington, DC: Center on Social Welfare Policy and Law.
Child Trends. December 2000. Web site: <http://www.childtrends.org>
Cohen, Jacob, ed. 1977. Statistical Power Analysis for the Behavioral Sciences, rev. ed. New York: Academic Press.
Cohen, Jacob, ed. 1988. Statistical Power Analysis for the Behavioral Sciences, 2nd ed. New York: Academic Press.
Coie, John D., and Kenneth A. Dodge. 1983. "Continuity of Children's Social Status: A Five-Year Longitudinal Study." Merrill-Palmer Quarterly 29: 261-282.
Connell, Jim P. 1990. The Research Assessment Package for Schools, Student Self-Report. Unpublished. University of Rochester.
Cooper, Harris, and Larry V. Hedges, eds. 1994. The Handbook of Research Synthesis. New York: Russell Sage Foundation.
Cowen, Emory L., Andreas Pederson, Haroutun Babigian, Louis D. Izzo, and Mary A. Trost. 1973. "Long Term Follow-up of Early Detected Vulnerable Children." Journal of Consulting and Clinical Psychology 41: 438-446.
Danziger, Sandra, Mary Corcoran, Sheldon Danziger, Colleen Heflin, Ariel Kalil, Judith Levine, Daniel Rosen, Kristin Seefeldt, Kristine Siefert, and Richard M. Tolman. 1999. Barriers to the Employment of Welfare Recipients. Ann Arbor: University of Michigan, Poverty Research and Training Center.
Dodoo, Martey. 2000. Evaluating Alternative Welfare-to-Work Approaches: Four-Year Impacts for Eleven Programs. Washington, DC: U.S. Department of Health and Human Services, Administration for Children and Families and Office of the Assistant Secretary for Planning and Evaluation; and U.S. Department of Education, Office of the Under Secretary and Office of Vocational and Adult Education.
Duncan, Greg, and Jeanne Brooks-Gunn, eds. 1997. Consequences of Growing Up Poor. New York: Russell Sage Foundation.
Duncan, Greg, Jeanne Brooks-Gunn, and Pamela Kato Klebanov. 1994. "Economic Deprivation and Early Childhood Development." Child Development 65(2): 296-318.
Duncan, Greg, and Katherine Magnuson. 2001. Off with Hollingshead: Socioeconomic Resources, Parenting, and Child Development. Unpublished manuscript.
Eberts, Randall W. 1997. "The Use of Profiling to Target Services in State Welfare-to-Work Programs: An Example of Process and Implementation." Working paper no. WP00098-052. Kalamazoo, MI: W. E. Upjohn Institute for Employment Research.
Families USA Foundation. 1999. Losing Health Insurance: Unintended Consequences of Welfare Reform. Washington DC. From web site: <http://www.familiesusa.org>
Farber, Henry, and Helen Levy. 2000. Recent Trends in Employer Sponsored Health Coverage: Are Bad Jobs Getting Worse? National Bureau for Economic Research. Working paper no. # 6709.
Farrell, Mary. 2000. Implementation, Participation Patterns, Costs, and Two-Year Impacts of the Detroit Welfare-to-Work Program. Washington, DC: U.S. Department of Health and Human Services, Administration for Children and Families and Office of the Assistant Secretary for Planning and Evaluation; and U.S. Department of Education, Office of the Under Secretary and Office of Vocational and Adult Education.
Farrington, David P. 1987. "Early Precursors of Frequent Offending." In James Q. Wilson and Glenn C. Loury, eds. From Children to Citizens: Families, Schools, and Delinquency Prevention. New York: Springer-Verlag.
Findlay, Steven, and Joel Miller. 1999. Down a Dangerous Path: The Erosion of Health Insurance Coverage in the United States. National Coalition on Health Care. From web site: <http://www.urbaninstitute.org>
Freedman, Stephen. 2000. Four-Year Impacts of Ten Programs on Employment Stability and Earnings Growth. Washington, DC: U.S. Department of Health and Human Services, Administration for Children and Families and Office of the Assistant Secretary for Planning and Evaluation; and U.S. Department of Education, Office of the Under Secretary and Office of Vocational and Adult Education.
Freedman, Stephen, Daniel Friedlander, Gayle Hamilton, JoAnn Rock, Marisa Mitchell, Jodi Nudelman, Amanda Schweder, and Laura Storto. 2000a. Evaluating Alternative Welfare-to-Work Approaches: Two-Year Impacts for Eleven Programs. Washington, DC: U.S. Department of Health and Human Services, Administration for Children and Families and Office of the Assistant Secretary for Planning and Evaluation; and U.S. Department of Education, Office of the Under Secretary and Office of Vocational and Adult Education.
Freedman, Stephen, Daniel Friedlander, Winston Lin, and Amanda Schweder. 1996. Five-Year Impacts on Employment, Earnings, and AFDC Receipt. GAIN Evaluation, Working paper no. 96.1. Prepared by the Manpower Demonstration Research Corporation for the U.S. Department of Health and Human Services.
Freedman, Stephen, Jean Knab, Lisa A. Gennetian, and David Navarro. 2000b. The Los Angeles Jobs-First GAIN Evaluation: Final Report on a Work First Program in a Major Urban Center. New York: Manpower Demonstration Research Corporation.
Freeman, Richard B., and David G. Blanchflower. 1999. Youth Employment and Joblessness in Advanced Countries. Chicago: University of Chicago Press.
Friedlander, Daniel. 1988. Subgroup Impacts and Performance Indicators for Selected Welfare Employment Programs. New York: Manpower Demonstration Research Corporation.
Friedlander, Daniel, and Gary Burtless. 1995. Five Years After: The Long-Term Effects of Welfare-to-Work Programs. New York: Russell Sage Foundation.
Friedlander, Daniel, Stephen Freedman, Gayle Hamilton, and Janet Quint. 1987. Final Report on Job Search and Work Experience in Cook County. New York: Manpower Demonstration Research Corporation.
Friedlander, Daniel, and Gayle Hamilton. 1993. The Saturation Work Initiative Model in San Diego: A Five-Year Follow-Up Study. New York: Manpower Demonstration Research Corporation.
Friedlander, Daniel, Gregory Hoerz, David Long, and Janet Quint. 1985a. Maryland: Final Report on the Employment Initiatives Evaluation. New York: Manpower Demonstration Research Corporation.
Friedlander, Daniel, Gregory Hoerz, Janet Quint, and James Riccio. 1985b. Arkansas: Final Report on the WORK Program in Two Counties. New York: Manpower Demonstration Research Corporation.
Gallup-Black, Adria. 1999. Using Audio-CASI to Collect Data on Domestic Violence. Mimeo. New York: Manpower Demonstration Research Corporation.
Gaquin, Deirdre A., and Mark S. Littman, eds. 1999. 1999 County and City Extra: Annual Metro, City, and County Data Book. Washington, DC: Bernan Press.
Gennetian, Lisa, and Cynthia Miller. 2000. Reforming Welfare and Rewarding Work: Final Report on the Minnesota Family Investment Program. Vol. 2: Effects on Children. New York: Manpower Demonstration Research Corporation.
Goldman, Barbara, Daniel Friedlander, and David Long. 1986. California: Final Report on the San Diego Job Search and Work Experience Demonstration. New York: Manpower Demonstration Research Corporation.
Greenberg, Mark. 1992. How States Can Reduce Welfare's Work Penalties: The "Fill-the-Gap" Option. Washington, DC: Center for Law and Social Policy.
Greenstein, Robert, and Jocelyn Guyer. 2001. "Supporting Work Through Medicaid and Food Stamps" In Rebecca Blank and Ron Haskins, eds. The New World of Welfare. Washington, DC: Brookings Institution Press.
Gresham, Frank M., and Stephen N. Elliot. 1990. Social Skills Rating System Manual. Circle Pines, MN: American Guidance Service.
Gueron, Judith M., and Edward Pauly. 1991. From Welfare to Work. New York: Russell Sage Foundation.
Guyer, Jocelyn, and Cindy Mann. 1999. Employed But Not Insured: A State-by-State Analysis of the Number of Low-Income Working Parents Who Lack Health Insurance. Washington, DC: Center on Budget and Policy Priorities.
Haimson, Joshua, and Alan Hershey. 1997. Getting Help to Stay Employed: The Use of Post-Employment Services. Princeton, NJ: Mathematica Policy Research.
Haimson, Joshua, Alan Hershey, and Anu Rangarajan. 1995. Providing Services to Promote Job Retention. Princeton, NJ: Mathematica Policy Research.
Haley, Jennifer, and Stephen Zuckerman. 2000. Health Insurance, Access and Use: United States, Tabulations from the 1997 National Survey of American Families. Washington, DC: Urban Institute. From web site: <http://www.nchc.org>
Hall, George E., and Deirdre A. Gaquin, eds. 1997. 1997 County and City Extra: Annual Metro, City, and County Data Book. Lanham, MD: Bernan Press.
Hamilton, Gayle. 1988. Interim Report on the Saturation Work Initiative Model in San Diego. New York: Manpower Demonstration Research Corporation.
Hamilton, Gayle. 1995. The JOBS Evaluation: Monthly Participation Rates in Three Sites and Factors Affecting Participation Levels in Welfare-to-Work Programs. Washington, DC: U.S. Department of Health and Human Services, Office of the Assistant Secretary for Planning and Evaluation.
Hamilton, Gayle. 2000. Do Mandatory Welfare-to-Work Programs Affect the Well-Being of Children? A Synthesis of Child Research Conducted as Part of the National Evaluation of Welfare-to-Work Strategies. Washington, DC: U.S. Department of Health and Human Services, Office of the Assistant Secretary for Planning and Evaluation and Administration for Children and Families; and U.S. Department of Education, Office of the Under Secretary and Office of Vocational and Adult Education.
Hamilton, Gayle, and Thomas Brock. 1994. The JOBS Evaluation: Early Lessons from Seven Sites. Washington, DC: U.S. Department of Health and Human Services, Administration for Children and Families and Office of the Assistant Secretary for Planning and Evaluation; and U.S. Department of Education, Office of the Under Secretary and Office of Vocational and Adult Education.
Hamilton, Gayle, Thomas Brock, Mary Farrell, Daniel Friedlander, and Kristen Harknett. 1997. Evaluating Two Welfare-to-Work Program Approaches: Two-Year Findings on the Labor Force Attachment and Human Capital Development Programs in Three Sites. Washington, DC: U.S. Department of Health and Human Services, Administration for Children and Families and Office of the Assistant Secretary for Planning and Evaluation; and U.S. Department of Education, Office of the Under Secretary and Office of Vocational and Adult Education.
Hamilton, Gayle, and Daniel Friedlander. 1989. Final Report on the Saturation Work Initiative Model in San Diego. New York: Manpower Demonstration Research Corporation.
Hamilton, Gayle, and Susan Scrivener. 1999. Promoting Participation: How to Increase Involvement in Welfare-to-Work Activities. New York: Manpower Demonstration Research Corporation.
Harknett, Kristen, and Lisa Gennetian. 2001. How an Earnings Supplement Affects the Marital Behavior of Welfare Recipients: Evidence from the Canadian Self-Suffiency Project. Working paper. Ottawa: Social Research and Demonstration Corporation.
Harvey, Elizabeth. 1999. "Short-Term and Long-Term Effects of Early Parental Employment on Children of the National Longitudinal Survey of Youth." Developmental Psychology 35(2): 445-459.
Haskins, Ron. 2001. "Effects of Welfare Reform on Family Income and Poverty." In Rebecca Blank and Ron Haskins, eds. The New World of Welfare. Washington, DC: Brookings Institution Press.
Hatcher, Larry. 1994. Step-by-Step Approach to Using the SAS System for Factor Analysis and Structural Equation Modeling. Cary, NC: SAS Institute.
Haveman, Robert, and Barbara Wolfe. 1995. "The Determinants of Children's Attainments: A Review of Methods and Findings." Journal of Economic Literature 33(1): 1829-1878.
Hoffman, Lois W., and Lise Youngblade. 1999. Mothers at Work: Effects on Children's Well-Being. Cambridge, UK: Cambridge University Press.
Holahan, John, and Niall Brennan. 2000. What Are the Adult Uninsured? Urban Institute. Assessing New Federalism Policy Brief, series B, no. B-14.
Hotz, V. Joseph, Guido Imbens, and Jacob A. Klerman. 2000. "The Long-Term Gains from GAIN: A Re-Analysis of the Impacts of the California GAIN Program." Working paper no. W8007. Cambridge, MA: National Bureau of Economic Research.
Kaplan, April. 1997. "Domestic Violence and Welfare Reform." Welfare Information Network Issue Notes. Vol. 1, no. 8 (September).
Kaplan, Jan. 1997. "Transitional Medicaid Assistance." Welfare Information Network Issue Notes. Vol. 1, no. 9 (December).
Kemple, James J., and Joshua Haimson. 1994. Florida's Project Independence: Program Implementation, Participation Patterns, and First-Year Impacts. New York: Manpower Demonstration Research Corporation.
Kemple, James J., Daniel Friedlander, and Veronica Fellerath. 1995. Florida's Project Independence: Benefits, Costs, and Two-Year Impacts of Florida's JOBS Program. New York: Manpower Demonstration Research Corporation.
Knab, Jean Tansey, Johannes M. Bos, Daniel Friedlander, and Joanna Weissman. 2001. Do Mandates Matter? The Effects of a Mandate to Enter a Welfare-to-Work Program. Prepared by the Manpower Demonstration Research Corporation for the U.S. Department of Health and Human Services.
Knab, Jean Tansey, and Stephen Freedman. 1996a. Preliminary Impacts on Employment, Earnings, and AFDC Receipt for the Detroit JOBS Program. JOBS Evaluation. Working paper no. 96.1. Prepared by the Manpower Demonstration Research Corporation for the U.S. Department of Health and Human Services.
Knab, Jean Tansey, and Stephen Freedman. 1996b. Preliminary Impacts on Employment, Earnings, and AFDC Receipt for the Oklahoma City JOBS Program. JOBS Evaluation, Working paper no. 96.3. Prepared by the Manpower Demonstration Research Corporation for the U.S. Department of Health and Human Services.
Knox, Virginia, Cynthia Miller, and Lisa A. Gennetian. 2000. Reforming Welfare and Rewarding Work: A Summary of the Final Report on the Minnesota Family Investment Program. New York: Manpower Demonstration Research Corporation.
Lamb, Michael. 1998. "Nonparental Child Care: Context, Quality, Correlates and Consequences." In Irving E. Siegel and K. Ann Renninger, eds. Handbook of Child Psychology, 4th ed. New York: Wiley.
Levine, Phillip B., and David J. Zimmerman. 2000. "Children's Welfare Exposure and Subsequent Development." Working paper no. 130. Evanston and Chicago: Northwestern University and University of Chicago, Joint Center for Poverty Research.
Lipsey, Mark W. 1990. Design Sensitivity: Statistical Power for Experimental Research. Newbury Park, CA: Sage.
Long, David, and Virginia Knox. 1985. Documentation of the Data Sources and Analytical Methods Used in the Benefit-Cost Analysis of the EPP/EWEP Program in San Diego. New York: Manpower Demonstration Research Corporation.
Lyon, Eleanor. 2000. "Welfare, Poverty and Abused Women: New Research and Its Implications." Mimeo. National Resource Center on Domestic Violence, Pennsylvania Coalition Against Domestic Violence.
Magnuson, Katherine, and Sharon M. McGroder. 2001. "From ABEs to 1,2,3s: The Effect of Maternal Education on Young Children's School Readiness." Paper presented at the Fourth Annual Welfare Reform Evaluation Conference, May 22-24, sponsored by the Administration for Children and Families in the U.S. Department of Health and Human Services, Washington, DC.
Martinson, Karin. 2000. The Experiences of Welfare Recipients Who Find Jobs. Washington, DC: U.S. Department of Health and Human Services, Office of the Assistant Secretary for Planning and Evaluation and Administration for Children and Families; and U.S. Department of Education, Office of the Under Secretary and Office of Vocational and Adult Education.
Masten, Ann S., J. Douglas Coatsworth, Jennifer Neemann, Scott D. Gest, Auke Tellegen, and Norman Garmezy. 1995. "The Structure and Coherence of Competence from Childhood Through Adolescence." Child Development 66(6): 1635-1659.
Mayer, Susan. 1997. What Money Can't Buy: Family Income and Family's Life Chances. Cambridge, MA: Harvard University Press.
McGroder, Sharon M. 2000. "Parenting Among Low-Income, African-American Single Mothers with Preschool-Age Children: Patterns, Predictors, and Developmental Correlates." Child Development 71(3): 752-771 STYLE="font-family: TmsRmn">.
McGroder, Sharon M., Martha J. Zaslow, Kristin A. Moore, and Suzanne M. LeMenestrel. 2000. Impacts on Young Children and Their Families Two Years After Enrollment: Findings from the Child Outcomes Study. Washington, DC: U.S. Department of Health and Human Services, Office of the Assistant Secretary for Planning and Evaluation and Administration for Children and Families; and U.S. Department of Education, Office of the Under Secretary and Office of Vocational and Adult Education.
McLanahan, Sara. 1997. "Parental Absence or Poverty: Which Matters More?" In Greg J. Duncan and Jeanne Brooks-Gunn, eds. Consequences of Growing Up Poor. New York: Russell Sage Foundation.
McLoyd, Vonnie C., Toby Epstein Jayaratne, Rosario Ceballo, and Julio Borquez. 1994. "Unemployment and Work Interruption Among African-American Single Mothers: Effects on Parenting and Adolescent Socioemotional Functioning." Child Development 65(2): 562-589.
Michalopoulos, Charles. 2001. "Sustained Employment and Earnings Growth: New Experimental Evidence on Earnings Supplements and Pre-Employment Services." In Richard Kazis and Marc S. Miller, eds. Low-Wage Workers in the New Economy. Washington, DC: Urban Institute Press.
Michalopoulos, Charles, and Gordon Berlin. 2001. "Financial Work Incentives for Low-Wage Workers." In Rebecca Blank and Ron Haskins, eds. The New World of Welfare. Washington, DC: Brookings Institution Press.
Michalopoulos, Charles, and Christine Schwartz. 2001. What Works Best for Whom: Impacts of 20 Welfare-to-Work Programs by Subgroup. Washington, DC: U.S. Department of Health and Human Services, Administration for Children and Families and Office of the Assistant Secretary for Planning and Evaluation; and U.S. Department of Education, Office of the Under Secretary and Office of Vocational and Adult Education.
Miller, Cynthia. 1997. "Updated MFIP Impacts." MDRC memo.
Miller, Cynthia, Virginia Knox, Lisa Gennetian, Martey Dodoo, JoAnna Hunter, and Cindy Redcross. 2000. Reforming Welfare and Rewarding Work: Final Report on the Minnesota Family Investment Program: Vol. 1: Effects on Adults. New York: Manpower Demonstration Research Corporation
Mincer, Jacob. 1974. Schooling, Experience, and Earnings. New York: Columbia University Press.
Moore, Kristin A., and Anne Driscoll. 1997. "Low-Wage Maternal Employment and Outcomes for Children: A Study." Future of Children 7(1): 122-127.
Moore, Kristin A. and Dana Glei. 1995. "Taking the Plunge: An Examination of Positive Youth Development." Journal of Adolescent Research 10(1): 15-40.
Moore, Kristin A., David E. Myers, Donna R. Morrison, Christine W. Nord, Brett V. Brown, and Barry Edmonston. 1993. "Age at First Birth and Later Poverty." Journal of Research on Adolescence 3(4): 393-422.
Morris, Pamela A., Aletha C. Huston, Greg J. Duncan, Danielle A. Crosby, and Johannes M. Bos. 2001. How Welfare and Work Policies Affect Children: A Synthesis of Research. New York: Manpower Demonstration Research Corporation.
Morris, Pamela A., and Charles Michalopoulos. 2000. The Self-Sufficiency Project at 36 Months: Effects on Children of a Program That Increased Parental Employment and Income. Ottawa: Social Research and Demonstration Corporation.
Mortimer, Jeylan T., Michael D. Finch, Seongryeol Ryu, Michael J. Shanahan, and Kathleen T. Call. 1996. "The Effects of Work Intensity on Adolescent Mental Health, Achievement, and Behavioral Adjustment: New Evidence from a Perspective Study." Child Development 67(3): 1243-1261.
Mullahy, John, and Barbara Wolfe. 2000. "Health Policies for the Nonelderly Poor." Focus 21(2): 32-37.
NICHD Early Child Care Research Network. 2000. "The Relation of Child Care to Cognitive and Language Development." Child Development 71(4): 960-980.
National Research Council and Institute of Medicine. 2000. From Neurons to Neighborhoods: The Science of Early Childhood Development. Committee on Integrating the Science of Early Childhood Development. Jack P. Shonkoff and Deborah A. Phillips, eds. Board on Children, Youth, and Families, Commission on Behavioral and Social Sciences and Education. Washington, DC: National Academy Press.
Ohls, James C., and Harold Beebout. 1993. The Food Stamp Program: Design Tradeoffs, Policy, and Impacts: A Mathematica Policy Research Study. Washington, DC: Urban Institute Press.
Parker, Jeffrey G., and Steven R. Asher. 1987. "Peer Relations and Later Personal Adjustment: Are Low-Accepted Children at Risk?" Psychological Bulletin 102(3): 357-389.
Pavetti, LaDonna A. 1992. "The Dynamics of Welfare and Work: Exploring the Process by Which Young Women Work Their Way Off Welfare." Prepared for the APPAM Annual Research Conference, October 29, Malcolm Wiener Center for Social Policy, John F. Kennedy School of Government, Harvard University.
Pavetti, Ladonna, and Dan Bloom. 2001. "State Sanctions and Time Limits." In Rebecca Blank and Ron Haskins, eds. The New World of Welfare. Washington, DC: Brookings Institution Press.
Petit, Gregory S., John E. Bates, Kenneth A. Dodge, and Darrell W. Meece. 1999. "The Impact of After-School Peer Contact on Early Adolescent Externalizing Problems As Moderated by Parental Monitoring, Perceived Neighborhood Safety, and Prior Adjustment." Child Development 70(3): 768-778.
Phillips, Deborah A., Miriam Voran, Ellen Kisker, Carollee Howes, and Marcy Whitebook. 1994. "Child Care for Children in Poverty: Opportunity or Inequity?" Child Development 65(2): 472-494.
Polachek, Solomon, and Stanley W. Siebert. 1993. The Economics of Earning. Cambridge, UK: Cambridge University Press.
Posner, Jill K., and Deborah L. Vandell. 1994. "Low-Income Children's After-School Care: Are There Beneficial Effects of After-School Programs?" Child Development 65(2): 440-456.
Primus, Wendell, Lynette Rawlings, Kathy Larin, and Kathryn Porter. 1999. The Initial Impacts of Welfare Reform on the Incomes of Single-Mother Families. Washington, DC: Center on Budget and Policy Priorities. From web site: <http://www.cbpp.org/8-22-99wel.pdf>
Quint, Janet C., Johannes M. Bos, and Denise F. Polit. 1997. New Chance: Final Report of a Comprehensive Program for Young Mothers in Poverty and Their Children. New York: Manpower Demonstration Research Corporation.
Quint, Janet, and Rebecca Widom. 2001. Post-TANF Food Stamp and Medicaid Benefits: Factors That Aid or Impede Receipt. New York: Manpower Demonstration Research Corporation.
Rangarajan, Anu, and Tim Novak. 1999. The Struggle to Sustain Employment: The Effectiveness of the Post-Employment Services Demonstration. Final Report. Princeton, NJ: Mathematica Policy Research.
Rangarajan, Anu, Peter Schochet, and Dexter Chu. 1998. Employment Experiences of Welfare Recipients Who Find Jobs: Is Targeting Possible? Princeton, NJ: Mathematica Policy Research.
Raphael, Jody. 1996. "Domestic Violence and Welfare Receipt: Toward a New Feminist Theory of Welfare Dependency." Harvard Women's Law Journal 19: 201-27.
Raphael, Jody, and Richard M. Tolman. 1997. Trapped by Poverty, Trapped by Abuse: New Evidence Documenting the Relationship Between Domestic Violence and Welfare. A research compilation from the Project for Research on Welfare, Work, and Domestic Violence. Chicago and Ann Arbor: Taylor Institute and the University of Michigan Research Development Center on Poverty, Risk, and Mental Health.
Riccio, James, George Cave, Stephen Freedman, and Marilyn Price. 1986. Virginia: Final Report on the Employment Services Program. New York: Manpower Demonstration Research Corporation.
Riccio, James, and Stephen Freedman. 1995. Can They All Work? A Study of the Employment Potential of Welfare Recipients in a Welfare-to-Work Program. New York: Manpower Demonstration Research Corporation.
Riccio, James, and Daniel Friedlander. 1992. GAIN: Program Strategies, Participation Patterns, and First-Year Impacts in Six Counties. New York: Manpower Demonstration Research Corporation.
Riccio, James, Daniel Friedlander, and Stephen Freedman. 1994. GAIN: Benefits, Costs, and Three-Year Impacts of a Welfare-to-Work Program. New York: Manpower Demonstration Research Corporation.
Riccio, James, Barbara Goldman, Gayle Hamilton, Karin Martinson, and Alan Orenstein. 1989. GAIN: Early Implementation Experiences and Lessons. New York: Manpower Demonstration Research Corporation.
Riger, Stephanie, Courtney Ahrens, and Amy Blickenstaff. 2001. "Measuring Interference with Employment and Education Reported by Women with Abusive Partners: Preliminary Data." In K. Daniel O'Leary and Roland D. Maiuro, eds. Psychological Abuse in Violent Domestic Relations. New York: Springer.
Riger, Stephanie, and Maryann Krieglstein. 2000. "The Impact of Welfare Reform on Men's Violence Against Women." American Journal of Community Psychology 28(5): 631-47.
Scholz, John Karl. 1996. "In-Work Benefits in the United States: The Earned Income Tax Credit." Economic Journal 106 (January): 156-169.
Scrivener, Susan, Gayle Hamilton, Mary Farrell, Stephen Freedman, Daniel Friedlander, Marisa Mitchell, Jodi Nudelman, and Christine Schwartz. 1998. Implementation, Participation Patterns, Costs, and Two-Year Impacts of the Portland (Oregon) Welfare-to-Work Program. Washington, DC: U.S. Department of Health and Human Services, Administration for Children and Families and Office of the Assistant Secretary for Planning and Evaluation; and U.S. Department of Education, Office of the Under Secretary and Office of Vocational and Adult Education.
Scrivener, Susan, and Johanna Walter. 2001. Evaluating Two Approaches to Case Management: Implementation, Participation Patterns, Costs, and Three-Year Impacts of the Columbus Welfare-to-Work Program. Washington, DC.: U.S. Department of Health and Human Services, Administration for Children and Families and Office of the Assistant Secretary for Planning and Evaluation; and U.S. Department of Education, Office of the Under Secretary and Office of Vocational and Adult Education.
State Policy Documentation Project. November 2000. Web site: <http://www.spdp.org >
Steinberg, Laurence, and Sanford M. Dornbusch. 1991. "Negative Correlates of Part-Time Employment During Adolescence: Replication and Elaboration." Developmental Psychology 27(2): 304-313.
Storto, Laura, Gayle Hamilton, Christine Schwartz, and Susan Scrivener. 2000. Oklahoma City's ET & E Program: Two-Year Implementation, Participation, Cost, and Impact Findings. Washington, DC: U.S. Department of Health and Human Services, Administration for Children and Families and Office of the Assistant Secretary for Planning and Evaluation; and U.S. Department of Education, Office of the Under Secretary and Office of Vocational and Adult Education.
Strawn, Julie, and Karin Martinson. 2000. Steady Work and Better Jobs: How to Help Low-Income Parents Sustain Employment and Advance in the Work Force. New York: Manpower Demonstration Research Corporation.
Sum, Andrew, Robert Taggart, and Neal Fogg. 1995. Dreams to Dust: The Deteriorating Labor Market Fortunes of Young Adult Men and Women in the United States, 1967-1994. Boston: Northeastern University, Center for Labor Market Studies.
Tout, Kathryn, Martha J. Zaslow, Angela R. Papillo, and Sharon Vandivere. 2001. "Early Care and Education: Work Support for Families and Developmental Opportunity for Young Children." Assessing the New Federalism. Occasional paper no. 51. Washington, DC: The Urban Institute.
Urban Institute. December 2000. National Survey of American Families. Web site: <http://www.newfederalism.urban.org>
U.S. Bureau of the Census. December 2000. Web site: <http://www.census.gov>
U.S. Department of Education, National Center for Education Statistics. 1995. Approaching Kindergarten: A Look at Preschoolers in the United States, 1995. NCES 95-280, by Nicholas Zill, Mary Collins, Jerry West, and Elvie G. Hausken. Washington, DC.
U.S. Department of Education, National Center for Education Statistics. 1997. The Condition of Education, 1997. NCES 97-388, by Thomas M. Smith, Beth Aronstamm Young, Yupin Bae, Susan P. Choy, and Nabeel Alsalam. Washington, DC.
U.S. Department of Education, National Center for Education Statistics. 2001. Dropout Rates in the United States: 1999. NCES 2001-022, by Phillip Kauffman, Jin Y. Kwon, Steve Klein, and Christopher Chapman. Washington, DC. From web site: <http://www.nces.ed.gov/pubs2001/2001022.pdf>
U.S. Department of Education, National Center for Education Statistics. December 2000. Web site: <http://www.nces.ed.gov>
U.S. Department of Health and Human Services. 1996. Characteristics and Financial Circumstances of AFDC Recipients, FY 1994. Washington, DC.
U.S. Department of Labor, Bureau of Labor Statistics. January 1998. Web site: <http://www.bls.gov>
U.S. Department of Labor, Bureau of Labor Statistics. October 2000. Web site: <http://www.bls.gov>
U.S. Department of Labor, Bureau of Labor Statistics. 2000. Employer Costs for Employee Compensation, 1986-99. Washington, DC. From web site: <http://stats.bls.gov/special.requests/ocwc/oclt/ect/ecbl0013.pdf>
U.S. Department of Education, National Center for Education Statistics. December 2000. Web site: <http://www.nces.ed.gov>
U.S. House of Representatives, Committee on Ways and Means. 1994. 1994 Green Book: Background Material and Data on Programs Within the Jurisdiction of the Committee on Ways and Means. Washington, DC: U.S. Government Printing Office.
U.S. House of Representatives, Committee on Ways and Means. 1996. 1996 Green Book: Background Material and Data on Programs Within the Jurisdiction of the Committee on Ways and Means. Washington, DC: U.S. Government Printing Office.
U.S. House of Representatives, Committee on Ways and Means. 1998. 1998 Green Book: Background Material and Data on Programs Within the Jurisdiction of the Committee on Ways and Means. Washington, DC: U.S. Government Printing Office.
U.S. House of Representatives, Committee on Ways and Means. 2000. 2000 Green Book: Background Material and Data on Programs Within the Jurisdiction of the Committee on Ways and Means. Washington, DC: U.S. Government Printing Office.
Vandell, Deborah L., and Janaki Ramanan. 1992. "Effects of Early and Recent Maternal Employment on Children from Low-Income Families." Child Development 63(4): 938-949.
Vandell, Deborah L., and Barbara Wolfe. 2000. Child Care Quality: Does It Matter and Does It Need to be Improved? Washington, DC: U.S. Department of Health and Human Services, Office of the Assistant Secretary for Planning and Evaluation.
Vernon-Feagans, Lynne, Diana C. Emanuel, and Ingrid Blood. 1997. "The Effects of Otitis Media and Quality of Daycare on Children's Language Development." Journal of Applied Developmental Psychology 18(3): 395-409.
Wilson, Julie Boatright, and David T. Ellwood. 1993. Welfare to Work Through the Eyes of Children: The Impacts on Children of Parental Movement from AFDC to Employment. Cambridge, MA: Malcolm Weiner Center for Social Policy, John F. Kennedy School of Government, Harvard University.
Wilson, Julie Boatright, David T. Ellwood, and Jeanne Brooks-Gunn. 1995. "Welfare-to-Work Through the Eyes of Children." In P. Lindsay Chase-Lansdale and Jeanne Brooks-Gunn, eds. Escape from Poverty: What Makes a Difference. New York: Cambridge University Press.
Woodcock, Richard W., and M. Bonner Johnson. 1989, 1990. Woodcock-Johnson Psycho-Educational Battery Revised. Allen, TX: DLM Teaching Resources.
Woodcock, Richard W., and Nancy Mather. 1989, 1990. "WJ-R Tests of Achievement: Examiner's Manual." In Richard W. Woodcock and M. Bonner Johnson, Woodcock-Johnson Psycho-Educational Battery Revised. Allen, TX: DLM Teaching Resources.
Yoshikawa, Hirokazu. 1995. "Long-Term Effects of Early Childhood Programs on Social Outcomes and Delinquency." Future of Children 5(3): 51-75.
Yoshikawa, Hirokazu. 1999. "Welfare Dynamics, Support Services, Mothers' Earnings, and Child Cognitive Development: Implications for Contemporary Welfare Reform." Child Development 70(3): 779-801.
Zaslow, Martha J. 1991. "Variation in Child Care Quality and Its Implications for Children." Journal of Social Issues 47(2): 125-138.
Zaslow, Martha J., and Carolyn A. Eldred, eds. 1998. Parenting Behavior in a Sample of Young Mothers in Poverty: Results of the New Chance Observational Study. New York: Manpower Demonstration Research Corporation.
Zaslow, Martha J., and Carol A. Emig. 1997. "When Low-Income Mothers Go to Work: Implications for Children." Future of Children 7(1): 110-116.
Zaslow, Martha J., Sharon M. McGroder, George Cave, and Carrie Mariner. 1999a. "Maternal Employment and Child Outcomes Among Families with Some History of Welfare Receipt." In Randy Hodson, series ed., and Toby L. Parcel, vol. ed. Research in the Sociology of Work Vol. 7, Work and Family. Stamford, CT: JAI Press.
Zaslow, Martha J., Sharon M. McGroder, Kristin A. Moore, and Suzanne M. LeMenestrel. 1999b. "Behavior Problems and Cognitive School Readiness Among Children in Families with a History of Welfare Receipt: Diverging Patterns and Their Predictors." Prepared for Society for Research in Child Development meetings, April 15, Albuquerque, NM.
Zaslow, Martha J., Sharon M. McGroder, Kristin A. Moore. 2000. Summary Report: Impacts on Young Children and Their Families Two Years After Enrollment: Findings from the Child Outcomes Study. Washington, DC: U.S. Department of Health and Human Services, Administration for Children and Families and Office of the Assistant Secretary for Planning and Evaluation; and U.S. Department of Education, Office of the Under Secretary and Office of Vocational and Adult Education.
Zaslow, Martha J., Kristin A. Moore, Jennifer L. Brooks, Pamela Morris, Kathryn Tout, Zakia Reed, and Carol A. Emig. Forthcoming. "Experimental Studies of Welfare Reform and Children." Future of Children.
Zaslow, Martha J., Kristin A. Moore, Donna Ruane Morrison, and Mary Jo Coiro. 1995. "The Family Support Act and Children: Potential Pathways of Influence." Children and Youth Services Review 17(1-3): 231-249.
Zaslow, Martha J., Erin Oldham, Ellen Magenheim, and Kristin A. Moore. 1998. "Welfare Families' Use of Early Childhood Care and Education Programs, and Implications for Their Children's Development." Early Childhood Research Quarterly 13(4): 535-563.