Congress requires DHHS, in collaboration with DOL, to evaluate the effectiveness of WtW initiatives. The evaluation is being conducted by Mathematica Policy Research, Inc., with assistance from two subcontractors, the Urban Institute and Support Services International, Inc. The evaluation addresses five key questions:
- What types and packages of services do WtW grantees provide? How do they compare to services already available under TANF or Job Training Partnership Act (JTPA) funding?
- What are the net impacts of various WtW program approaches on employment and on families' well-being?
- What challenges do grantees confront as they implement and operate WtW programs?
- Do the benefits of WtW programs outweigh their costs?
5. How well do PICs and other non-TANF organizations the primary vehicles for funding and operating WtW programs meet the challenge of serving those hardest to employ?
To address these questions, the evaluation plan includes three main components:
- Descriptive Assessment of All WtW Grantees. A mail survey of all formula and competitive grantees was administered in 1998, and a second survey will be completed in late 1999, providing a comprehensive overview of the program designs and activities offered, the target populations, the characteristics of participants, and, to the extent they are available, placement outcomes. All grant applications and state WtW plans submitted to DOL were also reviewed, and exploratory site visits were made to a selected group of grantees to develop a fuller understanding of program variations and to aid in the selection of sites for in-depth studies. The application and plan reviews and the exploratory visits form the basis of this report.
- In-Depth Process and Implementation Study. From 1999 to 2000, in-depth process analysis will be conducted focusing on 12 to 15 local grantees. Site visits will include discussions with staff of WtW programs and related agencies, focus groups with participants, and program observation. The analysis will identify implementation issues and challenges, and help to explain how programs achieve impacts estimated in some sites. A report on program implementation is scheduled to be issued in spring 2000.
- In-Depth Impact and Cost-Effectiveness Study. In three or four of these in-depth study sites, rigorous studies using random assignment designs will be conducted to determine what difference WtW programs make in employment and family well-being outcomes. This component is being conducted only in a subset of the 12 to 15 in-depth study sites, because few grantees are identifying more eligible candidates than they can serve, and such "excess demand" is a necessary precondition for the use of random assignment. Where the random assignment design is used, comparing outcomes for the two groups will indicate program impact and help identify successful program models. Comparisons will also be made between the net benefits of these impacts and the additional costs of delivering program services. Findings on program impacts will be reported in stages: mid-2001, late 2002, and mid-2003.
Under the BBA, Congress also authorized grants to American Indian tribes and Alaska native villages, and mandated that their programs be evaluated. Therefore, in addition to the main components listed above, a special process and implementation study focuses on documenting tribal welfare and employment systems, the supportive services they provide, and how tribes integrate funds from various sources to move their members from welfare to work. This study focuses on implementation experiences in 10 tribal sites, selected to include those with innovative designs and practices, signs of some success, and variation in location, size, and local labor market and economic conditions. Findings from the tribal program evaluation will be reported in fall 2000.
The rest of this report presents preliminary evaluation findings, drawing on additional information obtained since the first report based on the initial grantee survey. As indicated earlier, this report is based on two main sources of information: (1) two-day exploratory site visits to 22 local WtW programs (Table A.2), and (2) a review of grantee applications and state plans submitted to DOL.(4) The 22 local site visits occurred between November 1998 and June 1999; brief profiles of the 22 programs appear at the end of this volume as Appendix D.
|Source||Number of Programs Examined|
|Review of Applications Submitted by Competitive Grantees|
|Round 1 grantees awarded||51|
|Round 2 grantees awarded||75|
|Review of WtW Amendments to State TANF Plans|
|Formula grantees FY 1998 (states, territories, and District of Columbia)||48|
|Exploratory Site Visits to Local Grantees||22|
| Source: National Evaluation of the Welfare-to-Work Grants Program, WtW Grant Review Database, 1999.
Note: The following states did not participate in the formula grants program in FY 1998: Idaho, Mississippi, Ohio, South Dakota, Utah, and Wyoming.
In addition to the site visits, all 126 applications submitted to DOL that eventually resulted in awards of competitive WtW grants under Round 1 and Round 2 were reviewed, and all 48 WtW amendments to state TANF plans were reviewed.(5) The application and plan review is an ongoing task in the evaluation; as new grants are awarded, additional applications are reviewed.
The findings presented in this report should be regarded as preliminary rather than conclusive assessments, since the information is based on a small number of grantees and the exploratory visits were conducted only a short while after the programs started implementation. Policymakers should be cautious in interpreting these early experiences. Implementing the WtW grants program requires major system change, perhaps even greater and more time-consuming than could have been anticipated when the WtW program was authorized, and grantees will continue to learn and evolve.