The overall evaluation of the WtW grants program was designed to address five key questions:
- What types and packages of services do WtW grantees provide? How do they compare to services already available under TANF or Job Training Partnership Act (JTPA) funding?
- What are the net impacts of various WtW program approaches on employment and on families' well-being?
- What challenges do grantees confront as they implement and operate WtW programs?
- Do the benefits of WtW programs outweigh their costs?
- How well do PICs and other non-TANF organizations the primary vehicles for funding and operating WtW programs meet the challenge of implementing WtW programs for those hardest to employ?
The design of the evaluation has evolved since its inception, because of the slow pace of enrollment in WtW programs. This has resulted in a narrowing of the questions the evaluation can address. Under the modified design adopted by DHHS, the evaluation plan now includes two main components:
A Descriptive Assessment of All WtW Grantees. The mail survey of all grantees in 1998 and 1999 is providing an overview of program designs and activities, target populations, characteristics of participants, and, to the extent they are available, placement outcomes. Visits to several dozen grantees before the first survey helped develop a fuller understanding of program variations and provided a basis for selection of in-depth study sites. This report is based on the second survey.
In-Depth Process and Implementation Study. In 1999-2000, site visits are being conducted in 12 to 15 grantee sites, selected because of their innovative approaches, settings, or target groups or because they are typical of the most common WtW interventions. These visits include discussions with staff of WtW programs and related agencies, focus groups with participants, and program observation. The aim is to identify implementation issues and challenges, as well as lessons on program implementation. In most of these sites, follow-up data will be collected through surveys and administrative data and used for analysis of participants' activities in the programs and their employment outcomes. Analysis of program costs will also be conducted. A process and implementation report will be issued in spring 2000.
The original evaluation design called for impact and cost-effectiveness analysis to be conducted in about 10 of the in-depth study sites, based on a random assignment experimental design. This design would have made it possible to determine the difference WtW programs make in employment and family well-being outcomes. It now appears, however, that this component of the evaluation will be feasible in at most a few sites. The main barrier to conducting the impact study is the difficulty finding grantees that are identifying more eligible candidates than they can serve; such "excess demand" is a necessary precondition for the use of random assignment. However, in lieu of impact estimates, the "enhanced process analysis" described above will yield systematic follow-up data on the employment and social outcomes of participants in most of the in-depth study sites. Such findings will be reported in stages in mid-2001, late 2002, and mid-2003.
In addition to the components of the core evaluation, a special process and implementation study focuses on documenting welfare and employment systems operated by American Indian and Alaska Native grantees, the supportive services they provide, and how these tribal grantees integrate funds from various sources to move their members from welfare to work. Results of the tribal program evaluation will be reported in fall 2000.