The evaluation of the WtW grants program, conducted in 11 local sites,(10) had 4 core components:
- A descriptive assessment of all WtW grantees, based on two surveys administered to all WtW grantees nationwide to document program planning and early operations (Perez-Johnson and Hershey 1999; Perez-Johnson et al. 2000).
- A process and implementation analysis, based on exploratory site visits to 22 local WtW-funded programs (Nightingale et al. 2000), two rounds of in-depth visits to 11 of those sites (Nightingale et al. 2002), and follow-up telephone conversations with key administrators in each of the 11 sites (Nightingale et al. 2003).
- A program cost analysis in the in-depth study sites, documenting the total program costs and participant costs by service category and grantee site (Perez-Johnson et al. 2002).
- An enrollee outcomes analysis in the in-depth study sites, based on analysis of longitudinal data on individuals who enrolled in WtW over a period of approximately two years in each site.(11) A first report provided information on enrollees outcomes one year after program entry, based on a detailed follow-up survey of enrollees (Fraker et al. 2004). This final report adds information from a second survey of enrollees two years after entry, as well as two-year follow-up administrative data on welfare receipt, employment, and earnings.
The outcomes analysis is the source of most of the findings presented in this report. However, this report does reference selected key findings from the first three core components of the evaluation. In addition, Appendix G provides brief summaries of findings from those components.
In addition to the four core components, the evaluation included a detailed process and implementation study focused on tribal programs, as well as three special studies, each focused on either a specific WtW site or a specific segment of the WtW target population. The tribal study documented welfare and employment systems operated by American Indian and Alaska Native WtW grantees, the supportive services they provided, and how these tribal grantees integrated funds from various sources to move members from welfare to work (Hillabrant and Rhoades 2000; Hillabrant et al. 2001). Appendix A summarizes findings from the tribal study. The three special studies examined: (1) the provision of WtW services to noncustodial parents (Martinson et al. 2000), (2) a WtW-funded initiative in the state of Washington designed to increase child support payments (Perez-Johnson et al. 2003), and (3) two WtW programs in Philadelphia (VanNoy and Perez-Johnson 2004). Appendix G includes summaries of these studies.
Originally, this evaluation was to estimate, using an experimental design, the net impacts of the WtW grants program on participants and apply those estimates in an analysis of the programs costs and benefits. Slow enrollment in WtW programs rendered this evaluation design unworkable. Given service providers difficulties in meeting their enrollment goals, they were uniformly (and understandably) unwilling to allow the random assignment of enrollees to treatment and control groups, as the diversion of some eligible individuals into a control group where they would have received minimal or no services would have further hindered programs achievement of enrollment goals.
Given the impossibility of a rigorous experimental approach to estimating program impacts, DHHS consulted with its partners in the evaluations inter-agency workgroup DOL, the Department of Housing and Urban Development, and the Office of Management and Budget (OMB) and with MPR to develop an alternate evaluation design. The new design focused on analyzing participant outcomes such as employment, rather than the impact that the program had on outcomes. The alternate design and data collection instruments for all components of the evaluation received formal clearance from OMB.
A critical implication of this design change is that none of the findings presented in this report on outcomes should be interpreted as estimates of the net impacts of the local WtW programs that participated in the evaluation. This evaluation focused on program implementation and the extent to which enrollees entered and sustained employment and were able to leave the TANF rolls as intended by the program design. However, findings from the evaluation do not provide a valid basis to judge whether enrollees did so more or less than they would have in the absence of the WtW grants program.