Under the BBA, Congress requires DHHS, in collaboration with DOL, to evaluate the effectiveness of WtW initiatives. The evaluation is being conducted by Mathematica Policy Research, Inc., with assistance from two subcontractors, the Urban Institute and Support Services International, Inc. The evaluation will address five key questions:
- What are the types and packages of services WtW grantees provide?
- How do they compare to services already available under TANF or Job Training Partnership Act (JTPA) funding?What are the net impacts of various WtW program approaches on employment and on families' wellbeing?
- What challenges do grantees confront as they implement and operate WtW programs?
- Do the benefits of WtW programs outweigh their costs?
- How well do PICs and other nonTANF organizations the primary vehicles for funding and operating WtW programs meet the challenge of implementing WtW programs for those hardest to employ?
To address these questions, the evaluation plan includes three main components:
- Descriptive Assessment of All WtW Grantees. A mail survey of all formula and competitive grantees, in 1998 and 1999, will provide a comprehensive overview of program designs and activities offered, target populations, characteristics of participants, and, to the extent available, placement outcomes. Site visits will be made to a selected group of grantees to develop a fuller understanding of program variations and to aid in selection of sites for indepth studies. This report is based on responses to the first of the two grantee surveys. Findings from the second survey will be reported in early 2000.
- In-Depth Impact and CostEffectiveness Study. In 8 to 10 sites that agree to participate, rigorous studies using randomassignment designs will be conducted from 1999 to 2001 to determine what difference WtW programs make in employment and family wellbeing outcomes.4 Where the number of people eligible for the program exceeds WtW program capacity, they will be assigned to a program group or a control group. Comparing outcomes for the two groups will yield estimates of program impact and help identify successful program models. Comparisons will also be made between the net benefits of these impacts and the additional costs of delivering program services. Findings on program impacts will be reported in fall 2000 and again in 2002.
- In-Depth Process and Implementation Study. In 1999 and 2000, site visits will be conducted in 12 to 15 grantee sites, including impact evaluation sites and several others where impact analysis may not be possible but that are of interest for other reasons, including their innovative approaches, settings, or target groups. These visits will include discussions with staff of WtW programs and related agencies, focus groups with participants, and program observation. The study will identify implementation issues and challenges, as well as program details that help explain how programs achieve desired impacts. A report on program implementation is scheduled to be issued in late 1999.
In addition to these three core components, a special process and implementation study will focus on documenting tribal welfare and employment systems, the supportive services they provide, and how tribes integrate funds from various sources to move their members from welfare to work. Under the BBA, Congress also authorized grants to American Indian tribes and mandated that their programs be evaluated. This component of the evaluation will focus on implementation experiences in 10 sites, selected to include those with innovative designs and practices, signs of some success, and variation in location, size, and local labor market and economic conditions. Findings from the tribal program evaluation will be reported in fall 2000.
In the rest of this report, we present findings from the first grantee survey. The survey was mailed in early November 1998 to 598 actual and potential WtW grantees. Actual grantees included the 51 organizations that had received competitive grants directly from DOL in its first round of awards in May 1998. Potential grantees were mostly Service Delivery Agencies (SDAs) eligible for receiving an allocation of their state's formulabased WtW grant; 18 of these organizations had received competitive grants.5
The analysis presented is based on 415 responses provided by WtW grantee organizations. Responses were received from 422 of the 598 organizations included in the sample, for an overall response rate of 70.6 percent. However, three organizations were excluded from the analysis because they reported that they had refused their substate allocation of federal WtW formula funds. The responses for the remaining 4 organizations excluded from the analysis were received too late to be processed for inclusion in this report. Response rates by state, by type of grant, and overall are presented in the appendix.
The grantee survey data presented in this report are also complemented with information gathered through less formal discussions with grantees. Such discussions occurred as grantees called with questions about the survey and as evaluation staff pursued with interested grantees the possibility of their participation in the indepth component of the national evaluation.
The 415 respondents were generally similar, in organization type and sources of WtW funding, to the overall sample of 598 organizations to whom the survey was mailed (Table A.1). However, organizations that received competitive grants responded at a somewhat higher rate (78 percent) than the SDAs that account for almost all formula grantees (70 percent). As a result, organizations that received only formula grants comprised 91.5 percent of the overall survey sample, but 88.2 percent of grantee responses. In contrast, competitive grantees (including those that also received formula funds) made up 8.5 percent of the sample and 11.8 percent of grantee responses (Table A.1). Survey respondents were also similar in their levels of funding to nonrespondents although nonrespondents were more likely to be organizations with smaller WtW grants (Table A.2). We estimate that survey respondents account for 74.9 percent of the WtW funds awarded to the sample as a whole (not shown in table).
The design of the survey anticipated that many grantees would have little actual implementation experience to report on, and this expectation proved to be warranted. Many of the survey questions were designed to allow grantees to report on their early experience or their plans and projections, with regard to service strategies and types of services delivered, recruiting approaches and scale of enrollment, and demographic characteristics of the population served. During the survey, many grantees called with reservations about responding because their programs were not under way and, in some cases, even their plans remained uncertain; they were urged to respond with their best projection of where their WtW program was headed. Some grantees knew they would receive WtW funds and were aware of the amount of their substate formula allocation but had not yet actually received their funds. In general, as data presented later indicate, the number of participants enrolled in most WtW programs by the time of the survey was relatively small.
GRANTEE CHARACTERISTICS: OVERALL SURVEY SAMPLE
|Overall Survey Samplea
(n = 598)
(n = 415)
|Grant Type(s) Received|
|Formula WtW Grant||91.5||88.2c|
|Formula and Competitive Grants||3.3||5.3d|
|Source: National Evaluation of the WelfaretoWork Grants Program, First Grantee Survey (November 1998February 1999).
a This column presents information that was available on all local substate formula grantees and Round 1 competitive grantees from grantee lists provided by DOL in preparation for the survey.
b Other types of grantees include human services agencies, other public agencies, nonprofit communitybased organizations, universities and colleges, and an organization serving people with disabilities.
c Includes 75.9 percent of grantees that received formula funding only and 11.6 percent that received formula funding and a share of their state's discretionary funding. Also included are a few grantees (0.7 percent) that reported they had received only a share of the state's discretionary funding; since such grantees were not included in the survey sample, they most likely were expecting to receive substate formula funding but had not yet received the actual funds.
d Includes 4.1 percent of grantees that received formula and competitive grants only and an additional 1.2 percent that received these two types and also a share of their state's discretionary funding.
DISTRIBUTION OF SURVEY RESPONDENTS AND NONRESPONDENTS
|All Grants||Formula Grants||Competitive Grants|
|Less than $250,000||6.8||11.3||7.6||12.0||0.0||0.0|
|$250,000 to $999,999||50.3||47.8||55.7||50.7||2.4||0.0|
|$1,000,000 to $2,999,999||28.5||28.3||29.1||28.6||23.8||22.2|
|$3,000,000 or more||14.4||12.6||7.6||8.7||73.8||77.8|
|Average grant amount||$1,829,890||$1,581,522||$1,599,499||$1,434,891||$3,848,558||$4,025,375|
|Source: For formula grants, WelfaretoWork state plans submitted to DOL for FY 1998 (various dates). For competitive grants, grantees' applications to DOL (various dates).
Note: This table classifies grants by funding amount rather than grantees, and the data from state plans on which it drew were incomplete. As a result, the sample size does not match the sample of 598 with which the survey began. Funding information was obtained from state plans and competitive grant applications for a total of 569 grants, but 20 of the grantees received both formula and competitive funding, so 549 grantee organizations are represented in the table. Funding information for the remaining 49 grantees was not available. For four states (Alabama, Georgia, Kentucky, and Washington) substate allocations of formula funding were not available (42 SDAs). For 7 additional SDAs in other states, the formula allocation could not be determined from the state plans.
This first evaluation report must therefore be regarded as a very early look at the nature of the programs that will emerge, the services they will provide, and projections concerning the success they will achieve in moving participants into the workforce. The second survey, in fall 1999, will provide a more complete picture of the WtW grants program. The indepth study component of the evaluation, focusing on the selected 15 sites, will provide deeper insights into the issues confronted in implementing WtW services and their impacts on participants' employment success.
1. Under the Workforce Investment Act of 1998, new workforce investment boards (WIBs) will replace PICs and WDBs, providing in all states a means for coordinating workforce investment, adult education and literacy, and vocational rehabilitation services through OneStop Centers.
2. The three problems specified by the BBA are (1) lack of a high school diploma or general equivalency diploma (GED) and low reading and math skills, (2) substance abuse problems, and (3) a poor work history.
3. U.S. Department of Health and Human Services Home Page. "Change in Welfare Caseloads as of September 1998." January 27, 1999.
4. Outcome measures identified in the legislation include placements in unsubsidized employment (including those lasting six or more months), placements in the private and public sectors, the earnings of individuals who obtain employment, and the average WtW program expenditures per placement. The evaluation will gather evidence on these and other outcomes, including the types of services WtW participants receive and their receipt of TANF benefits.
5. The survey was mailed to JTPA SDAs in all states, except the six that had turned down WtW formula funding (Idaho, Mississippi, Ohio, South Dakota, Utah, and Wyoming). In 12 cases, SDAs were replaced as survey respondents by alternative WtW administrative entities approved under waiver procedures allowed under the law. Only Round 1 competitive grantees were included, but recipients of competitive grants awarded under the second round of funding (in November 1998) and the third round of funding (in September 1999) will be included in the second grantee survey, in fall 1999. The survey was mailed late (in January 1999) to Alaska, because of delays in processing information on the alternative entities substituting for PICs as grantees.