The Department of Health and Human Services' efforts to study measures of welfare program outcomes began with a requirement of the Family Support Act of 1988. The resulting report acknowledged that performance under the welfare program's Job Opportunities and Basic Skills Training (JOBS) program should not be measured solely by levels of activity or participation, but it stopped short of making recommendations for specific performance standards and measures of outcomes (HHS, 1994). While enactment of the Government Performance and Results Act (GPRA) of 1993 has increased the use of performance measurement systems at all levels of government, critical issues, as discussed below, still need to be resolved before making recommendations for the best set of outcomes for performance measurement and standards, particularly when they are linked to financial consequences (penalties or bonuses).
In developing this report, we set out to accomplish several objectives. First, we reviewed the literature on performance measurement specifically as it applies to and is used in welfare and welfare-to-work programs. Second, we analyzed the participation requirements under the TANF program (as well as the JOBS participation requirements) to determine the benchmark against which potential outcome measures for evaluating the success of state block grant programs in helping recipients move from welfare to work would be compared. In order to take advantage of the experience gained through implementation of the TANF participation rate requirements, we delayed this analysis until the participation rate data for the first year of TANF implementation became available in summer 1999. Third, we sought input and advice from a wide range of individuals and organizations (including state representatives, researchers and advocates) to get their sense of the goals of TANF for which outcome measures would be appropriate and their suggestions for potential measures. Fourth, starting with the suggestions we received during our consultation process, we constructed a representative list of possible outcome measures, including an analysis of the data and measurement issues that would affect their usefulness as alternatives to the minimum participation rates. Finally, this report stops short of making recommendations for specific measures; rather, it provides a framework for policymakers to use in determining whether outcome-based performance measures should be used as a substitute for or in conjunction with the minimum participation rates, and if so, in selecting among potential measures.
Our immediate objective in preparing this report is to identify for policymakers the plethora of factors affecting state performance that might be impacted by outcome-based measures in general, and some specific outcome measures in particular. As noted above, the legislative request for this report mandated that outcome-based performance measures be evaluated as an alternative to the minimum participation rate requirements under current law. Because there are financial penalties linked to these participation rate requirements, we looked for outcome-based performance measures that are sufficiently robust to justify linking them to financial consequences.
Our goal is to present the strengths and weaknesses of different approaches to outcome-based performance measurement and different performance measures and highlight the tradeoffs that must be made, such as between measuring the long-term success of a program and having results quickly in order to provide timely feedback on program performance. In particular, we discuss in some detail the data limitations that constrain the choices of possible performance measures.
Chapter I of this report provides a brief summary of the evolution and application of performance measurement techniques for government-wide accountability under GPRA, with a special focus on the history of performance measurement in welfare-to-work programs. It also highlights the current system of penalties and rewards under the TANF program.
In Chapter II we discuss the basic elements of an outcome-based performance measurement system, and identify a number of issues that need to be taken into consideration in applying such a system to the TANF program.
Chapter III presents an illustrative but not exhaustive set of possible alternative outcomes measures for the TANF program, most of which were discussed at a consultation session we held in July 1999. These examples are presented both to illustrate the range of issues that need to be considered in developing an outcome-based performance measurement system for TANF, and because measures similar to these are likely to be among the candidates for inclusion in such a system. For each item addressed, we describe the performance measure and its relationship to the TANF program and examine various measurement issues that arise in defining how the measure would be calculated, issues related to the availability, quality and timeliness of data to calculate the measure, and fairness issues related to whether some states would be advantaged or disadvantaged by a particular measure.
Finally, in Chapter IV, drawing from our examination, we offer some conclusions to guide future considerations of instituting outcomes measures to evaluate states' success in implementing TANF.
This report also includes five appendices. Appendix A provides the results of our literature review of the use of outcome-based performance measures in welfare and workforce development programs. Appendix B summarizes the meeting we hosted in July 1999 to consult with states, researchers and advocates on potential outcomes measures for evaluating the success of the states in moving individuals out of the welfare system through employment as an alternative to the minimum participation rates. Although summarized in this Appendix, the views of consultation participants are incorporated throughout the report. Appendix C includes a description of the participation requirements for welfare recipients under both TANF and the predecessor JOBS program, and compares the two. In Appendix D we identify select data sources (both survey and administrative) for potential outcome measures and describe the characteristics of each. A bibliography is included as Appendix E.
It is our hope and intention that this report provides relevant information on strategies for evaluating states' success in implementing TANF to inform upcoming discussions pertinent to the program's reauthorization.