The following report is submitted pursuant to section 107 of the Personal Responsibility and Work Opportunity Reconciliation Act of 1996 (P.L. 104-193; PRWORA), which provides that the Secretary of the Department of Health and Human Services (HHS) shall conduct a study of alternative outcomes measures under the Temporary Assistance for Needy Families (TANF) program and submit a report on the findings of that study to the Senate Finance Committee and the House Ways and Means Committee.
Sec. 107. STUDY ON ALTERNATIVE OUTCOMES MEASURES(a) STUDY.-- The Secretary shall, in cooperation with the States, study and analyze outcomes measures for evaluating the success of the States in moving individuals out of the welfare system through employment as an alternative to the minimum participation rates described in section 407 of the Social Security Act. The study shall include a determination as to whether such alternative outcomes measures should be applied on a national or a State-by-State basis and a preliminary assessment of the effects of section 408(a)(7)(C) of such Act(1).
This report has been prepared by staff in the Department of Health and Human Services, Administration for Children and Families (ACF) and the Office of the Assistant Secretary for Planning and Evaluation (ASPE) with assistance from contractor staff.
[ Go to Contents ]
The Department of Health and Human Services' efforts to study measures of welfare program outcomes began with a requirement of the Family Support Act of 1988. The resulting report acknowledged that performance under the welfare program's Job Opportunities and Basic Skills Training (JOBS) program should not be measured solely by levels of activity or participation, but it stopped short of making recommendations for specific performance standards and measures of outcomes (HHS, 1994). While enactment of the Government Performance and Results Act (GPRA) of 1993 has increased the use of performance measurement systems at all levels of government, critical issues, as discussed below, still need to be resolved before making recommendations for the best set of outcomes for performance measurement and standards, particularly when they are linked to financial consequences (penalties or bonuses).
In developing this report, we set out to accomplish several objectives. First, we reviewed the literature on performance measurement specifically as it applies to and is used in welfare and welfare-to-work programs. Second, we analyzed the participation requirements under the TANF program (as well as the JOBS participation requirements) to determine the benchmark against which potential outcome measures for evaluating the success of state block grant programs in helping recipients move from welfare to work would be compared. In order to take advantage of the experience gained through implementation of the TANF participation rate requirements, we delayed this analysis until the participation rate data for the first year of TANF implementation became available in summer 1999. Third, we sought input and advice from a wide range of individuals and organizations (including state representatives, researchers and advocates) to get their sense of the goals of TANF for which outcome measures would be appropriate and their suggestions for potential measures. Fourth, starting with the suggestions we received during our consultation process, we constructed a representative list of possible outcome measures, including an analysis of the data and measurement issues that would affect their usefulness as alternatives to the minimum participation rates. Finally, this report stops short of making recommendations for specific measures; rather, it provides a framework for policymakers to use in determining whether outcome-based performance measures should be used as a substitute for or in conjunction with the minimum participation rates, and if so, in selecting among potential measures.
Our immediate objective in preparing this report is to identify for policymakers the plethora of factors affecting state performance that might be impacted by outcome-based measures in general, and some specific outcome measures in particular. As noted above, the legislative request for this report mandated that outcome-based performance measures be evaluated as an alternative to the minimum participation rate requirements under current law. Because there are financial penalties linked to these participation rate requirements, we looked for outcome-based performance measures that are sufficiently robust to justify linking them to financial consequences.
Our goal is to present the strengths and weaknesses of different approaches to outcome-based performance measurement and different performance measures and highlight the tradeoffs that must be made, such as between measuring the long-term success of a program and having results quickly in order to provide timely feedback on program performance. In particular, we discuss in some detail the data limitations that constrain the choices of possible performance measures.
Chapter I of this report provides a brief summary of the evolution and application of performance measurement techniques for government-wide accountability under GPRA, with a special focus on the history of performance measurement in welfare-to-work programs. It also highlights the current system of penalties and rewards under the TANF program.
In Chapter II we discuss the basic elements of an outcome-based performance measurement system, and identify a number of issues that need to be taken into consideration in applying such a system to the TANF program.
Chapter III presents an illustrative but not exhaustive set of possible alternative outcomes measures for the TANF program, most of which were discussed at a consultation session we held in July 1999. These examples are presented both to illustrate the range of issues that need to be considered in developing an outcome-based performance measurement system for TANF, and because measures similar to these are likely to be among the candidates for inclusion in such a system. For each item addressed, we describe the performance measure and its relationship to the TANF program and examine various measurement issues that arise in defining how the measure would be calculated, issues related to the availability, quality and timeliness of data to calculate the measure, and fairness issues related to whether some states would be advantaged or disadvantaged by a particular measure.
Finally, in Chapter IV, drawing from our examination, we offer some conclusions to guide future considerations of instituting outcomes measures to evaluate states' success in implementing TANF.
This report also includes five appendices. Appendix A provides the results of our literature review of the use of outcome-based performance measures in welfare and workforce development programs. Appendix B summarizes the meeting we hosted in July 1999 to consult with states, researchers and advocates on potential outcomes measures for evaluating the success of the states in moving individuals out of the welfare system through employment as an alternative to the minimum participation rates. Although summarized in this Appendix, the views of consultation participants are incorporated throughout the report. Appendix C includes a description of the participation requirements for welfare recipients under both TANF and the predecessor JOBS program, and compares the two. In Appendix D we identify select data sources (both survey and administrative) for potential outcome measures and describe the characteristics of each. A bibliography is included as Appendix E.
It is our hope and intention that this report provides relevant information on strategies for evaluating states' success in implementing TANF to inform upcoming discussions pertinent to the program's reauthorization.
[ Go to Contents ]
Over the years, a range of terms has been used to describe the different types of performance measures used to gauge program success. Some studies have tried to achieve consensus on definitions of key terms in performance measurement, particularly for use in welfare-to-work and other employment programs (Brown and Corbett, 1997; Hatry, 1999; Martin and Kettner, 1996; Midwest Welfare Peer Assistance Network, 1999; U.S. Department of Health and Human Services, 1994). In particular, these studies draw a crucial distinction between process measures and outcome measures.
While this distinction is conceptually important, in practice, there is often some uncertainty about whether a particular measure should be considered a process or an outcome measure. This is not merely the result of confusing terminology, but reflects the reality that there is often a continuum between pure process measures and pure outcome measures. For example, in a program for teen parents, the number of people served and the cost per participant are process measures. Depending on the specific goals of the program, outcome measures might include the fraction of participants who have not had a subsequent child two years later, or the fraction of participants who are employed. Measures that fall in the middle of the continuum might include the fraction of participants who attend high school, the fraction of participants who have received a certificate of General Educational Development (GED), or the fraction who meet the program's internal definition of successful completion. Such intermediate goals are sometimes referred to as "interim outcome measures" because they represent an important milestone even though they are not the ultimate goal of the program. Other sources refer to such measures as "outputs."
Performance measurement, or the measurement of the results (or outcomes) and efficiency of services or programs (Hatry, 1999), has been the subject of growing interest at all levels of government in recent years. In particular, there has been a recent movement toward increased use of outcome measures, rather than process measures. A number of broad trends have contributed to this growth.
In response to critics who have expressed skepticism about the value of government services, many providers of government services have turned to outcome-based measures in order to prove the utility of their efforts. Specifically, this new accountability focus now requires providers to show not just that they have in fact spent the public money on the activities for which it was designated, and not just that they have been efficient in serving as many people as possible with the available funds (process measures), but also that the statutory goals of the programs are being met and that recipients are better off as a result (outcome measures).
To some degree, this trend represents the spread of techniques used in the private sector - including a focus on measurement of product quality and customer satisfaction and on the establishment of numerical targets for improvement. Similar techniques had also been used in the Defense Department since the 1950s to compare the expected costs and effectiveness of various proposed weapons systems. However, such techniques had not often been applied to the provision of social services. Many providers of social services had only rudimentary capacities to track what happened to the recipients of their services.
With the enactment of the GPRA in 1993, Congress required all federal agencies to identify the goals of their programs and to report annually on their progress in achieving these goals. GPRA seeks to shift the focus of federal management and decisionmaking from a preoccupation with process measures such as the number of tasks completed or units of service provided to a more direct consideration of the results or outcomes of programs - that is, the real differences the tasks or services provided make in people's lives (Hinchman, 1997).
The recent devolution of policy and program design and funding to the state and local level has also increased the attention paid to performance measurement. In a number of areas, including human services policy, federal policy makers have created block grants, which give states great flexibility in their use of funds within broad program parameters. In such an environment, the most logical way to hold states accountable for their use of public funds is to monitor program outcomes.
The increased emphasis on holding public agencies accountable for the attainment of program goals and the outcomes of their clients is also reflected in the numerous state initiatives to develop and use performance measures. In some states, the welfare agencies are participating in comprehensive performance measurement systems which focus on establishing indicators or benchmarks of progress toward goals across programs and agencies. In other states, performance measures are used internally by agencies to monitor the performance and accountability of local offices or of contractors.
However, the shift toward use of outcome-based performance measures has occasioned some controversy, particularly when financial consequences have been attached to agencies' success or failure in achieving specified targets or standards. The major reason is that even the most effective programs are only one element out of many that affect participants' outcomes. As Forsythe (2000) notes, "almost by definition, high-level outcome measures track social changes that are influenced by factors that are not under the direct control of operating agencies," such as the overall state of the economy, the underlying social and demographic characteristics of participants, and societal attitudes about the roles of men and women. Most program administrators are understandably leery of being measured - and possibly rewarded or penalized - on results which they can not fully control. Yet, the only measures which are totally under the control of program operators are process measures, such as the number of clients served. This issue is discussed in more detail below.
[ Go to Contents ]
Interest in outcome-based performance measures in the employment training and welfare-to-work arenas has been growing for quite some time. Since 1982, the U.S. Department of Labor has required states and local service agencies receiving funding under the Job Training Partnership Act (JTPA) program to report data on client outcomes and has provided corresponding incentives and sanctions on the basis of that outcome data. For adult JTPA participants, the key performance measures were the employment rate and average weekly earnings during the 13th week after program exit. These outcomes were measured both for all adult participants and for the subset of participants who were also welfare recipients. The Workforce Investment Act of 1998 (WIA), which replaced JTPA, includes a performance measurement system which builds on the JTPA model.
The Family Support Act of 1988 (P.L. 100-485), which created the JOBS program, also established work participation rate requirements for the states to meet. Prior to the establishment of JOBS, the primary system for holding states accountable for their use of AFDC funds was the Quality Control (QC) system. QC focused on the accuracy of the eligibility determinations and benefit calculations. The Family Support Act emphasized participation in work activities and established a participation rate requirement as a measure of programs' success in engaging recipients in work-related activities, primarily education and training. States were required to engage seven percent of non-exempt recipients in activities in Fiscal Year (FY) 1990, rising to 20 percent by FY 1995. (More detail on the JOBS participation rate requirements and a comparison of them to the participation rate requirements under TANF are provided in Appendix C.)
In the Family Support Act, Congress asked HHS to develop recommendations for performance standards regarding "specific measures of outcomes" beyond simple measures of levels of activity or participation. The resultant report, completed in 1994, acknowledged the importance of developing an outcome-based system of performance measures, but raised some critical issues that the Department believed "needed to be addressed and dealt with prior to using outcomes as the basis for performance measurement and standards." (HHS, 1994) These included:
The 1994 report did not recommend specific outcome-based performance measures, but rather presented a workplan for a process to refine existing participation rates, develop outcome-based performance measures and standards, and strengthen accountability mechanisms, including modifications to the Quality Control system. This proposed course of action was overtaken by the enactment of PRWORA and the implementation of the new TANF block grants.
[ Go to Contents ]
If state variation in JOBS program objectives (considering the relatively prescriptive federal mandates) contributed to difficulty in recommending a system of outcome-based performance measures under JOBS, then passage of PRWORA added to the problem's complexity. PRWORA eliminated the cash welfare (AFDC) and job training (JOBS) programs and replaced them with a block grant called Temporary Assistance for Needy Families (TANF). Its purpose was to increase state flexibility in providing assistance to needy families within the framework of four broad Congressionally mandated goals. These goals are to:
States have total flexibility in setting priorities among these goals as they choose how to spend their TANF block grant. There is no requirement that states spend equal amounts on each of these goals, or even that they spend any funds on a given goal. To date, the majority of TANF funds have been spent on cash and work-based assistance, which was the primary use of funds allowed under AFDC. Most of the remaining funds have been spent on work activities, child care, and other work-based supports. States also have full control over such aspects of the program as benefit levels, eligibility, and the design and sequencing of work activities. In order to protect state flexibility, Congress prohibited HHS from regulating state behavior unless specifically required under the law.
In counterpoint to this broad state flexibility, in designing TANF, Congress was quite prescriptive in some specific areas - such as time limits, sanctions, and requirements imposed on teen parents. These mandates are backed up by financial penalties, under which states may lose a portion of their block grant allocation for such violations as failure to participate in the income and eligibility verification system, failure to maintain a certain level of historic funding effort, or failure to comply with the five-year time limit on assistance. These penalties are all attached to process measures, and are designed to ensure compliance with Congressional priorities.
The penalty which has attracted the most attention so far is for failure to meet the work participation rate requirement, under which states must engage a target percentage of all recipients (with very limited exceptions) in work and work-related activities. As described in detail in Appendix C, the list of activities which may be counted under this requirement is more restrictive than the countable activities under JOBS. This participation rate requirement is becoming more challenging over time, as both the hours of participation required in order to be counted and the target participation rate rise each year. In the first three years of the TANF program, all states have achieved the required all-families participation rate, but 19 states failed to achieve the higher target for two-parent families in FY 1997, 14 failed to do so in FY 1998 and 8 failed to do so in FY 1999. (See Table 2 and Table 3 in Appendix C.)
While TANF does not include any penalties based on outcome-based performance measures, it is worth noting that the participation rate does have some aspects of an outcome measure, in that most of the people whom the states can count toward the rate are working in unsubsidized employment, which is key to one of the most fundamental goals of TANF - requiring families to make efforts to work. Moreover, states receive credit toward the participation rate for the degree to which their caseloads have declined since 1995, to the extent that these changes were not caused by changes in eligibility. Thus, the participation rate also rewards states that have moved families off welfare.
The bonuses under TANF are more outcome-oriented. Under one bonus, the Bonus for Reductions in Out-of-Wedlock Births, Congress provided up to $100,000,000 per year in bonuses for up to five states that demonstrate the greatest decreases in out-of-wedlock births, so long as those states also have a reduction in the abortion rate from FY 1995. Under a second bonus, the High Performance Bonus, Congress appropriated an average of $200,000,000 per year for five years to reward the highest performing states in achieving the goals of TANF. Congress did not specify the measures to be used, but required HHS to develop a formula for scoring states' performance, in consultation with the National Governors' Association and the American Public Human Services Association (formerly the American Public Welfare Association). For bonuses awarded in FYs 1999-2001, the measures used are primarily related to the first two goals of TANF: job entry and success in the workforce, measured by a weighted combination of earnings gains and job retention. Awards are provided both for absolute performance and for improvement compared to the previous year. States choose whether to compete for any or all of these work measures.
Under the Final Rule, published in the Federal Register on August 30, 2000 (65 FR 52814), for High Performance Bonuses awarded in FYs 2002 and 2003, states will also have the opportunity to compete under a measure of family formation and stability and five measures of states' success in supporting work and self-sufficiency by providing eligible families with health insurance (through Medicaid and SCHIP), food stamps, and child care subsidies (HHS, 2000(a)).
There is some overlap between the mandate for this report and the requirement to reward with a High Performance Bonus states identified as high performers, based on formulas that measure states' performance in operating their TANF programs. This report takes into account the input received through the High Performance Bonus consultation process, as well as lessons learned in developing the High Performance Bonus interim guidance and proposed and final rules. We also include the High Performance Bonus measures among those discussed in detail in Chapter III, as these measures could well serve as the building blocks for a system of outcome-based performance measurement.
However, the two requirements differ in some important respects. First, the High Performance Bonus measures needed to be selected and implemented in a narrow time frame, with limited possibility of developing new data sources or using past performance to establish benchmarks. More importantly, in considering outcome-based performance measures as an alternative to the work participation requirements, this report examines the use of such measures for penalties as well as for bonuses, and explores the issues which would need to be addressed if outcome-based performance measures were to be used as the primary mechanism for holding states accountable for their use of federal TANF funds.
[ Go to Contents ]
In keeping with the Department's commitment to a broad consultation strategy used throughout the development of TANF-related regulations, HHS hosted a consultation meeting on July 21, 1999 with representatives from states and research and advocacy organizations. The purpose of the meeting was to identify and discuss some potential outcomes measures that could be used to evaluate state performance as an alternative to the minimum participation rates. All states were invited to send representatives, as were a substantial number of research and advocacy organizations with whom the Department has consulted on other TANF issues, including the development of state guidance on the High Performance Bonus. About half the states participated, as well as almost 20 research and advocacy organizations and several federal agencies. A list of states and organizations represented is included in Appendix B, along with a more detailed summary of the consultation meeting.
The objectives of the consultation were to discuss:
A major topic of discussion by the group was whether outcome-based performance measures should be focused only on the goal of moving recipients off welfare through employment, or whether they should address the other goals of TANF as well. Participants suggested measures representing a wide range of outcomes of interest, falling roughly into nine broad categories:
Most of the participants were open in principle to measuring state performance across the wider range of TANF goals, but there was no consensus on which goals were most important. The first two categories of outcomes (i.e., work participation and employment and poverty and movement to self-sufficiency) were the focus of most of the discussion, along with identification of some specific measures that might be used to promote achievement of the desired outcomes. Discussion of the broader data issues affecting the measures (e.g., is it available, how much does it cost to collect it, is it accurate and precise at the state level, is it reported frequently and in a timely manner?) was limited, although in general, states expressed reservations about any measures that would increase their data collection and reporting burden or change their data requirements in the short term.
Efforts to identify the "preferred" and "least favorite" measures among those generated through the brainstorming session revealed fairly substantial differences between the group of state representatives and the group of representatives from research and advocacy organizations. State representatives favored a limited set of core measures, such as those used for the first two years of the High Performance Bonus under the interim guidance. They suggested that states should have the option whether to compete on additional measures, such as progression along the poverty continuum, a measure of the percent of those required to work who have earnings, and welfare recidivism. On the other hand, the researchers and advocates favored multiple measures in order to reflect the wide range of possible goals under TANF, such as a measure of labor market success, a broader measure of program participation, a measure of extreme child poverty, and a measure of the provision of supportive services. There was widespread agreement among all participants, however, that the two-parent work participation rate was a "least favorite" measure.
The summary of the highlights of the consultation, which was shared with all participants and consultation invitees, is at Appendix B.
1. Section 408(a)(7)(C) of the Social Security Act, as amended by PRWORA, provides for an exemption from the 60 month time limit for individuals who have been "battered or subjected to extreme cruelty." See page 17 of this report for a discussion of the domestic violence provisions of TANF. [return to text]
Main page of report
Contents of report
Home Pages:
Human Services Policy (HSP)
Assistant Secretary for Planning and Evaluation
(ASPE)
Administration for Children and Families
(ACF)
U.S. Department of Health and Human Services
(HHS)
Updated: 02/03/04