In 1994, the Department of Health and Human Services issued a report on the use of outcome measures to evaluate welfare-to-work programs. This report was issued in response to a Congressional mandate to study alternatives to the work activity participation rates required under the Job Opportunities and Basic Skills Training (JOBS) program. The 1994 report on outcome-based performance measures was generally supportive of the concept of outcome measures, but expressed concern about possible unintended consequences. The 1996 welfare reform act mandated a further examination of outcome-based measures for evaluating states’ success in moving individuals from welfare to work, in light of the dramatic program changes the law put in place. In recent years, outcome-based performance measurements have become more common generally. Many states have adopted their own internal outcome-based performance measurement and incentive systems. A total of 47 states submitted data in 1999 in order to compete for the first year of the High Performance Bonus.
Participation rates were initially designed to introduce a sense of reciprocal obligation into a welfare system that, heretofore, had been based on issuing benefits – henceforth, at least some portion of the caseload would be required to do something in exchange for receiving benefits. With the advent of time limits, participation rates also provided a means of ensuring that clients received services that would promote self-sufficiency.
Today’s environment could not have been foreseen when the welfare reform law passed in 1996:
On July 21, 1999, the Department hosted a consultation meeting with representatives from states and research and advocacy organizations to initiate our formal study and analysis of potential outcomes measures for evaluating the success of the states in moving individuals out of the welfare system through employment, as an alternative to the minimum participation rates. Consultation participants discussed TANF’s overall goals related to work that should be advanced through performance measures and, in turn, some specific measures that might be used to promote those goals without creating perverse incentives. A wide range of potentially desirable outcomes were identified through a brainstorming process – both from the narrower perspective of targeting employment successes solely, and from the broader perspective of recognizing success in meeting any of the purposes of TANF. Potential measures fell roughly into the following broad categories:
The majority of the discussion focused on the first two of these categories.
There was general agreement among consultation participants that there is interest in what is happening to both current participants and leavers. However, many technical concerns were raised:
Most participants agreed that in an ideal world, employment stability and earnings progression should be measures are also outcomes of interest along with job entry., but that there are some concerns regarding the selection of specific measures. They re was some discussion about discussed the problems of selecting an appropriate time frame for measuring employment stability and wage progression. There was general consensus that outcome measures lose their effectiveness as incentives if there is too great a lag between the time the actions are taken and the calculation of outcome measures – whether tied to penalties or bonuses. Yet, it is unrealistic to expect significant wage progression in a short period of time, and job retention is more meaningful over longer periods of time. Six months was suggested as a compromise by one researcher, but some states argued that it was important to use the same timeframe (13 weeks) used for performance measurement under WIA.
Consultation participants were sharply divided on the issue of whether poverty would be an appropriate measure. Researchers and advocates almost universally argued in favor of a measure of poverty and income, while state representatives argued against it on the grounds that it is not reasonable to expect TANF agencies to solve the poverty problem. Everyone recognized that AFDC never served more than a fraction of the poor population. It was generally concluded that a measure of extreme poverty (under 50 percent of the poverty level) would be more likely to capture changes caused by TANF than the basic poverty rate.
Following the brainstorming process to identify desirable outcomes and potential measures, attendees were divided into two groups – one consisting of state representatives, and the other consisting of researchers and advocates. Each group discussed the measures proposed in the brainstorming session and identified its most and least favorite measures. The results are summarized in the table below.
There was an overall broad consensus that, in a program with as many different objectives as TANF, no single measure could adequately capture all of the variety of possible state actions. In general, researchers and advocates were, therefore, inclined toward a wide range of measures. Mindful of the administrative burden involved, state representatives supported the idea of a menu of possible measures from which states could choose the ones under which they wish to compete.
Researchers and advocates supported the inclusion of a broad population measure, such as the rate of extreme child poverty. They argued that, since TANF is not an entitlement, it is essential to include a measure that reflects states’ choices of whom to serve under TANF, not just their success with the population served. By contrast, state representatives strongly opposed such measures. They argued that population measures are driven by many factors over which state TANF agencies have little or no control. Researchers and advocates agreed that it would be important to adjust the measures to reflect the different circumstances and demographics in each state. These might include statistical models (such as under the Job Training Partnership Act (JTPA) performance measurement system), negotiated standards (such as under the Tribal TANF program), or measures that reflect improvements over time.
| Most Favorite Measures | Least Favorite Measures | |
|---|---|---|
| State Representatives |
|
|
| Researchers And Advocates |
|
|
There was a great deal of ambivalence about the appropriate role of process measures. On the one hand, states thought federally established process measures were inappropriate for a block grant such as TANF. (There was much more interest in process measures at the state level as a management tool for program administrators.) At the same time, state representatives were deeply concerned about being evaluated using outcome measures over which they did not have full control. There was some support expressed for a two-stage system, under which the federal government would look first at outcomes and consider process measures only for those states that had not met their outcome performance goals. Such an approach would allow high-performing states to operate under reduced federal oversight, while states that did all the right things but had poor outcomes due to events beyond their control would be protected from penalties. Participants generally agreed that there was a continued role for a participation rate measure, but that the list of activities that count as participation should be expanded to give states credit for engaging recipients in education and other activities such as mental health services or substance abuse treatment, when needed.
States generally opposed the addition of measures for which there are no baseline data to indicate what is a reasonable level of performance. This was due both to a natural concern about how standards would be set in the absence of such data (the two-parent family participation rate was cited as an example of an unreasonable standard) and to a sense that outcome measures lose their effectiveness as incentives if states do not know whether they have a reasonable chance at a bonus or if they are in danger of a penalty. This is particularly true for a rank-based system, such as the current High Performance Bonus, since states have no little idea of how their performance compares to that of their peers.
States were uniformly concerned about the burden of data collection. They recommended that any new measures be based on either national survey data or on the administrative data that states are already required to collect and report under TANF. They also urged that performance measures under TANF be made consistent with those under other related programs, particularly the workforce development programs under WIA. They asked that no changes be made to data collection requirements at least until reauthorization, so that they could stabilize their systems.
The question of performance penalties versus performance bonuses was discussed at some length. State representatives generally supported bonuses rather than penalties. Some basic principles were enunciated:
There was a general sense that in order for bonuses to have incentive effects, all states should be able to compete. This argues for systems where all states can receive awards if they achieve at the appropriate level, as opposed to systems where only a small number of top performers receive awards. Some participants argued for rating states based on improvement, rather than absolute performance, but others opposed this, saying that in the long run it was unrealistic to expect continual improvement. Others suggested that adjustments to reflect state economic circumstances and demographics could help resolve this problem, or that states could be organized into comparable groups and then ranked against their peers.
List of Participating States and Organizations
Participating States
Participating Organizations
Main page of report
Contents of report
Home Pages:
Human Services Policy (HSP)
Assistant Secretary for Planning and Evaluation
(ASPE)
Administration for Children and Families
(ACF)
U.S. Department of Health and Human Services
(HHS)
Updated: 02/06/01