On July 21, 1999, the Department hosted a consultation meeting with representatives from states and research and advocacy organizations to initiate our formal study and analysis of potential outcomes measures for evaluating the success of the states in moving individuals out of the welfare system through employment, as an alternative to the minimum participation rates. Consultation participants discussed TANFs overall goals related to work that should be advanced through performance measures and, in turn, some specific measures that might be used to promote those goals without creating perverse incentives. A wide range of potentially desirable outcomes were identified through a brainstorming process both from the narrower perspective of targeting employment successes solely, and from the broader perspective of recognizing success in meeting any of the purposes of TANF. Potential measures fell roughly into the following broad categories:
- work participation and employment;
- poverty and movement to self-sufficiency;
- requirements for two-parent families;
- duration of welfare receipt;
- caseload reductions;
- child outcomes;
- supportive services;
- customer satisfaction; and
- educational outcomes.
The majority of the discussion focused on the first two of these categories.
Work Participation and Employment
There was general agreement among consultation participants that there is interest in what is happening to both current participants and leavers. However, many technical concerns were raised:
- Due to differences across state TANF programs, leavers are not the same population from state to state. Therefore, it is unfair to compare the leavers populations across states. Does it make sense to have the base be all recipients at a given point in time?
- How should multiple exits and entries within a specified period be treated? Should we exclude people who return to welfare?
- In considering possible measures, discussants were unclear what treatment should be measured. In the Workforce Investment Act (WIA), the treatment is services. What is the TANF treatment receipt of cash assistance?
- If employment were a measure, what should count as employment? Would one dollar of earnings in a quarter constitute employment?
Most participants agreed that in an ideal world, employment stability and earnings progression should be measures are also outcomes of interest along with job entry., but that there are some concerns regarding the selection of specific measures. They re was some discussion about discussed the problems of selecting an appropriate time frame for measuring employment stability and wage progression. There was general consensus that outcome measures lose their effectiveness as incentives if there is too great a lag between the time the actions are taken and the calculation of outcome measures whether tied to penalties or bonuses. Yet, it is unrealistic to expect significant wage progression in a short period of time, and job retention is more meaningful over longer periods of time. Six months was suggested as a compromise by one researcher, but some states argued that it was important to use the same timeframe (13 weeks) used for performance measurement under WIA.
Poverty and Movement to Self Sufficiency
Consultation participants were sharply divided on the issue of whether poverty would be an appropriate measure. Researchers and advocates almost universally argued in favor of a measure of poverty and income, while state representatives argued against it on the grounds that it is not reasonable to expect TANF agencies to solve the poverty problem. Everyone recognized that AFDC never served more than a fraction of the poor population. It was generally concluded that a measure of extreme poverty (under 50 percent of the poverty level) would be more likely to capture changes caused by TANF than the basic poverty rate.
Following the brainstorming process to identify desirable outcomes and potential measures, attendees were divided into two groups one consisting of state representatives, and the other consisting of researchers and advocates. Each group discussed the measures proposed in the brainstorming session and identified its most and least favorite measures. The results are summarized in the table below.
There was an overall broad consensus that, in a program with as many different objectives as TANF, no single measure could adequately capture all of the variety of possible state actions. In general, researchers and advocates were, therefore, inclined toward a wide range of measures. Mindful of the administrative burden involved, state representatives supported the idea of a menu of possible measures from which states could choose the ones under which they wish to compete.
Researchers and advocates supported the inclusion of a broad population measure, such as the rate of extreme child poverty. They argued that, since TANF is not an entitlement, it is essential to include a measure that reflects states choices of whom to serve under TANF, not just their success with the population served. By contrast, state representatives strongly opposed such measures. They argued that population measures are driven by many factors over which state TANF agencies have little or no control. Researchers and advocates agreed that it would be important to adjust the measures to reflect the different circumstances and demographics in each state. These might include statistical models (such as under the Job Training Partnership Act (JTPA) performance measurement system), negotiated standards (such as under the Tribal TANF program), or measures that reflect improvements over time.
|Most Favorite Measures||Least Favorite Measures|
|Researchers And Advocates||
There was a great deal of ambivalence about the appropriate role of process measures. On the one hand, states thought federally established process measures were inappropriate for a block grant such as TANF. (There was much more interest in process measures at the state level as a management tool for program administrators.) At the same time, state representatives were deeply concerned about being evaluated using outcome measures over which they did not have full control. There was some support expressed for a two-stage system, under which the federal government would look first at outcomes and consider process measures only for those states that had not met their outcome performance goals. Such an approach would allow high-performing states to operate under reduced federal oversight, while states that did all the right things but had poor outcomes due to events beyond their control would be protected from penalties. Participants generally agreed that there was a continued role for a participation rate measure, but that the list of activities that count as participation should be expanded to give states credit for engaging recipients in education and other activities such as mental health services or substance abuse treatment, when needed.
States generally opposed the addition of measures for which there are no baseline data to indicate what is a reasonable level of performance. This was due both to a natural concern about how standards would be set in the absence of such data (the two-parent family participation rate was cited as an example of an unreasonable standard) and to a sense that outcome measures lose their effectiveness as incentives if states do not know whether they have a reasonable chance at a bonus or if they are in danger of a penalty. This is particularly true for a rank-based system, such as the current High Performance Bonus, since states have no little idea of how their performance compares to that of their peers.
States were uniformly concerned about the burden of data collection. They recommended that any new measures be based on either national survey data or on the administrative data that states are already required to collect and report under TANF. They also urged that performance measures under TANF be made consistent with those under other related programs, particularly the workforce development programs under WIA. They asked that no changes be made to data collection requirements at least until reauthorization, so that they could stabilize their systems.
The question of performance penalties versus performance bonuses was discussed at some length. State representatives generally supported bonuses rather than penalties. Some basic principles were enunciated:
- A core set of the most critical measures should have both penalties and bonuses attached.
- It is more appropriate to give bonuses than to impose penalties when states have less control over outcomes.
- Caution is needed when considering penalties for measures that are not highly accurate. In particular, thresholds are bad in a "noisy: data system, because narrow real differences can have large consequences. (The TANF requirement for higher state spending levels (MOE) when participation requirements are not met was cited as example.)
- It is essential to develop reasonable standards if penalties are to be assessed. They should be based on baseline data, but not on a simple national average.
- Penalties typically result in an adversarial relationship between the states and the federal government. Historically, when sanctions have been imposed, legal battles have followed, resulting in high costs in both money and time for both sides. Is this worth it?
There was a general sense that in order for bonuses to have incentive effects, all states should be able to compete. This argues for systems where all states can receive awards if they achieve at the appropriate level, as opposed to systems where only a small number of top performers receive awards. Some participants argued for rating states based on improvement, rather than absolute performance, but others opposed this, saying that in the long run it was unrealistic to expect continual improvement. Others suggested that adjustments to reflect state economic circumstances and demographics could help resolve this problem, or that states could be organized into comparable groups and then ranked against their peers.
List of Participating States and Organizations
- District of Columbia
- New Jersey
- New York
- Rhode Island
- South Carolina
- West Virginia
- American Federation of State, County and Municipal Employees
- California Welfare Directors Association
- Childrens Defense Fund
- Census Bureau
- Center on Budget and Policy Priorities
- Center for Law and Social Policy
- Domestic Policy Council
- Johns Hopkins University, Institute for Policy Studies
- Mathematica Policy Research, Inc.
- National Association of Counties
- National Association on Welfare Research and Statistics
- National Organization for Women, Legal Defense and Education Fund
- National Center for Children in Poverty Research Forum
- National Governors Association
- Office of Management and Budget
- The Urban Institute
- U.S. Department of Agriculture, Food and Nutrition Service
- U.S. Department of Health and Human Services, Health Care Financing Administration
- U.S. Department of Labor