Standards identify expected levels of performance and provide the basis for assessing whether states are achieving program goals and, therefore, should be rewarded or penalized. The standards included in an outcome-based performance measurement system should be challenging yet achievable. Standards appear to be most likely to affect states at the margin - those which are in danger of being penalized or within striking range of receiving a bonus. In a study of JTPA programs, for example, Dickinson and West (1988) found that about 42 percent of the local operating entities they studied tried to maximize their measured performance, one-fourth tried only to slightly exceed their standards, and about one-third tried merely to meet their standards in order to avoid program sanctions. If a standard is set too low, it loses its effectiveness as an incentive for states to improve their performance. If it is set too high, states are likely to be put off by the unreasonable standard. Depending on the consequences for failure, states are likely to either simply give up trying to achieve the unreasonable standard or look for ways to get around it. For example, it appears that the high participation rate requirements for two-parent families on TANF have caused several states to change the way they provide assistance to such families by using state maintenance of effort (MOE) dollars rather than federal TANF funds.(3)
It is extremely difficult to determine an appropriate standard without baseline data on past performance. When data for a specific measure have never been collected or analyzed before, neither state nor federal policymakers are likely to know what would be a reasonable level of performance. In developing the TANF High Performance Bonus, HHS dealt with this issue by rewarding the top states in each category, rather than by establishing a fixed standard. It is still too early to tell whether this approach of rewarding the top performers will motivate the broad middle range of states to improve their performance. One encouraging sign, however, is that in the first year of the High Performance Bonus, a wide range of states (46) elected to submit data to compete for a bonus on one or more of the four measures.
The national average is another method that is used to set a standard, as is the case with the Quality Control system for the Food Stamp Program. At the consultation, state representatives expressed opposition to this approach. They objected to not knowing their performance target up front, and to the possibility that a state could find itself penalty-liable in a given year without experiencing any change in its performance, due simply to changes in other states' performance. (The same concerns would apply to a system that penalized the bottom n performers under a measure.)
One important issue is whether to establish a single nationwide performance standard for each measure or to adjust standards to account for differences in economic and demographic circumstances among states. In the past, different federal programs have chosen different options. Within TANF, we have examples of both absolute standards (the state work participation rates) and negotiated standards (the participation rates under the Tribal TANF program). JTPA used a regression model that took into account economic and demographic factors to adjust its performance standards for each state and for local areas. WIA provides for negotiated performance standards at both the state and sub-state level. Elements that must be considered in the negotiations process include: how the standards compare to other areas, taking into account economic and demographic factors and program design; the extent to which the standards promote continuous performance improvement; and the extent to which the standards assist the program in achieving a high level of customer satisfaction. (The use of outcome-based performance measures in these and other welfare and workforce development programs is discussed in more detail in Appendix A.)
One of the concerns that has been raised about modifying standards to reflect differences in demographic conditions is that it reduces the incentive for states to provide appropriate services to those populations identified as "hard-to-serve." The TANF program takes a unique approach to this issue with respect to the domestic violence hardship exemption. Section 408(a)(7)(C) of the Social Security Act, as amended by PRWORA, permits states to exempt victims of domestic violence from the time limit and, under regulations implementing that provision, from the work requirements. Individuals receiving an exemption from work participation rates or the time limit due to domestic violence are not removed from the initial calculations. However, if a state fails to meet the work participation rate requirements or exceeds the cap on time limit extensions, and can show that this failure is due to provision of good cause domestic violence waivers, HHS may grant reasonable cause relief from the penalties. States may only receive this relief if they have adopted the Family Violence Option and are providing appropriate services to individuals granted waivers. To date, no state has needed this relief.
Under the current participation rate requirements, similar relief is not provided to states that fail to meet the standards due to exemptions provided to individuals with other barriers to employment, such as mental health issues or substance abuse. In particular, states do not receive credit for engaging recipients in appropriate services that are not among the list of specific countable work-related activities.
A different approach, which does not directly adjust for economic and demographic conditions but has some of the same effects, is to reward states for improvements rather than (or in addition to) absolute levels of performance. This approach was taken by HHS in developing the TANF High Performance Bonus measures. This gives states that have performed poorly in the past a strong incentive to improve, even if they are unlikely to achieve results that place them in the ranks of higher-performing states. Moreover, since demographic conditions do not change very much from year to year, improvements are likely to be caused by changes in program operations rather than by underlying conditions.
While states participating in the consultation were generally receptive to the notion of basing some bonuses on improvement, some expressed concern about standards that have incremental increases each year. A few states that began their welfare reform efforts early on, under waiver policies, felt that they were approaching the maximum realistic levels of work participation and should not be penalized if they did not continue to improve.
States also expressed a great deal of concern at the consultation about rigid thresholds for penalties that create "cliffs" in which a small difference in outcomes could result in the imposition of large penalties. This is a particular concern where the data are believed to be "noisy" and error-prone. The current TANF regulations illustrate one way in which such threshold effects can be minimized - the amount of the penalty assessed for failure to achieve the minimum participation rate requirements is proportional to the degree of the failure. However, there is still a cliff under the statute because a state's failure to meet the participation rate, by whatever margin, results in its "maintenance of effort" (MOE) funding requirement increasing from 75 percent to 80 percent.