Alternative Outcome Measures:
TANF Block Grant

Chapter IV:
Conclusions

[ Main Page of Report ]

Contents

1. Performance measures are only one element of a comprehensive system of program monitoring, research, and evaluation.
2. Before specific measures can be identified, it is critical to reach agreement among all major stakeholders on appropriate program goals.
3. There is no perfect outcome-based performance measurement system. Therefore, it would be appropriate to build gradually upon an existing system.
4. Using outcome measures does not mean abandoning useful process measures, especially because of problems in attributing outcomes to program interventions.

The Department of Health and Human Services is fully committed to the concept of outcome-based performance measures. Measuring accomplishments rather than activities or processes is an important step toward ensuring that both federal agencies and our state and local partners maintain focus on the goals of the programs and are accountable for producing results. An emphasis on outcomes is particularly appropriate for a decentralized program such as TANF, where the federal government does not prescribe detailed program parameters.

This commitment is reflected in the Department's overall strategic planning process that links all activities to the goals established under the GPRA plan. In addition, states have been given outcome-based incentives under TANF through the High Performance Bonus and the Bonus for Reductions in Out-of-Wedlock Births. Although these bonus systems provide only financial incentives for positive outcomes without penalizing negative outcomes, and the GPRA plan has neither bonuses nor penalties attached to it, these systems are, nonetheless, important components of outcome-based performance measurement and lay the groundwork upon which future enhancements will be based.

The question is no longer whether outcome-based performance measurement is a valuable tool, but rather how to incorporate outcome-based measures into an overall accountability system. The following observations are offered to help guide this development process. We hope that these observations will prove useful during upcoming discussions on evaluating states' success under TANF as the program is considered for reauthorization.

1.    Performance measures are only one element of a comprehensive system of program monitoring, research and evaluation.

Outcome-based performance measures are part of a broader system of program information. As such, they should not be viewed as the way to answer all of our questions about the effects of a program. Neither should they be expected to substitute for other kinds of studies that provide different kinds of information.

For example, policymakers and researchers are greatly interested in whether individuals leaving welfare are succeeding in retaining employment. This question can best be answered with long-term data, covering periods of a year or more. However, such a time frame may be inappropriate for a performance measurement system, which must provide timely feedback to program operators. Similarly, cost constraints may prohibit the inclusion of performance measures based on data that can be collected only through detailed surveys. Therefore, it is usually necessary to supplement the data reported through a performance measurement system with more intensive studies to address questions that are outside the range of such a system. However, such surveys are generally conducted only intermittently and in limited geographic areas. Performance measurement, on the other hand, provides ongoing information about programs which cannot be obtained from such surveys.

In the welfare-to-work arena, this report has shown that it is difficult to identify performance measures that are reliable measures of the "value-added" benefits of a program - that is, measures that reflect outcomes beyond those that would have occurred without the program in place. This difficulty is compounded for a program such as TANF, where state policies and strategies vary widely, as do state capacities, economic conditions and caseload characteristics. To truly understand program effectiveness, it is important to supplement performance measures with other evaluation activities. Together these efforts will provide a more thorough and accurate assessment of program performance.

Used properly, performance measurement systems and rigorous evaluations complement each other. Performance measures can be a valuable tool for generating hypotheses about the relationships between interventions and outcomes. These hypotheses can then be tested through in-depth evaluations of program effectiveness in a limited number of research sites. Similarly, the findings from research and evaluation can be used to refine performance measurement systems and improve their value for monitoring and motivating performance.

2.    Before specific measures can be identified, it is critical to reach agreement among all major stakeholders on appropriate program goals.

Several studies on the development of outcome-based performance measures have stressed that identifying and reaching consensus on program goals is one of the first and most critical steps (Dyer, 1994; Horsch, 1996 (b); Yates, 1997). This aspect of building performance measurement systems can be both challenging and time consuming - a study of GPRA implementation found that developing long-term strategic goals and translating them into specific performance measures was one of the most difficult elements of building a system (GAO, 1997). This process generally requires stakeholders to engage in consensus-building to define broadly shared visions of what program goals are important and what strategies are required to achieve them. As part of this effort, it is also critical to ensure that there is buy-in from top leadership at the outset (Bittner, 1998; Hatry, 1999).

The authorizing legislation clearly articulates four goals for TANF, and states enjoy wide discretion in setting their own program priorities from among those goals and in choosing how to spend their block grant funds. There is a broad consensus among stakeholders that work is a central focus of TANF. This consensus is reflected by the near-universal first-year performance competition among states on the various work-related measures of the TANF High Performance Bonus. Many states have also adopted internal goals on measures of their success in placing welfare recipients in jobs and assisting them in retaining employment and increasing their wages.

Agreement among stakeholders on goals beyond work is less universal. States are not required to spend equal amounts of block grant funding on each of the four TANF goals nor even to spend funds on each of the goals. To date, states have exercised their discretion by funding activities as diverse as programs focused on cash and work-based assistance; programs that include work activities, child care, and other work-based supports; programs that support the formation and maintenance of two-parent families; programs focused on preventing out-of-wedlock pregnancies; programs that expand supports for all working poor families, whether or not they have previously received cash assistance; and various combinations of these activities.

Only when goals have been agreed upon, does it make sense to discuss specific measures and related performance standards. Even if it is not possible to achieve full consensus, it is still important to include all stakeholders in the discussion, in order to obtain a real-world perspective on what is operationally feasible, what is the potential for unintended consequences, etc.

3.    There is no perfect outcome-based performance measurement system. Therefore, it would be appropriate to build gradually upon an existing system, e.g., the TANF High Performance Bonus system, and to monitor it for fairness and to prevent unintended consequences.

This report has shown that outcome-based performance measures - particularly in the welfare-to-work arena - are susceptible to some shortcomings. Some measures may be influenced by factors other than the program, such as the state of the economy or the composition of the welfare caseload, and it may be too complicated or costly to detect the statistical impact or adjust for these factors. Other measures may elicit an unintended response by states or localities - such as creaming. Data problems make it impossible to use some desirable measures, because the data are too costly to collect, or not available at the state level.

This does not mean that it is not worthwhile to develop outcome-based performance measures. On the contrary, the experiences of the welfare and workforce development systems in developing and using these measures have shown that there are methods to work around these issues. But it is appropriate to proceed by building on existing measures and data systems, rather than starting from scratch with ambitious new measures.

Approaches that may be adopted include:

• As recommended by Bartik (1996), select measures that meet a "minimum" fairness threshold, that cannot be manipulated by the client mix or program data in ways that do not increase the "value-added" of the program.
• Use moderate or limited bonuses and penalties. Some researchers suggest keeping the level of bonuses or sanctions relatively weak unless and until a clear link with program effectiveness can be established (Barnow, 1999). Others find that attaching high stakes to accountability systems - particularly early in their development - may lead to a perception that the accountability systems are arbitrary or unreasonable and thereby undermine support for them (Bartik, 1996; Horsch, 1996(a)).
• Collect baseline data on state performance for some time period before establishing standards for a particular measure, particularly when penalties are involved. For example, a system of outcome-based performance measures could initially be based on those measures established under the TANF High Performance Bonus, which reward the top performers on a measure without establishing a minimum performance threshold. Data for new measures might be collected first for information purposes only, then attached to bonuses, and only later, attached to penalties.
• Implement a long term program of developing additional data sources that accurately - and within a reasonable cost - measures the desired outcomes most precisely. In some cases, it may be appropriate to add additional elements to the TANF data reporting system. In other cases, it may be desirable to add questions to large-scale national surveys.

4.    Using outcome measures does not mean abandoning useful process measures, especially because of problems in attributing outcomes to program interventions.

It is not necessary to make an either/or choice between outcome-based performance measures and process measures, such as the work participation rate. Along with outcome measures, participation measures are useful for several reasons:

• To maintain a level of federal direction over the types and hours of activities in which welfare recipients participate. The participation rate under TANF places a strong emphasis on actual work, whether subsidized or unsubsidized, and allows education, training, and job search to count only under strictly limited circumstances. At the July 1999 consultation, many state representatives and researchers spoke out in favor of continuing a participation rate, but expanding the range of allowable activities.
• To ensure that a high percentage of recipients is receiving services. This is particularly critical now that welfare has become a time-limited program; if the mandatory work participation rates are not enforced, there is a danger that people will reach their time limits and lose cash assistance without ever having received services or the experience in work to help them move to self-sufficiency. Together, the participation rate target and the definition of which individuals are counted (e.g., whether recipients who are exempt from participation should be excluded from the calculation) determine what fraction of all recipients will be expected to be engaged in work-related activities.⁽⁷⁾
• To give states some credit for engaging recipients in work-related activities even if, due to state economic conditions or the characteristics of the recipient caseload, they do not achieve the desired levels of outcomes (e.g., unsubsidized employment). It may take several years of services before the "hardest-to-serve" recipients can achieve unsubsidized employment and participation rate requirements can reward states for helping such recipients to achieve intermediate steps, such as participation in substance abuse treatment or community service employment.

Regardless of motivation, outcome-based performance measures and work participation rates are not mutually exclusive.

As our experience with the High Performance Bonus system has shown, implementing an outcome-based performance measurement system under TANF would not be easy, neither for those who would set the standards nor for those whose performance would be measured against them. As these conclusions indicate, it is important to proceed judiciously, with an understanding of the real-world limits of such a system as well as of its strengths. But, as Secretary Shalala said, speaking at the University of Michigan in 1997:

[B]y focusing on outcomes we do more than fulfill our moral obligation... we force ourselves to use scarce resources wisely; to develop objective standards that we can use to demand accountability; and to put ourselves in a position to achieve even better results in the future.⁽⁸⁾

Endnotes

7.  Note that the target participation rate will always be lower than the fraction of the caseload that is mandated to participate, because some recipients will inevitably be between activities, or noncompliant. [return to text]

8.  Remarks by: Donna E. Shalala, Secretary of Health and Human Services, Fedele and Iris Fauri Memorial Lecture, University of Michigan, Ann Arbor, Michigan, September 18, 1997. [return to text]

Where to?

Top of page
Contents

Main page of report
Contents of report

Home Pages:
Human Services Policy (HSP)
Assistant Secretary for Planning and Evaluation (ASPE)
Administration for Children and Families (ACF)
U.S. Department of Health and Human Services (HHS)

Updated: 02/06/01