This report has shown that outcome-based performance measures - particularly in the welfare-to-work arena - are susceptible to some shortcomings. Some measures may be influenced by factors other than the program, such as the state of the economy or the composition of the welfare caseload, and it may be too complicated or costly to detect the statistical impact or adjust for these factors. Other measures may elicit an unintended response by states or localities - such as creaming. Data problems make it impossible to use some desirable measures, because the data are too costly to collect, or not available at the state level.
This does not mean that it is not worthwhile to develop outcome-based performance measures. On the contrary, the experiences of the welfare and workforce development systems in developing and using these measures have shown that there are methods to work around these issues. But it is appropriate to proceed by building on existing measures and data systems, rather than starting from scratch with ambitious new measures.
Approaches that may be adopted include:
- As recommended by Bartik (1996), select measures that meet a "minimum" fairness threshold, that cannot be manipulated by the client mix or program data in ways that do not increase the "value-added" of the program.
- Use moderate or limited bonuses and penalties. Some researchers suggest keeping the level of bonuses or sanctions relatively weak unless and until a clear link with program effectiveness can be established (Barnow, 1999). Others find that attaching high stakes to accountability systems - particularly early in their development - may lead to a perception that the accountability systems are arbitrary or unreasonable and thereby undermine support for them (Bartik, 1996; Horsch, 1996(a)).
- Collect baseline data on state performance for some time period before establishing standards for a particular measure, particularly when penalties are involved. For example, a system of outcome-based performance measures could initially be based on those measures established under the TANF High Performance Bonus, which reward the top performers on a measure without establishing a minimum performance threshold. Data for new measures might be collected first for information purposes only, then attached to bonuses, and only later, attached to penalties.
- Implement a long term program of developing additional data sources that accurately - and within a reasonable cost - measures the desired outcomes most precisely. In some cases, it may be appropriate to add additional elements to the TANF data reporting system. In other cases, it may be desirable to add questions to large-scale national surveys.