Report on Alternative Outcome Measures: Temporary Assistance for Needy Families (TANF) Block Grant . Defining Performance Measurement

12/01/2000

An outcome or results-oriented system for measuring program success represents a shift from traditional approaches to accountability, which typically involve tracking inputs and processes. While laws like PRWORA and WIA require the use of outcome-based performance measures, many other human service programs are also increasingly using this type of accountability system. This change in emphasis stems in part from the Government Performance and Results Act (GPRA) enacted by Congress in 1993. Seeking to promote improved government performance and greater confidence in public programs, GPRA established a government wide requirement for agencies to identify agency and program goals and to report on their results in achieving those goals. The increasing use of performance measures in all types of human service programs has prompted a number of researchers to examine the goals and the defining elements of measures of program performance  commonly known as performance measurement systems.

The Urban Institute (1980) defines performance measurement systems as the regular collection and reporting of program information in three areas  their efficiency, quality, and effectiveness (Urban Institute, 1980). According to Martin and Kettner (1996), measuring the efficiency of a welfare-to-work program, for instance, involves assessing the amount of service provided and the number of clients completing the program and comparing these measures against the costs involved. Measuring quality involves the assessment of the nature of services provided and tries to maximize the quality of services provided in relation to program inputs. Measures of effectiveness focus on outcomes  also referred to as results or accomplishments  of programs, such as the number of individuals who find jobs through an employment program. As described more extensively below, in both the welfare and workforce development systems, an emphasis has been placed on measuring the effectiveness of programs rather than their efficiency or quality. This is usually what is meant by outcome-based performance measurement.

Studies have also explored how performance measurement systems can be used to fulfill a variety of purposes (Bartik, 1996; Behn, 1991; Brown and Corbett, 1997; Hatry, 1999; U.S. Department of Health and Human Services, 1994). Several goals of performance measurement systems have been recognized, each designed to make programs publicly accountable for their operations:

  • To assure basic fiscal integrity in the expenditure of public funds and their use for specified, authorized activities. This includes ensuring funds are spent on appropriate individuals or services.
  • To provide for sound management of program services by supplying information on program operations and outcomes. Performance measurement systems can provide information to program managers regarding whether the program is achieving critical program objectives beyond fiscal accountability  such as who is served in the program, what types of services are provided, and the results of the programs efforts. This information can be used to inform management and policy decisions.
  • To monitor program outcomes. Performance measures provide information that allow higher levels of government or program managers to track how well a program is doing in achieving specific goals or objectives.
  • To motivate better performance by managers and line workers associated with the program. Some performance measures are designed to motivate staff to achieve particular goals that may be important to the program.

These goals are not mutually exclusive, but different goals may require different types of performance measures (Bartik, 1996). For example, to identify how to improve the effectiveness of programs, the measures selected have to be an accurate gauge of the programs effectiveness and may be required to be linked to information on operational strategies so it is known why particular approaches are effective. In contrast, to motivate local offices and staff, measures must be timely and understandable and linked to the allocation of resources. This indicates a need for a variety of different performance measures  particularly if the system has multiple goals.

Over the years, a range of terms has been used to describe the different types of performance measures used to gauge program success  and these terms are often used interchangeably, although they have varying connotations and meanings. Some studies have tried to achieve consensus on useful ways for defining performance measurement-related terms, particularly for use in welfare-to-work and other employment programs (Brown and Corbett, 1997; Hatry, 1999; Martin and Kettner, 1996; Midwest Welfare Peer Assistance Network, 1999; U.S. Department of Health and Human Services; 1994).

  • Outcome measures. Outcome performance measures focus on the results of a policy or program and are generally related to the goals the program hopes to achieve. In most cases, these measures focus on the outcomes for a group of individuals involved in the program. In welfare-to-work programs, key outcome measures include job placement rates, employment retention rates, or wage rates. Further, some types of outcome measures  such as General Educational Development (GED) certificate attainment rates  are referred to as "interim outcome measures" because they represent an important milestone even though they may not be the ultimate goal of the program.
  • Process measures. Process measures address administrative or operational activities of the program. These types of measures usually reflect the "means" to getting to an end result  rather than the goal itself. Examples of process measures include participation rates reflecting the type and level of service received through the program, the percentage of applications for assistance which are acted upon in a timely manner, and the percentage of cases in which the cash benefits are calculated accurately.
  • Indicators. Indicators are measures of behavior, status, or condition that can be tracked over time and across people. Examples of indicators include state marriage and divorce rates or poverty rates. Indicators typically track the behavior or situations of broad population groups.
  • Performance standards. These are numerical "goals" or standards established for a performance measure, such as a 70 percent employment rate or a 25 percent participation rate.

Clearly, performance measurement systems can be used to meet a variety of goals and can be measured in different ways. Given these issues, careful consideration must be given to the design of performance measurement systems that use outcome measures. There are also issues that are specific to the design of outcome-based performance measures in welfare-to-work programs these are discussed below.