Public programs focused on improving the employment levels and earnings among economically disadvantaged groups have witnessed an increasing use of outcome-based measures to determine program success. These programs use measures focusing on "results" to gauge program success and to hold public agencies accountable for achieving certain goals. The Personal Responsibility and Work Opportunity Reconciliation Act of 1996 (PRWORA) — which changed the nature of the welfare system by devolving program responsibility to the states, enacting restrictions on the amount of time individuals can receive assistance, and requiring recipients to engage in work quickly — required the U.S. Department of Health and Human Services (HHS) to reward states for the success of their cash assistance and welfare-to-work programs (known as the Temporary Assistance for Needy Families (TANF) program) based on their performance on a range of outcome-based measures.
The workforce development system also uses outcome-based performance measures to determine program success. The Workforce Investment Act of 1998 (WIA) – which consolidates and streamlines a range of employment and training services for economically disadvantaged individuals — requires the state and local workforce development agencies that operate the program to meet specific outcome-based performance measures and provides incentives to do so.(1) The WIA performance measurement system builds on the one developed under the Job Training Partnership Act (JTPA). Administered by the U.S. Department of Labor, JTPA was the principal federal job training program prior to WIA for economically disadvantaged youth and adults, dislocated workers, and others who faced significant barriers to employment.
This paper describes the experiences of programs designed to improve the employment prospects and earnings of economically disadvantaged adults — particularly welfare-to-work programs — in using outcome-based performance measures. To provide context, the paper begins with a review of the literature on the goals and defining elements of performance measurement systems. Next, the paper identifies issues that are critical to address when developing and using outcome-based performance measures in welfare-to-work programs. It also reviews studies that have described and assessed the use of outcome-based performance measures in the welfare and workforce development systems at the federal level and then turns to a discussion of similar initiatives at the state level. The paper concludes with a discussion of the lessons drawn from these experiences.
[ Go to Contents ]
An outcome or results-oriented system for measuring program success represents a shift from traditional approaches to accountability, which typically involve tracking inputs and processes. While laws like PRWORA and WIA require the use of outcome-based performance measures, many other human service programs are also increasingly using this type of accountability system. This change in emphasis stems in part from the Government Performance and Results Act (GPRA) enacted by Congress in 1993. Seeking to promote improved government performance and greater confidence in public programs, GPRA established a government wide requirement for agencies to identify agency and program goals and to report on their results in achieving those goals. The increasing use of performance measures in all types of human service programs has prompted a number of researchers to examine the goals and the defining elements of measures of program performance — commonly known as performance measurement systems.
The Urban Institute (1980) defines performance measurement systems as the regular collection and reporting of program information in three areas — their efficiency, quality, and effectiveness (Urban Institute, 1980). According to Martin and Kettner (1996), measuring the efficiency of a welfare-to-work program, for instance, involves assessing the amount of service provided and the number of clients completing the program and comparing these measures against the costs involved. Measuring quality involves the assessment of the nature of services provided and tries to maximize the quality of services provided in relation to program inputs. Measures of effectiveness focus on outcomes — also referred to as results or accomplishments — of programs, such as the number of individuals who find jobs through an employment program. As described more extensively below, in both the welfare and workforce development systems, an emphasis has been placed on measuring the effectiveness of programs rather than their efficiency or quality. This is usually what is meant by outcome-based performance measurement.
Studies have also explored how performance measurement systems can be used to fulfill a variety of purposes (Bartik, 1996; Behn, 1991; Brown and Corbett, 1997; Hatry, 1999; U.S. Department of Health and Human Services, 1994). Several goals of performance measurement systems have been recognized, each designed to make programs publicly accountable for their operations:
These goals are not mutually exclusive, but different goals may require different types of performance measures (Bartik, 1996). For example, to identify how to improve the effectiveness of programs, the measures selected have to be an accurate gauge of the program’s effectiveness and may be required to be linked to information on operational strategies so it is known why particular approaches are effective. In contrast, to motivate local offices and staff, measures must be timely and understandable and linked to the allocation of resources. This indicates a need for a variety of different performance measures — particularly if the system has multiple goals.
Over the years, a range of terms has been used to describe the different types of performance measures used to gauge program success — and these terms are often used interchangeably, although they have varying connotations and meanings. Some studies have tried to achieve consensus on useful ways for defining performance measurement-related terms, particularly for use in welfare-to-work and other employment programs (Brown and Corbett, 1997; Hatry, 1999; Martin and Kettner, 1996; Midwest Welfare Peer Assistance Network, 1999; U.S. Department of Health and Human Services; 1994).
Clearly, performance measurement systems can be used to meet a variety of goals and can be measured in different ways. Given these issues, careful consideration must be given to the design of performance measurement systems that use outcome measures. There are also issues that are specific to the design of outcome-based performance measures in welfare-to-work programs – these are discussed below.
[ Go to Contents ]
Outcome-based performance measures — particularly measures that use the employment and earnings of program participants to gauge program success – are increasingly common among welfare and workforce development programs. However, studies have highlighted a number of issues that need to be addressed when these types of measures are used. (Barnow, 1999; Bartik, 1996; U.S. Department of Health and Human Services, 1994). These issues — which are each discussed in more detail below — include an inconsistent relationship between outcomes and program effectiveness, a need to ensure that measures are fair and equitable, the possibility of unintended consequences, and the problem of multiple goals. Some of these issues stem from an absence of answers that research is able to provide at this time, while others are due to a growing body of evidence that suggests the inherent challenges of designing outcome-based performance systems for welfare-to-work programs.
One concern regarding the use of outcome-based performance measures to reflect program success is that specific measures that are commonly used — such as increasing employment rates or earnings — often do not accurately measure the "added value" of the programs (Bartik, 1996; Barnow, 1999; U.S. Department of Health and Human Services, 1994).
Research on welfare caseload dynamics has shown the natural movement of welfare recipients on and off welfare (Bane and Ellwood, 1983; Pavetti, 1993). The findings show that a large proportion of welfare recipients exit welfare after relatively short periods of time, while a substantial minority remain on welfare for longer spells. Some of those who leave do so due to employment; others leave for reasons related to marriage, remarriage, or further changes in their personal or economic situation; and still others leave for reasons that are not known. The studies also show that, in most cases, a large majority of those who leave welfare do so on their own, without either the benefit of an employment program or the requirement to participate in one. This movement off welfare and into employment represents what might be called a baseline or "natural" outcome unrelated to the operations of a welfare-to-work program.
The role of an employment program is not necessarily to achieve high outcome rates — but to add to the outcomes that would normally occur. To be judged successful, a program must exceed — or "add value to" the natural outcome rates. A program could do so in a number of ways. It could either move people more quickly into jobs and off welfare than they otherwise would have (through job search activities). Or it could assist getting people jobs who would not otherwise have gotten them, such as by providing job training or specialized services for clients facing difficult barriers (i.e., domestic violence, substance abuse). The strict enforcement of participation requirements may also cause some individuals to leave cash assistance rather than participate.
Evaluations of welfare-to-work programs have found that there is not a strong correlation between the "value-added" by the employment program and the attainment of high outcomes on employment-related measures. The 1994 Report to Congress by the U.S. Department of Health and Human Services examined random assignment studies of the welfare-to-work programs that operated in the 1980s. These evaluations were specifically designed to measure the "added value" — or impact — of programs targeted at welfare recipients in increasing the earnings and reducing the welfare dependency of those referred to an employment program (the program group) compared to an identical group of individuals (the control group) who did not receive program services. This review found that those programs that performed well on specific outcomes measures related to moving people into jobs and off welfare did not necessarily have greater success — in terms of program impacts — than those who did not perform well on the measures.
More recent data from the National Evaluation of Welfare-to-Work Strategies (NEWWS) (formerly the JOBS evaluation) confirms this finding (Freedman et al.,2000). This study included random assignment studies of welfare-to-work programs in seven sites; for simplicity, results from five sites are discussed here.(2) As Table 1 shows, even though Columbus had the highest employment rate (50.2 percent), the "added value" by the program is lower than the other sites in the evaluation (3.5 percent). Moreover, the outcomes in Portland, which had a substantially higher "added value" were similar to several of the other sites. Thus, in this case, outcomes do not serve as a good "proxy" for added value, and an assessment of the relative effectiveness of the programs based solely on outcomes would have been mistaken.
| County |
Longitudinal Participation Rate |
Percent Employed After Two Years |
"Added Value" (Difference) |
|
|---|---|---|---|---|
| Program Group | Control Group | |||
| Atlanta | 73.8% | 42.8% | 38.5% | +4.4% |
| Columbus | 52.1% | 50.2% | 46.7% | +3.5% |
| Grand Rapids | 69.0% | 47.2% | 43.1% | +4.1% |
| Portland | 61.1% | 46.2% | 35.3% | +10.9% |
| Riverside | 43.8% | 31.3% | 27.1% | +4.2% |
This table also shows that participation rates — a process measure — also are not good proxies for program impacts. The participation rates shown on Table 1 are longitudinal measures — which report the proportion of individuals who participated in the program at least one day within a two-year period — and are calculated differently than the monthly participation rates required under JOBS and TANF(3) (which report the number of individuals who participate a certain number of hours per week each month). These results show that the program with the highest participation rate, Atlanta (73.8 percent), had a very similar added value to the site with the lowest participation rate (43.8 percent).
Table 2 shows the outcomes and "added value" for a different measure — earnings over a two-year period — with similar results. This table shows that sometimes outcomes can be correlated with the level of added value. The Portland program group achieved both a high level of earnings ($7,133) and the highest level of added value ($1,842). However, the relationship is not consistent. Columbus had similar earnings to Portland ($7,569) but its added value was dramatically lower ($677). In addition, the site with the lowest earnings — Riverside ($5,488) — had the second highest added value ($1,276). Similar to the findings with employment rates, longitudinal participation rates are not correlated with the added value of programs on earnings measures.
| County |
Longitudinal Participation Rate |
Average Total Earnings Over Two Years |
"Added Value" (Difference) |
|
|---|---|---|---|---|
| Program Group | Control Group | |||
| Atlanta | 73.8% | $5,820 | $5,006 | $813 |
| Columbus | 52.1% | $7,569 | $6,892 | $677 |
| Grand Rapids | 69.0% | $5,674 | $4,639 | $1,035 |
| Portland | 61.1% | $7,133 | $5,291 | $1,842 |
| Riverside | 43.8% | $5,488 | $4,213 | $1,276 |
Research on workforce development programs has found similar results. Barnow (1999) found a weak correspondence between program impacts and measured performance in the JTPA program. In examining the 16 sites in the National JTPA Evaluation (evaluated using a random assignment design), this study found that the relationship between program performance on employment-related measures and program impact was positive but statistically insignificant.
This evidence suggests that a system of performance measurement that focuses on outcomes may not necessarily lead programs to increase their added value. Rather, it could reward the substantial amount of normal employment activity by welfare recipients rather than the program’s added value: programs with higher (or lower) outcome rates overall may simply reflect the higher (or lower) natural outcomes. However, controlled evaluations, which are the best way to measure program impacts are generally too expensive and time-consuming to rely on for ongoing feedback and monitoring of programs.
As discussed above, there is a natural rate at which welfare recipients find jobs with no assistance from employment programs. Studies have found that this natural rate is due to the influence of several factors over which state and local managers have little control, including the state’s economic conditions and the demographics of the welfare caseload (Barnow, 1999; Bartik, 1996; U.S. Department of Health and Human Services, 1994). An important dimension of performance measurement systems is holding states accountable for performance that is within their control — not for factors for which they can be expected to have little or no responsibility.
Different state and local welfare-to-work programs operate within significantly different labor markets and under economic conditions that are diverse and highly variable. This may have a significant effect on the outcomes produced by the state or local program. For example, because there are fewer jobs available, a program operating in a depressed economy may place fewer recipients in jobs than one functioning in a booming labor market. In this case, the economy may be a key factor in explaining the difference between the states’ results, not the effectiveness of the employment program.
States and localities also have different and changing welfare caseloads in terms of overall size, demographics, and other local factors. For example, some states have a relatively high proportion of very disadvantaged recipients (e.g., those lacking educational credentials or employment histories) on their cash assistance caseload. Because it is more difficult to employ this group, a state in this situation could appear less successful, based on a job-related outcome measure, than a state that served a more job-ready population. In this case, a state’s performance would, at least in part, be driven by the composition of the cash assistance caseload.
Other important factors that affect the outcomes of a state’s employment program are the cash assistance benefit levels and income disregard policies. Cash assistance benefit levels for a family of three with no income range from $120 per month in Mississippi to $923 per month in Alaska (Gallagher et al. 1997). Earnings disregards — the amount of income disregarded when calculating the benefit level — also vary from small, decreasing disregards in some states to others that disregard all income as long as the family is below the poverty line. As a result, some states will perform better on certain outcome measures — such as the number of individuals who leave cash assistance due to employment — because of their grant level and earnings disregard policies rather than because of their program’s performance. While states could change their benefit levels and earnings disregards so that they fared better on certain types of performance measures, that is not the goal of a performance measurement system. (See below for a discussion of unintended consequences). Instead, the goal is to develop measures that treat states equitably regardless of their benefit and earnings disregard levels.
These findings suggest that it is important to recognize the role that uncontrollable factors can have on performance measures and to develop mechanisms that ascribe differences in outcomes to the right factors. This process of ensuring that standards are fair and equitable across states is known as "leveling the playing field." Later sections of this paper discuss some of the mechanisms that have been developed to address this issue, including regression-adjusting measures to reflect differences in economic conditions and welfare caseloads, standards of performance that are negotiated to reflect local conditions, and using measures of improvement rather than absolute levels of performance.
Another issue that can be encountered in developing outcome-based performance measures is the creation of unintended consequences (Barnow, 1999; Bartik, 1996; U.S. Department of Health and Human Services, 1994). This means that an unintended behavior is created when trying to achieve a certain result. The most prevalent example of this in the world of employment and training programs is known as "creaming." When only the outcomes for clients enrolled in the program are considered, programs can enhance their performance on employment and earnings measures by serving those clients who are most "job ready" and who, with minimal program assistance, are most likely to become employed on their own. Programs may also have a disincentive to focus on the hard-to-serve clientele — who may actually be more in need of the services provided by the program — because it would affect their ability to achieve a certain level on performance measures.
There may be additional unintended or adverse consequences as well. For example, a focus on caseload reduction can lead to incentives to divert recipients from receiving assistance or to lower grant levels or earnings disregards so individuals leave assistance more quickly when they find jobs.
A final issue to address in developing an outcome-based system of performance measurement grows out of the relatively broad purposes of welfare-to-work programs. While the overall goal is to move individuals into employment and/or off cash assistance, there are a number of objectives that could be pursued to achieve this goal. For example, some programs may emphasize finding "better jobs" or jobs with benefits or higher wages while others may emphasize moving individuals quickly into jobs regardless of their wage level. Moreover, some programs may use their TANF programs to reduce poverty, for example by providing income support for needy families as long as they remain poor — even if this means the families receive assistance for a longer period. Others may view self-sufficiency and reduced dependency as the primary goal and reduce the level of support provided in order to make work a necessity.
Studies of welfare programs in the 1980's found that, depending on the objectives administrators identify and the service and management strategies they adopt, programs move in substantially different directions with different results. For example, one study found that a range of programs was successful depending on the specific measures used — some achieved high employment rates, others had larger earnings gains, and still others were more cost-effective (U.S. Department of Health and Human Services, 1994). However, achievement of one outcome did not necessarily correlate with achievement on the other outcomes. Thus, it is important to take care in identifying and promoting one particular program objective over another.
[ Go to Contents ]
In spite of the potential challenges in using outcome-based performance measures in welfare-to-work programs, at the federal level, both welfare and workforce development programs have increasingly focused on the use of these measures to gauge program success and effectiveness. This section discusses the evolution of each of these outcome-based performance based systems — and, to the extent possible, discusses how they have addressed the specific challenges discussed above. In addition, the paper examines similarities as well as differences between these two systems.
Welfare-to-work programs administered by the U.S. Department of Health and Human Services are increasingly shifting toward using outcome-based performance measures to gauge program success. The welfare-to-work program that preceded the TANF program — the Job Opportunities and Basic Skills Training (JOBS) program — did not explicitly require the federal government to establish outcome-based performance measures. Rather, this program primarily relied on two process measures — participation rates and the proportion of funds spent on long-term welfare recipients — as its primary measures of program accomplishments. Some studies found that the JOBS program was not sufficiently focused on employment, in part due to the nature of its performance measurement system (U.S. General Accounting Office, 1994, 1995).
The legislation governing the JOBS program did require the U.S. Department of Health and Human Services to develop recommendations for outcome-based measures. In its 1994 Report to Congress (U.S. Department of Health and Human Services, 1994), HHS developed a timeframe for developing these measures – with measures to be put in place in 1996. However, the passage of PRWORA in 1996 superseded these plans. Possible performance measures mentioned in the 1994 Report to Congress included: percent of the cash assistance caseload that received aid for more than a specified period, the JTPA performance measures (see below), increases in employment and earnings of program participants after leaving the JOBS program, and retention of JOBS participants in unsubsidized employment.
The statute governing the TANF program contains more explicit guidance concerning the development of outcome-based performance measures. Like the JOBS statute, PRWORA also detailed participation rates states are required to meet. However, it also required HHS to develop a "high performance bonus" to reward states based on their success in attaining the goals of the act and to distribute a bonus to reward states based on their success in reducing out-of-wedlock births. For the high performance bonus, the law gave HHS — working with the states — discretion over what measures should be used. Congress was much more specific regarding the performance measure for out-of-wedlock births.
For the initial three years of the high performance bonus, HHS developed interim guidance that included outcome-based performance measures that reflect states’ performance in moving individuals from welfare to work (U.S. Department of Health and Human Services, 1998, 1999, and 2000). The guidance included four key work measures for the high performance bonus: (1) the job entry rate; (2) the success in the workforce rate (includes measures of both job retention and earnings gains); (3) the increase in job entry rate; and (4) the increase in the success in the work force rate. States use quarterly Unemployment Insurance (UI) records and other administrative data to calculate these measures. Bonuses are awarded to the ten states with the best performance on each measure.
In the final rule for bonuses to be awarded in FYs 2002 and 2003, HHS retained the work measures (but changed the data source to the Federal Parent Locator Service/National Directory of New Hires) and added a measure on family formation and stability (using Census Bureau data), and three measures of states’ success in supporting work and self-sufficiency by providing low-income working families with health insurance (using data submitted by the states), food stamps (using Census Bureau data), and child care assistance (using Census Bureau and data submitted by the states). Awards totaling $200 million per year will be awarded for bonus years 1999-2003.
The TANF program also includes a bonus to decrease out-of-wedlock births. For this bonus, the five states with the largest decrease in the ratio of out-of-wedlock births to total births (which also have a reduction in their abortion rates) will receive a bonus.(4) A total of $100 million per year is available for this bonus.
While it is too early to assess the effects of the performance measurement system for the TANF program, it is important to note that the welfare system includes a number of mechanisms to deal with the potential issues in using outcome-based measures discussed above. By including measures based on program improvement, the TANF program adjusts somewhat — although imperfectly – for the lack of level playing field. Using a program improvement measure allows states that may be facing difficult economic conditions or serving a difficult caseload that would not otherwise receive a bonus to obtain an award. The system also uses a range of measures — including job placement, job retention, and earnings progression — to gauge success. This provides some opportunity for programs with different or multiple goals to compete for a bonus. In addition, the work measures for the high performance bonus include both working adults who leave TANF as well as those who remain on TANF. This dual focus reduces the impact of state program design and payment standards on state performance.
Finally, because the measures are based on the performance of all cash assistance recipients, the likelihood of "creaming," i.e., serving only the most employable welfare recipients, is reduced. The TANF participation rates, which also require participation by a broad segment of the welfare population, also serve to counterbalance the potential creaming effect of the measures.
The WIA (and its predecessor JTPA) program, administered by the U.S. Department of Labor (DOL), provides a range of employment-related services to different types of disadvantaged individuals including adults, welfare recipients, and youth. The JTPA program, in particular, has had extensive experience using outcome-based performance measures to gauge the success of its employment and training programs. Because of this longer experience, more studies have been conducted on both the experiences and effects of the JTPA performance measurement system than on performance measurement within welfare programs. This section describes the JTPA and WIA performance measurement systems, as well as findings on the effectiveness of these systems.
Several studies have examined the experience of the JTPA program in developing and using outcome-based performance measures (Bartik, 1994; Barnow, 1999; Dickinson and West, 1988; Zornitsky and Rubin; 1988). When JTPA was enacted in 1982, the legislation included specific requirements for outcome-based performance standards. As described by Barnow (1999), the JTPA system had two primary goals: to monitor how well the state and local levels of government were performing in achieving the goals and objectives of the law and to improve performance by giving program operators incentives to achieve these goals and objectives.
Under JTPA, the U.S. Department of Labor was responsible for determining the performance measures for the local Service Delivery Areas (SDAs) — the entities that operated the program at the local level. The primary role of states was to decide how bonus money should be distributed among the SDAs and how any performance-based sanctions should be imposed. In addition to the federally-set performance measures, states could propose supplementary performance measures to be used for allocating bonuses.
The JTPA performance measurement system had four core measures for adults and relied on survey data collected from program participants to calculate these measures. (Administrative data were used to compute performance on two youth measures.) SDAs were expected to meet or exceed performance standards – specific thresholds were set at the federal level. Based on the most recent program experiences for all SDAs, DOL set the standards for the core performance measures at levels where 75 percent of the SDAs would be expected to exceed these minimum performance expectations. The measures for adults are listed below; the national standards for 1996/97 are noted in parentheses (Barnow, 1999):
The JTPA performance measurement system included both monetary rewards and programmatic sanctions for SDAs that exceeded or failed to meet the performance standards. States were given control over which SDAs received positive and negative incentives. Up to five percent of JTPA funds were set aside to be used by states to reward SDAs who exceeded the performance standards. SDAs that exceeded the standards could receive additional funding, and activities undertaken with those funds could be exempted from performance standards. Thus, good performance provided SDAs with more flexibility to try new approaches or to serve more at-risk groups (Barnow, 1999). On the negative side, programs that failed to meet the standards set for them for in two consecutive years were subject to reorganization by the Governor (meaning the program could be restructured or restaffed).
In its early years of implementation, the JTPA performance measurement system was criticized because it promoted creaming and other unintended consequences (Barnow, 1992; Bartik, 1994; Zornitksy and Rubin, 1988). Unlike TANF, each SDA had some control over who was enrolled in program services and participation in the program was voluntary. This resulted in a stronger potential for creaming. Studies found that while the creaming tendencies were not universal across all SDAs, the standards did result in a focus on the less disadvantaged in some localities. Dickinson and West (1988) found that the JTPA performance standards did not prevent SDAs that had a strong commitment to serving hard-to-serve groups from targeting and serving those groups. In addition, Heckman et al. (1996) found that JTPA case workers accepted the least employable applicants into the program in spite of their effect on performance standards — in part due to the fact that they preferred to assist the most disadvantaged clients. However, Dickinson and West (1988) also found that those SDAs that had the strongest focus on meeting the standards were also less likely to serve disadvantaged groups.
In addition to enrolling the most advantaged among those eligible for program services, the JTPA system was also thought to be encouraging SDAs to offer low-cost services (an early JTPA performance measure included measures of program costs per program terminee), such as job search assistance, rather than more intensive services such as long-term training. Concerns were raised because more intensive and longer-term training was believed to have a greater impact on earnings in the long run. (Barnow, 1999)
In response to these issues, the 1992 amendments to JTPA required states to adjust their performance standards to reflect differences in economic conditions and in the demographic characteristics of the program participants in each SDA (this had previously been at the discretion of the state). States were allowed to use DOL adjustment factors or an alternative procedure approved by DOL.(5) In addition to providing a mechanism to level the playing field, it was intended that these adjustments would give states incentives to serve more disadvantaged individuals (Barnow, 1999). The amendments also prohibited any performance measures based on costs.
To some extent these efforts appear to have mitigated creaming in the JTPA program. While local SDAs did not always understand how the adjustment model would affect their performance on the measures (Barnow, 1992; Zornitsky and Rubin, 1988), Dickinson and West (1988) found — in a study done before the adjustment model was mandatory — that the SDAs that did use the adjustment model significantly increased the percentage of disadvantaged groups served.
Overall, the effect of the outcome-based standards on program performance in JTPA is mixed. As noted above, Barnow (1999) did not find a strong link between program effectiveness and performance on the JTPA standards. In addition, there appears to be considerable variation in the extent to which the performance standards influenced the local SDAs. Dickinson and West (1988) found, for example, that about 42 percent of the SDAs they studied tried to maximize their measured performance, one-fourth tried only to exceed their standards slightly, and about one-third tried merely to meet their standards in order to avoid program sanctions.
The variation in how SDAs responded to the performance standards may in part be due to the decentralized nature of the JTPA performance system — where SDAs faced differing financial incentives. Because some states rewarded only the best performers while others distributed funds broadly among all SDAs that met or exceeded standards (Barnow, 1999), SDAs faced differing levels of financial incentives. In addition, because reorganization is such an extreme measure, states often used their discretionary authority to modify the standards so that poor performers would not fail in two consecutive years (Barnow, 1999). The fact that few SDAs actually faced the most severe penalties could have also influenced their response to the measures.
Building on the system developed under its predecessor, the WIA statute also places a strong emphasis on outcome-based performance standards. WIA requires that a comprehensive performance accountability system be developed with the following components: a focus on results defined by "core indicators" of performance; measures of "customer satisfaction" with programs and services; a strong emphasis on the continuous improvement of services; annual performance levels and improvement plans developed during negotiations with federal, state, and local partners; and awards and sanctions based on state and local performance.
The WIA performance system continues some aspects of the JTPA system — but with some critical differences (DOL, 1999(a); DOL, 1999(b); DOL, 2000). Table 3 provides a comparison of the performance measures used under the two systems. The performance measures for adults include: entry into unsubsidized employment; retention in unsubsidized employment six months after entry into employment; earnings gains in unsubsidized employment six months after job entry; and attainment of a recognized credential in relation to the achievement of educational or occupational skills, by those who enter into unsubsidized employment. Customer satisfaction will be measured based on the responses of both program participants and employers. States may also develop additional indicators of performance. Unlike JTPA which relied on a survey of program participants, because of the reduced cost associated with data collection, states are required to use quarterly wage records (UI data) to compute performance on employment-related measures.
| JTPA Performance Measures | WIA Performance Measures |
|---|---|
|
|
WIA requires that the expected levels of performance on each core indicator be negotiated between the Department of Labor and individual states — an approach that is different from the SDA-level standards used in JTPA. The agreed-upon level of performance for each state must reflect how it compares with other states (taking into account differences in economic conditions, participant characteristics, and the proposed service mix and strategies). Each local workforce investment area can negotiate with the state and reach agreement on the local level of performance expected on each core indicator, taking similar factors into account.
Like JTPA, WIA has an incentive system with both rewards and sanctions. Rather than providing incentive grants to states, if a state fails to meet the adjusted levels of performance in two consecutive years, the state allocation can be reduced by up to five percent. The Department of Labor is required to award an incentive grant to each state that exceeds its performance levels for WIA (as well as those required for the Vocational and Applied Technology Education Act (Perkins Act)). States must set aside part of their allocation to provide incentive grants (or bonuses) to localities, at the discretion of the Governor. Localities that fail to meet the core indicators of performance for two consecutive years may be required to reorganize.
Because the population that can be served under WIA is broader than that served under JTPA, WIA does not require states to use a statistical model to adjust their performance standards. Instead, WIA levels the playing field by providing for negotiated performance standards at both the state and local level. Among the factors which must be considered in the negotiations process are how the levels compare to other state or local programs, taking into account such factors as economic and demographic characteristics and service design. (Other factors include promoting continuous improvement in performance measures and attaining a high level of customer satisfaction). Statistical analyses may be taken into account as part of these negotiations. In addition, by including a relatively broad range of measures to gauge program performance – including customer satisfaction and credential attainment — and allowing states to add additional measures, the WIA system has some accommodation for programs with different goals.
Overall, the welfare and workforce development performance measurement systems have some common elements. The most striking similarity is the type of core measures used — both systems use measures based on the job placement rate, earnings progression, and job retention (at least for a preliminary period under TANF). Both systems also rely primarily on a bonus system for rewarding states on the selected outcome-based performance measures.
However, there are several key distinctions between the two systems. First, TANF combines the bonuses for achievement of outcome-based performance measures with penalties based on process measures, such as the work participation rate requirements. WIA links financial penalties as well as bonuses to the performance measures. Second, the new WIA system relies on standards that are negotiated at the federal and state levels. In addition, the WIA/JTPA system also allows adjustments to performance expectations based on economic conditions and demographic characteristics. In contrast, the welfare system relies on overall rankings of states and measures of improvement. Finally, compared to TANF, the workforce development system is more uniformly decentralized. Under WIA, awards and sanctions apply to both states and localities and the state plays a major role in how funds are distributed within the state. Under TANF, states have the discretion to decide if funds should be distributed at all to local agencies.
[ Go to Contents ]
As discussed in the previous section, the federal government has played a strong role in the development of outcome-based performance standards for both the welfare and workforce development systems. In addition to these federal efforts, however, states have also become increasingly involved in developing their own performance measurement systems. This section discusses state efforts to build these systems — which range from very broad efforts that cover a range of human service programs to more narrow efforts focused on welfare or workforce development programs.
State initiatives to develop their own performance measurement systems are in part due to changes at the federal level that transfer additional responsibilities to states and local units of government. This devolution of responsibility poses new challenges for public administrators, who not only have to manage programs but also maintain and expand existing information systems needed to monitor program performance.
The design and use of performance measurement systems vary widely by state. One study (Horsch, 1996(a)) developed several useful dimensions for describing state efforts to build these systems.
There are clearly a number of different approaches at the state level for developing outcome-based performance measures. At one end are comprehensive, cross-sector systems which focus on establishing indicators or benchmarks of progress toward certain goals. Rather than measuring outcomes associated with specific programs (such as welfare-to-work or WIA programs), these states use measurable social goals — such as reducing poverty or lowering the teen birth rate — to monitor the overall effectiveness of public programs, to set goals, and/or to coordinate efforts across agencies (Brown et al, 1997; Yates, 1997).
Oregon developed the first and most comprehensive effort (Dyer, 1996; Lewis and Dunkle, 1996; Popovich, 1996) — although Florida, Minnesota, and Vermont have embarked on similar efforts.(6) As summarized in Chart 1, Oregon has put substantial energy into reorganizing government activities at all levels to take a more goals-oriented approach. This was done through a multi-year collaborative process involving stakeholders at all levels including state agencies, local community leaders, the business community, and citizens. While public agencies were not consolidated as part of this effort, individual agencies use the indicators to guide policy and contracting decisions.
While some states have developed these relatively broad performance measurement systems, others focused more specifically on welfare and/or workforce development programs. For example, in Ohio, the state is giving counties block grants to administer the TANF programs and using some of the same performance measures as those used at the federal level on TANF to gauge success (Yates, 1997). Counties that exceed performance standards in the areas of a work participation rate and out-of-wedlock births receive greater funding flexibility and earn financial incentives.
Wisconsin established performance standards for its private contractors that administer the state’s TANF program. Contractors are required to meet a base level of performance to remain in compliance with the contract requirements and receive bonuses for exceeding the standards. Standards are established for: employment rates, wage rates, job retention, the provision of appropriate basic educational services (to certain types of individuals), the availability of employer-provided health insurance, and basic skills/job skills attainment (an optional measure — for bonuses only).
A study of state welfare-to-work performance systems completed before the 1996 welfare law found that over half had statewide performance standards (GAO, 1995). Most of these states used employment rates as the key performance measure, although some also used other measures such as wage rates, job retention, and education and training achievement. Another study found state variation in whether uniform performance standards for welfare-to-work were set for local offices or negotiated with the state, whether they were adjusted using JTPA standards, and whether rewards and sanctions were significant or relatively informal (APWA, 1994).
Other states are developing performance measures and standards that focus more specifically on workforce development programs. Almost half the states are using outcome-based performance measures and standards to assess and monitor the performance measures that cut across the entire workforce development system — including JTPA/WIA, the Employment Service (ES), vocational education, and others (Hyland, 1997; Simon, 1998). Some of these efforts include TANF programs as well. As an example, the California legislature enacted a statewide system to evaluate the performance of all its publicly-funded workforce development programs including JTPA (now WIA), TANF, the Employment Service, vocational education, community colleges, and rehabilitation programs (State of California, 1999) (see Chart 1 for details).
Some states sought to effect change to their workforce development system through a common policy framework for designated programs established by an interprogram or interagency team (North Carolina and Illinois). Others achieved this goal by reorganizing their programs into a single administrative entity (Texas and Iowa).
Overall, many states are moving forward on developing outcome-based performance measures. A few states have developed relatively comprehensive systems that track social indicators rather than the outcomes of specific programs. However, others are more narrowly focused on welfare or workforce development programs. These efforts generally complement the development of outcome-based measures occurring at the federal level.
|
Oregon Oregon’s results-based accountability system for children and families is part of a comprehensive statewide system. With the goal of creating a more results-oriented, decentralized governance system, Oregon first developed a strategic vision (known as Oregon Shines) with a task force of over 150 individuals identifying broad goals for the state. The state then established – through a broad participatory process — a total of 92 specific benchmarks to measure whether the state was making progress in achieving these goals (known as Oregon Benchmarks). Benchmarks were established in key policy areas related to the strategic vision — including education and workforce development, reducing welfare dependency and increasing self-sufficiency, and protecting the health of children. Indicators for measuring these benchmarks were then developed and goals set for each indicator. Specific goals include reducing the welfare caseload from 40,000 to 33,000 through self-sufficiency efforts; reducing the percentage of children living in poverty from 11 percent to 6 percent; and reducing the first time demand for public assistance among young adults by decreasing rates of teen pregnancy, teen drug use, and juvenile crime and increasing school graduation and placement rates. Oregon Option — an effort designed to break down barriers and facilitate cooperation between levels of government and across agencies and to promote an outcome-based approach to planning — was also a key component of this state’s effort.
California California has a number of separate but inter-related efforts to track the well-being of children and families. The state tracks indicators in several areas including education, health, and workforce development. In the area of workforce development, the California legislature enacted in 1996 a statewide system to evaluate the performance of all its publicly-funded workforce programs including JTPA (now WIA), TANF, ES, vocational education, community colleges, and rehabilitation programs. With a goal of being fully operational by 2001, the outcome measures in this system include: rate of employment, length of employment retention, earnings before and after program participation, rate of change in Unemployment Insurance (UI) status, rate of change in the number of individuals moving from tax receiver to tax payer, and rate of advancement to higher education. |
1. WIA mandates the consolidation of specific employment and training programs administered by the Departments of Labor (DOL; the Job Training Partnership Act (JTPA) and the Employment Service), community colleges, other vocational and adult education providers, vocational rehabilitation providers as well as employment and training activities provided through the Community Services Block Grant, the Department of Housing an Urban Development, and the Veteran's Administration. Welfare-to-work activities provided under TANF are not required to be part of the workforce development system, but they may be, and in many states and localities are, included. However, grantees under the Welfare-to-Work (WtW) program administered by DOL are mandatory partners under WIA. [return to the text]
2. In four of the sites — Atlanta, Columbus, Grand Rapids, and Riverside — random assignment occurred to two different program groups. Results from only one of the programs in each of these sites is included here. In Atlanta, Grand Rapids, and Riverside, results for the Labor Force Attachment programs (an approach stressing quick entry into the labor market) were included on the table rather than the Human capital Development program (an approach stressing an investment in upfront education and training). In columbus, results for the Traditional Case Management program (an approach that used separate staff to provide income eligibility and JOBS case management responsibilities rather than the Integrated program (an approach that combined the responsibility of income eligibility and case management functions into one worker) were included. Results for the programs not included on the table were similar to those that were and would not have changed the overall rankings on outcomes or impacts. In addition, results from Detroit and Oklahoma were not included — including these sites would not have changed the overall conclusions drawn from the table. [return to the text]
3. Data on participation rates in the NEWWS evaluation were obtained from Hamilton, et al., 1997 (Atlanta, Columbus, and Grand Rapids); Scrivner, et al., 1998 (Portland) and Brock and Hartnett, 1998 (Columbus). [return to the text]
4. In FY 1999, the states that received bonuses were Alabama, California, the District of Columbia, Massachusetts, and Michigan. [return to the text]
5. Adjustment factors used in the JTPA system included: percent of participants who were female, percent age 55 or more, percent high school dropout, percent African-American, percent cash assistance recipient, percent long-term cash assistance recipient, percent basic skills deficient, percent with disabilities, percent lacking significant work history, percent not in the labor force, the local employment rate, the three-year growth rate of earnings in retail and wholesale trade, and annual earnings in retail and whole trade. [return to the text]
6. For more information on Oregon and other states developing and using comprehensive indicators to measure program performance, see Brown, et al., 1997. [return to the text]
Main page of report
Contents of report
Home Pages:
Human Services Policy (HSP)
Assistant Secretary for Planning and Evaluation
(ASPE)
Administration for Children and Families
(ACF)
U.S. Department of Health and Human Services
(HHS)
Updated: 02/12/01