As discussed in the previous section, the federal government has played a strong role in the development of outcome-based performance standards for both the welfare and workforce development systems. In addition to these federal efforts, however, states have also become increasingly involved in developing their own performance measurement systems. This section discusses state efforts to build these systems which range from very broad efforts that cover a range of human service programs to more narrow efforts focused on welfare or workforce development programs.
State initiatives to develop their own performance measurement systems are in part due to changes at the federal level that transfer additional responsibilities to states and local units of government. This devolution of responsibility poses new challenges for public administrators, who not only have to manage programs but also maintain and expand existing information systems needed to monitor program performance.
The design and use of performance measurement systems vary widely by state. One study (Horsch, 1996(a)) developed several useful dimensions for describing state efforts to build these systems.
- Cross-sector vs. sector-specific models. A few states are developing cross-sector outcome-based systems that cover many or all of a states programs focused on children and families. This approach typically begins with a statewide strategic plan and statewide goals cutting across a number of different agencies and programs. Many more states have begun to develop systems as sector-specific efforts such as those focused on employment and training programs or programs for youth.
- Level of participation. The participation of political appointees, agency personnel, and citizens in the design of the system also varies among the states. Some states have chosen an inclusive participatory process incorporating the viewpoints of people from a number of different areas, including citizens and line staff. For other states, the planning process involves state agency-level staff.
- Type of collaboration. Collaboration may occur across government agencies, between state and local entities, or both. At the state-level, some have reorganized their agencies by bringing together those that focus on the achievement of similar results, while others have established formal or informal bodies to coordinate performance measurement systems. The extent and the nature of the involvement of localities also varies some may be involved in the development of the state effort, others may define their own results within a framework established at the state level, and others may allow locally-determined as well as state measures.
- Use of results. State systems also differ in the nature of the results they identify and the process used to articulate the important results they wish to achieve. Some states have developed comprehensive models on how results will be identified, collected, and reported. Other states have begun developing their systems only by identifying, collecting, and reporting those measures which meet their immediate needs. Still others have identified a comprehensive set of measures, but are not currently collecting and reporting all of them because of time, data, or resource constraints.
- Types of measures. Measures can be articulated at the child/family/community level, the agency level, or the program level. The level of the measures generally dictates who is responsible for achieving the results.
There are clearly a number of different approaches at the state level for developing outcome-based performance measures. At one end are comprehensive, cross-sector systems which focus on establishing indicators or benchmarks of progress toward certain goals. Rather than measuring outcomes associated with specific programs (such as welfare-to-work or WIA programs), these states use measurable social goals such as reducing poverty or lowering the teen birth rate to monitor the overall effectiveness of public programs, to set goals, and/or to coordinate efforts across agencies (Brown et al, 1997; Yates, 1997).
Oregon developed the first and most comprehensive effort (Dyer, 1996; Lewis and Dunkle, 1996; Popovich, 1996) although Florida, Minnesota, and Vermont have embarked on similar efforts.(6) As summarized in Chart 1, Oregon has put substantial energy into reorganizing government activities at all levels to take a more goals-oriented approach. This was done through a multi-year collaborative process involving stakeholders at all levels including state agencies, local community leaders, the business community, and citizens. While public agencies were not consolidated as part of this effort, individual agencies use the indicators to guide policy and contracting decisions.
While some states have developed these relatively broad performance measurement systems, others focused more specifically on welfare and/or workforce development programs. For example, in Ohio, the state is giving counties block grants to administer the TANF programs and using some of the same performance measures as those used at the federal level on TANF to gauge success (Yates, 1997). Counties that exceed performance standards in the areas of a work participation rate and out-of-wedlock births receive greater funding flexibility and earn financial incentives.
Wisconsin established performance standards for its private contractors that administer the states TANF program. Contractors are required to meet a base level of performance to remain in compliance with the contract requirements and receive bonuses for exceeding the standards. Standards are established for: employment rates, wage rates, job retention, the provision of appropriate basic educational services (to certain types of individuals), the availability of employer-provided health insurance, and basic skills/job skills attainment (an optional measure for bonuses only).
A study of state welfare-to-work performance systems completed before the 1996 welfare law found that over half had statewide performance standards (GAO, 1995). Most of these states used employment rates as the key performance measure, although some also used other measures such as wage rates, job retention, and education and training achievement. Another study found state variation in whether uniform performance standards for welfare-to-work were set for local offices or negotiated with the state, whether they were adjusted using JTPA standards, and whether rewards and sanctions were significant or relatively informal (APWA, 1994).
Other states are developing performance measures and standards that focus more specifically on workforce development programs. Almost half the states are using outcome-based performance measures and standards to assess and monitor the performance measures that cut across the entire workforce development system including JTPA/WIA, the Employment Service (ES), vocational education, and others (Hyland, 1997; Simon, 1998). Some of these efforts include TANF programs as well. As an example, the California legislature enacted a statewide system to evaluate the performance of all its publicly-funded workforce development programs including JTPA (now WIA), TANF, the Employment Service, vocational education, community colleges, and rehabilitation programs (State of California, 1999) (see Chart 1 for details).
Some states sought to effect change to their workforce development system through a common policy framework for designated programs established by an interprogram or interagency team (North Carolina and Illinois). Others achieved this goal by reorganizing their programs into a single administrative entity (Texas and Iowa).
Overall, many states are moving forward on developing outcome-based performance measures. A few states have developed relatively comprehensive systems that track social indicators rather than the outcomes of specific programs. However, others are more narrowly focused on welfare or workforce development programs. These efforts generally complement the development of outcome-based measures occurring at the federal level.
Oregons results-based accountability system for children and families is part of a comprehensive statewide system. With the goal of creating a more results-oriented, decentralized governance system, Oregon first developed a strategic vision (known as Oregon Shines) with a task force of over 150 individuals identifying broad goals for the state. The state then established through a broad participatory process a total of 92 specific benchmarks to measure whether the state was making progress in achieving these goals (known as Oregon Benchmarks). Benchmarks were established in key policy areas related to the strategic vision including education and workforce development, reducing welfare dependency and increasing self-sufficiency, and protecting the health of children. Indicators for measuring these benchmarks were then developed and goals set for each indicator. Specific goals include reducing the welfare caseload from 40,000 to 33,000 through self-sufficiency efforts; reducing the percentage of children living in poverty from 11 percent to 6 percent; and reducing the first time demand for public assistance among young adults by decreasing rates of teen pregnancy, teen drug use, and juvenile crime and increasing school graduation and placement rates. Oregon Option an effort designed to break down barriers and facilitate cooperation between levels of government and across agencies and to promote an outcome-based approach to planning was also a key component of this states effort.
California has a number of separate but inter-related efforts to track the well-being of children and families. The state tracks indicators in several areas including education, health, and workforce development. In the area of workforce development, the California legislature enacted in 1996 a statewide system to evaluate the performance of all its publicly-funded workforce programs including JTPA (now WIA), TANF, ES, vocational education, community colleges, and rehabilitation programs. With a goal of being fully operational by 2001, the outcome measures in this system include: rate of employment, length of employment retention, earnings before and after program participation, rate of change in Unemployment Insurance (UI) status, rate of change in the number of individuals moving from tax receiver to tax payer, and rate of advancement to higher education.
1. WIA mandates the consolidation of specific employment and training programs administered by the Departments of Labor (DOL; the Job Training Partnership Act (JTPA) and the Employment Service), community colleges, other vocational and adult education providers, vocational rehabilitation providers as well as employment and training activities provided through the Community Services Block Grant, the Department of Housing an Urban Development, and the Veteran's Administration. Welfare-to-work activities provided under TANF are not required to be part of the workforce development system, but they may be, and in many states and localities are, included. However, grantees under the Welfare-to-Work (WtW) program administered by DOL are mandatory partners under WIA.
2. In four of the sites Atlanta, Columbus, Grand Rapids, and Riverside random assignment occurred to two different program groups. Results from only one of the programs in each of these sites is included here. In Atlanta, Grand Rapids, and Riverside, results for the Labor Force Attachment programs (an approach stressing quick entry into the labor market) were included on the table rather than the Human capital Development program (an approach stressing an investment in upfront education and training). In columbus, results for the Traditional Case Management program (an approach that used separate staff to provide income eligibility and JOBS case management responsibilities rather than the Integrated program (an approach that combined the responsibility of income eligibility and case management functions into one worker) were included. Results for the programs not included on the table were similar to those that were and would not have changed the overall rankings on outcomes or impacts. In addition, results from Detroit and Oklahoma were not included including these sites would not have changed the overall conclusions drawn from the table.
3. Data on participation rates in the NEWWS evaluation were obtained from Hamilton, et al., 1997 (Atlanta, Columbus, and Grand Rapids); Scrivner, et al., 1998 (Portland) and Brock and Hartnett, 1998 (Columbus).
4. In FY 1999, the states that received bonuses were Alabama, California, the District of Columbia, Massachusetts, and Michigan.
5. Adjustment factors used in the JTPA system included: percent of participants who were female, percent age 55 or more, percent high school dropout, percent African-American, percent cash assistance recipient, percent long-term cash assistance recipient, percent basic skills deficient, percent with disabilities, percent lacking significant work history, percent not in the labor force, the local employment rate, the three-year growth rate of earnings in retail and wholesale trade, and annual earnings in retail and whole trade.
6. For more information on Oregon and other states developing and using comprehensive indicators to measure program performance, see Brown, et al., 1997.