There are increasing expectations across the Federal Government that agency programs, with a given set of resources and through a series of actions and decisions, will produce the desired outcomes and products for the intended audiences or beneficiaries. This study provides an assessment of the Health Resources and Services Administration's (HRSA) current capacity to respond to such expectations in compliance with the Government Performance and Results Act (GPRA) of 1993. The findings and recommendations in this study address the development of the kind of performance management system needed to respond to the governmentwide search for greater effectiveness.
HRSA commissioned this study to assess its ability to develop and implement a performance measurement and management system in response to the GPRA. Under GPRA, each Federal agency is required to develop comprehensive strategic plans as well as annual performance plans that set specific performance goals for each program activity. Moreover, each agency must report annually on actual performance as it compares with goals. The specific objectives of the study were to provide information about the current status (as of summer 1995) of the development of performance indicators in the four HRSA bureaus and in the Office of Rural Health Policy (ORHP); the adequacy of data and data sources for applying the indicators; and the potential for the agency to use the performance measures in strategic planning or other efforts requiring indicators and data that can be aggregated across its organizational units.
A variety of economic and political forces have increased emphasis on performance in both the public and private sectors. This larger societal phenomenon has produced converging political, budgetary, and management environments that collectively result in a systematic search for effective Federal programs. One of the basic underlying purposes of GPRA is to provide information about the comparative performance of the many Federal programs and agencies competing for budgetary resources, so that available resources may be allocated to those with the best return on the Nation's investment.
Under GPRA, each Federal agency is required to report annually (beginning in March 2000) on actual program performance as it compares with program goals. A preliminary set of performance measures for program and spending is to be developed for fiscal 1997 budget submissions, with actual performance data obtained from fiscal 1995 to fiscal 1997. The intent of GPRA is to motivate Federal agencies to identify programs that are effective in achieving goals and to reallocate resources away from those that are ineffective. GPRA focuses on the quality of performance, quantity of products, timeliness of products, and program efficiency.
The Lewin Group undertook this study for HRSA, organizing the approach around a central question: Can this organization, with a given set of resources, through a series of actions and decisions, produce outputs that have the desired effects and outcomes for the intended audiences or beneficiaries? Preliminary design activities included meetings with HRSA staff, a literature review, and other background research.
An interview protocol was developed and administered to each of the four bureaus and ORHP. Information was solicited about each program's objectives, inputs (e.g., dollars, staff), processes (e.g., training approaches), outputs (e.g., number of people trained), impacts (e.g., number of trainees who would not have found comparable work), outcomes (e.g., number of trainees who find and retain work), and current use of performance measures and indicators. In addition, a wide variety of program materials was collected and reviewed. Draft assessments were developed and shared with staff from the relevant bureaus and offices to ensure that the information was complete and accurate. Findings from the interviews and background research were synthesized, analyzed, and then compiled into assessments of the program clusters and individual programs or budget line items.
The study used an overall assessment approach and analytical framework to document the linkages and commonalities among various HRSA organizational components with respect to resources, processes, products, outcomes, and populations served. Application of a program logic model that illustrates the rationale, structure, and division of labor of each program was critical to the assessments. The program logic model uses a matrix to represent the interactions and relationships among a program's objectives, illustrating the flow of inputs, activities, outputs, outcomes, and impacts required to achieve them.
HRSA's current program structures and measurement efforts form a base for a HRSA-wide measurement system. As part of this system, significant indicators that are common to many programs could be used to measure achievement of agencywide goals. The study identified three principles that should underlie the performance measurement strategy:
- The measurement strategy should emphasize a coordinated, interdependent system of health care resources, rather than a disparate set of programs.
- The measurement strategy should be clearly linked to strategic planning, program activities, evaluation planning and findings, and budgeting.
- The measurement strategy should be developed incrementally, with initial focus on the GPRA requirements that are easiest to meet.
The study found that it is both feasible and desirable to use a standard approach to measuring program performance across HRSA programs. In addition, progress in measuring and collecting performance data is quite varied across HRSA programs and program clusters. Within most programs, there is inadequate linkage along the continuum of performance measurement, with a general emphasis on producing outputs rather than evaluating program performance or outcomes. The study also concludes that data and data sources are not treated as strategic HRSA resources and that approaches to specification, collection, and use of data vary widely. Furthermore, the budget does not provide resources specifically for the data collection and evaluation necessary to build and sustain the kind of performance measurement system recommended in the study.
Recommendations are made for both short- and long-term actions. Short-term recommendations include the appointment of a central coordinator responsible for performance management efforts; the establishment of intra-agency working groups to specify performance measures, data collection, support strategies, and involvement of service beneficiaries; the provision of resources; and the provision of technical assistance tailored to specific HRSA components for developing and implementing systems of performance measurement.
Long-term recommendations address the development of a performance management system infrastructure. First, the HRSA strategic planning process should be continued and refined. Second, the performance management process needs to be linked to the budget process, to evaluation, and to grants and contracts management. Third, mechanisms need to be developed to incorporate feedback and data from grantees and populations served by HRSA. Fourth, some further restructuring of HRSA programs will be needed to facilitate performance measurement in light of HRSA-wide strategic priorities. Finally, the commitment and ongoing support of senior management is needed if staff members are to undertake the changes necessary to implement and sustain the performance system.
Use of Results
The study produced a framework for assessing HRSA functions and related performance measures and for how individual program activities and outcomes relate to the Agency's overall strategic goals. Subsequent technical assistance to HRSA on performance indicator development is proceeding, based on needs identified in this initial status report. The logic models developed for the four bureaus and ORHP are serving as an internal guide to the assessment of program inputs, outputs, and outcomes. Assistance to individual bureaus to expand staff capacity in performance measurement began at the end of fiscal 1995. The results of this study have broader applications as well. Many Federal agencies could use the performance measurement framework to assess the linkages between their strategic goals and individual program components, activities, and performance. The identified performance measures, the recommended questions on assessing program effectiveness, and the information on how to develop performance indicators are all valuable tools for any Federal agency developing planning and performance measurement strategies in compliance with GPRA.
Office of Planning, Evaluation, and Legislation
Karen Thiel Raykovich, Ph.D.
PIC ID: 5954
The Lewin Group, Fairfax, VA