This evaluation reports on Phase II of a three-phase effort to develop a fully integrated computerized data system to facilitate planning and evaluation of the programs administered by the Bureau of Health Professions. Phase I assessed indicators of the Bureau's progress in meeting the goals and objectives defined in its strategic plan. Phase II analyzed the feasibility and appropriateness of the goals, outcomes, and indicators developed by the Bureau and identified strategies the agency can use in refining them with periodic input from grantees in the field. A performance monitoring system was proposed to facilitate ongoing program management. The system described in this study builds on the Bureau of Health Professions Grants Management Application System. The study recommended that the proposed performance monitoring system be piloted with several programs before it is fully implemented.
The overall purpose of this project was to help the Bureau of Health Professions develop a set of outcome-based performance measures and design a performance monitoring system, both to measure whether supported programs are meeting national health workforce objectives and to identify necessary program course corrections. The following issues were addressed in the study:
- Can the outcomes and indicators identified to monitor program progress in meeting the agency's goals be reliably collected and analyzed?
- Do grantees believe that the identified goals, outcomes, and key indicators reflect the most valuable contributions of their projects?
- What data sources and collection tools are currently available?
- What kind of system should be developed to support the requirements of the Government Performance Results Act (GPRA) of 1993 and of other performance monitoring and reporting measures?
The Bureau of Health Professions provides both policy leadership and support for enhancement of the health professions workforce and development of its educational infrastructure. This evaluation follows earlier work to assess and refine a set of goals with respect to workforce quality, supply, diversity, and distribution, as well as the outcomes and indicators of performance identified to measure and monitor progress toward those goals. This strategic planning process was undertaken by the Bureau in response to internal and external pressures for more effective targeting of scarce Federal resources toward those programs and activities that support and have a demonstrable effect on national workforce priorities. The development of a more explicit outcome-oriented system that identifies measures of performance related to Bureau-funded efforts will help address such concerns and will be very useful in future planning and program management.
The cross-cutting goals developed by the Bureau of Health Professions address (1) the development of a health care workforce that has the mix of competencies and skills needed to deliver cost-effective, quality care; (2) the need for educational programs that will yield professionals who can meet the needs of vulnerable populations; (3) the need for cultural diversity in the health professions; and (4) the need to stimulate and monitor the education system's ability to respond to the changing demands of the health care marketplace. Expected outcomes and indicators of success were developed and refined for each goal as part of phaseI of this evaluation effort. This evaluation was undertaken to address the next set of questions that needed to be examined by the Bureau in order to develop a performance measurement and management system that could link individual grantee level information in all of the Bureau's programs to one or more of the cross-cutting goals, associated outcomes, indicators, and performance-related functions.
The investigators employed several approaches to achieve the evaluation objectives. Among these were ongoing discussions among evaluators, Bureau of Health Professions program administrators, and other agency officials and committees whose input would inform the design of the performance evaluation system. The evaluators also consulted outside experts representing a range of health workforce training perspectives on feasibility issues related to data collection from grantees for program monitoring and evaluation purposes. An initial survey of data sources was then undertaken to explore existing systems that may be applicable to the ongoing data collection needs of the Bureau's performance monitoring system.
This report focuses on the design of a Comprehensive Performance Monitoring System (CPMS). The study calls for a system that will provide information to answer a basic performance question: Can the Bureau of Health Professionsþwith available funding and guiding legislation and through planned and funded grant activitiesþmeet national health workforce objectives for targeted populations? In capturing the information necessary to answer this question, the system would track essential inputs, processes, outputs, and outcomes for Bureau programs. The system would provide that information in regular reports to internal decisionmakers and to the Bureau's external customers, including Congress and grantees. The CPMS should support the following functions: monitoring and measuring performance goals; analyzing and assessing indicator data; identifying successes and problems; reviewing key program processes; and identifying opportunities.
A detailed set of recommended steps was developed for the Bureau's preparation for implementing the CPMS. These include completing final specifications of cross-cutting indicator definitions; specifying performance benchmarks and a process for assessment; identifying external indicators that must be tracked (e.g., market interest, availability); identifying sources of data and pilot testing new data collection; specifying hardware and software requirements of the system; and developing a pilot version of the system for beta testing.
The processes essential for maintaining the functions of the CPMS will require data, hardware, software, and human resources. The analytic and reporting functions of the system would be largely automatic once designed and implemented. Obtaining key information resources will require collecting data that are readily available and developing data collection instruments or surveys to collect data that are not currently reported by all grantees on a systematic basis. Other data collection resources managed outside of the Bureau may also be integrated into the system. The CPMS could exist in a variety of forms, ranging from paper files and reports to computer data base systems. The study recommended that an add-on module linked to the Grants Management Application System may yield the most efficient system. The system could be built as a relational data base, with linkage variables for cross-referencing data. The types of resources that would be needed for a computer-based performance monitoring system could include computer hardware; system software, including data base system software; and trained software and hardware systems support staff to develop, maintain, and update the capabilities of the system as needed. The data base could be on a stand-alone computer or could reside on a network.
The study concluded that in the final phase of this effort, the Bureau of Health Professions must address the questions necessary to perform key functions of the CPMS for each of the cross-cutting goals, outcomes, and indicators. Furthermore, the Bureau must decide what data elements should be collected from grantees to support the CPMS. Many data elements may already be collected and captured in the Grants Management Application System; thus, the CPMS should be integrated with this system and the staff with these responsibilities should coordinate their efforts.
Use of Results
This Bureau of Health Professions study on performance monitoring to support program management and Bureau-wide planning also fits into the context of broader efforts within the Health Resources and Services Administration and HHS to conduct strategic planning for resource investment and compliance with the GPRA. Policymakers and program staff in other Federal agencies within and outside HHS can benefit from the efforts described in this evaluation.
Bureau of Health Professions
PIC ID: 5497
The Lewin Group, Fairfax, VA