Performance Improvement 2000. Evaluation Management


The management of HHS evaluations, which are carried out on a regular basis by the HHS agencies and offices and coordinated by ASPE, involves four basic functions: (1) evaluation planning and coordination, (2) project management, (3) quality assurance, and (4) dissemination of evaluation reports. A description of each function in general terms follows.

Evaluation Planning and Coordination

The HHS agencies, ASPE, the Office of the Inspector General (OIG), and OPHS develop evaluation plans annually in concert with HHSs program planning, legislative development, and budgeting cycles. Plan development is coordinated by ASPE. Each agency or office plan generally states the evaluation priorities, or projects under consideration for implementation. Typically, HHS evaluation priorities include: congressionally-mandated program evaluations, evaluations of Secretarial program or policy initiatives, assessments of new programs and ones that are candidates for reauthorization, and evaluations that support program performance management and accountability.

More specifically, HHS evaluation planning activities are coordinated with three department-wide planning initiatives: (1) strategic planning and performance management according to the Government Performance and Results Act (GPRA) of 1993; (2) annual research, demonstration and evaluation budget planning in conjunction with the President’s Budget; and (3) annual evaluation planning for the use of the PHS one-percent evaluation set-aside authority.

First, HHS evaluation activities support the Department’s strategic planning and performance management activities in several ways. Completed evaluation studies are used in shaping the specific HHS strategic goals and objectives. Evaluation findings provide an important source of information or evidence about the success of various HHS programs or policies that collectively make up the strategies to achieve the goals and objectives. The HHS Strategic Plan highlights evaluations that document efficacy or effectiveness of strategic programs or policies and lists future evaluations that will benefit strategic planning. HHS agencies are also using findings from their evaluations to support GPRA annual performance reporting to Congress and program budget justifications.

Secondly, Congress requests that HHS coordinate all of its research, demonstration, and evaluation programs to ensure that the results of these projects address HHS program goals and objectives. ASPE and the Assistant Secretary for Management and Budget work together with HHS agencies to provide the Congress with a special annual research, demonstration, and evaluation budget plan that coincides with the preparation of the President’s fiscal year budget. The plan outlines HHS agency research, demonstration, and evaluation priorities as related to the Department’s strategic goals and objectives.

Thirdly, those agencies and offices that use the PHS one-percent evaluation set-aside authority—AHRQ, CDC, HRSA, NIH, ASPE, OPHS, SAMHSA—submit a formal plan to ASPE, which coordinates and develops the individual plans into the HHS report to the Congress on the use of the PHS one-percent authority. This report must be submitted to the Congress before HHS can implement the plan.

Project Management

The design and management of evaluation projects at HHS is principally decentralizedthe HHS agencies, OIG, and ASPE all are responsible for executing annual evaluation plans, developing evaluation contracts, and disseminating and applying evaluation results. Even within agencies while there is some oversight responsibility and execution capability in the Office of the Director or Administratorthe various subunits (centers, institutes, and bureaus) conduct much of the day-to- day evaluation activity.

The OIG performs independent evaluations through its Office of Evaluations and Inspections (OEI). The OEI’s mission is to improve HHS programs by conducting inspections that provide timely, useful, and reliable information and advice to decision makers. This information (findings of deficiencies or vulnerabilities and recommendations for corrective action) is usually disseminated through inspection reports issued by the Inspector General. A summary of individual inspection reports and other OIG reports can be viewed on the Internet at OEI also provides technical assistance to HHS agencies in conducting their evaluations.

Quality Assurance and Improvement

Most evaluation projects are developed at the program level. The initial quality review is generally conducted by a committee of agency- or office-level policy and planning staff members. Before a project is approved, however, it is also reviewed for technical quality by a second committee with expertise in evaluation methodology. Technical review committees follow a set of criteria for quality evaluation practice established by each agency. Some HHS agencies also have external evaluation review committees composed of evaluation researchers and policy experts from universities and research centers.

Since HHS began reporting to Congress in 1995 on evaluations completed and in-progress through the Performance Improvement report series, the Department has focused more attention on the improving the quality of evaluation studies performed every year. The Evaluation Review Panel, convened yearly to recommend evaluations for highlighting in Chapter II of this report, has contributed insights to HHS evaluation officers on the strengths and challenges of ensuring quality evaluation studies. HHS evaluation officers have had opportunities to discuss these strengths and challenges with the Panel and to identify steps to improve agency evaluation capacity.

Based on these discussions, ASPE initiated two evaluation capacity-building projects in

FY 2000: a department-wide evaluation training needs assessment survey and establishment of an evaluation consultant contract mechanism to assist agency program managers.

Dissemination of Evaluation Reports

Maintaining report libraries and distributing information on evaluation results is an important component of HHS evaluation management. The Departments information and reports on major evaluations is available centrally through the website of the HHS Policy Information Center (PIC). The website address is: The PIC website offers users an opportunity to search (by key word searching or by selected program or policy topics) the departmental evaluation data base and electronic report library maintained by ASPE. As an information data base and library resource, the PIC contains nearly 7,000 completed and in-progress evaluation and policy research studies conducted by the Department, as well as key studies completed outside of HHS by the U.S. General Accounting Office (GAO) and private foundations.

Typically, the results of HHS evaluations are disseminated through targeted distribution of final reports, articles in refereed journals, and presentations at professional meetings and conferences. Although individual HHS agencies have primary responsibility for disseminating results, the ASPE will continue to its Department-wide efforts to expand dissemination of evaluation results to the larger research and practice communities through centralized computer communications and publications like the PIC website and database.

1 - FDA programs are principally authorized by legislation other than the PHS Act, specifically the authority of the Agriculture, Rural Development, Food and Drug Administration and Related Agencies Appropriations Act. IHS programs are authorized under the Indian Health Care Improvement Act and the Indian Self-Determination Act, and appropriated under the Department of the Interior and Related Agencies Appropriations.