Performance Improvement 2002. Evaluation Management


Evaluation Management

The management of HHS evaluations, which are carried out on a regular basis by the HHS agencies and offices and coordinated by ASPE, involves four basic functions: (1) evaluation planning and coordination, (2) project management, (3) quality assurance, and (4) dissemination of evaluation reports. A description of each function in general terms follows.

Evaluation Planning and Coordination

The HHS agencies, ASPE, the Office of the Inspector General (OIG), and OPHS develop evaluation plans annually in concert with HHS’s program planning, legislative development, and budgeting cycles. Plan development is coordinated by ASPE. Each agency or office plan generally states the evaluation priorities, or projects under consideration for implementation. Typically, HHS evaluation priorities include: congressionally-mandated program evaluations, evaluations of Secretarial program or policy initiatives, assessments of new programs and ones that are candidates for reauthorization, and evaluations that support program performance management and accountability.

More specifically, HHS evaluation planning activities are coordinated with three department-wide planning initiatives. First, HHS evaluation activities support the Department’s strategic planning and performance management activities in several ways. Completed evaluation studies are used in shaping the specific HHS strategic goals and objectives. Evaluation findings provide an important source of information or evidence about the success of various HHS programs or policies that collectively make up the strategies to achieve the goals and objectives. The HHS Strategic Plan highlights evaluations that document efficacy or effectiveness of strategic programs or policies and lists future evaluations that will benefit strategic planning. HHS agencies are also using findings from their evaluations to support GPRA annual performance reporting to Congress and program budget justifications.

Secondly, Congress requests that HHS coordinate all of its research, demonstration, and evaluation (RD&E) programs to ensure that the results of these projects address HHS program goals and objectives. ASPE and the Assistant Secretary for Budget, Technology and Finance work together with HHS agencies to provide the Congress with a special annual research, demonstration, and evaluation budget plan that coincides with the preparation of the President’s fiscal year budget. The plan outlines HHS agency research, demonstration, and evaluation priorities as related to the Department’s strategic goals and objectives. ASPE plans to use the RD&E reporting mechanism as a research planning format.

The newly-formed Research Coordination Council (chaired by the ASPE, and containing representatives of HHS agencies), will foster greater interactions among its research programs. The Council will also begin to streamline research and evaluate Department-wide research priorities to ensure greater efficiencies in research, demonstration, and evaluation. The Council will strengthen HHS research coordination and planning around key Departmental priorities and themes.

Thirdly, those agencies and offices that use the PHS one-percent evaluation set-aside authority-AHRQ, CDC, HRSA, NIH, ASPE, OPHS, SAMHSA-submit a formal plan to ASPE, which coordinates and develops the individual plans into the HHS report to the Congress on the use of the PHS one-percent authority. This report must be submitted to the Congress before HHS can implement the plan.

Project Management

HHS agencies, OIG, and ASPE all are responsible for executing annual evaluation plans, developing evaluation contracts, and disseminating and applying evaluation results. All agencies and their subunits are encouraged to coordinate with each other on research/evaluation project planning and release of final reports that are likely to relate to work of other HHS agencies. Within agencies - while there is some oversight responsibility and execution capability in the Office of the Director or Administrator - the various subunits (centers, institutes, and bureaus) conduct much of the day-to-day evaluation activity.

The OIG performs independent evaluations through its Office of Evaluations and Inspections (OEI). The OEI’s mission is to improve HHS programs by conducting inspections that provide timely, useful, and reliable information and advice to decision makers. This information (findings of deficiencies or vulnerabilities and recommendations for corrective action) is usually disseminated through inspection reports issued by the Inspector General. A summary of individual inspection reports and other OIG reports can be viewed on the Internet at <>. OEI also provides technical assistance to HHS agencies in conducting their evaluations.

Quality Assurance and Improvement

Most evaluation projects are developed at the program level. The initial quality review is generally conducted by a committee of agency- or office-level policy and planning staff members. Before a project is approved, however, it is also reviewed for technical quality by a second committee with expertise in evaluation methodology. Technical review committees follow a set of criteria for quality evaluation practice established by each agency. Some HHS agencies also have external evaluation review committees composed of evaluation researchers and policy experts from universities and research centers.

Since HHS began reporting to Congress in 1995 on evaluations completed and in-progress through the Performance Improvement report series, the Department has focused more attention on improving the quality of evaluation studies performed every year. The HHS Evaluation Review Panel, convened yearly to recommend evaluations for highlighting in Chapter II of this report, has contributed insights to HHS evaluation officers on the strengths and challenges of ensuring quality evaluation studies. HHS evaluation officers have had opportunities to discuss these strengths and challenges with the Panel and to identify steps to improve agency evaluation capacity.

Based on these discussions, ASPE has initiated two evaluation capacity-building projects. The first project was to establish an evaluation consultant Blanket Purchase Agreement (BPA) contract mechanism to assist agency program managers in obtaining short term assistance in developing evaluation designs or monitoring the performance quality of evaluation contractors. The second project was to conduct a department-wide evaluation training needs assessment survey, which indicates training needs in various areas.

Dissemination of Evaluation Reports

Maintaining report libraries and distributing information on evaluation results is an important component of HHS evaluation management. The Department’s information and reports on major evaluations is available centrally through the website of the HHS Policy Information Center (PIC), located at the following address: <>. The PIC website offers users an opportunity to search (by key word or by selected program or policy topics) the departmental evaluation database and electronic report library maintained by ASPE. As an information database and library resource, the PIC contains over 8,000 completed and in-progress evaluation and policy research studies conducted by the Department, as well as key studies completed outside of HHS by the U.S. General Accounting Office (GAO) and private foundations.

Typically, the results of HHS evaluations are disseminated through targeted distribution of final reports, articles in refereed journals, and presentations at professional meetings and conferences. Although individual HHS agencies have primary responsibility for disseminating results, the ASPE will continue its Department-wide efforts to expand dissemination of evaluation results to the larger research and practice communities through centralized computer communications and publications like the PIC website and project information database.

[1] — FDA programs are principally authorized by legislation other than the PHS Act, specifically the authority of the Agriculture, Rural Development, Food and Drug Administration and Related Agencies Appropriations Act. IHS programs are authorized under the Indian Health Care Improvement Act and the Indian Self-Determination Act, and appropriated under the Department of the Interior and Related Agencies Appropriations.