Performance Improvement 2006. Evaluation Management

Management of evaluations carried out by HHS agencies and offices and coordinated by ASPE involves:

(1) planning and coordination, (2) project oversight, (3) quality assurance, and (4) dissemination of results (Figure 3, Evaluation Management Cycle). A description of each function follows.

Evaluation Mangement Cycle 

Figure 3

• Evaluation Planning and Coordination

  • HHS agencies, ASPE, the Office of the Inspector General (OIG), and OPHS develop evaluation plans annually in concert with HHS program planning, legislative development, and budgeting cycles. Plan development is coordinated by ASPE. Each agency or office evaluation plan generally states the evaluation priorities, or projects under consideration for implementation. Typically, HHS evaluation priorities include: congressionally-mandated program evaluations, evaluations of Secretarial program or policy initiatives, assessments of new programs and ones that are candidates for reauthorization, and evaluations that support program performance management and accountability.

    HHS evaluation planning activities are coordinated with three department-wide planning initiatives. First, HHS evaluation activities support the Department’s strategic planning and performance management activities in several ways. Completed evaluation studies are used in shaping the specific HHS strategic goals and objectives. Evaluation findings provide important sources of information and evidence about the success of various HHS programs or policies. The HHS Strategic Plan highlights evaluations that document efficacy or effectiveness of strategic programs or policies and lists future evaluations that will benefit strategic planning. HHS agencies use findings from their evaluations to support the Government Performance and Results Act (GPRA) annual performance reporting to Congress and program budget justifications.

    Second, Congress requests that HHS coordinate and report to Congress regarding all of its research, demonstration, and evaluation (RD&E) programs to ensure that the results of these projects address HHS program goals and objectives. ASPE and the Assistant Secretary for Budget, Technology and Finance work together with HHS agencies to provide the Congress with a special annual research, demonstration, and evaluation budget plan that coincides with the preparation of the President’s fiscal year budget. The plan outlines HHS agency research, demonstration, and evaluation priorities as related to the Department’s strategic goals and objectives (Figure 4, Evaluation Reporting Cycle).

    Evaluation Reporting Cycle

    

    Figure 4

     

    The Secretarially-created Research Coordination Council (chaired by the ASPE, and containing representatives of HHS agencies) fosters greater interactions among the research programs. The Council’s work includes streamlining research and evaluating Department-wide research priorities to ensure greater efficiencies in research, demonstration, and evaluation. The Council seeks to strengthen HHS research coordination and planning around key Departmental priorities and themes.

    Third, the Secretary must report to the Congress his plan for using PHS evaluation set-aside funds before implementing the plan (Figure 4). Those agencies and offices that use the PHS evaluation set-aside authority—ACF, AHRQ, CDC, HRSA, NIH, ASPE, OPHS, SAMHSA—submit a formal plan to ASPE, which coordinates and develops the individual plans into the HHS report to the Congress on the use of the PHS authority.

• Project Oversight

  • HHS agencies, ASPE, and the OIG execute annual evaluation plans that involve developing evaluation contracts and disseminating and applying evaluation results. All agencies and their subunits (centers, institutes, and bureaus) coordinate with each other on research and evaluation project planning and release of final reports that relate to work of other HHS agencies. While there is some oversight responsibility and execution capability in the Office of the Director or Administrator for each agency, the various agency subunits conduct much of the day-to-day evaluation activity.

    The OIG performs independent evaluations through its Office of Evaluations and Inspections (OEI). OEI’s mission is to improve HHS programs by conducting inspections that provide timely, useful, and reliable information and advice to decision makers. These findings of deficiencies or vulnerabilities and recommendations for corrective action are usually disseminated through inspection reports issued by the Inspector General. A summary of individual inspection reports and other OIG reports can be viewed on the World Wide Web at http://oig.hhs.gov/reports.html. OEI provides technical assistance to HHS agencies in conducting their evaluations.

• Quality Assurance and Improvement

  • Most evaluation projects are developed at the program or office level. A committee of agency- or office- level policy and planning staff members generally conducts the initial quality review. Before a project is approved, a second committee reviews it for technical quality with expertise in evaluation methodology. Technical review committees follow a set of criteria for quality evaluation practice established by each agency. ASPE, for example, has a formalized peer review process in which experienced evaluators on staff review, discuss and approve all proposed research projects before they are submitted for funding. Some HHS agencies have external evaluation review committees composed of evaluation experts from universities and research centers.

    Since HHS began reporting to Congress in 1995 on completed evaluations through the Performance Improvement report series, the Department has focused attention on improving the quality of evaluation studies performed. An Evaluation Review Panel, convened periodically, has contributed insights to HHS evaluation officers on the strengths and challenges of ensuring quality evaluation studies. HHS evaluation officers have had opportunities to discuss these strengths and challenges and identify steps to improve agency evaluation projects.

• Dissemination of Evaluation Reports

  • Maintaining on-line electronic report libraries and distributing information on evaluation results is an important component of HHS evaluation management. The Department’s information and reports on major evaluations is available centrally through the website of the HHS Policy Information Center (PIC), located at the following Web address: http://aspe.hhs.gov/pic. ASPE’s PIC website offers users an opportunity to search – by key word, selected program, or policy topics, – the departmental evaluation report database and electronic report library maintained by ASPE. The PIC contains over 8,500 completed and in-progress evaluation and policy research studies conducted by the Department, as well as key studies completed outside of HHS by the U.S. Government Accountability Office (GAO) and private foundations.

    The results of HHS evaluations are disseminated through targeted distribution of final reports, articles in refereed journals, and presentations at professional meetings and conferences. Although individual HHS agencies have primary responsibility for disseminating results, ASPE continues its Department-wide efforts to expand dissemination of evaluation results to the larger research and practice communities through e-mail lists, e-newsletters, and publications available on the PIC Web site.

View full report

Download

"pi_2006.pdf" (pdf, 1.39Mb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®