Performance Improvement 1997. Evaluation Management

02/01/1997

The management of HHS evaluations, carried out on a regular basis by the HHS agencies and offices and coordinated by ASPE, involves five basic functions:

  1. Evaluation planning and coordination
  2. Project management
  3. Quality assurance
  4. Dissemination of evaluation reports
  5. Ensuring effective use of evaluation results

A description of each function in general terms follows. Additional information on the individual HHS agencies, ASPE, and OPHS evaluation functions can be found in chapter III.

Evaluation Planning and Coordination

The HHS agencies, ASPE, the Office of the Inspector General (OIG), and OPHS develop evaluation plans annually, in concert with HHS's program planning, legislative development, and budgeting cycles. Plan development is coordinated by ASPE. Each agency or office plan generally states the evaluation priorities or projects under consideration for implementation. Typically, HHS evaluation priorities include congressionally mandated program evaluations, evaluations of Secretarial program or policy initiatives, and assessments of new programs, programs that are candidates for reauthorization, or programs for which key budget decisions are anticipated.

HHS evaluation planning activities are required more specifically in two ways. First, those agencies and offices that use the PHS 1-percent evaluation set-aside authority--AHCPR, CDC, HRSA, NIH, ASPE, OPHS, SAMHSA--submit a formal plan to ASPE, which coordinates and develops the individual plans into the HHS report to Congress on the use of the PHS 1-percent authority. This report must be submitted to Congress before HHS can implement the plan.

Second, Congress has asked HHS to coordinate all of its research, demonstration, and evaluation programs to ensure that the results of these projects address HHS's program goals and objectives. ASPE and the Assistant Secretary for Management and Budget now work with HHS agencies to provide Congress with a special annual research, demonstration, and evaluation budget plan that coincides with the preparation of the President's fiscal year budget. The plan outlines each HHS agency's research, demonstration, and evaluation priorities as related to overall HHS program goals and objectives.

Project Management

The design and management of evaluation projects at HHS is primarily decentralized--the various HHS agencies, OIG, and ASPE are all responsible for executing annual evaluation plans, developing evaluation contracts, and disseminating and applying evaluation results. Even within agencies, while there is some oversight responsibility and execution capability in the Office of the Director or Administrator, the various subunits (centers, institutes, bureaus) conduct much of the day-to-day evaluation activity.

OIG performs independent evaluations through its Office of Evaluations and Inspections (OEI). OEI's mission is to improve HHS programs by conducting inspections that provide timely, useful, and reliable information and advice to decisionmakers. This information (findings of deficiencies or vulnerabilities and recommendations for corrective action) is usually disseminated through inspection reports issued by the Inspector General. A summary of individual inspection reports and other OIG reports can be viewed on the Internet at http://www.sbaonline.sba.gov/ignet. OEI also provides technical assistance to HHS agencies in conducting their evaluations.

Quality Assurance

Most evaluation projects are developed at the program level. The initial quality review is generally conducted by a committee of agency- or office-level policy and planning staff members. Before a project is approved, however, it is also reviewed for technical quality by a second committee of staff who are skilled in evaluation methodology. Technical review committees follow a set of criteria for quality evaluation practice established by each agency. Some HHS agencies also have external evaluation review committees composed of evaluation researchers and policy experts from universities and research centers. More details on the quality assurance procedures for the various HHS agencies, ASPE, and OPHS are presented in chapter III.

Dissemination of Evaluation Reports

Maintaining report libraries and distributing information on evaluation results is an important component of HHS evaluation management. Project information and reports are continuously submitted to the HHS Policy Information Center (PIC)--the departmental evaluation data base and library maintained by ASPE. As an information data base and library resource, the PIC contains nearly 7,000 completed and in-progress evaluation and policy research studies conducted by the Department, as well as key studies completed outside HHS by the U.S. General Accounting Office and private foundations.

Typically, the results of HHS evaluations are disseminated through targeted distribution of final reports, articles in refereed journals, and presentations at professional meetings and conferences. Although the individual HHS agencies have primary responsibility for disseminating results, there is a departmentwide effort under way to expand dissemination to the larger research and practice communities through centralized computer communications and publications. Abstracts of all studies maintained in the PIC evaluation data base are on the Internet through the HHS World Wide Web server, at the following URL: http://aspe.hhs.gov/PIC/gate2pic.htm. Information is available on completed and in-process projects; it includes the name and telephone number of the HHS official responsible for the project.

In addition, HHS is widely distributing copies of its first departmentwide annual report on evaluation (Performance Improvement 1996: Evaluation Activities of the U.S. Department of Health and Human Services). The report's theme of performance improvement represents the numerous changes and initiatives throughout HHS to increase the effectiveness and efficiency of health and human services programs. This report to Congress summarizes the findings of HHS evaluations completed during FY 1995. In that year, HHS agencies produced 148 evaluation reports and supported more than 200 evaluation projects in progress. The report provides summaries or abstracts of the completed project reports and contacts for further information.

In addition to providing the report to members of Congress, copies were sent to State and local public health and social services officials, various associations for schools of public health, social welfare, and social work, and other national health and social policy research and practice associations. A similar dissemination strategy is being used for this report, the second annual HHS report on evaluations. Both reports, Performance Improvement 1996 and Performance Improvement 1997, are also available on the previously described HHS home page in three computer formats for downloading information.

Ensuring Effective Use of Evaluation Results

HHS is committed to ensuring that evaluations yield a high return on the investment of available program funds. In the last decade, HHS evaluations were used primarily by program managers for internal purposes of improving program operations and efficiency. The major outcome or impact evaluations conducted by HHS were generally congressionally mandated and involved a substantial commitment of evaluation resources. In the 1990's, however, the demand for more program outcome and impact evaluations has increased, largely in response to fulfilling HHS's responsibility for implementing reform legislation. To understand the effects of these transformations on HHS programs at the system and individual levels will require a substantial investment of evaluation resources in outcome or impact studies. In addition, HHS is giving priority to evaluation efforts related to implementation of the Government Performance and Results Act (GPRA), which concerns all HHS programs. The expectations for GPRA are that by FY 1999 HHS will have in place the capacity to measure the performance of its programs with data that are complete, accurate, and consistent enough to support decision making at various management levels. The measures should be tied to program performance goals and should have the capability of measuring program results in achieving those goals.