Performance Improvement 1999. Appendix A - Description of the HHS Evaluation Program

02/01/1999

Evaluation plays an integral role in carrying out the HHS mission. Assessing various aspects of agency program performance allows staff to identify means of improving that performance. The HHS evaluation function has three goals: (1) to provide information on HHS programs that helps government officials and members of the Congress make decisions related to programs, policies, budgets, and strategic planning; (2) to help HHS managers improve program operations and performance; and (3) to disseminate evaluation results and methodological tools useful to the larger health and human services community of State and local officials, researchers, advocates, and practitioners for improving the performance of their programs.

This chapter describes the organization and operation of evaluation at HHS. It first provides an overview of the kinds of evaluation activities supported by HHS agencies and then describes the resources and funding mechanisms used to support them. The chapter details HHS evaluation management, including planning procedures, project management, quality assurance, and dissemination of results. The chapter concludes with a discussion of effective uses of HHS evaluations, with illustrations from projects completed in the past year.

HHS EVALUATION ACTIVITIES

HHS defines evaluation as the assessment of program performance (efficiency, effectiveness, and responsiveness) through the analysis of data or information collected systematically and ethically, and the effective use of resulting information in program or policy decisionmaking and program management. This definition encompasses a range of evaluation activities, such as those listed below.

  • Outcome evaluations measure the immediate or intermediate effects of a program with respect to the stated goals or objectives.
  • Impact evaluations look at the broader and long-term results, intended or unintended, of a program on populations or institutions involved.
  • Implementation or process evaluations assess the nature of program inputs and outputs and their relationship to the stated goals and objectives.
  • Policy assessments examine health policies with respect to their development, implementation, or their impact on public health or program activity.
  • Cost-benefit or cost-effectiveness analyses develop methodology and its application to assess the relationship of program results to program costs (direct and indirect), often in comparison with alternative programs.
  • Survey data analyses evaluate the results of HHS programs or policies by analyzing data obtained from surveys.
  • Performance measurement and data systems identify and test the validity and reliability of process, output, and outcome indicators to measure the performance of programs and develop data systems supporting implementation of the Government Performance and Results Act (GPRA) of 1993.
  • Simulations and models use computer simulations and modeling techniques to analyze the impact of policy changes on service delivery systems and beneficiaries.
  • Management studies examine the effectiveness or efficiency of the administration or operation of HHS programs and offices.
  • Evaluation syntheses integrate the results from multiple independent evaluation studies within a defined program or policy area in a fashion that improves the accessibility and application of those results.
  • Evaluation feasibility studies assess the clarity and importance of program goals and objectives, the consensus of program stakeholders on the potential utility of evaluation information, and the availability of relevant performance data before committing to a full-scale program evaluation.
  • Evaluation design projects procure assistance developing of an evaluation design, measurement tools, or analytic models in preparation for fully implementing an evaluation.
  • Instrument development projects develop evaluation instruments (design, measurement, or analytic) for a specific HHS program or for general use by the health and human services community.
  • Evaluation technical assistance helps HHS officials and grantees with any aspect of evaluation planning, project design implementation analysis, or use of results.
  • Evaluation dissemination identifies target audiences and mechanisms to inform program constituencies and evaluation stakeholders about evaluation results.
  • Evaluation training/conferences maintain the professional skills and expertise of evaluation staff through training opportunities, and promote the dissemination of HHS evaluations through conferences.

EVALUATION RESOURCES

The second mechanism for evaluation funding is the legislative set-aside authority which permits the Secretary of HHS to use a proportion of overall program funds for evaluation purposes. The largest of such set-aside authorities is one established for evaluations conducted by several agencies of the U.S. Public Health Service (AHCPR, CDC, HRSA, NIH, and SAMHSA), ASPE, and the Office of Public Health and Science in the Office of the Secretary. The mechanism is called the 1-percent evaluation set-aside legislative authority, which is provided for in Section 241 of the Public Health Service (PHS) Act. This authority was established in 1970, when the Congress amended the Act to permit the HHS Secretary to use up to 1 percent of appropriated funds to evaluate authorized programs. Section 241 limits the base from which 1 percent of appropriated funds can be reserved for evaluations of programs authorized by the PHS Act. Excluded are funds appropriated for FDA, IHS1, and certain other programs that are managed by PHS agencies but not authorized by the Act (e.g., HRSA's Maternal and Child Health Block Grant and CDC's National Institute for Occupational Safety and Health). In FY 1998, HHS invested more than $35 million in 1-percent evaluation funds to carry out evaluation activities related to relevant public health programs. The FY 1998 Labor, HHS, and Education Appropriations Act provided that an additional $115 million in 1-percent evaluation funds be used to support data collection and analysis activities in AHCPR and CDC's National Center for Health Statistics.

EVALUATION MANAGEMENT
The management of HHS evaluations, which are carried out on a regular basis by the HHS agencies and offices and coordinated by ASPE, involves four basic functions: (1) evaluation planning and coordination, (2) project management, (3) quality assurance, and dissemination of evaluation reports. A description of each function in general terms follows. Additional information on the individual HHS agencies, ASPE, and OPHS evaluation functions is found in chapter III.

Evaluation Planning and Coordination
The HHS agencies, ASPE, the Office of the Inspector General (OIG), and OPHS develop evaluation plans annually in concert with HHS's program planning, legislative development, and budgeting cycles. Plan development is coordinated by ASPE. Each agency or office plan generally states the evaluation priorities, or projects under consideration for implementation. Typically, HHS evaluation priorities include congressionally mandated program evaluations, evaluations of Secretarial program or policy initiatives, and assessments of new programs, programs that are candidates for reauthorization, or programs for which key budget decisions are anticipated.

More specifically, HHS evaluation planning activities involve preparing two reports to the Congress. First, those agencies and offices that use the PHS 1-percent evaluation set-aside authority--AHCPR, CDC, HRSA, NIH, ASPE, OPHS, SAMHSA--submit a formal plan to ASPE, which coordinates and develops the individual plans into the HHS report to the Congress on the use of the PHS 1-percent authority. This report must be submitted to the Congress before HHS can implement the plan.

Second, the Congress requests that HHS coordinate all of its research, demonstration, and evaluation programs to ensure that the results of these projects address HHS program goals and objectives. ASPE and the Assistant Secretary for Management and Budget work together with HHS agencies to provide the Congress with a special annual research, demonstration, and evaluation budget plan that coincides with the preparation of the President's fiscal year budget. The plan outlines each HHS agency's research, demonstration, and evaluation priorities as related to its strategic goals and objectives.

Project Management
The design and management of evaluation projects at HHS is principally decentralized--the HHS agencies, OIG, and ASPE all are responsible for executing annual evaluation plans, developing evaluation contracts, and disseminating and applying evaluation results. Even within agencies-- while there is some oversight responsibility and execution capability in the Office of the Director or Administrator--the various subunits (centers, institutes, and bureaus) conduct much of the day-to-day evaluation activity.

The OIG performs independent evaluations through its Office of Evaluations and Inspections (OEI). The OEI's mission is to improve HHS programs by conducting inspections that provide timely, useful, and reliable information and advice to decisionmakers. This information (findings of deficiencies or vulnerabilities and recommendations for corrective action) is usually disseminated through inspection reports issued by the Inspector General. A summary of individual inspection reports and other OIG reports can be viewed on the Internet at http://www.sbaonline.sba.gov/ignet. OEI also provides technical assistance to HHS agencies in conducting their evaluations.

Quality Assurance
Most evaluation projects are developed at the program level. The initial quality review is generally conducted by a committee of agency- or office-level policy and planning staff members. Before a project is approved, however, it is also reviewed for technical quality by a second committee with expertise in evaluation methodology. Technical review committees follow a set of criteria for quality evaluation practice established by each agency. Some HHS agencies also have external evaluation review committees composed of evaluation researchers and policy experts from universities and research centers. More details on the quality assurance procedures for the various HHS agencies, ASPE, and OPHS are presented in chapter III.

Dissemination of Evaluation Reports
Maintaining report libraries and distributing information on evaluation results is an important component of HHS evaluation management. Project information and reports are continuously submitted to the HHS Policy Information Center (PIC)--the departmental evaluation data base and library maintained by ASPE. As an information data base and library resource, the PIC contains nearly 7,000 completed and in-progress evaluation and policy research studies conducted by the Department, as well as key studies completed outside of HHS by the U.S. General Accounting Office (GAO) and private foundations.

Typically, the results of HHS evaluations are disseminated through targeted distribution of final reports, articles in refereed journals, and presentations at professional meetings and conferences. Although individual HHS agencies have primary responsibility for disseminating results, there is a departmentwide effort under way to expand dissemination to the larger research and practice communities through centralized computer communications and publications. Abstracts of all studies maintained in the PIC evaluation data base are now accessible on the Internet at http://www.hhs.gov. Information is available on completed projects, including the name and telephone number of an HHS official responsible for the project.

In addition, HHS widely distributes copies of the annual evaluation report series, beginning with Performance Improvement 1995 and continuing to this report, through the HHS website. For convenience, these reports can be accessed on the Internet at: http://aspe.hhs.gov/PIC/gate2pic.htm.

FDA programs are principally authorized by legislation other than the PHS Act, specifically the authority of the Agriculture, Rural Development, Food and Drug Administration and Related Agencies Appropriations Act. IHS programs are authorized under the Indian Health Care Improvement Act and the Indian Self-Determination Act, and appropriated under the Department of the Interior and Related Agencies Appropriations