Evaluation plays an integral role in carrying out the HHS mission. Assessing various aspects of agency program performance allows staff to identify means of improving that performance. The HHS evaluation function has three goals: (1) to provide information on HHS programs that helps government officials and members of the Congress make decisions related to programs, policies, budgets, and strategic planning; (2) to help HHS managers improve program operations and performance; and (3) to disseminate evaluation results and methodological tools useful to the larger health and human services community of State and local officials, researchers, advocates, and practitioners for improving the performance of their programs.
This appendix presents the organization and operation of evaluation at HHS. Described first is the definition and basic purposes of evaluation at HHS, followed by information the resources and funding mechanisms used to support evaluations. The last section of the appendix provides more details on various aspects of the management of HHS evaluation activities, such as the role played in departmental planning initiatives, how evaluation projects are managed; how HHS agencies ensure technical quality of evaluation projects, and promote effective dissemination and use of evaluation results.
DEFINITION AND PURPOSE OF EVALUATION AT HHS
HHS defines evaluation as the assessment of the performance (efficiency, effectiveness, and responsiveness) of HHS programs or strategies through the analysis of data or information collected systematically and ethically, and the effective use of resulting information in strategic planning, program or policy decision making and program improvement. As mentioned in the foreword of this report, evaluations serve one or more of the following purposes:
Program effectiveness provides a way to determine the impact of HHS programs on achieving intended goals and objectives.
Performance measurement is the primary mechanism used to monitor annual progress in achieving departmental strategic and performance goals. To support performance measurement, we are investing evaluation funds to develop and improve performance measurement systems and the quality of the data that supports those systems.
Environmental assessment is the way we understand the forces of change in the health and human services environment that will influence the success of our programs and the achievement of our goals and objectives. In turn, this understanding allows us to adjust our strategies and continue to deliver effective health and human services.
Program management reflects the need of program managers to obtain information or data helpful for effectively designing and managing a program. These evaluations generally focus on developmental or operational aspects of program activities and provide understanding of services delivered and populations served.
Evaluation activities of the various HHS agencies are largely supported through two funding mechanisms: direct use of program funds and use of special legislative set-aside authorities for evaluation. The first is a common mechanism by which programs managers have discretionary authority to use appropriated program funds to support contracts that will design, implement, and analyze evaluation data. In some cases, a programs legislative authority calls for a specially mandated evaluation, and program funds are used directly to support the evaluation.
The second mechanism for evaluation funding is the legislative set-aside authority which permits the Secretary of HHS to use a proportion of overall program funds for evaluation purposes. The largest of such set-aside authorities at HHS is one established for evaluations conducted by several agencies of the U.S. Public Health Service (AHRQ, CDC, HRSA, NIH, and SAMHSA), ASPE, and the Office of Public Health and Science (OPHS) in the Office of the Secretary. The mechanism is called the 1-percent evaluation set-aside legislative authority, which is provided for in Section 241 of the Public Health Service (PHS) Act. This authority was established in 1970, when the Congress amended the Act to permit the HHS Secretary to use up to 1 percent of appropriated funds to evaluate authorized programs. Section 241 limits the base from which 1 percent of appropriated funds can be reserved for evaluations of programs authorized by the PHS Act. Excluded are funds appropriated for FDA, IHS1, and certain other programs that are managed by PHS agencies but not authorized by the Act (e.g., HRSAs Maternal and Child Health Block Grant and CDCs National Institute for Occupational Safety and Health).
The management of HHS evaluations, which are carried out on a regular basis by the HHS agencies and offices and coordinated by ASPE, involves four basic functions: (1) evaluation planning and coordination, (2) project management, (3) quality assurance, and (4) dissemination of evaluation reports. A description of each function in general terms follows.
Evaluation Planning and Coordination
The HHS agencies, ASPE, the Office of the Inspector General (OIG), and OPHS develop evaluation plans annually in concert with HHSs program planning, legislative development, and budgeting cycles. Plan development is coordinated by ASPE. Each agency or office plan generally states the evaluation priorities, or projects under consideration for implementation. Typically, HHS evaluation priorities include: congressionally-mandated program evaluations, evaluations of Secretarial program or policy initiatives, assessments of new programs and ones that are candidates for reauthorization, and evaluations that support program performance management and accountability.
More specifically, HHS evaluation planning activities are coordinated with three department-wide planning initiatives. First, HHS evaluation activities support the Departments strategic planning and performance management activities in several ways. Completed evaluation studies are used in shaping the specific HHS strategic goals and objectives. Evaluation findings provide an important source of information or evidence about the success of various HHS programs or policies that collectively make up the strategies to achieve the goals and objectives. The HHS Strategic Plan highlights evaluations that document efficacy or effectiveness of strategic programs or policies and lists future evaluations that will benefit strategic planning. HHS agencies are also using findings from their evaluations to support GPRA annual performance reporting to Congress and program budget justifications.
Secondly, Congress requests that HHS coordinate all of its research, demonstration, and evaluation programs to ensure that the results of these projects address HHS program goals and objectives. ASPE and the Assistant Secretary for Management and Budget work together with HHS agencies to provide the Congress with a special annual research, demonstration, and evaluation budget plan that coincides with the preparation of the Presidents fiscal year budget. The plan outlines HHS agency research, demonstration, and evaluation priorities as related to the Departments strategic goals and objectives.
Thirdly, those agencies and offices that use the PHS one-percent evaluation set-aside authorityAHRQ, CDC, HRSA, NIH, ASPE, OPHS, SAMHSAsubmit a formal plan to ASPE, which coordinates and develops the individual plans into the HHS report to the Congress on the use of the PHS one-percent authority. This report must be submitted to the Congress before HHS can implement the plan.
The design and management of evaluation projects at HHS is principally decentralizedthe HHS agencies, OIG, and ASPE all are responsible for executing annual evaluation plans, developing evaluation contracts, and disseminating and applying evaluation results. Even within agencies while there is some oversight responsibility and execution capability in the Office of the Director or Administratorthe various subunits (centers, institutes, and bureaus) conduct much of the day-to- day evaluation activity.
The OIG performs independent evaluations through its Office of Evaluations and Inspections (OEI). The OEIs mission is to improve HHS programs by conducting inspections that provide timely, useful, and reliable information and advice to decision makers. This information (findings of deficiencies or vulnerabilities and recommendations for corrective action) is usually disseminated through inspection reports issued by the Inspector General. A summary of individual inspection reports and other OIG reports can be viewed on the Internet at <http://www.hhs.gov/oig/oei>. OEI also provides technical assistance to HHS agencies in conducting their evaluations.
Quality Assurance and Improvement
Most evaluation projects are developed at the program level. The initial quality review is generally conducted by a committee of agency- or office-level policy and planning staff members. Before a project is approved, however, it is also reviewed for technical quality by a second committee with expertise in evaluation methodology. Technical review committees follow a set of criteria for quality evaluation practice established by each agency. Some HHS agencies also have external evaluation review committees composed of evaluation researchers and policy experts from universities and research centers.
Since HHS began reporting to Congress in 1995 on evaluations completed and in-progress through thePerformance Improvement report series, the Department has focused more attention on the improving the quality of evaluation studies performed every year. The HHS Evaluation Review Panel, convened yearly to recommend evaluations for highlighting in Chapter II of this report, has contributed insights to HHS evaluation officers on the strengths and challenges of ensuring quality evaluation studies. HHS evaluation officers have had opportunities to discuss these strengths and challenges with the Panel and to identify steps to improve agency evaluation capacity.
Based on these discussions, ASPE has initiated two evaluation capacity-building projects. The first project was to establish an evaluation consultant Blanket Purchase Agreement (BPA) contract mechanism to assist agency program managers in obtaining short terms assistance in developing evaluation designs or monitoring the performance quality of evaluation contractors. The second project is conducting a department-wide evaluation training needs assessment survey. The results will be available in the fall of 2002.
Dissemination of Evaluation Reports
Maintaining report libraries and distributing information on evaluation results is an important component of HHS evaluation management. The Departments information and reports on major evaluations is available centrally through the website of the HHS Policy Information Center (PIC), located at the following address: <http://aspe.hhs.gov/pic/>. The PIC website offers users an opportunity to search (by key word searching or by selected program or policy topics) the departmental evaluation database and electronic report library maintained by ASPE. As an information database and library resource, the PIC contains nearly 7,000 completed and in-progress evaluation and policy research studies conducted by the Department, as well as key studies completed outside of HHS by the U.S. General Accounting Office (GAO) and private foundations.
Typically, the results of HHS evaluations are disseminated through targeted distribution of final reports, articles in refereed journals, and presentations at professional meetings and conferences. Although individual HHS agencies have primary responsibility for disseminating results, the ASPE will continue to its Department-wide efforts to expand dissemination of evaluation results to the larger research and practice communities through centralized computer communications and publications like the PIC website and project information database.
- FDA programs are principally authorized by legislation other than the PHS Act, specifically the authority of the Agriculture, Rural Development, Food and Drug Administration and Related Agencies Appropriations Act. IHS programs are authorized under the Indian Health Care Improvement Act and the Indian Self-Determination Act, and appropriated under the Department of the Interior and Related Agencies Appropriations. (back)