| |
Office of the Assistant Secretary |
Policy Information Center |
The mission of the U.S. Department of Health and Human Services (HHS) is to enhance the health and well-being of Americans by providing for effective health and human services and by fostering strong, sustained advances in the sciences underlying medicine, public health, and social services. The following HHS agencies and offices accomplish this goal through their program activities and performance evaluations.
The Assistant Secretary for Planning and Evaluation (ASPE), located in the Office of the Secretary, coordinates evaluation activities throughout HHS.
Evaluation plays an integral role in carrying out the HHS mission. Assessing various aspects of agency program performance allows staff to identify means of improving that performance. The HHS evaluation function has three goals: (1) to provide information on HHS programs that helps government officials and members of the Congress make decisions related to programs, policies, budgets, and strategic planning; (2) to help HHS managers improve program operations and performance; and (3) to disseminate evaluation results and methodological tools useful to the larger health and human services community of State and local officials, researchers, advocates, and practitioners for improving the performance of their programs. This last goal is very important to HHS. Its agencies have an important obligation to foster the development of new knowledge about the effectiveness of health and human services programs and interventions and evaluation tools for use by the larger health and human services community. Although the findings and recommendations of HHS evaluations are usually first used by the Administration and the Congress, they can also be applied by others in the research and practice communities to improve program performance at the State and community levels. The purpose of this report is to disseminate information about recent HHS evaluations and to make sure that the potential for wider application is realized.
This Chapter describes the organization and operation of evaluation at HHS. It first provides an overview of the kinds of evaluation activities supported by HHS agencies and then describes the resources and funding mechanisms used to support them. The Chapter details HHS evaluation management, including planning procedures, project management, quality assurance, and dissemination of results. The Chapter concludes with a discussion of effective uses of HHS evaluations, with illustrations from projects completed in the past year.
HHS defines evaluation as the assessment of program performance (efficiency, effectiveness, and responsiveness) through the analysis of data or information collected systematically and ethically, and the effective use of resulting information in program or policy decisionmaking and program management. This definition encompasses a range of evaluation activities, such as those listed below.
Evaluation activities of the various HHS agencies are largely supported through two funding mechanisms: direct use of program funds and use of special legislative set-aside authorities for evaluation. The first is a common mechanism by which programs managers have discretionary authority to use appropriated program funds to support contracts that will design, implement, and analyze evaluation data. In some cases, a program's legislative authority calls for a specially mandated evaluation, and program funds are used directly to support the evaluation.
The second mechanism for evaluation funding is the legislative set-aside authority which permits the Secretary of HHS to use a proportion of overall program funds for evaluation purposes. The largest of such set-aside authorities is one established for evaluations conducted by several agencies of the U.S. Public Health Service (AHCPR, CDC, HRSA, NIH, and SAMHSA), ASPE, and the Office of Public Health and Science in the Office of the Secretary. The mechanism is called the 1-percent evaluation set-aside legislative authority, which is provided for in Section 241 of the Public Health Service (PHS) Act. This authority was established in 1970, when the Congress amended the Act to permit the HHS Secretary to use up to 1 percent of appropriated funds to evaluate authorized programs. Section 241 limits the base from which 1 percent of appropriated funds can be reserved for evaluations of programs authorized by the PHS Act. Excluded are funds appropriated for FDA, IHS(1), and certain other programs that are managed by PHS agencies but not authorized by the Act (e.g., HRSA's Maternal and Child Health Block Grant and CDC's National Institute for Occupational Safety and Health). In FY 1997, HHS invested more than $35 million in 1-percent evaluation funds to carry out evaluation activities related to relevant public health programs. The FY 1997 Labor, HHS, and Education Appropriations Act provided that an additional $96 million in 1-percent evaluation funds be used to support data collection and analysis activities in AHCPR and CDC's National Center for Health Statistics.
The management of HHS evaluations, which are carried out on a regular basis by the HHS agencies and offices and coordinated by ASPE, involves four basic functions:
1. Evaluation planning and coordination
2. Project management
3. Quality assurance
4. Dissemination of evaluation reports.
A description of each function in general terms follows. Additional information on the individual HHS agencies, ASPE, and OPHS evaluation functions is found in Chapter III.
Evaluation Planning and Coordination
The HHS agencies, ASPE, the Office of the Inspector General (OIG), and OPHS develop evaluation plans annually in concert with HHS's program planning, legislative development, and budgeting cycles. Plan development is coordinated by ASPE. Each agency or office plan generally states the evaluation priorities, or projects under consideration for implementation. Typically, HHS evaluation priorities include congressionally mandated program evaluations, evaluations of Secretarial program or policy initiatives, and assessments of new programs, programs that are candidates for reauthorization, or programs for which key budget decisions are anticipated.
More specifically, HHS evaluation planning activities involve preparing two reports to the Congress. First, those agencies and offices that use the PHS 1-percent evaluation set-aside authority--AHCPR, CDC, HRSA, NIH, ASPE, OPHS, SAMHSA--submit a formal plan to ASPE, which coordinates and develops the individual plans into the HHS report to the Congress on the use of the PHS 1-percent authority. This report must be submitted to the Congress before HHS can implement the plan.
Second, the Congress requests that HHS coordinate all of its research, demonstration, and evaluation programs to ensure that the results of these projects address HHS program goals and objectives. ASPE and the Assistant Secretary for Management and Budget work together with HHS agencies to provide the Congress with a special annual research, demonstration, and evaluation budget plan that coincides with the preparation of the President's fiscal year budget. The plan outlines each HHS agency's research, demonstration, and evaluation priorities as related to its strategic goals and objectives.
Project Management
The design and management of evaluation projects at HHS is principally decentralized--the HHS agencies, OIG, and ASPE all are responsible for executing annual evaluation plans, developing evaluation contracts, and disseminating and applying evaluation results. Even within agencies-- while there is some oversight responsibility and execution capability in the Office of the Director or Administrator--the various subunits (centers, institutes, and bureaus) conduct much of the day-to-day evaluation activity.
The OIG performs independent evaluations through its Office of Evaluations and Inspections (OEI). The OEI's mission is to improve HHS programs by conducting inspections that provide timely, useful, and reliable information and advice to decisionmakers. This information (findings of deficiencies or vulnerabilities and recommendations for corrective action) is usually disseminated through inspection reports issued by the Inspector General. A summary of individual inspection reports and other OIG reports can be viewed on the Internet at http://www.sbaonline.sba.gov/ignet. OEI also provides technical assistance to HHS agencies in conducting their evaluations.
Quality Assurance
Most evaluation projects are developed at the program level. The initial quality review is generally conducted by a committee of agency- or office-level policy and planning staff members. Before a project is approved, however, it is also reviewed for technical quality by a second committee with expertise in evaluation methodology. Technical review committees follow a set of criteria for quality evaluation practice established by each agency. Some HHS agencies also have external evaluation review committees composed of evaluation researchers and policy experts from universities and research centers. More details on the quality assurance procedures for the various HHS agencies, ASPE, and OPHS are presented in Chapter III.
Dissemination of Evaluation Reports
Maintaining report libraries and distributing information on evaluation results is an important component of HHS evaluation management. Project information and reports are continuously submitted to the HHS Policy Information Center (PIC)--the departmental evaluation data base and library maintained by ASPE. As an information data base and library resource, the PIC contains nearly 7,000 completed and in-progress evaluation and policy research studies conducted by the Department, as well as key studies completed outside of HHS by the U.S. General Accounting Office (GAO) and private foundations.
Typically, the results of HHS evaluations are disseminated through targeted distribution of final reports, articles in refereed journals, and presentations at professional meetings and conferences. Although individual HHS agencies have primary responsibility for disseminating results, there is a departmentwide effort under way to expand dissemination to the larger research and practice communities through centralized computer communications and publications. Abstracts of all studies maintained in the PIC evaluation data base are now accessible on the Internet at http://www.os.HHS.gov. Information is available on completed projects, including the name and telephone number of an HHS official responsible for the project.
In addition, HHS widely distributes copies of the annual evaluation report series, beginning with Performance Improvement 1995 and continuing to this report, through the HHS website. For convenience, these reports can be accessed on the Internet at: http://aspe.os.HHS.gov/PIC/gate2pic.htm.
ENSURING EFFECTIVE USE OF EVALUATION RESULTS
HHS is committed to ensuring that evaluations yield a high return on the investment of available program funds. Today, the Department's evaluation resources are used in several ways related to strategic planning, program, and policy development. Performance measurement and data systems are the primary mechanisms used to monitor progress in achieving departmental goals and objectives as well as specific program outcomes. Effectiveness of programs and strategies are in-depth evaluation studies to understand how HHS strategies and programs are linked to performance goals and objectives. Environmental assessments are prospective evaluation studies that assess how changes in the larger society affect the Department's programs and strategies. Program management and support evaluations are used to improve the management of health and social service programs and the quality of the Department's evaluation efforts. Each of these four uses, with examples of current HHS evaluations, is described below.
Performance Measurement and Data Systems
Implementation of the HHS strategic plan and the performance plans of the PHS agencies, pursuant to the Government Performance and Results Act (GPRA) of 1993, requires the Department to focus a portion of its evaluation activities on the development of performance objectives and measures and information systems necessary to produce the data needed to assess progress toward achieving its goals. During FY 1997, evaluation priorities of the individual HHS agencies included a number of projects serving this purpose. For example, ACF completed a project for Head Start Programs to develop outcome performance measures (e.g., how these children benefit from the program), in addition to process measures (e.g., how many teachers have the appropriate credentials?) (6693). FDA completed its fifth and final report on performance data for the Prescription Drug User Fee Act (PDUFA), documenting the improvements obtained in the last few years in speeding up the process for drug reviews and approvals (6079.2). ASPE produced a report synthesizing the state-of-the-art performance measurement for public health and developed recommendations for measures that could be used to monitor Federal-State performance partnership grants being considered for the future (6177). An example of a major survey providing nationally representative data is one completed last year by HRSA on the users of community health centers (CHC) and the services they were provided (5737). It produced, for the first time, nationally representative data on CHC users that is comparable to similar data on health care use from the National Health Interview Survey. The data are now being analyzed to address questions about users' race and ethnicity, health risk behaviors, most prevalent diagnoses, and services used.
Effectiveness of Programs and Strategies
Implementation of the Department's strategic and performance plans also requires evaluative information on how well the programs and strategies are working. Most of the HHS agency evaluations completed in FY 1997 provide this information. For example, HCFA submitted a report to the Congress on the quality of care in the Medicaid program (6302). It assessed the variations in the rate of performance of selected treatments and procedures on Medicaid beneficiaries for small areas within and between the States. Medical records in a sample of hospitals were also examined for three conditions: pediatric asthma, hysterectomy, and complicated delivery. Overall, the report concluded that care was considered adequate and comparable to that received by privately insured patients. Another example of a focused program effectiveness study is one conducted by HRSA on rural applications of telemedicine (5749). The project included a mail survey of all non-Federal hospitals located outside metropolitan areas in the summer of 1995, plus a short survey for hospitals doing only teleradiology and a longer survey for hospitals applying telemedicine to purposes beyond radiology. The study demonstrated that although rural telemedicine is in the earliest stages of development, it is expanding rapidly. HRSA is using the results to design a common telemedicine evaluation instrument for HRSA grantees and for consideration by other Federal sponsors, as well as in shaping the Rural Telemedicine Grant Program. In terms of evaluating research programs, NIH conducted an evaluation of the NIH Consensus Conference as an effective mechanism for translating research into practice (6284).
Environmental Assessment
The Department's strategic plan acknowledges that achievement of its goals and objectives is contingent on many external factors that are beyond the Department's control. For example, the managed care revolution is having a significant impact on access to services and health outcomes for those populations traditionally served by the public health system. Understanding the impact of these environmental forces on public health programs and the customers they serve is essential for assessing and adjusting the Department's goals and strategies in the future. A number of evaluations in FY 1997 addressed these environmental issues.
For example, AHCPR conducted a study to help clarify its future priorities in outcomes and effectiveness research and to provide insights regarding future opportunities for public-private partnerships in this area (6385). The study found that many private sector organizations conduct effectiveness research, often to monitor health care performance against accreditation standards. AHCPR will also use the findings to help guide its support to developing and validating effectiveness and outcome research tools used by the private sector. In another example, ASPE last year analyzed market barriers to the development of pharmacotherapies for substance abuse and addiction, particularly for abuse of and addiction to cocaine (6694). This report, featured in Chapter II, found several critical market barriers that must be taken into account in future HHS program and policy development. Since the substance abuse treatment market relies heavily upon State and Federal reimbursement, most substance abuse treatment services are subsumed under the mental health benefits of entitlement programs. Drug companies are reluctant to rely upon this kind of reimbursement in an age of shrinking budgets for mental health services.
Program Management and Support
Effective management of programs that achieve departmental goals and objectives is essential to the success of those programs. In FY 1997, the evaluations completed and in progress include a number of priorities devoted to assessing and improving the management of health and social service programs. For example, HCFA last year looked at the implementation of Ambulatory Patient Groups (APGs)--the Medicaid outpatient prospective payment system that groups patients for payment purposes rather than paying on a cost basis (6320). In this case study of Iowa's implementation of the APG system and an analysis of the reimbursement methodology, success was reported in reducing outpatient costs, where that was the immediate goal, and the system encouraged higher-cost facilities to reduce costs and reward lower-cost facilities. In another example of studies to improve program operations, OPHS conducted a Healthy People 2000 stakeholder study to help plan and prepare for the Healthy People 2010 goals and objectives (6491), also featured in Chapter II. This evaluation reviewed the successes and failures of the Healthy People 2000 benchmarks, with the hope of making this national framework of performance indicators on health status more results-oriented. The focus group participants recommended new communication avenues to make Healthy People 2010 available to more professionals and community leaders.