MISSION: To generate and disseminate information that improves the health care system.
AHCPR Evaluation Program
AHCPR has designed a portfolio that responds to the evaluation needs for the following:
- Comprehensive information to assess AHCPR's effectiveness in meeting its major and long-term priorities and goals.
- Information from "fast-track" or quick turnaround projects to respond to critical AHCPR, Public Health Service, and HHS concerns.
- Information from internal evaluations to improve the efficiency with which AHCPR performs its work.
Evaluation efforts are built into virtually all AHCPR program activities. The evaluation mechanisms used by the agency include targeted evaluation studies undertaken through contracts or grants; peer review of grant applications and technical review of contract proposals for scientific integrity; obtaining information on the usefulness of AHCPR research efforts and findings through AHCPR's User Liaison Program, which provides information to State policymakers, health departments, and officials; and other targeted efforts such as focus groups and surveys to provide baseline information and assist the design of future AHCPR work.
AHCPR established a two-tier system for assessing proposed evaluation projects. The process begins with an executive-level review to assess policy relevance and relative priority of proposed projects. The review is conducted by the Administrator and senior staff. The second, or technical merit, review assesses policy-relevant project proposals for feasibility, soundness of design, costs, potential importance of the findings, and relation to ongoing evaluation activities. The second review is conducted by the Task Force on AHCPR Evaluation Projects, a group consisting of one individual with evaluation expertise from each office and center in AHCPR.
Summary of Fiscal 1995 AHCPR Evaluations
The 12 AHCPR evaluation projects completed in fiscal 1995 reflect AHCPR's mission of increasing scientific knowledge for improved clinical decisionmaking and for organizing public and private systems of health care delivery. The projects are part of AHCPR's efforts to develop information that will improve health care quality, reduce health care costs, and enhance access to health services; respond to information needs of consumers, clinical practitioners, payers, and policymakers; and promote dissemination and use of new information about health care systems, methods, and technologies.
The first group of projects consists of four studies to advance methodologies for evaluating and improving the quality of clinical care. The first project, a contract with the Center for Clinical Quality Evaluation in Washington, D.C., translated three AHCPR-supported clinical practice guidelines (on urinary incontinence, acute postoperative pain, and benign prostatic hyperplasia, or BPH) into clinical performance measurement systems. The project found it is possible to develop valid and reliable guideline-based performance measures from AHCPR- supported guidelines, and the structure of guideline recommendations can have a strong influence on developing such measures. The project also found that targeted educational efforts conducted by Medicare peer review organizations and based on the BPH guideline resulted in improved performance on specific aspects of clinical care.
The second project, a contract with Rand of Santa Monica, California, developed a quality review system based on the AHCPR- supported cataract guideline. This review system was designed for use in a range of settings including ophthalmologists in fee-for- service private practice, salaried ophthalmologists on staff at model health maintenance organizations (HMOs) or other managed care organizations, and ophthalmologists providing eye care services under capitated contracts or subcontracts with capitated primary care groups or other managed care organizations. The project found that it is feasible to develop performance measures based on the cataract guideline and that a medical record-based review system can be very useful for estimating guideline adherence and can highlight variations in practice and opportunities for quality improvement.
The third project, also conducted by Rand, developed a review system based on the guideline for prediction and prevention of pressure ulcers. The pressure ulcer guideline review system was developed and tested in collaboration with the Department of Veterans Affairs (VA) and was tested among VA and non-VA hospitals. The study revealed that because pressure ulcer care is multidisciplinary and recommends daily performance of many behaviors, the review systems required collection of data elements from a variety of data sources over a statistical sampling of hospital days.
In addition to developing clinical performance measures from science-based guidelines, these first three studies pointed to the need for organized and comprehensive information on the types of clinical performance measures in existence as well as the need for methods for evaluating the structure and quality of these measures. The fourth project therefore developed a typology or framework for collecting and evaluating clinical performance measures. AHCPR's fiscal 1995 measurement typology projects, described in chapter II, established a common language that can be used to compare measures. It also identified areas where additional clinical quality measures are needed.
Another category of projects completed in fiscal 1995 responds to the information needs of consumers, clinical practitioners, payers, and policymakers. The first of these projects, the Survey Design Project (SDP), developed prototype survey modules to obtain consumer assessments of their access to care, use of services, health outcomes, and patient satisfaction. The SDP forms the basis of AHCPR's Consumer Assessments of Health Plans Study (CAHPS), a five-year project to develop and test questionnaires that assess health plans and services, produce easily understandable reports for communicating survey data to consumers, and evaluate the usefulness of these reports for consumers in selecting health care plans and services. The second project in this category, using a qualitative case study approach, identified factors related to the selection, implementation, and use of information management systems. Project findings guided the development of an AHCPR research solicitation focused on understanding computerized decision support systems for clinical quality improvement. The third project in this category focused on improved clinical information from managed care settings. Conducted by the Institute for Health Policy Studies at the University of San Francisco, this study assessed the feasibility of using managed care data to assess the differential use of selected tests and procedures. The final project in this category developed a directory of minority health and human services data resources that can be used by researchers and others interested in examining data that contain race and ethnic identifiers.
The final group of projects completed in fiscal 1995 concentrated on evaluating the usefulness of AHCPR-supported clinical practice guidelines. A project conducted by George Washington University developed analytical designs and methods that could be used to evaluate the process of developing clinical practice guidelines. The project identified key variables related to the process of guideline development and assessed the feasibility of measuring the variables across guidelines. The next study, conducted by the American Institutes for Research, collected detailed and structured information from guideline users on issues associated with implementation of guidelines and their perceived impact. Approximately 100 physicians in office-based settings, physicians in academic settings, and nurses participated in 10 focus groups across the United States and provided information on how better to construct and disseminate information on improving clinical practice. A third project, conducted by the Institute of Medicine, described and evaluated methods for setting priorities for guideline topics. The final report in this category, AHCPR Clinical Practice Guidelines Program: Report to Congress, summarizes the Agency's evaluation work to date and highlights future directions for the Agency's work.
These findings, as well as those from other AHCPR-supported evaluation projects, suggest that the methods for introducing and implementing guidelines into clinical settings are extremely important in fostering conformance with science-based clinical recommendations. As a result of findings from this and other projects, AHCPR will no longer be developing clinical practice guidelines but instead will focus on science-based partnerships to improve health care delivery. This new three-faceted strategy will continue AHCPR's commitment to the synthesis of scientific and medical effectiveness information, establish public-private partnerships to make guidelines and other quality-related information accessible through a national clearinghouse, and continue to support research and evaluation focusing on methods to integrate scientific and medical effectiveness information into everyday clinical practice.
AHCPR Evaluations in Progress
AHCPR's ongoing evaluation activities also concentrate on generating information to improve the health care system. In the area of information for clinical quality improvement, AHCPR has supported two followup efforts to the fiscal 1995 measurement typology project, described in chapter II. The first is the development of CONQUEST 1.0, the COmputerized Needs-Oriented QUality Measurement Evaluation SysTem. The CONQUEST project builds on its predecessor project in three ways. First, the project evaluates and strengthens the framework developed in the typology project by verifying the data with measure developers. Second, it creates an interlocking database of clinical conditions with coded information from AHCPR-supported and other clinical practice guidelines, so users can identify practice recommendations and link them to clinical quality measures. Third, the project creates a computerized system with a user- friendly interface to link measures to clinical information and guide the selection of measures. A related project currently under way involves evaluating this product by convening users to pilot test CONQUEST and to participate in focus groups about its usefulness.
Another project under way in fiscal 1996 builds on AHCPR's consumer choice work. The survey design project conducted by Research Triangle Institute and CAHPS develop surveys to collect data on consumers' satisfaction with access to and use of health services and methods to disseminate this information for improved consumer choice. Through the Survey Users Network, a related contract, AHCPR will conduct needs assessments with users to obtain information on how they use consumer surveys as well as their technical assistance needs in issues such as sampling, survey administration, data analysis, and reporting. The needs assessments conducted under the survey will help inform AHCPR of the CAHPS' work.
New Directions for AHCPR Evaluation
In the coming years AHCPR will focus its evaluation activities on three general areas. The first area is the need to develop AHCPR- wide performance measures that can be used to evaluate the quality of the Agency's work as well as to inform policymaking, budget planning, and program management. Efforts are under way to develop evaluation designs that will identify key customers or users of AHCPR information, identify their information needs, and obtain feedback on the usefulness of the Agency's products and reports.
The second general area centers on building AHCPR's research portfolio to translate research findings into forms of information that actively help consumers, practitioners, payers, and others make effective health care decisions. This area builds on research related to decision-support systems and hospital information systems, as well as psychosocial and organizational research examining factors that influence provider practice and clinical quality improvement.
The third area will build on AHCPR's work to examine and evaluate changes in the health care system, and how those changes affect access to care, use of services, quality of care, and patient outcomes. Through the use of targeted evaluation projects, analyses of data from the Medical Expenditure Panel Survey, and other media, this area of research will help HHS evaluate the effect of policy changes. For example, one area might focus on comprehensive evaluations and evaluation syntheses of the effect of managed care on vulnerable populations--and specifically, the impact of system changes such as the Section 1115 demonstration waivers on Medicaid beneficiaries, on health professionals, and on public health organizations and services within and across States.