MISSION: To generate and disseminate information that improves the health care system.
AHCPR Evaluation Program The Agency for Health Care Policy and Research (AHCPR) has designed a portfolio that responds to three types of evaluation needs:
- The need for comprehensive information to assess the Agency's effectiveness in meeting its major and long-term priorities and goals.
- The need for information from fast-track or quick-turnaround projects to respond to critical Agency, PHS, and departmental concerns.
- The need for information from internal evaluations to improve the efficiency of key program areas.
To address these evaluation needs, an evaluation component is built into each AHCPR project. The evaluation mechanisms, which vary depending on the project, include special evaluation studies; peer review of grant applications and subsequent review by the National Advisory Council for Health Care Policy, Research, and Evaluation; the User Liaison Program (which provides information on the value of our research to State policymakers); and other evaluation efforts such as focus groups to provide feedback on AHCPR products and to provide baseline information to assist in the design of future evaluation projects.
AHCPR received delegated authority to review evaluation projects in 1992 and, as part of delegated review, established a formal structure to review proposed evaluation projects that coincides with the review of concepts for all other Agency projects. Thus, review of proposed evaluation contracts is integrated into the Agency's formal planning and budget process.
AHCPR has implemented a two-tier process for reviewing evaluation projects eligible for 1 percent set-aside funds. The process begins with an executive evaluation review to assess the policy relevance and relative priority of proposals submitted. This review is conducted by the Administrator and senior staff. The second, or technical, review assesses policy-relevant project proposals for feasibility, soundness of design, costs, potential importance of the findings, and relation to ongoing evaluation activity. This second review is conducted by the Task Force on AHCPR Evaluation Projects, a group consisting of one individual with evaluation expertise from each Office and Center.
Summary of FY 1994 Evaluations During fiscal year (FY) 1994, AHCPR completed six evaluations on two major topics: examining the process of developing clinical practice guidelines, and monitoring and evaluating health care delivery.
One of the statutory responsibilities of AHCPR is to arrange for the development, review, and revision of clinical practice guidelines. AHCPR-supported guidelines may be used by physicians, educators, and health care practitioners to help determine how diseases, disorders, and other health conditions can most effectively and appropriately be prevented, diagnosed, treated, and managed. The development of guidelines relates to AHCPR's strategic plan goal to determine what works best in clinical practice and to translate that knowledge into clinical decisionmaking. The first three AHCPR-supported evaluation studies completed in FY 1994 focused on strengthening the methodology underlying the guidelines by examining the strengths and limitations of current methods, and by identifying improved approaches and how they might be implemented.
The first project examined the methods used to conduct peer and pilot reviews for six of the guidelines. Peer and pilot review methods are used before a guideline is released to ensure its methodological soundness and scientific accuracy as well as to obtain information on its clarity, organization, and format. The study culminated in several recommendations for improving the quality of peer and pilot review.
Two other studies focused on methods for incorporating treatment cost analyses into the guideline development process. One examined the methodologies and data sources used in cost studies for eight of the guidelines and examined some of the difficulties in making cost estimates about various guideline treatment options. The other focused on identifying databases that could be used to provide information to generate cost estimates. This study found that no single database can be used to estimate the costs of recommendations in clinical practice guidelines. It is necessary to assemble data from several sources to estimate costs because claims databases sometimes lack information on certain services, and there are significant questions about the accuracy of data in various databases.
Information from these three studies has been used to inform two AHCPR reports to Congress on methods for developing guidelines and methods for including cost information in the guidelines. The findings have also been used to formulate a more uniform approach to guideline development based on what works best from the various approaches used to date.
The second area of emphasis among the AHCPR evaluations completed in FY 1994 relates to the Agency's strategic plan goal to monitor and evaluate health care delivery. Critical to fulfilling this goal is research that can be useful to policymakers by providing a greater understanding of the effects of emerging delivery systems and changes in the health care market structure. One such project focused on methods and issues related to the implementation and integration of advanced clinical information systems. The report presented a design for studying integrated health care information systems, including issues such as study feasibility, site selection, research design, and a data collection plan.
Another study examined inappropriate extension of hospitalization and the role of discharge planning, and identified research needs and potential existing data sources. Major findings include the following: 5 percent of extended hospital stays are not medically necessary; the cost of these stays was conservatively estimated at $420 million in 1987 dollars. Reasons for extended hospital stay include lack of sufficient nursing home beds and other alternate placement settings, delays in qualifying patients for posthospital benefits, and delays in appointing conservators for incompetent patients. Discharge planning appears to have only a small impact on extended hospital stays.
The final evaluation completed in FY 1994 assessed the availability of research on the cost-effectiveness of managed care health plans. The study found managed care plans had lower hospital utilization, greater use of less costly alternatives to expensive procedures and tests, greater use of preventive measures, comparable quality of care, and somewhat lower enrollee satisfaction generally, but greater satisfaction with cost. The study recommended additional research on managed care performance, including more comprehensive research projects to determine what works in managed care, rather than just whether managed care works.
Evaluations in Progress AHCPR currently supports seven evaluations that focus on its mission to generate and disseminate information that improves the health care system. AHCPR's ongoing projects provide information that will facilitate improvements in health care quality by advancing the science of quality measurement. In addition, AHCPR continues its commitment to improving the quality of AHCPR-supported clinical practice guidelines.
Two prominent FY 1995 projects relate to improving the science of quality measurement and improvement. The first, the Measurement Typology Project, was designed to collect and summarize information on clinical quality measures. These are technical quality measures used to estimate the extent to which health care providers deliver services that are appropriate for each patient's condition; whether services are provided safely, competently, and in an appropriate timeframe; and what the outcomes are from the services provided. The project summarizes 40 clinical performance measure sets--which contain a total of 1,287 performance measures--and provides a framework for evaluating the usefulness of various measures. The next phase of this project will expand and refine the typology, which ultimately will provide the foundation for an ongoing national resource of validated clinical quality measures that will assist in quality measurement and improvement.
The second project, to be completed in FY 1995, was designed to focus on consumer information needs. It developed a model consumer survey to examine how consumers perceive the quality of health care they receive, including their attitudes toward their access to care, use of specific health services, and health outcomes. This project and the Measurement Typology Project are complementary efforts to facilitate evaluation of quality from the perspectives of clinical quality assessment and consumer satisfaction.
AHCPR is also continuing its commitment to improving its clinical practice guidelines. Three projects focus on methods for improving the efficiency and effectiveness of guideline development, and three large-scale targeted evaluation projects will be completed during FY 1995. These projects will provide information on the usefulness of the guidelines for quality measurement and improvement.
An evaluation design study conducted by the George Washington University Center for Health Policy is examining methods for studying the process of guideline development. A follow-on study being conducted by the American Institutes for Research (AIR) is applying these evaluation design principles to examine factors that influence providers' views of the quality of various guidelines. Some key findings of the George Washington University study compare AHCPR's guidelines to those sponsored by other organizations. AHCPR's guidelines are found to address a broader audience (including a wider range of clinicians and clinical settings and a wider range of nonclinicians, including patients, families, payers, regulators, and purchasers); to have very detailed and comprehensive written policies and procedures for their development; to have more specialized expert resources on which to draw; and to use multiple methods for external review.
These differences have important implications for evaluating the credibility, usability, and potential impact of the AHCPR-supported guidelines on provider practice. Recommendations from this study, as well as those from the AIR follow-on project, are expected to be used to improve guideline development. AHCPR is also undertaking a project with the Institute of Medicine to examine optimal methods for selecting guideline topics.
In addition to projects to improve the guideline development process, AHCPR has sponsored a number of targeted evaluation projects. Through the use of evaluation contracts, cooperative agreements, and intramural research projects, AHCPR has supported evaluative efforts for guidelines on acute postoperative pain; urinary incontinence; benign prostatic hyperplasia; cataracts in adults; prediction and prevention of pressure ulcers; and depression in primary care.
Two targeted evaluation projects, to be completed in FY 1995, will provide important information on the feasibility of using the guidelines for quality measurement and improvement, as well as for insight into the difficulties of implementing guidelines. The first project develops, implements, and evaluates quality and utilization review criteria and educational outreach based on AHCPR-supported clinical practice guidelines. The project uses five Medicare Peer Review Organizations (PROs) to develop criteria based on three guidelines (urinary incontinence, acute postoperative pain, and benign prostatic hyperplasia [BPH]); develop and test training materials to use the criteria for case review; pilot test the criteria to assess intra- and inter-rater reliability; apply guideline-based review criteria; and compare guideline-based review with the review systems currently used by PROs. The PROs are playing an integral role in developing, implementing, and evaluating alternative educational outreach strategies based on the BPH guideline.
The second project is building on AHCPR's existing efforts to evaluate clinical practice guidelines by translating AHCPR-supported guidelines into medical review criteria for use in a variety of care settings including hospitals, health maintenance organizations (HMOs), ambulatory clinics, and physicians' offices. This project, like the project described in the previous paragraph, will assess the usefulness of alternative educational interventions for disseminating the guidelines and review criteria and for changing provider practice behavior. One component of the project focuses on cataract in adults, and the other on pressure ulcers in adults. The criteria for this project will be developed with the Department of Veterans Affairs (VA) and will be tested in a number of VA facilities. In a follow-on project, AHCPR and the VA will test alternative methods for disseminating the guideline and using the criteria as tools to assess the effect of guidelines on practice patterns.
New Directions for Evaluation AHCPR is in the process of developing a new strategic plan that reexamines current activities in relation to a rapidly changing health care marketplace. This new direction recognizes that AHCPR's programs must generate the information and tools needed to improve health care delivery and its outcomes. In addition to generating new knowledge, AHCPR's portfolio will focus on translating research findings into forms of information that actively assist consumers, practitioners, payers, and others in making effective health care decisions.
AHCPR is also responding to new challenges to government to become more efficient, flexible, creative, and effective. The Agency recognizes the need for evaluation activities to be linked to the planning process and to yield information that can inform policymaking, budget planning, and program management.
In the coming years AHCPR will continue to support research in the area of quality measurement and improvement. A Request for Applications, titled Consumer Assessments of Health Plans Study (CAHPS), was announced in the NIH Guide in May 1995. The proposed study will build on the consumer survey project described above. The project will demonstrate and evaluate the use of consumer surveys to assess health plans. AHCPR will also build on work in the area of clinical performance measurement by expanding the Measurement Typology Project to incorporate guideline-based clinical performance measures into the measures inventory.