MISSION: To generate and disseminate information that improves the health care system.
The Evaluation Program within the Agency for Health Care Policy and Research (AHCPR) is designed to respond to three types of evaluation needs:
- the need for comprehensive information to assess the agency's effectiveness in meeting its major and long-term priorities and goals
- the need for information from "fast-track" or quick turnaround projects to respond to critical agency and departmental concerns
- the need for information from internal evaluations to improve the efficiency with which the agency performs its work
To address these needs, evaluation components are built into virtually all AHCPR activities. Among the evaluation mechanisms used by the agency are targeted evaluation studies undertaken through contracts or grants; peer review of grant applications and technical review of contract proposals for scientific rigor and integrity; efforts to obtain feedback from "customers" on the usefulness of AHCPR research efforts; feedback from AHCPR's User Liaison Program (which provides information and technical assistance to State policymakers, health departments, and officials); and other targeted efforts, such as focus groups and surveys to provide baseline information, inform the design of future agency work, and assess progress toward goals.
AHCPR received delegated authority to review evaluation projects in 1992. In carrying out that authority, it established a two-tier system for assessing proposed evaluation projects eligible for 1-percent set-aside funds. The first phase of the process, an executive-level review assessing the policy relevance and relative priority of proposed projects, is conducted by the administrator and senior staff. The second, a technical merit review, assesses policy-relevant proposals for technical feasibility, soundness of design, costs, potential importance of the findings, and relation to ongoing evaluation activities. This second review is conducted by the Task Force on AHCPR Evaluation Projects, a group consisting of one individual with evaluation expertise from each Office and Center in the agency.
Summary of Fiscal Year 1996 Evaluations
The four high-priority evaluation projects completed in FY 1996 directly reflect the agency's commitment to enhancing the scientific underpinnings of the Nation's efforts to measure and improve the quality of health care services. Each provides information to enhance the quality of the health care system: information that can improve clinical quality, inform consumer choice, and focus future research on what works best in health care.
Evaluating Practice Guidelines and Developing Science-Based Measures of Clinical Quality
The first of the four completed projects assessed the feasibility of deriving clinical quality evaluation and improvement tools from evidence-based practice guidelines. The project responds to the country's increasing demand for valid, reliable, science-based indicators of clinical quality that can be used in a variety of health care settings. A contract with the Center for Clinical Quality Evaluation (formerly the American Medical Review Research Center) in Washington, DC, translates three practice guidelines (for urinary incontinence, acute postoperative pain, and benign prostatic hyperplasia) into performance measures that can be used to determine the extent to which clinical care is consistent with scientific recommendations and to target specific opportunities for quality improvement.
With the assistance of Medicare Peer Review Organizations (PRO's) from four States, the project developed guideline-based measures and tested them for use among the Medicare population. The PRO's also served as educational facilitators in a quasi-experimental design component of the project focusing on the introduction of the benign prostatic hyperplasia (BPH) guideline in four "continuum-of-care" settings (clinics and provider groups providing both ambulatory and inpatient care).
The project found that it is feasible to develop guideline-based performance measures that could be used to evaluate and improve clinical performance. The BPH guideline was favorably received by both physicians and patients, and improvements in clinical performance made for specific areas were the focus of educational interventions.
Enhancing the Science of Clinical Quality Measurement
The second project, the Computerized Needs-Oriented Quality Measurement Evaluation System (CONQUEST 1.0), enhances the science of clinical quality measurement and evaluation by creating a standardized and rigorous approach to understanding performance measures, as well as for identifying, evaluating, and selecting measures most appropriate to the users' needs. The project collected 53 sets of clinical performance measures (tools that assess the delivery of clinical services) and summarized their content and construction so that the measures could be compared directly on factors such as their data needs, the aspect of clinical care each measure evaluates, their scoring and analysis considerations, reliability and validity testing, and the extent of their use to date. The information collected is summarized in a "measure" data base.
In addition to information on clinical performance measures, the project summarizes information on 50 common clinical conditions into a "condition" data base. Information in the condition data base includes age groups affected by the condition; forms and prevalence of the condition; cost and utilization associated with the condition; comorbidities, risk factors, and potentially preventable adverse events associated with the condition; information on clinical services used to diagnose and treat the condition; and information on provider settings and health care professionals associated with the primary condition. A key feature of the condition data base is its inclusion of recommendations from clinical practice guidelines and findings from medical effectiveness research. Information from the condition data base can be used to identify performance measures for a given condition, facilitate the use of individual performance measures for broader quality measurement and improvement uses, and interpret findings from performance data relative to information from practice guidelines and medical effectiveness research.
Initial response to CONQUEST 1.0 has been extremely favorable; the agency has distributed over 3,000 copies of the first version of the software and user manual and is currently developing a "run time" version that will function in Windows 95. In addition, the agency is building on this initial effort by sponsoring a multiyear contract to develop a Quality Measurement Network, a clearinghouse and technical resource for those interested in identifying and using measures to evaluate and improve clinical performance.
Helping Consumers Understand Health Care Quality Information and Evaluate Health Care Plans
The third project, described in chapter II, develops an approach for States to use in evaluating the effect of health care delivery changes on access to, quality of, and satisfaction with health care. Through the AHCPR Rural Health Research Center at the University of Washington, and its subcontractor the Oregon Health Sciences University, the Oregon Consumer Scorecard (OCS) Project assists the State of Oregon in developing an approach to evaluate access to and quality of care provided under the Oregon Health Plan. The Oregon Health Plan is a State-supported health insurance program designed to provide universal access to care for low-income residents. The project assisted in the development and pilot testing of measures of clinical quality and consumer satisfaction and assessed methods for communicating quality information to consumers in a way that would help them evaluate and select health care plans.
The project found that the information needs of consumers and health professionals differ markedly. While consumers judge health plans on such issues as how they gain access to specialty services for various acute or chronic conditions, health care professionals use population-based performance measures to judge quality of care. These population-based measures focus more extensively on clinical processes and outcomes; while these measures currently may have little meaning to most consumers, they have great potential in shaping the quality of health care and health plan accountability. The project found that consumers are interested in a variety of formats for reviewing health plan data and that the availability of a "personal guide" or a trained individual to assist consumers in understanding information would be highly beneficial. With respect to health plan information needs, the project found that health plans face a "data burden" that is costly and could be relieved by the establishment of uniform standards that are used by all health plans and purchasers.
In addition to providing technical and substantive assistance to the State of Oregon, the project develops the science base for statewide quality assessment and reporting, an effort which can be used to help other States evaluate quality of care.
Synthesizing Information From AHCPR-Sponsored Research
The last of the four projects is a multiyear effort to track and synthesize findings from AHCPR-sponsored research in the Medical Treatment Effectiveness Program (MEDTEP). AHCPR has been working with Walcoff and Associates, Inc., of Fairfax, Virginia, to collect and analyze information on MEDTEP research, including clinical practice guidelines. The project has collected information from published professional literature; presentations to and proceedings of conferences and meetings; trade journals in the health care delivery field; reports of congressional testimony and deliberation; reports of State and local legislation or executive testimony and deliberation; magazine and newspaper articles; radio and television programs; and reports of other Federal agencies. During the period of the project, findings from these tracking efforts have been published in the AHCPR-sponsored newsletters Research Activities and MEDTEP Update. Most recently the project has produced comprehensive summaries of MEDTEP projects, a bibliography of Publications of the Patient Outcomes Research Teams projects, and a data base of information on these AHCPR-sponsored projects.
Evaluations in Progress
Ongoing evaluations at AHCPR focus on three broad areas: environmental scans to identify research needs and potential partnership opportunities, evaluations of ongoing agency work, and evaluation planning and design studies to guide future efforts. These activities are part of AHCPR's emphasis on developing and implementing measures of agency performance and on ensuring that agency-sponsored research efforts are most useful to a wide range of audiences.
Assessing the Health Care Environment
Three projects examine ongoing health care policy and research activities in the public and private sectors. These "environmental scans" help AHCPR identify emerging health policy and research issues, facilitate agency strategic planning, and avoid duplication and waste of resources. They also identify areas in which Federal resources, tools, data, and methodologies can help answer questions about the effects of health care policy changes on the cost, quality, and access to health care services, as well as the effectiveness of those services.
One project builds on the work of the Department of Health and Human Services Data Council's Committee on Health Data Standards and the Information Infrastructure Task Force's Health Information Working Group to collect and summarize information on the information standards activities of various Federal agencies. This project will update a previous report, Current Activities of Selected Health Care Informatics Organizations. Another project will identify and describe the range of outcomes research activities conducted by various private-sector organizations and focus on approaches for identifying research topics, outcomes research methodologies, and translation of findings into clinical practice. A third project in this group will identify approaches to achieving productive public-private research collaborations. This effort will use publicly available data to identify examples of public-private collaborations and to examine such facets of the project as methods for identifying research topics, treatment of investment and financial records, and reporting research results.
Evaluating Ongoing Agency Work
Three projects in the agency's current portfolio evaluate aspects of AHCPR's ongoing research program activities. The first is a readership survey assessing customer satisfaction with selected AHCPR publications. The study focuses on the AHCPR publication with largest distribution, Research Activities, and will assess both the format and content of the publication. Subsequent surveys will target other agency products, with the objective of developing more effective means of communicating agency research findings. Another project synthesizes the results of AHCPR research aimed at implementing and evaluating of clinical practice guidelines, physician behavior change strategies, and other quality improvement tools. The third project in this category evaluates AHCPR's training activities for minority health services researchers. It will also examine other (non-AHCPR) minority health services research efforts to identify desirable elements and characteristics of success.
Evaluation Designs and Planning Studies
Four projects are ongoing in the area of evaluation design and planning. These projects assess the current state of knowledge on a given health care topic area, identify research gaps, and develop and test approaches and indicators that could be useful for assessing the quality of agency-sponsored programs. The first project, an assessment of data release strategies, examines how data from AHCPR's Medical Expenditure Panel Survey and other national data bases have been used to support health policy and health services research. The project will delineate the strengths and weaknesses of the various data sets and identify ways to increase their utility. It will also provide needs assessment information essential for maximizing the usefulness of AHCPR's large-scale data efforts.
Another project examines the feasibility of compiling an encounter-level managed-care research data base by identifying currently available public- and private-sector data and by identifying data analysis and data release problems. The third project in this category focuses on better understanding research needs in the area of appropriateness of care. This study will explore the published literature on the causes of diagnostic inefficiency in primary care, interventions designed to address these inefficiencies, and the effect of interventions on the quality of care. The project will focus on errors in selected high-cost, high-prevalence areas of health care for which there may be considerable uncertainty about diagnosis, but accepted treatments. The final study in this category examines published and unpublished studies on the effects of consumer health informatics on patient decision making.
New Directions for Evaluation
In FY 1998 and beyond, AHCPR will focus its evaluation activities in three general areas: the development and use of agencywide performance measures, evaluation projects targeted at effective communication and implementation of health care quality and effectiveness findings, and evaluation of the effect of health care policy changes on the cost and quality of services, access to services, and the effectiveness of those services.
The first area emphasizes agencywide performance measures that can evaluate the quality of the agency's work and inform policy making, budget planning, and program management. Agencywide performance measures will help AHCPR address requirements of the Government Performance and Results Act, providing information that will improve the agency's performance in key program areas. Projects initiated in FY 1997 will produce and test measures that can be applied in FY 1998 and beyond. In addition to these efforts, the agency will be evaluating the effect of changes in programmatic operations. One example of this type of effort is a proposed evaluation of the effect of recent improvements to the AHCPR grants process.
The second area centers on building AHCPR's portfolio of research to translate research findings into forms of information that actively assist consumers, practitioners, payers, and others in making effective health care decisions. Of particular interest to the agency are needs assessment and formative evaluation efforts related to the new Evidence-Based Practice Centers program. This new initiative, which will replace AHCPR's guideline development program, creates research centers that will conduct systematic reviews and synthesize scientific evidence related to the effectiveness of various approaches, procedures, and technologies. These evidence syntheses can be used by practitioners and health care organizations to develop clinical practice guidelines and other quality improvement tools. Evaluation activities will first focus on assessing user needs in terms of topics, formats, and technical assistance. Subsequent evaluation activities will assess the usefulness of the new products for clinical quality improvement.
The third area will build on AHCPR's work to examine and evaluate changes in the health care system and how those changes affect access to care, use of services, quality of care, and patient outcomes. As the agency continues improving the Medical Expenditure Panel Survey, new evaluation questions and opportunities arise on such issues as the efficiency of the new design and its effectiveness in answering key health policy questions. The agency will be evaluating aspects of the survey process, including data support contracts and approaches to evaluating modeling capacities.