MISSION: To generate and disseminate information that improves the health care system.
The Evaluation Program within the Agency for Health Care Policy and Research (AHCPR) is used to respond to three types of information needs:
- The need to determine the Agency's effectiveness in meeting its long-term goals and priorities
- The need to improve the efficiency of the Agency's program performance through internal evaluations; and
- The need to respond to critical Agency and departmental concerns via "fast-track" or quick-turnaround assessments.
To address these needs, evaluation components are built into virtually all major AHCPR programmatic activities and a wide variety of "freestanding" projects are undertaken as needs are identified. Among the evaluation mechanisms used by the Agency are targeted evaluation studies undertaken through contracts or grants; efforts to obtain feedback from "customers" on the usefulness of AHCPR research efforts, including such mechanisms as focus groups and surveys; and feedback from AHCPR's User Liaison program (which provides information and technical assistance to State policymakers, health departments, and officials). Information gathered from AHCPR's evaluation-related activities will be used in the annual performance plans developed in response to the Government Performance and Results Act starting in FY 1999.
All "freestanding" evaluation activities undergo two levels of review. First, brief evaluation proposals are developed by staff and undergo review by affected senior managers. Those proposals approved by senior managers are submitted for review by the administrator. The administrator evaluates proposals for policy relevance, priority, cost effectiveness, and timeliness. Those that receive the administrator's approval are then methodologically developed and centrally reviewed for technical merit, including technical feasibility, costs, and relation to ongoing evaluation activities.
Summary of Fiscal Year 1997 Evaluations
Outcomes and Effectiveness Research in the Private Sector
To help clarify AHCPR's future priorities in outcomes and effectiveness research (OER) and to provide insights regarding future opportunities for public-private partnerships in this area, AHCPR undertook an analysis and evaluation of private-sector OER activities (6385). The study found that a wide variety of private-sector organizations report performing OER, often for the purposes of monitoring health care performance against certain standards, e.g., accreditation standards. In addition, private-sector organizations conduct OER primarily to improve their strategic and financial positions. More specifically, health product companies usually focus their OER efforts on new drugs and devices, while less attention is given to other health care interventions, particularly those that are not new or those that are used primarily for traditionally underserved or vulnerable populations. AHCPR has supported the development and validation of many of the research tools used in private-sector OER. The report found little evidence of a redundant effort between publicly and privately supported OER and reinforced the need for public support for continued methodologic development in OER.
Drawing upon research previously performed in the U.S., Canada, and the United Kingdom, a recent study reported that informatics tools and decision aids affect patient decisionmaking about medical screening and treatment (6376). More specifically, researchers found that patients who use informatics tools are more satisfied with the decisionmaking process and have more knowledge of treatment alternatives than patients who do not use the tools. The researchers recommend that the effects of informatics tools on a full range of outcomes be assessed (e.g., patient-provider communication, treatment selection, health behavior, clinical and health status outcomes, and health care costs). It is important that factors influencing patient use of informatics tools be identified as well as the cost-effectiveness of different types of patient informatics tools.
As part of its effort to measure and improve quality of care, AHCPR completed two related evaluations of CONQUEST--the Computerized Needs-Oriented Quality Measurement System (5961.1 and 5961.2). CONQUEST is a prototype software system for collecting and evaluating clinical performance measures, and organizing them into a classification scheme that presents essential information in a standard format. The two evaluations identified short-term, intermediate, and long-range strategies for CONQUEST's continued development. While finding CONQUEST 1.0 to be a good starting point, the evaluations recommended that future versions of CONQUEST be expanded to include more measures, more conditions, and more detail on measure construction/specifications. In addition, the evaluations indicated that CONQUEST should be modified to make it more intuitive and user-friendly. Many of the recommendations were incorporated into CONQUEST 2.0.
Managed Care Organizations
To help determine the extent to which AHCPR and other organizations will be able to analyze data from managed care organizations (MCOs), AHCPR completed work on a project to determine the feasibility of creating a managed care encounter-level data base (6374). The rapid growth of managed care is changing the ability of health services researchers and policy analysts to answer fundamental questions about access, utilization, cost, and quality of care. Data that were previously produced as a "by-product" of the fee-for-service system are either no longer available or vary in comprehensiveness and accuracy. One major finding of this study was that MCO data bases vary substantially in the availability and comprehensiveness of encounter-level data and in the ability to link these data with administrative and financial data. MCOs increasingly view their information systems, and the data they store, as proprietary, and there are a number of disincentives for MCOs to share their data. In addition, there are no industrywide definitions and reporting standards. Despite these barriers, MCOs are interested in exploring ways to work with Federal agencies to facilitate the availability of data for the purposes of research.
AHCPR has used the study findings to 1) identify future opportunities for public-private partnerships; 2) solicit feedback from multiple constituents on models that the Agency could employ to facilitate improvements in, and accessibility of, encounter-level data; 3) assess current AHCPR supply-side data initiatives (e.g., Healthcare Cost and Utilization Project); 4) formulate a longer-term encounter-level data strategy; and 5) inform and support related activities in the Secretary's Quality Initiative and the President's Commission on Consumer Protection and Quality.
AHCPR conducted a customer survey to determine satisfaction with its publications and their dissemination (6375). The effort found that AHCPR's monthly newsletter, Research Activities, is well-received as it is currently written, presented, and distributed. A large percentage of readers keep the publication as a reference source and would prefer to continue to receive the publication in hard copy format, rather than via the World Wide Web (where it is also currently available). Overall, the survey results showed that the publication has a relatively loyal following. The study also found that AHCPR's patient/consumer guides are most frequently requested by phone and delivered in a timely manner. In addition, recipients are pleased with the services of AHCPR's publications clearinghouse and had very positive reactions to many of the guides' different features. This information will be used as performance data in AHCPR's Government Performance and Results Act Plan and will serve as a baseline for any future surveys fielded to determine satisfaction with changes made to Research Activities. During FY 1997, AHCPR also completed projects on evaluating minority health services research training activities (6384), identifying private sector organizations as potential research partners (6386), developing a workplan for preparing an inventory of performance measures currently being used or considered by HHS agencies (6691), and evaluating AHCPR's small conference grant program (6692).
Evaluations in Progress
Grant Application and Review Processes
AHCPR is conducting an evaluation of its grants (large and small) application receipt and review processes (6696). The evaluation includes a combination of qualitative and quantitative techniques that will document the extent to which changes made to the grants application submission and peer review processes have resulted in improved efficiency and effectiveness from the perspectives of applicants, reviewers, and Agency staff.
Data Collection Evaluation
AHCPR is undertaking a project to assess the usefulness to policymakers and researchers of AHCPR-supported major data collection efforts, such as the Medical Expenditure Panel Survey, the Health Care Cost and Utilization project, and the HIV Cost and Services Utilization Survey. More specifically, it will examine methods to improve survey operations and timely data release. The study will examine how other Federal agencies, as well as private foundations, conduct various aspects of their large-scale surveys (6383).