Agency for Healthcare Research and Quality

Mission

To generate and disseminate information that improves the health care system.

Evaluation Program

The Evaluation Program within the Agency for Healthcare Research and Quality (AHRQ) provides information on the Agency’s effectiveness in meeting its long‑term goals and priorities, the efficiency of the Agency’s program performance through internal evaluations, and critical Agency and departmental concerns via “fast‑track” or quick‑turnaround assessments.

Evaluation components are built into virtually all major AHRQ programmatic activities and a wide variety of “freestanding” projects are undertaken as needs are identified. Among the evaluation mechanisms used by the Agency are targeted evaluation studies undertaken through contracts or grants; efforts to obtain feedback from “customers” on the usefulness of AHRQ research efforts, including such mechanisms as focus groups and surveys; and feedback from AHRQ’s User Liaison program (which provides information and technical assistance to state policymakers, health departments, and officials). Information gathered from AHRQ’s evaluation‑related activities are used in the annual performance plans developed in response to the Government Performance and Results Act.

Completed Evaluations

Evaluation of Performance of the Medical Expenditure Panel Survey: Final Report on Tasks 1 and 2

The purpose of this project was to evaluate key aspects of the Medical Expenditure Panel Survey (MEPS) in terms of data quality, cost, and timeliness.  The evaluation focused on areas in which there have been programmatic concerns, especially those in which the data now exists for evaluation and which would be implemented in a timely way to inform quality enhancements for MEPS.  In the current design of the MEPS, the Medical Provider Component (MPC) was not intended to support inference to the population of providers in the U.S.  The research questions motivating MEPS have focused on household members as the unit of analysis and have concerned the health care received by this population.  Data were obtained from medical providers only because the household members themselves are not reliable sources of data on health care use and expenditures. Where appropriate, the evaluation identified priorities for redesign, and outlined potential cost-effective alternatives.
PIC ID: 7401.1; CONTACT: Michael Hagan, 301-594-1354; PERFORMER: Research Triangle Institute, Research Triangle Park, NC

Review of Existing Reports on Health Care Quality to Aid the Development of the National Quality Report

This review was designed to summarize for AHRQ what others have done in reporting on health care quality and, at the same time, to draw lessons from those works for the national quality report (NQR).  The purpose was to review existing health care quality reports for content and style, to identify elements of reports that should be adopted and replicated in the NQR, and to determine approaches to be avoided.  Included for review were reports developed by any organization, whether international organizations, foreign countries, state governments, purchasing coalitions, health plans, or other groups.  Key insights were:  (1) Audience: For the NQR, a primary audience should be defined early in the work. (2) Purpose: It will be important to articulate clearly the purpose of the report and its specific goals. (3) Content: These areas need to be considered for the NQR --populations covered, aggregations used, comparisons made, level of disclosure, and statistics presented. (4) Special Populations:  The NQR may have to reanalyze existing data sources to obtain population subgroup comparisons and use static assessments of health care quality for special populations from special studies in the literature. (5) Data Sources:  Existing reports provided few ready private sources of data for the NQR. (6) Media and Style: The NQR should be produced in both published and Internet versions, with the Internet version containing a full-text “PDF” format.  (7) Evaluation: AHRQ may want to consider both Internet feedback and a scientific evaluation of the NQR after its release.
PIC ID: 7715; CONTACT: Diana Dodd, 301-594-1543; PERFORMER: Medstat Group, Washington, DC

In-Progress Evaluations

Development of Point of Service Assessments for the Office of Management

This evaluation will be undertaken to develop point-of-service assessments for the Office of Management in the areas of personnel, administrative support, information technology, telecommunications, and contract general support services.
PIC ID: 7485; EXPECTED COMPLETION: FY 2002; CONTACT: Barry Flaer, 301-594-1433; PERFORMER: Washington Consulting Group, Bethesda, MD

Evaluation of Accounting and Financial Management Services

The contractor is to provide on an as needed basis, consulting services for accounting, auditing, financial reporting review, financial management, analysis, planning, performance measurement, and other related services.
PIC ID: 7487; EXPECTED COMPLETION: FY 2002; CONTACT: Barry Flaer, 301-594-1433; PERFORMER: Barents Group, KPMG Peat Marwick LLP, Washington, DC

Evaluation of Centers for Education and Research on Therapeutics (CERTS)

This task is designed to evaluate the impact of the CERTs program and to determine if the program is moving toward its goals as designed in the RFA. The program goal is to develop free-standing continuing education centers which conduct research, educate health care providers and other decision makers, and improve the quality of care while reducing costs with a concomitant decrease in errors and adverse effects.   There will also be a focus on the effectiveness and necessity of specific administrative/organization components of the CERTs program, namely the activities of the Coordinating Center and the Steering Committee.  The products will include an evaluation of progress and products of the first year and one-half of the centers, the Coordinating Center and the Steering Committee, and will help identify accomplishments to date and new ideas for implementation in the CERTs II initiative.
PIC ID: 7686; EXPECTED COMPLETION: FY 2002; CONTACT: Lynn Bosco, 301-594-2416; PERFORMER: Washington Consulting Group, Bethesda, MD

Evaluation of Sources of Information Used to Track the Impact of AHRQ-Sponsored Research

The purpose of this task is to compare the effectiveness of five approaches to determining the impact of AHRQ funded research, tools and data.
PIC ID: 7685; EXPECTED COMPLETION: FY 2002; CONTACT: Kevin Murray, 301-594-6128; PERFORMER: Center for Health Policy Studies, Columbia, MD

Feasibility Study for an Applied Patient Safety Training & Outplacement Program (Patient Safety Improvement Corps)

This project is designed to determine the feasibility of establishing, operating, and maintaining an applied patient safety training and outplacement program.
PIC ID: 7683; EXPECTED COMPLETION: FY 2002; CONTACT: Marge Keyes, 301-594-1824; PERFORMER: Research Triangle Institute, Research Triangle Park, NC

Implementation Planning Study for Integration of Medical Event Reporting Input and Data Structure for Reporting to AHRQ, CDC, CMS & FDA

The purposes of this project are: (1) to identify existing and developing data collection activities within the four participating DHHS partners, where they are located, how the information is stored (including specifications for the data storage), and similarities and redundancies in the current data infrastructures in the four Agencies; (2) to examine the conceptual and technological issues, complexity, and costs involved in development of a single interface for collecting data related to medical error and patient safety risks through the integrating existing and developing DHHS reporting systems; and, (3) to identify and evaluate advantages and disadvantages of options for the future development of a web-based integrated reporting interface among the existing and developing data collection systems in the four partnering agencies.  AHRQ and its DHHS partners intend to apply the report of this implementation planning study task order to future procurement to continue the data and reporting integration development process.
PIC ID: 7681; EXPECTED COMPLETION: FY 2002; CONTACT: James Battles, 301-594-9892; PERFORMER: MEDSTAT Group, Cambridge, MA

Patient Safety Measures

This project will develop two baseline patient safety evaluation measures:  1) safety culture assessment and 2) a survey on medical event reporting systems.  These instruments shall be used to determine context and antecedent conditions existing prior to initiation or at the beginning of implementation of AHRQ’s patient safety research initiative.
PIC ID: 7679; EXPECTED COMPLETION: FY 2002; CONTACT: James Battles, 301-594-9892; PERFORMER: Barents Group, KPMG Peat Marwick LLP, Washington, DC

Health Services Research Education: Assessing Customer Satisfaction and Program Needs

This is an evaluation of certain elements of a national training program in health services research funded by the Agency for Healthcare Research and Quality (AHRQ).  The project will provide AHRQ with information about present and anticipated needs in health services research training and an assessment of the level of satisfaction that past trainees from the institutionally based training programs have had with their experiences during training, and with the adequacy of these training activities relative to trainees’ career expectations and requirements.
PIC ID: 7397; EXPECTED COMPLETION: FY 2002; CONTACT: Debbie Rothstein, 301-594-2918; PERFORMER: Battelle Corporation, Arlington, VA

An Evaluation of Quality Measures and Timely and Cost Effective Strategies to Implement the Design Enhancements to the Medical Expenditure Panel Survey (MEPS)

The purpose of this project is to evaluate quality measures that should be added to the MEPS and to evaluate timely and cost effective strategies that implement design enhancements to the MEPS supportive of the Agency’s national quality measurement initiative.
PIC ID: 7401.2; EXPECTED COMPLETION: FY 2002; CONTACT: Thomas W. Reilly, 410-786-2196; PERFORMER: Barents Group, KPMG Peat Marwick LLP, Washington, DC

Development and Analysis of HCUP - The Healthcare Cost and Utilization Project

The purpose of this project is to support the Agency by designing, collecting, developing and analyzing a system of information on hospitals and their patients suitable for national studies.
PIC ID: 7693; EXPECTED COMPLETION: FY 2005; CONTACT: Jenny Schnaier, 301-594-6827; PERFORMER: Medstat Group, Washington, DC

Evaluation of the HCUP Central Distributor & HCUP Net

The purpose of this evaluation is to assess two dissemination mechanisms of products produced as part of the HCUP.   The first dissemination mechanism is the HCUP Central Distributor, a mechanism established to disseminate the State Inpatient Databases (SID), one in a family of databases produced as part of HCUP.  The second dissemination mechanism is HCUPnet, an interactive web-based system designed to generate tables of outcomes by diagnoses and procedures based on queries from users.   The purpose of this task is to provide information about how well the HCUP Central Distributor is satisfying its original objective and to provide recommendations for improvements.
PIC ID: 7684; EXPECTED COMPLETION: FY 2002; CONTACT: Herbert Wong, 301-594-6822; PERFORMER: Center for Health Policy Studies, Columbia, MD

Maintain and Expand the Healthcare Cost and Utilization Project (HCUP)

The objectives for this contract are: 1) to obtain encounter-level data from statewide information sources; 2) to maintain and expand a uniform, multi-state health care database for health services research and health policy analysis, and 3) to make these data available to a broad set of public and private users.  This contract will extend the capabilities of the HCUP national data system to assist researchers and decision-makers at the national, state, and community levels by expanding HCUP to include more states and more settings of care, and to develop tools to make better use of administrative data.
PIC ID: 7694; EXPECTED COMPLETION: FY 2005; CONTACT: Jenny Schnaier, 301-594-6827; PERFORMER: Medstat Group, Washington, DC

Validating the HCUP Patient Safety Indicators

This proposal extends work that is currently being conducted on the first two modules of the next generation of HCUP Quality Indicators (QI) developed by Stanford University. The development of reliable and valid QI for HCUP is a high priority and this proposal aids in this effort.  The current validation proposal was funded to evaluate approximately 45 measures developed by Stanford.  The proposed project will validate approximately 30 additional measures that are currently being developed by Stanford focusing on complications or adverse medical events.
PIC ID: 7672; EXPECTED COMPLETION: FY 2003; CONTACT: Anne Elixhauser, 301-594-6815; PERFORMER: Research Triangle Institute, Research Triangle Park, NC

Validating the HCUP Quality Indicators

This project will seek to validate the HCUP Quality Indicators (QIs) against information available from data sources other than hospital administrative data.  The accuracy and validity of the HCUP QIs will be explored by comparing results obtained from the QI software with information from other data sources such as medical records, outpatient administrative data, and laboratory data.
PIC ID: 7671; EXPECTED COMPLETION: FY 2002; CONTACT: Anne Elixhauser, 301-594-6815; PERFORMER: Research Triangle Institute, Research Triangle Park, NC

Innovative Approaches to Training Clinicians for Bioterrorist Attacks

This project will develop innovative approaches to training clinicians for bioterrorist attacks, and creative methods for evaluating their effectiveness.  Such approaches must be applicable on a broad scale and must be adapted to the information needs and current challenges confronting busy clinicians.
PIC ID: 7680; EXPECTED COMPLETION: FY 2002; CONTACT: Carolyn M. Clancy, 301-594-1485; PERFORMER: Research Triangle Institute, Research Triangle Park, NC

Empirical Analysis of a Guideline-developed Clinical Performance Measure–the Example of Pediatric Gastroenteritis Development

Assessing and categorizing whether actual care conforms to clinical practice guidelines is an increasingly common approach to measuring appropriateness and quality of health services.   Often, however, the process of care that is to be categorized simply as conforming or not may involve several different components or dimensions.  This project will analyze existing data concerning one example of such a summary measure, conformance with American Academy of Pediatrics recommendations regarding gastroenteritis.
PIC ID: 7037.1; EXPECTED COMPLETION: FY 2002; CONTACT: Steve Fox, 301-597-1485; PERFORMER: MEDSTAT Group, Cambridge, MA

Criteria for Determining Disability in Speech-Language Disorders

This task order is designed to develop an evidence report.  The project title was nominated by the Social Security Administration (SSA), which will serve as a partner to the PC for this task order.  The plan is to use the evidence report to assist the SSA in updating its Listing of Impairments for Speech-Language Disorders, and revising its disability policy, as may be appropriate.  The SSA Office of Disability will represent the SSA on this task order.
PIC ID: 7688; EXPECTED COMPLETION: FY 2002; CONTACT: Kevin Murray, 301-594-6128; PERFORMER: Research Triangle Institute, Research Triangle Park, NC

Evaluating AHRQ’s Low-Income Research Portfolio, 1989-2000

This task order is designed to identify AHRQ’s contribution to the field of health services research for low-income populations; to evaluate the impact this contribution has had on health care policy and practice; and, ultimately, outcomes for low-income health services research.
PIC ID: 7687; EXPECTED COMPLETION: FY 2002; CONTACT: Robin Weinick, 301-594-2010; PERFORMER: Center for Health Policy Studies, Columbia, MD

IDS Solutions for Transferring Medication Data Across Patient Care Settings

This proposal is important because it presents a reasonable methodology to examine and improve methods of data transfer between treatment settings.  This is a vital concern to clinicians because information about the medications a patient is taking and which medications he/she is sensitive (or allergic) to must be accurately transferred to insure patient safety.
PIC ID: 7682; EXPECTED COMPLETION: FY 2002; CONTACT: Kelly Morgan, 301-594-1782; PERFORMER: Research Triangle Institute, Research Triangle Park, NC