|
Office of the Assistant Secretary for Planning and Evaluation |
| |
|
Policy Information Center |
MISSION: To generate and disseminate information that improves the health care system.
Evaluation Program
The Evaluation Program within the Agency for Healthcare Research and Quality (AHRQ) provides information on the Agency's effectiveness in meeting its long-term goals and priorities, the efficiency of the Agency's program performance through internal evaluations, and critical Agency and departmental concerns via "fast-track" or quick-turnaround assessments.
Evaluation components are built into virtually all major AHRQ programmatic activities and a wide variety of "freestanding" projects are undertaken as needs are identified. Among the evaluation mechanisms used by the Agency are targeted evaluation studies undertaken through contracts or grants; efforts to obtain feedback from "customers" on the usefulness of AHRQ research efforts, including such mechanisms as focus groups and surveys; and feedback from AHRQ's User Liaison program (which provides information and technical assistance to State policymakers, health departments, and officials). Information gathered from AHRQ's evaluation-related activities are used in the annual performance plans developed in response to the Government Performance and Results Act.
All "freestanding" evaluation activities undergo two levels of review. First, brief evaluation proposals are developed by staff and undergo review by senior managers. Those proposals approved by senior managers are submitted for review by the director. The director evaluates proposals for policy relevance, priority, cost effectiveness, and timeliness. Those that receive the approval of the director are then methodologically developed and centrally reviewed for technical merit, including technical feasibility, costs, and relation to ongoing evaluation activities.
The Computerized Needs Oriented Quality Measurement Evaluation System (CONQUEST) is a quality improvement software tool that provides information on 1,200 clinical performance measures, both process and outcomes, on more than 50 conditions. The Agency for Healthcare Research and Quality (AHRQ) promotes CONQUEST as a tool for improving the quality of health care provided to the U.S. population through dissemination of the software. Its more immediate objectives are to improve the state of clinical performance measurement and to promote greater standardization in clinical performance measurement. This evaluation was requested by AHRQ to provide information on how CONQUEST 2.0 was being received and used by the audiences that the Agency had targeted and reached (e.g., health plans, health care providers, policy makers, employers, and other government agencies). The study found that at the time the survey was fielded, only about 10 percent of recipients were actually using CONQUEST. One of the barriers to use was that people did not have the time to learn how to use the software and many suggested that more than a brief demonstration and self-instructional materials need to be available if persons are to be encouraged to use the software. Other suggestions and recommendations include making the software a web-based operation, training for first-time users and developing a better description of what the CONQUEST software is and what it is not.
AGENCY SPONSOR: Center for Quality Measurement and Improvement
FEDERAL CONTACT: Marge Keyes, 301-594-1824
PIC ID: 7010
PERFORMER: Research Triangle Institute, Research Triangle Park, NC
The Agency for Health Care Policy and Research (AHCPR) supports 12 Evidence-based Practice Centers (EPCs) to undertake scientific analyses and evidence syntheses on high-priority topics. The science syntheses-evidence reports and technology assessments produced by the EPCs provide the scientific foundation for public and private organizations to use in developing and implementing their own practice guidelines, performance measures and other tools to improve the quality of health care and make decisions related to the effectiveness or appropriateness of specific health care technologies. This project was an evaluation of these reports aimed at tracking the use and perceptions of the EPC products to improve future products from the program. The primary method used to study the EPC Program was a survey of collaborators to the Evidence-Based Practice Centers and of potential users. The organizations surveyed included quality improvement organizations, health care providers, health-related schools, health-related professional associations, government organizations, and government research agencies. The findings support and illustrate the usefulness in the reports and that the EPC program should continue to aim to produce high-quality comprehensive reviews on important clinical topics that lead to concise and clear results.
AGENCY SPONSOR: Center for Practice and Technology Assessment
FEDERAL CONTACT: Margaret Coopey, 301-594-4022
PIC ID: 7037
PERFORMER: Medstat Group, Washington, DC
This study was undertaken to study the differences in quality between ultrasound services furnished by individuals who are credentialed versus those who are not. The study examined and evaluated difference in error rates, resulting complications, and patient outcomes to determine any quality differences that can be correlated with the differences in credentialing status. The researchers assessed a public meeting discussion on existing evidence of the effects of sonographer credentialing, conducted a preliminary literature review and examined the data collected and maintained by the two dominant sonographer credentialing organizations to conduct an analysis of credentialing effects on quality. The public meeting discussions and the literature review indicated very little evidence exists directly relating sonographer credentialing outcomes in a quantifiable meaningful fashion. Further, although the credentialing organizations maintain detailed information specific to sonographers, the amount of additional data required to control for other confounding factors that would ultimately affect the outcome of a particular course of treatment would require a fairly extensive research project, possibly including primary data collection. Based on the findings, the report recommends no further studies of this issue by AHRQ.
AGENCY SPONSOR: Agency for Healthcare Research and Quality
FEDERAL CONTACT: Harry Handelsman, 301-594-4025
PIC ID: 7477
PERFORMER: Barents Group, KPMG Peat Marwick LLP, Washington, DC
This effort was comprised of a literature review and synthesis of the empirically-based evidence on the effectiveness of interventions to increase enrollment of low-income children and families in public health insurance programs. The primary purpose of the literature review was to enhance understanding of what is known, and what is yet to be known, about the types of outreach interventions that are likely to be most successful for increasing enrollment in the State Child Health Insurance Program (S-CHIP) and Medicaid programs. The literature review identifies and reviews nine published evaluations of outreach activities designed to increase awareness of, or enrollment in, public insurance programs. The review also identifies and describes 17 formal evaluations underway of outreach efforts to improve enrollment in Medicaid and SCHIP programs. The review found that rigorous evaluations of the types of outreach activities that are frequently considered to be more effective for overcoming barriers to enrollment in SCHIP, Medicaid, or other public insurance programs are largely absent from the published literature. Research is deficient with regard to outreach strategies targeting various subgroups of children, community variables that might facilitate or mitigate outreach strategies, and the cost-effectiveness of alternative approaches. The report concludes with options for prioritizing future research in this area.
AGENCY SPONSOR: Center for Cost and Financing Studies
FEDERAL CONTACT: Peggy McNamara, 301-594-6826
PIC ID: 7399
PERFORMER: Barents Group, KPMG Peat Marwick LLP, Washington, DC
The goals of this project were to provide the Agency for Health Care Policy and Research (AHCPR) with information about present and anticipated needs in health services research training that are not currently being met or are only being met partially by the AHCPR-funded program; and An assessment of the level of satisfaction that past trainees from the institutionally-based health services research training program had with their experiences during training, and with the adequacy of these training activities relative to trainees' career expectations and requirements. The evaluation consisted of (1) a needs assessment for the national training program in health services research; and (2) an assessment of customer satisfaction among former trainees with the education, research, and mentoring features of this program. The assessments were focused on four broad areas of inquiry, which are: Infrastructure, Community, Visibility, and Future Directions. Focus group discussions were held among the key stakeholders of the training programs, other focus groups were conducted with program directors, program faculty, current trainees and employers of health services researchers, and a questionnaire was used to determine customer satisfaction with the trainee experience. Overall, respondents were satisfied with the quality of their training experience; 77 percent were satisfied with their program's visibility; large numbers were satisfied with many aspects of the infrastructure, the mentoring from the faculty, the duration of the stipend, and the academic programs: There was not, however, a strong sense of community among AHCPR trainees.
AGENCY SPONSOR: Agency for Healthcare Research and Quality
FEDERAL CONTACT: Debbie Rothstein, 301-594-2918
PIC ID: 7011
PERFORMER: Battelle Columbus Laboratories, Columbus, OH
Following the Congressional mandate stipulating that AHRQ establish an Office of Priority Populations (OPP) to conduct and support research, evaluations and demonstration projects, the Lewin Group was requested to develop a range of organizational and management options to establish that office. The goal of the OPP will be to determine the etiology of health disparities among priority populations and to provide solutions that will begin to close the gap. Priority populations include low income groups, minorities, women, children, elderly, those with special needs, and rural and inner-city persons. The Lewin Group conducted an extensive two-month review of AHRQ internal staff, agency representatives and external groups to propose a set of recommendations on the establishment of this new office. They recommend that the Office have a Director with a speciality in one of the priority areas and additional staff of 10 to 12 individuals. The Director shall have four persons reporting directly to his/her and who can address the crosscutting needs of rural, inner-city, special needs, and low-income sub-groups. The development of the OPP, they recommend, should focus on: strategic planning activities including identifying the vision, goals and objectives for priority populations research; infrastructure building focusing on development of internal and external communication channels; budget development; and staff development.
AGENCY SPONSOR: Agency for Healthcare Research and Quality
FEDERAL CONTACT: Wendy Perry, 301-594-2453
PIC ID: 7476
PERFORMER: Lewin Group, San Francisco, CA
The Agency for Healthcare Research and Quality (AHRQ) requested this study to determine, in Phase I, the feasibility of creating an electronic bulletin board (EBB) that would facilitate forming health services research partnerships. Phase II will occur only if the EBB should prove feasible. Through structured discussions with web managers of Internet sites and focus groups of health services researchers, it was found that there is little need for or desire to use the proposed EBB to facilitate the establishment of research partnerships at the present time. Informants indicated a preference for continuing to identify candidates for research partnerships through personal-professional networks or literature reviews. Alternatives that AHRQ may wish to consider that the report suggests might achieve some of the objectives of the proposed EBB are to: include the EBB as an application on an existing web site for health services research; develop and promote a listserv around partnership issues thereby creating a user community that is interested in the proposed EBB; develop a referral network with organizations that currently promote partnerships in health services research to broker those requests for partnerships coming to AHRQ; and compile and disseminate information about funding for health services research and promote the creation of more requests for applications and requests for proposals that require (and reward) the creation of partnerships.
AGENCY SPONSOR: Agency for Healthcare Research and Quality
FEDERAL CONTACT: Cindy Brach, 301-594-1824
PIC ID: 7404
PERFORMER: Battelle Corporation, Arlington, VA
The Agency for Healthcare Research and Quality (AHRQ), through its User Liaison Program (ULP), synthesizes and disseminates health care research results to State and local health policymakers in order to provide them with the "information and tools with which informed health policy decisions can be made." Among the "products" offered through ULP are two-and-a-half day policy-thematic workshops, somewhat longer skill-oriented workshops, and, more recently, somewhat briefer Masters Series seminars targeted to high-level policymakers, as well as written and electronic material. The primary goal of this evaluation was to demonstrate how participants used the information and materials they obtained at the workshops. The evaluation also examined whether or not ULP is appropriately targeted toward audiences most likely to use ULP products effectively. The evaluation had two components. First, an analysis of 1999 workshop participants to determine whether or not the workshops were attracting high-level participants. In the second part of the evaluation, phone interviews were conducted with 50 participants of 1999 workshops. The report found that respondents consider ULP workshops to be high-quality, informative, interactive learning events. Most respondents have used the information they gained at the workshop, either to enhance their knowledge base on health matters or in a direct application to their work. The complete final report can be found on the web at this address:
AGENCY SPONSOR: Agency for Healthcare Research and Quality
FEDERAL CONTACT: Steve Seitz, 301-594-2066
PIC ID: 7512
PERFORMER: Abt Associates, Inc., Cambridge, MA and Urban Institute, Washington, DC
In line with its new emphasis on special populations, including the disabled, AHRQ contracted to review the last ten years of Agency-funded work on individuals with disabilities to determine the areas of emphasis of the projects in order to assess the breadth and depth of the AHRQ disabilities-research portfolio.
EXPECTED DATE OF COMPLETION:09/15/2001
AGENCY SPONSOR: Center for Organization and Delivery Studies
FEDERAL CONTACT: Michael Hagan, 301-594-1354
PIC ID: 7484
PERFORMER: Center for Health Policy Studies, Columbia, MD
The purpose of this project is to evaluate quality measures that should be added to the MEPS and to evaluate timely and cost effective strategies that implement design enhancements to the MEPS supportive of the Agency's national quality measurement initiative.
EXPECTED DATE OF COMPLETION:09/15/2001
AGENCY SPONSOR: Center for Organization and Delivery Studies
FEDERAL CONTACT: Thomas W. Reilly, 410-786-2196
PIC ID: 7401.2
PERFORMER: Barents Group, KPMG Peat Marwick LLP, Washington, DC
The purpose of this project is to provide information that will assist in developing an institutional component to the Consumer Assessment of Health Plans (CAHPS) survey. CAHPS is a set of surveys that ask consumers about their experiences with their health plans. This project will conduct a review of the literature, identifying existing relevant surveys in the field and conducting and documenting a meeting of experts in the development and implementation of CAHPS surveys.
EXPECTED DATE OF COMPLETION:09/30/2001
AGENCY SPONSOR: Center for Quality Measurement and Improvement
FEDERAL CONTACT: Charles Darby, 301-594-2050
PIC ID: 7493
PERFORMER: Barents Group, KPMG Peat Marwick LLP, Washington, DC
This evaluation will be undertaken to develop point-of-service assessments for the Office of Management in the areas of personnel, administrative support, information technology, telecommunications, and contract general support services.
EXPECTED DATE OF COMPLETION:09/30/2001
AGENCY SPONSOR: Agency for Healthcare Research and Quality
FEDERAL CONTACT: Barry Flaer, 301-594-1433
PIC ID: 7485
PERFORMER: Washington Consulting Group, Bethesda, MD
Assessing and categorizing whether actual care conforms to clinical practice guidelines is an increasingly common approach to measuring appropriateness and quality of health services. Often, however, the process of care that is to be categorized simply as conforming or not may involve several different components or dimensions. This project will analyze existing data concerning one example of such a summary measure, conformance with American Academy of Pediatrics recommendations regarding gastroenteritis, for the purpose of evaluating the extent to which the overall measurements of whether practices conform may be sensitive to choices between alternative approaches for assessing the different components or dimensions of care.
EXPECTED DATE OF COMPLETION:09/30/2001
AGENCY SPONSOR: Center for Outcomes and Effectiveness Research
FEDERAL CONTACT: Steve Fox, 301-597-1485
PIC ID: 7037.1
PERFORMER: MEDSTAT Group, Cambridge, MA
The contractor is to provide on an as needed basis, consulting services for accounting, auditing, financial reporting review, financial management, analysis, planning, performance measurement, and other related services.
EXPECTED DATE OF COMPLETION:09/30/2001
AGENCY SPONSOR: Agency for Healthcare Research and Quality
FEDERAL CONTACT: Barry Flaer, 301-594-1433
PIC ID: 7487
PERFORMER: Barents Group, KPMG Peat Marwick LLP, Washington, DC
This report summarizes the findings of an effort to identify and describe data sources that can be used to conduct research on the characteristics of health care markets. The project was also undertaken to determine whether a comprehensive ongoing data system that describes health care markets can be constructed. To provide this information, the project conducted a literature review of health care market research and developed two data collection tools to ensure consistency and accuracy of the data collection. Because of their broad availability, the report recommends the use of both zip code level data sets and county level data to develop a data system that describes health care markets. The inventory of data sources undertaken in this project is the foundation for a more comprehensive feasibility study which should include an investigation of the possible uses for such a data system. Identification of the research questions to be answered by the system's data will be important for identifying the databases to be included in the system as well an investigation of data definitions to determine whether data sets are compatible when combined.
EXPECTED DATE OF COMPLETION:10/31/2001
AGENCY SPONSOR: Agency for Healthcare Research and Quality
FEDERAL CONTACT: Michael Hagan, 301-594-1354
PIC ID: 7402
PERFORMER: Center for Health Policy Research, Denver, CO
The purpose of the project is to evaluate key aspects of the MEPS in terms of data quality, cost, and timeliness. The evaluation will focus on areas in which there have been programmatic concern, especially those in which the data now exist for evaluation and which would be implemented in a timely way to inform quality enhancements for MEPS. Where appropriate, the evaluation will identify priorities for redesign, and outline potential cost-effective alternatives.
EXPECTED DATE OF COMPLETION:09/30/2001
AGENCY SPONSOR: Agency for Healthcare Research and Quality
FEDERAL CONTACT: Michael Hagan, 301-594-1354
PIC ID: 7401
PERFORMER: Research Triangle Institute, Research Triangle Park, NC
The National Guideline Clearinghouse (NGC) is a web-based source of clinical practice guidelines. Guidelines are available in both abstract and full-text with links to original sources. Guidelines can be downloaded in various formats and mechanisms. It offers comparisons of guidelines that cover similar topic areas, with major interventions and areas of agreement and disagreement. A structured program evaluation will be completed on the National Guideline Clearinghouse to look at what works and doesn't work and how it can be improved. The evaluation will: (1) identify and survey the intended NGC users to understand the quality and usefulness of NGC products, and how the information is being applied to health care delivery; (2) identify and survey the intended users who have not yet accessed the NGC to determine why they are not using the site; (3) evaluate the website functionality, content and linkages to identify opportunities to improve the products and services; and (4) provide a limited assessment of the impact of the NGC on the development of guidelines and quality improvement initiatives to affect clinical behavior.
EXPECTED DATE OF COMPLETION:09/30/2001
AGENCY SPONSOR: Center for Practice and Technology Assessment
FEDERAL CONTACT: Margaret Coopey, 301-594-4022
PIC ID: 5955
PERFORMER: Lewin Group, San Francisco, CA
This is an evaluation of certain elements of a national training program in health services research funded by the Agency for Healthcare Research and Quality (AHRQ). The project will provide AHRQ with information about present and anticipated needs in health services research training and an assessment of the level of satisfaction that past trainees from the institutionally based training programs have had with their experiences during training, and with the adequacy of these training activities relative to trainees' career expectations and requirements.
EXPECTED DATE OF COMPLETION:09/30/2001
AGENCY SPONSOR: Center for Cost and Financing Studies
FEDERAL CONTACT: Debbie Rothstein, 301-594-2918
PIC ID: 7397
PERFORMER: Battelle Corporation, Arlington, VA
This project will describe and analyze the methods used by other federal agencies to monitor the results and impact of the research they fund; to contrast it with ongoing AHRQ activities to achieve this purpose; and to present options for improving AHRQ activities consistent with AHRQ's level of resources.
EXPECTED DATE OF COMPLETION:09/30/2001
AGENCY SPONSOR: Office of Health Care Information
FEDERAL CONTACT: Sandra Isaacson, 301-594-2067
PIC ID: 7494
PERFORMER: Lewin Group, San Francisco, CA
The purpose of this contract is to do a comprehensive literature search, along with an Internet search, for information on quality reporting programs in the U.S. and internationally. That information will be analyzed to see if there are any common themes or best practices among other reporting entities which may serve to inform work on the National Quality Report.
EXPECTED DATE OF COMPLETION:09/30/2001
AGENCY SPONSOR: Center for Quality Measurement and Improvement
FEDERAL CONTACT: Diana Dodd, 301-594-1543
PIC ID: 7491
PERFORMER: MEDSTAT Group, Cambridge, MA
This project will review and synthesize the scientific literature over the past decade for advances in rehabilitation of patients with chronic impairments. (The focus vision impairment as an example, and on measures of improvement in functioning). To assess the rate and success of adoption of rehabilitation strategies in various delivery systems, as well as obstacles to successful adoption, within different models of medical practice and health benefit plans. To recommend next steps for AHRQ in building a research base to address the needs of private and public decision-makers on improving rehabilitation for chronic conditions, with specific application to vision impairments.
EXPECTED DATE OF COMPLETION:09/30/2001
AGENCY SPONSOR: Center for Organization and Delivery Studies
FEDERAL CONTACT: Bernard Friedman, 301-594-6819
PIC ID: 7495
PERFORMER: Lewin Group, San Francisco, CA