Performance Improvement 1998. National Institutes of Health


MISSION: To sponsor and conduct medical research that leads to better health for all Americans.

Evaluation Program

The National Institutes of Health (NIH) generates scientific knowledge that leads to improved health. This is done by conducting medical research in its intramural laboratories and by supporting research in universities, medical and health professional schools, and other health research organizations. NIH fosters the widespread dissemination of the results of medical research, facilitates the training of research investigators, and ensures the viability of the research infrastructure. The NIH Evaluation Program is an integral part of how NIH sponsors and conducts medical research.

The complexities and challenges of evaluating and assessing fundamental science have become more widely recognized as science agencies have begun implementation of the Government Performance and Results Act (GPRA). The GPRA stresses the use of evaluation to develop measures for reporting on program results. However, measures for evaluating the results of fundamental science have limitations. These are discussed in the March 1997 General Accounting Office report, Measuring Performance: Strengths and Limitations of Research Indicators. The report compares private versus public research and discusses the strengths and limitations of indicators used to measure research. The NIH evaluation program is addressing these challenges as part of its strategy for implementing the GPRA.

Philosophy and Priorities

The NIH Evaluation Program provides information to assist the NIH Director and Institute, Center, and Division (ICD) Directors in determining whether NIH goals and objectives are being achieved and to help guide policy development and program direction. Evaluations are planned and conducted from two sources of funds: 1-percent set-aside evaluation funds used to fund trans-NIH projects, and ICD program funds used for program evaluations for use by various committees, working groups, task forces, workshops, conferences, and symposia to assist the ICD in program management and development. This approach ensures that planning and priority setting specific to the mission of each ICD are fully developed and implemented and that there is central leadership for developing crosscutting initiatives and promoting collaboration among the ICDs.

NIH's major evaluation priority areas fall within three broad program areas: basic research, research training and career development, and facilities maintenance. These broad program areas form the basis for the strategies and activities that NIH undertakes to achieve the goals identified in the GPRA planning process.

Policies and Operations

A distinguishing feature of the NIH Evaluation Program is the utilization of a variety of evaluation strategies that include the use of national advisory councils, boards of scientific counselors, consensus development conferences, and ad hoc committees that help to chart scientific directions and select the most promising research to support.

A two-tier system is used to review project requests that will use 1-percent evaluation set-aside funding. The first tier involves a review and recommendations by the NIH Technical Merit Review Committee (TMRC) on the technical aspects of project proposals and whether a project fits within HHS guidelines for use of the set-aside fund. The second tier involves the NIH Evaluation Policy Oversight Committee (EPOC), which considers TMRC recommendations, conducts policy level reviews, and makes final funding recommendations to the NIH Director or his designee.

Summary of Fiscal Year 1997 Evaluations

Completed evaluations addressed a cross-section of the NIH research program, training, and career development. Three of the NIH evaluations completed are highlighted below.

A blue-ribbon panel of experts, the National Institute of Mental Health (NIMH) Intramural Research Program Planning Committee (IRPPC), was charged with evaluating and, as necessary, redefining the role of the Intramural Research Program (IRP) in fulfilling the mission of the NIMH. The IRPPC report, Finding the Balance: Report of the National Institute of Mental Health Intramural Research Program (IRP) Planning Committee (6697), provided the NIMH Director and Scientific Director with tools for fundamental renewal of the IRP. For example, the IRPPC recommended the consolidation of the Neuroscience Center of St. Elizabeth's Hospital with the NIMH IRP program at the Bethesda campus. Such a consolidation will provide better access to the special facilities the Bethesda campus offers, more opportunities for collaboration, and better access to patients.

Insuring a strong scientific infrastructure--

with a suitable talent base of highly qualified, highly trained, and diverse investigators capable of yielding the scientific discoveries of the future--and excellent research facilities is essential for medical progress. To increase the number of underrepresented minorities entering careers in biomedical research, the National Institute of General Medical Sciences (NIGMS) established two "bridges" programs to encourage minority students to go beyond the associate degree to obtain the bachelor's degree, or beyond the masters degree to obtain a doctorate. NIGMS established an Electronic Student Tracking and Reporting system to capture the information needed to assess the educational outcomes of students participating in the Bridges Program. The results from this evaluation, Evaluation of the Bridges to the Future Program (6093), indicated relatively high rates of transfer from 2-year to 4-year institutions, or from M.S. to Ph.D. programs, when compared to national norms. As a result, NIGMS is continuing support of this program.

To examine how research results translate into practice, A Study to Determine the Changes of Physician Prescribing of Antimicrobial Agents, Omeprazole, and H2-Receptor Antagonists for Peptic Ulcer Disease Two Years Following the NIH Consensus Conference on Helicobacter Pylori (6284) was designed to evaluate the effect of the consensus conference in prescribing patterns of physicians for the treatment of peptic ulcer disease (PUD), focusing on prescriptions for antimicrobial and antisecretory agents and H2-receptor antagonists. In particular, the study focused on whether the recommendations of a February 1994 NIH Consensus Development Conference advocating the use of antimicrobial agents for PUD patients with H. pylori infection were coincident with a change in physician prescribing patterns. The study revealed that two years after the highly publicized conference on the eradication of H. pylori, antimicrobial agents were not widely prescribed for the treatment of PUD within the Pennsylvania Medicaid population. The need to eradicate H. pylori to reduce and, ultimately, eliminate the recurrence of PUD in association with the findings of this research suggest that physicians in this specific population do not appear to be using national recommendations based on state-of-the-art information and clinical studies pertaining to PUD, as presented by the NIH. Restricting the study to Medicare patients limits its generalizability and could be related to issues of equity and quality of care for a disadvantaged population. If findings indicate significant cost savings associated with the use of the NIH recommended drug regimens in the treatment of PUD, this approach could be applicable to other populations.

Other NIH evaluations were completed on the following topics: the status of biomedical research facilities (3562.2); National Research Service Award Research Training Programs (5571); the National Heart, Lung and Blood Institute (NHLBI) article citation and patent references study (5577); the Rare Disease Clinical Research Database (6043); laboratory animal use, facilities, and resources (6048); promoting safe and effective genetic testing (6090); user satisfaction of the Physician Data Query data base (6092); research instrumentation needs (6121); toxicology and environmental health information resources (6534); evaluation of the international cooperative bio-diversity groups (6641); and the Federal Interagency Forum on Aging-Related Statistics (6652).

Evaluations in Progress

NIH has over two dozen evaluation projects in progress, as listed in Appendix B. Several of the major evaluations, soon to be completed, are summarized below.

An ongoing study that will provide information in the research program area is the Surgeon General's Report on Oral Health (6861). This study will provide information on how the public health elements of education, daily preventive self-care, and collaboration between individuals and their care providers can help produce a high level of health in a particular area. In spite of tremendous oral health progress throughout the last five decades, data demonstrate that results are distributed unevenly across populations. There still remain populations and communities with devastating levels of oral, dental, and craniofacial diseases. This report will identify specific steps that individuals and communities can take to improve their oral health status.

Another study under way in the research program area is a project titled, Evaluation of Simplification of the Human Resources Management System at the National Institutes of Health (6862). The Secretary has designated NIH as a pilot project for a period of 5 years to simplify personnel management and personnel administration policies and procedures. The main platform for the evaluation is the delegation of human resources management requirements and responsibilities; the long-range thrust is to strengthen NIH management and integrate human resources management into strategic planning. The evaluation seeks to determine the linkage between the impact of the human resources management delegations and the achievement of major NIH goals, such as advancing research to improve health and prevent disease.

An ongoing study that will provide information in the research training and career development area is, Study of National Needs for Biomedical and Behavioral Research Personnel (6860). This congressionally-mandated study determines the overall national need for research personnel in the biomedical, behavioral, and clinical sciences; and assesses the efforts of the National Research Service Awards (NRSA) programs in ensuring an adequate supply of qualified investigators. Earlier studies have included recommendations concerning both the number of NRSA research training positions needed in the future and the importance of systematic program monitoring and assessment. These recommendations have been used by NIH in formulating program priorities and budget requests, and by the Congress during the appropriations process. For example, it is not uncommon for congressional appropriations reports to cite the number of positions recommended by this study and to appropriate funds accordingly.

In the facilities area, NIH collaborates with the National Science Foundation (NSF) to assess the condition of existing facilities and identify needs for new and refurbished research facilities nationwide. Since its inception in 1988, the biennial Survey of Scientific and Engineering Research Facilities at Colleges and Universities (6863), prepared jointly by NIH and NSF, has provided considerable data on the amount, condition, and capital project activity by the Nation's research-performing institutions. For example, the 1996 Status of Biomedical Facilities reports that 47 percent of all biomedical research-performing institutions classified the amount of biological science research space as inadequate, and 51 percent indicated that they had an inadequate amount of medical science research space.