Performance Improvement 1997. National Institutes of Health


MISSION: To sponsor and conduct medical research that leads to better health for all Americans.

Evaluation Program

The National Institutes of Health (NIH) develops scientific knowledge that leads to improved means to prevent illness, cure disease, and treat disability. This is done by conducting medical research in its intramural laboratories and supporting research in universities, medical and health professional schools, hospitals, and other health research organizations. NIH fosters the widespread dissemination of the results of medical research, facilitates the training of research investigators, and ensures the viability of the research infrastructure. The NIH Evaluation Program is an integral part of how NIH manages its support and conduct of medical research.

Philosophy and Priorities

NIH evaluation activities assess program performance (efficiency, effectiveness, responsiveness); analyze both quantitative and qualitative results based on those assessments; and use the resulting information in policy making, strategic planning, budgeting, and program development and management.

The quantity and diversity of diseases, disorders, and biological systems in the NIH portfolio make strategic planning and evaluation at NIH a complex task that is continuously evolving. Priorities are set and research programs are planned and evaluated at two levels: the Institutes and Centers (IC's); and centrally by the NIH Director, with whom the ultimate responsibility lies for the course of NIH-funded medical research.

This two-level approach ensures that planning and priority setting specific to the mission of each IC are fully developed and implemented with a clear vision and within the fiscal constraints set by the IC budget and that there is central leadership for developing cross-cutting initiatives and promoting collaboration among the IC's. The NIH Evaluation Program provides information to assist the NIH Director and IC Directors in determining whether NIH goals and objectives are being achieved and to help guide policy development and program direction.

Policies and Operations

A distinguishing feature of the NIH Evaluation Program is the extent to which it employs a variety of evaluation strategies and inputs that include the use of national advisory councils, boards of scientific counselors, consensus development conferences, and ad hoc committees that help to chart scientific directions and select the most promising research to support.

Program evaluations are funded through both a trans-NIH mechanism, the 1-percent set-aside authority, and by individual IC program funds. A two-tier system is used to review project requests to use set-aside funding. The first tier involves a review and recommendations by the NIH Technical Merit Review Committee (TMRC) on the technical aspects of project proposals and whether a project fits within HHS guidelines for use of the set-aside fund. The second tier involves the NIH Evaluation Policy Oversight Committee (EPOC), which considers TMRC recommendations and makes final funding recommendations to the Director of NIH or his or her designee. It also conducts policy-level concept reviews of proposals for NIH-wide evaluation studies; establishes the overall NIH set-aside budget; and oversees the process. EPOC recommendations are approved by the NIH Director or his or her designee. Concurrently, the IC's fund program evaluations from their budgets that are used by various committees, working groups, task forces, workshops, conferences, and symposia to assist them and the NIH in program management and development.

One of the ways NIH sets priorities and evaluates its programs is discussion with extramural grantees, intramural investigators and other NIH staff, members of Congress and the Administration, and members of the public, including professional societies and voluntary health organizations. These individuals and groups provide valuable input on pressing public health needs, important scientific opportunities, knowledge gaps, and the balance between patient-oriented and laboratory research.

Summary of Fiscal Year 1996 Evaluations

The evaluations completed in FY 1996 addressed a cross-section of the NIH research program, including bench research, clinical research, a demonstration project, and drug research. Examples of some of these evaluations are described below.

Evaluation of NIH Shared Instrumentation Grant Programs: Reports From Users documents the results of surveys of users of costly, state-of-the-art instruments purchased through the National Center for Research Resources shared instrumentation program. The program provides equipment that is to be shared among mainly NIH-funded investigators (termed major users), and other researchers who do not have an NIH grant. The study found that 16,050 scientists nationwide used these instruments in 1993; that of these, 69 percent were NIH-supported investigators; and that three-quarters of the major users believed the instruments were essential to their research. The study also found that 70 percent of 11-year-old equipment was still in use, and that 4 out of 5 major users of this older equipment believed the equipment was adequate for their research.

Moving from bench science, the report Clinical Research in the National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK) examined how much clinical research was being conducted in NIDDK's research portfolio, under two definitions of clinical research. The first definition required that the patient and the physician had a consultation; this was the narrow definition. The broader definition, adopted by the Advisory Committee to the NIH Director on Clinical Research, subsumed the narrow definition and also included areas of epidemiology, behavioral studies of patients, outcomes research, health services research, and technology development. The report found that in FY 1995, NIDDK spent 31 percent of its extramural budget on clinical research using the narrow definition and 37 percent of its budget using the broader definition; NIDDK spent 29 percent of the intramural budget on clinical research using the narrow definition and 33 percent using the broader definition.

Turning to evaluation of a demonstration project, the report Navajo Alcohol Rehabilitation Demonstration Program examined a federally funded program to provide culturally sensitive treatment for Navajo Native Americans who suffer from alcoholism and the effects of alcohol abuse. It is the only program to use Navajo-speaking counselors in the treatment. The evaluation found a higher level of participation in culturally sensitive therapies after completion of an inpatient treatment program, low use of aftercare services, a high program completion rate, and a low readmission rate. Data on long-term effects were not available.

Another way to treat chemical dependency is medication. The report Development of Medications for the Treatment of Opiate and Cocaine Addictions: Issues for the Government and Private Sector examines the National Institute of Drug Abuse's program to coordinate and encourage academic, private, and Federal regulatory involvement in developing and marketing new medications to treat drug abuse. The study found that pharmacotherapy for the treatment of drug addiction has received little attention; only two additional drugs beyond methadone have been approved, both of which were developed at least 20 years ago; and there is no approved medication for the treatment of cocaine addiction. The report noted major disincentives to pharmaceutical research and development for antiaddiction medications, among which is a lack of sustained Federal leadership. The report gives recommendations to help remove these barriers.

Evaluations in Progress

NIH supports a number of evaluations in progress. They range from small- to large-scale assessments, from evaluability studies to comprehensive evaluations. A few examples to illustrate the range include an examination of the research grant application and award processes; school health programs in Grades Kþ12; evaluation of laboratory animal use, facilities, and resources; and development of a data base for NIH-supported rare disease research.

The study Career Status and Satisfaction With NIH Research Grant Application and Award Processes: A Sample Survey of FY 1992 R01 and R29 Applicants will survey individuals who applied to NIH for an R01 or R29 grant in FY 1992. The purpose is to obtain information on the current status of these individuals. Career progress and research productivity will be assessed by comparing: (1) funded principal investigators versus others supported on NIH grants; (2) principal investigators funded on grants from non-NIH sources; and (3) those who are unsuccessful in getting any support for their research. Respondents will also be asked to assess their satisfaction with the grants application and award process and to provide their opinions on specific modifications NIH is considering for improving policies and procedures.

The study Comprehensive School Health Programs in Grades Kþ12, conducted by the Institute of Medicine, will assess grade-school health education, school health promotion and disease prevention, and school-based health care services from kindergarten through high school. The study will identify and examine model programs to formulate strategies for successful implementation of school health programs at national, State, and local levels.

NIH is responsible for the nationwide infrastructure for animals in research, which includes not only laboratory animals, but also the resources and facilities required to ensure their availability, appropriate use, and humane care. The purpose of Evaluation of Laboratory Animal Use, Facilities and Resources--Phase II is to provide NIH and other government agencies with information and objective data to inform the current national discussion--involving the press, animal rights activists, scientists, and others--about laboratory animal use.

Congress requires NIH to report annually on clinical research it is sponsoring for rare diseases. Development of a Database for NIH-Supported Rare Disease Research will establish a data base of NIH-sponsored rare disease clinical research. The Oak Ridge Institute for Science and Education will provide the instrument for NIH to determine the presence of existing data bases and the nature and extent of data on rare disease studies funded by the NIH. The contractor will review the completed instruments and will develop and provide recommendations on estimated costs and on the best ways to make the data base accessible to the biomedical research community, practicing physicians and other health professionals, and the public.

New Directions for Evaluation

NIH will continue to engage in a wide variety of evaluation activities. Priority setting for future NIH evaluation activities includes input from a number of sources including the HHS Strategic Plan, the NIH Government Performance and Results Act Performance Plan, and the NIH Strategic Plan--all in the developmental phase; the NIH Director's Areas of Emphasis; the Administration's High-Priority Areas; recommendations of the NIH IC Directors; and the recommendations of the EPOC. As a result, NIH will give priority funding to the following evaluation areas of study.

  • Medical Research--This area focuses on issues such as the economic impact of medical research; clinical research including the impact of managed care; the health of special populations; and the development of measures for medical research accomplishments.
  • Data Collection and Dissemination--This area focuses on issues such as expanded data collections to address emerging priorities; the health information dissemination infrastructure and strategy; and technology transfer.
  • Research Training--This area focuses on issues involving national needs for medical research personnel; the quality of NIH training programs; and diversity among trainees.
  • Research Infrastructure--This area focuses on issues involving the maintenance and construction of buildings and facilities; computers in biomedical research and medicine; and informatics in medical research.

NIH evaluation studies provide a rational basis for managerial decision making and responding to public concerns for accountability in government. Additionally, it is through such studies that the NIH is able to determine progress in meeting its scientific objectives, strengthening research and administrative activities, and contributing to the mission of NIH to sponsor and conduct research that leads to better health for all Americans.