Performance Improvement 1996. National Institutes of Health

02/01/1996

MISSION: To sponsor and conduct medical research that leads to better health for all Americans.

NIH Evaluation Program

NIH develops scientific knowledge that leads to improved means to prevent illness, cure disease, and treat disability. It accomplishes its mission by conducting medical research in its own intramural laboratories and supporting research in universities, medical and health professional schools, hospitals, and other health research organizations. NIH fosters the widespread dissemination of the results of medical research, facilitates the training of research investigators, and ensures the viability of the research infrastructure. The NIH Evaluation Program is an integral part of how NIH manages its support and conduct of medical research.

Philosophy and Priorities

NIH evaluation activities assess program performance (efficiency, effectiveness, responsiveness); analyze both quantitative and qualitative results based on those assessments; and use the resulting information in policymaking, strategic planning, budgeting, and program development and management.

The quantity and diversity of diseases, disorders, and biological systems in the NIH portfolio make strategic planning and evaluation at NIH a complex task that is continuously evolving. Priorities are set and research programs are planned and evaluated at two levels: the institutes and centers (ICs), and centrally by the NIH Director, with whom the ultimate responsibility lies for the course of NIH-funded medical research.

This two-level approach ensures that planning and priorities specific to the mission of each IC are fully developed and implemented with a clear vision and within the fiscal constraints set by the IC budget, and that there is central leadership for developing crosscutting initiatives and promoting collaboration among the ICs. The NIH Evaluation Program provides information to assist the NIH Director and IC directors in determining whether NIH goals and objectives are being achieved and to help guide policy development and program direction.

Policies and Operations

A distinguishing feature of the NIH Evaluation Program is the extent to which it employs a variety of evaluation strategies and inputs that include the use of national advisory councils, boards of scientific counselors, consensus development conferences, and ad hoc committees that help chart scientific directions and select the most promising research.

Program evaluations are funded through both a trans-NIH mechanism, the 1 percent set-aside authority, and by individual IC program funds. A two-tier system reviews project requests to use set-aside funding. The first tier involves review and recommendations by the NIH Technical Merit Review Committee (TMRC) on the technical aspects of project proposals and on whether a project fits within HHS guidelines for set-aside funding. The second tier involves the NIH Evaluation Policy Oversight Committee (EPOC) that considers TMRC recommendations and makes final funding recommendations to the NIH Director or designee. It also conducts policy-level concept reviews of proposals for NIH-wide evaluation studies, establishes the overall NIH set-aside budget, and oversees the process. EPOC recommendations are approved by the NIH Director or designee. Concurrently, the ICs fund program evaluations from their budgets; these evaluations are used by various committees, working groups, task forces, workshops, conferences, and symposia for program management and development.

An important characteristic of NIH's Evaluation Program is solicitation from a number of sources to provide information to the NIH Director and the IC directors. Discussions are continuously held with extramural grantees, intramural investigators and other NIH staff, members of Congress and the Administration, and members of the public, including professional societies and voluntary health organizations. These individuals and groups provide valuable input on pressing public health needs, important scientific opportunities, knowledge gaps, and the balance between patient-oriented and laboratory research.

Summary of Fiscal 1995 NIH Evaluations

The evaluations completed in fiscal 1995 addressed a cross- section of the NIH research program, as demonstrated by the following sample of studies.

"Fetal Alcohol Syndrome: Diagnosis, Epidemiology, Prevention, and Treatment" responded to the congressional mandate that a study of fetal alcohol syndrome (FAS) and related birth defects be carried out. A complete description of this study, funded by the National Institute of Alcohol Abuse and Alcoholism, is in chapter II of this report.

"Directions in Nursing Research Training" summarizes information on National Institute of Nursing Research (NINR) (1) funding that permits nurse scientists to follow research career paths; (2) recommendations for nursing research training in a National Research Council report; and (3) the views and recommendations of the scientific community. The report also discusses research training issues in the broad context of Federal and academic perspectives. The report recommends that NINR disseminate the report widely, analyze the research career paths of NINR trainees and fellows to determine the criteria for successful training experiences, emphasize the need to increase the number of nurses with doctoral degrees, and support research consortia and partners.

"Cancer at a Crossroads: A Report to Congress for the Nation" describes the results of an evaluation undertaken at the request of Congress to assess the achievements of the National Cancer Program, identify barriers to reducing the burden of cancer, and recommend future research and program directions. A subcommittee appointed to address these issues concluded that the strongest strategy for a renewed "war on cancer" should (1) apply currently available knowledge about cancer prevention and care to all segments of the population; (2) increase support for transnational research that develops basic cancer knowledge into preventive strategies, new technologies, and effective treatments; and (3) increase support for basic cancer research to ensure the continued flow of new discoveries that lead to better cancer prevention and care.

"Measuring Social Inequalities in Health" provides the results of a National Institute of Child Health Development conference held to develop recommendations for improving or changing measures of socioeconomic gradients in federally supported health data sets. The conferees recommended (1) the availability of existing social class data be publicized, and (2) researchers be encouraged to analyze the data. The conference also provided investigators wishing to apply for NIH funds with improved tools for implementing the revised guidelines for the inclusion of women and minorities in clinical research.

"Report on Outreach Activities of the National Library of Medicine" details a 5-year review of National Library of Medicine (NLM) outreach activities in response to congressional concerns that hospitals and health professionals in isolated areas lack access to recent scientific and biomedical information. The report found that (1) the number of outreach projects had increased from 16 in 1989 to almost 300 at the end of 1994; (2) approximately 30,000 user codes issued in 1989 had increased to nearly 100,000 by 1994; (3) 4 million searches of the NLM databases in 1989 increased to 7 million searches in 1994; and (4) similar changes have occurred in the use of the "Grateful Med" system.

"Support for Bioengineering Research" responded to a request from Congress that the HHS Secretary, acting through the NIH Director, conduct a detailed inventory of sources and amounts of public and private funding for basic bioengineering research in fiscal 1993. The report recommended (1) establishing an Interagency Bioengineering Coordinating Committee; (2) including basic bioengineering research within appropriate intramural programs; and (3) providing, through the Federal Register, a "comment period" notice to suggest research topics for inclusion in the annual Small Business Innovation Research Omnibus solicitation.

NIH Evaluations in Progress

NIH currently supports more than 30 evaluations. They range from small- to large-scale assessments, from evaluative studies to comprehensive evaluations. Examples illustrating this range include the following:

"Evaluation of the National Research Service Award (NRSA) Research Training Program" is the first phase of a longer term evaluation effort directed at examining the extent to which the objectives of NIH/NRSA programs are being met. It is an update and extension of earlier work performed by the National Academy of Sciences published in a 1984 report titled "Career Achievements of NIH Predoctoral Trainees and Fellows." The study will address the major evaluation questions frequently posed by constituencies of NIH research training; make efficient use of extant data relevant to these evaluation questions; and identify gaps in existing databases, thus guiding the development of the data-collection efforts planned for the second and third stages of the evaluation.

"Implementation Phase for an Evaluation of the Minority Access to Research Careers (MARC) Honors Undergraduate Research Training Program" will evaluate the degree to which the intended goals of the MARC undergraduate program are being met and will result in the establishment of a general information database on the program and its trainees. Information is being collected on the programs implemented at MARC institutions, the trainees, the training experience, and former trainees' subsequent education and career paths.

"Evaluation of NIH Implementation of Section 491 of the PHS Act Mandating a Program of Protection of Research Subjects" is examining NIH's implementation of a program to ensure adequate protection to all individuals who are used as research subjects. The study is attempting (1) to determine the impact of the program on the administrative operations and research activities of awardee institutions and (2) to examine information on some financial costs, administrative burden, levels of effort, and selected program measures related to the protection of human subjects, the education of researchers, and the facilitation of research.

The Task Force on Genetic Testing is examining issues surrounding the use and regulation of genetic tests. The objectives of the task force are to review the state of the art of genetic testing; examine the strengths and weaknesses of current practices and policies relating to testing; and, if needed, recommend changes or policy options to ensure that the public is protected so that only the appropriate tests are done by qualified laboratories.

The study, "Methodology To Assess the Impact of National Heart, Lung and Blood Institute (NHLBI) Research," is examining alternative measures to demonstrate the impact of basic research in terms of reductions in disease mortality and morbidity. The objectives of this study are to introduce the concept of examining patients as an additional measure for assessing the impact of NIH-supported research; to develop a methodology to use patients in evaluation research; and to apply the methodology to an NHLBI project to compare the outcomes for individual research project grants funded under requests for applications with individual research grants funded from unsolicited applications.

The study, "Characterization of User Population and User Satisfaction of the Physician Data Query (PDQ) Database," is being conducted to obtain information on PDQ database users and their satisfaction with the comprehensive cancer database. The objectives of the study are to survey PDQ database users to characterize who is using the database, and to determine how the information is used and if users are satisfied with the information they receive and with the method of retrieval (e.g., CD-ROM, online, or hard copy).

The "Evaluation of Laboratory Animal Use, Facilities, and Resources" will acquire, analyze, aggregate, and report objective data for fiscal 1993 on the number and characteristics of animals used and on animal-related facilities and resources at those institutions and organizations that have animal welfare assurances currently on file with the NIH Office of Protection From Research Risks.

The "Survey of Academic Research Equipment" responds to a congressional directive under Public Law 96-44, Section 7, that the National Science Foundation, in cooperation with NIH, conducts an ongoing instrumentation survey and issue biennial reports. The purpose of the study is to assess national trends in medical research instrumentation.

New Directions for NIH Evaluation

NIH will continue to engage in a wide variety of evaluation activities. Priorities for future evaluation activities reflect input from a number of sources: the HHS Strategic Plan, the NIH Strategic Plan, and the NIH GPRA Performance Plan--all in the developmental phase; the NIH Director's Areas of Emphasis; the Administration's High Priority Areas; recommendations of NIH/IC directors; and recommendations of the Evaluation Policy and Oversight Committee. As a result, NIH will give priority funding to the following areas:

  • Medical Research. This area focuses on issues such as the economic impact of biomedical research, clinical research including the impact of managed care, the health of special populations, and the development of measures for medical research accomplishments.
  • Information Dissemination. This area focuses on issues such as expanded data collections to address emerging priorities, the health information dissemination infrastructure and strategy, and technology transfer.
  • Research Training. This area focuses on issues involving national needs for medical research personnel, the quality of NIH training programs, and diversity among trainees.
  • Research Infrastructure. This area focuses on issues involving the maintenance and construction of buildings and facilities, computers in biomedical research and medicine, and informatics in medical research.

NIH evaluation studies provide a rational basis for managerial decisionmaking and responding to public concerns for accountability in government. In addition, it is through such studies that NIH is able to determine its progress in meeting scientific objectives, strengthening research and administrative activities, and contributing to its mission of sponsoring and conducting research that leads to better health for all Americans.