Performance Improvement 1996. Centers for Disease Control and Prevention and Agency for Toxic Substances and Disease Registry


MISSION: To promote health and quality of life by preventing and controlling disease, injury, and disability.

Agency for Toxic Substances and Disease Registry (ATSDR)

MISSION: To prevent exposure and adverse human health effects and diminished quality of life associated with exposure to hazardous substances from waste sites, unplanned releases, and other sources of pollution in the environment.

CDC Evaluation Program

CDC places high priority on evaluations seeking to answer policy, program, and strategic planning questions. Performance improvement studies are of particular interest and importance to CDC. Evaluation studies are developed and selected based on CDC's eight strategies to achieve its mission. These strategies are as follows:

  • Monitor health.
  • Detect and investigate health problems.
  • Conduct research to enhance prevention.
  • Develop and advocate sound public health policies.
  • Implement prevention methods.
  • Promote healthy behaviors.
  • Foster safe and healthful environments.
  • Provide leadership and training.

CDC's evaluation philosophy is consistent with its overall focus on the practice and application of sound public health strategies. This orientation is exemplified by CDC's "bottom up" nature of project development. Projects are conceptualized, developed, and monitored by CDC's public health professionals who are most closely aligned with the everyday practice of public health--the programmatic staff.

Annually, the Director of CDC provides guidance to the various center, institute, and office (CIO) directors on 1 percent set- aside evaluation activities. This memorandum generally includes information about the types of studies to be carried out with 1 percent evaluation funds. Each proposal undergoes multiple levels of review. Initial review is conducted by the Office of Program Planning and Evaluation (OPPE). Subsequent reviews are completed by the Office of the Assistant Secretary for Planning and Evaluation staff. Study authors are provided with comments, questions, and recommendations made by reviewers. In addition to providing their responses, authors may revise their proposals at this time.

A panel of CDC evaluators, scientists, and program managers are convened to review and rank proposals. Review criteria include (1) relevance to prevention effectiveness, (2) relative importance of the public health problem being addressed, (3) probability that the proposed project will accomplish its objectives, and (4) extent to which other CDC programs will derive benefit from the project. Results from the panel review are converted into a comprehensive ranking that is provided to the Director of CDC. Final funding decisions are made at that time.

Finally, staff within OPPE work closely with program staff to ensure development of a clear statement of work for selected projects. Prior to initiation of procurements, a final ad hoc review of the project statement of work is completed.

ATSDR Evaluation Program

ATSDR receives its funds from the U.S. Environmental Protection Agency/Superfund appropriations rather than Public Health Service appropriations; therefore ATSDR does not receive a 1 percent evaluation set-aside. Nevertheless, ASTDR is responding to the changes mandated in its program planning and evaluation efforts by the National Performance Review and the Government Performance and Results Act (GPRA). To meet those requirements, ATSDR staff members modified ATSDR's planning process by incorporating implementation strategies and outcome/performance measures.

Prominent issues addressed in the new planning system emphasize ATSDR's priority to improve the health of people affected by hazardous substances polluting the environment. Using exposure assessments and demographic data to identify people at risk and, more directly, assessing/addressing the concerns of the Agency's customers represent some of its improvements. The new planning system provides the basis for measuring ATSDR performance and making systematic improvements a part of its internal evaluation activities.

Summary of Fiscal 1995 CDC Evaluations

Fourteen evaluations were completed in fiscal 1995. Of this number, 10 were funded through the 1 percent evaluation set- aside, and 4 were funded with program dollars. These studies are of four types: data policy/surveillance studies, program evaluations, evaluations of educational interventions, and organizational assessments.

Data Policy/Surveillance Studies

Five studies sought to assess the implications of policy decisions on various existing or proposed data/surveillance systems. One of these studies addressed automated analysis of the four major data systems maintained by the National Center for Health Statistics: vital statistics, health examination, health interview, and health care. The primary domain of this project was health care and health interview data. Another study assessed the feasibility of developing an injury risk factor surveillance system; a third project entailed the conduct of two distinct surveys to explore issues related to influenza vaccination coverage.

Program Evaluations

Three program evaluation studies were completed in fiscal 1995. These studies included evaluations of the Medical Examiner/Coroner Information-Sharing Program, the Fatality and Assessment and Control Program, and the Lead Poisoning Prevention Program (LPPP). These program evaluations were context-specific. For example, the evaluation of LPPP examined an emerging trend in the program--the effects of managed care reforms on childhood lead poisoning prevention (CLPP) programs, focusing on CLPP programs funded by CDC.

Evaluations of Educational Interventions

Of the three evaluations of educational interventions completed in fiscal 1995, two of them focused on guides to diabetes education. One guide, written in Spanish, was evaluated for the appropriateness of its use for diabetic patients in Hispanic populations. The second diabetes educational intervention focused on health care providers. This project provided information to the Division of Diabetes Translation on the impact of three different provider interventions related to the dissemination of a provider guide. The final study evaluated the progress and activities conducted by the 71 HIV education programs funded by CDC's Division of Adolescent and School Health through cooperative agreements with State and local education agencies.

Organizational Assessments

Three organizational assessments were carried out in fiscal 1995. Two of these studies addressed management issues of discrete programmatic areas: immunization and women's health. The third study sought to develop a comprehensive evaluation strategy that can be incorporated into planning budget and legislative processes in the National Center for Chronic Disease Prevention and Health Promotion.

CDC Evaluations in Progress

During fiscal 1995, 18 evaluation studies were in progress. These studies consist of three types: evaluations of data policy/surveillance systems, program evaluations, and assessments of specialized aspects of proposed programs.

Evaluations of Data Policy/Surveillance Systems

Six studies are under way that relate to either data management or reporting systems, or both. These studies include evaluations of racial and ethnic identification data, surveillance data for drug-resistant pneumococcal infections, and an assessment of the National Nosocomial Infections Surveillance (NNIS) Program.

In the study of racial and ethnic identification data, three major activities will be performed to improve the data quality of natality data systems. These activities include (1) a systematic evaluation of how respondents for birth records interpret current questions about their ethnic and racial identity; (2) an assessment of how respondents classify themselves on birth certificates regarding the current racial and ethnic categories used in the National Center for Health Statistics' natality statistics data systems; and (3) a comparison of respondents' interpretation and classification in response to the current categories of how they would interpret questions and classify themselves by using other racial and ethnic categories under consideration by the Office of Management and Budget.

The study evaluating CDC's current surveillance system for drug- resistant streptococcus pneumoniae will evaluate the system's hospital-based sentinel surveillance system for sensitivity and representativeness. Active population-based surveillance for invasive pneumococcal infections in two geographically distinct areas will be conducted. Surveillance areas for this evaluation are either adjacent to a sentinel surveillance hospital or in a community with one or more hospitals that are sufficiently large enough to serve as a comparison for the rest of the community.

Finally, the major purpose of the assessment of the NNIS Program is to develop a methodology for assessing NNIS hospitals' accuracy in identifying and reporting nosocomial infections in NNIS hospitals and to assess the utility of criteria used to diagnose these infections. Specifically, the study is designed to (1) confirm the presence of nosocomial infections, by site, in patients previously reported to NNIS as infected; (2) confirm the absence of infections, by site, in a hospital-selected population of patients who were at risk for acquiring nosocomial infections but for whom no infections were reported; (3) identify certain risk factors and nosocomial infection case-finding criteria to the identified infection site; and (4) determine whether consistent findings can be achieved by two similarly trained observers who apply specific nosocomial infection case-finding criteria to a retrospective review of a given medical record.

Program Evaluations

Seven major program evaluations are currently in progress. Included in this number are evaluations of the National Institute for Occupational Safety and Health's Health Hazard Evaluation (HHE) Program, the Field Epidemiology Training Program (FETP), the National Laboratory and Training Network (NLTN), and the Dengue Hemorrhagic Fever Prevention and Control Program in San Juan, Puerto Rico.

The HHE study is a process evaluation. As such, the overall purpose of the study is to develop a tool and procedure by which NIOSH can conduct an ongoing evaluation of the effectiveness of the HHE Program, which responds to 400 to 500 requests for onsite health hazard evaluations each year from employers; employees; employee representatives; or Federal, State, or local agencies.

By contrast, two of the studies mentioned are outcome evaluations. The evaluation of FETP will ascertain whether the program has achieved its objectives to train public health professionals in applied epidemiologic skills, to promote the sustainability of autonomous FETPs, and to develop a global network of national programs.

The evaluation of NLTN will assess the degree to which NLTN achieves its goals and mission. Study questions fall into five general categories: (1) offerings related to needs of laboratories and their staff, (2) quality of the training provided, (3) impact of the training provided, (4) quality of outreach and marketing, and (5) barriers to training.

Finally, the evaluation of the Dengue Hemorrhagic Fever Prevention and Control Program will assess three major components of the program: (1) proactive disease surveillance, (2) education of the medical community, and (3) community-based mosquito control.

Assessments of Specialized Aspects of Proposed Programs

Five specialized studies relating to discrete aspects of current or proposed programs are in progress. These studies include an evaluation of tuberculosis (TB) outreach worker activities as well as a community-context study seeking to understand minors' access to tobacco products. Two studies related to violence prevention are also included in this category. One of these studies relates to suicide in Native American communities, and the other considers various implementation aspects of youth violence prevention programs.

Each of the studies mentioned has as its core a desire to understand particular human behaviors and motivations. A clear understanding of these issues must be present before wholehearted program and fiscal commitments are made. For example, the Advisory Committee for the Elimination of TB recommended increased funding for TB outreach and workers conducting the outreach. However, prior to committing resources to this strategy, CDC must have a clear understanding of how outreach work is currently conducted, and what outreach workers do in their everyday work. It is through the evaluation of TB outreach workers that the National Center for HIV, STD, and TB Prevention will be provided with baseline data about how outreach workers function.

Similarly, a community context study is also being conducted that will help refine CDC's understanding of the relationships between public policies prohibiting minors' access to tobacco, the implementation and enforcement of such policies, tobacco-vendor perceptions and actions that may influence the sale of tobacco products to minors, and the use of tobacco products by minors.

Finally, two evaluations addressing violence prevention issues are also under way. The first study is being conducted jointly by CDC and the Indian Health Service. The study will evaluate surveillance systems, process indicators, and outcomes of four multifaceted suicide prevention programs. The second violence prevention study will identify key factors in the successful development and implementation of CDC-funded youth violence prevention programs. As part of the study, a document will be developed that will assist individuals, groups, or communities as they implement their own youth violence prevention programs.

New Directions for CDC Evaluation

CDC's evaluation program will continue to support studies aimed at improving public health programs. This focus on improvement is consistent with the congressional intent of GPRA. Efforts to develop and implement performance indicators are currently under way at CDC. Similarly, as programs have been developing and implementing performance indicators, projects that assess the effectiveness and efficacy of such indicators have been initiated. CDC's evaluation priorities in upcoming years will be based on five crosscutting strategies that address the programmatic needs of its 11 CIOs.

Strengthen Core Public Health Functions

A number of key activities at CDC center around its commitment to strengthen the Nation's core public health functions. Tangible results of this commitment are reflected by CDC's work with State and local health departments and other partners throughout the country to obtain information necessary for monitoring and evaluating health, conducting epidemiologic and laboratory studies, developing new technologies, and providing training and technical assistance to help ensure that States and communities can protect the health of their citizens. Ongoing and proposed evaluation studies seek to answer how well CDC is fulfilling this commitment.

Enrich Its Capacity To Respond to Urgent Threats to Health

Epidemiologic investigations and laboratory work done by CDC and State and local health departments enable CDC to address urgent threats to public health in a timely and effective manner. Similarly, in a world increasingly threatened by emerging infections, CDC's leadership role in this area is critical. In fiscal 1996, a number of evaluation studies address important programmatic issues related to infectious disease.

Develop Nationwide Prevention Strategies

One of CDC's key roles is to translate knowledge about effective methods of preventing disease and injuries into nationwide strategies that reach people in communities throughout the country. Several studies under way seek to assess the extent to which guidelines developed by CDC impact disease prevention activities throughout the Nation.

Promote Women's Health

Recognizing that women's health issues have not always received the attention they have warranted, CDC has identified promotion of women's health as one of its top priorities. Although a number of projects are funded by CDC's Office on Women's Health, there are several ongoing evaluation studies that address important women's health issues. An example of one such study is an evaluation of guidelines for preventing perinatal HIV infection by measuring how recommendations are translated into routine clinical practice. Successful implementation of recommendations depends on the ability to maximize the opportunities for HIV- infected women to learn their HIV infection status, to be offered and receive preventive therapy, and to gain access to health and social services for themselves and their infants.

Invest in the Nation's Youth

CDC's commitment to investing in the Nation's youth is exemplified by its school health education programs. However, CDC's investment is not limited to school-based intervention programs. For example, one recently initiated study is designed to assess the effectiveness of various teen pregnancy prevention interventions. This study will result in a systematic methodology to identify and evaluate community intervention components that show promise based on the behavioral science theory, available evaluation information, and consensus of leaders in the field.