Performance Improvement 1997. Agency for Toxic Substances and Disease Registry


MISSION: To prevent exposure and adverse human health effects and diminished quality of life associated with exposure to hazardous substances from waste sites, unplanned releases, and other sources of pollution in the environment.

CDC Evaluation Program

The Centers for Disease Control and Prevention (CDC) places high priority on evaluations seeking to answer policy, program, and strategic planning questions. Performance improvement studies are of particular interest and importance to the agency. Evaluation studies are developed and selected based on CDC's eight strategies to achieve its mission:

  • Monitor health.
  • Detect and investigate health problems.
  • Conduct research to enhance prevention.
  • Develop and advocate sound public health policies.
  • Implement prevention methods.
  • Promote healthy behaviors.
  • Foster safe and healthful environments.
  • Provide leadership and training.

CDC's evaluation philosophy is consistent with its overall focus on the practice and application of sound public health strategies. This orientation is exemplified by the agency's "bottom up" nature of project development. Projects are conceptualized, developed, and monitored by the public health professionals at CDC who are most closely aligned with the everyday practice of public health--program staff.

On an annual basis, the Director of CDC provides guidance to the various Center, Institute, and Office Directors on 1-percent set-aside evaluation activities. This memorandum generally includes information about the types of studies to be carried out with 1-percent evaluation funds. Each proposal undergoes multiple levels of review. Initial review is conducted by the Office of Program Planning and Evaluation. Subsequent reviews are completed by staff within the Office of the Assistant Secretary for Planning and Evaluation. Study authors are provided with comments, questions, and recommendations made by reviewers. In addition to providing their responses, authors are given the opportunity to revise their proposals at this time.

A panel of CDC evaluators, scientists, and program managers meets to review and rank proposals. Review criteria include (1) relevance to prevention effectiveness; (2) relative importance of the public health problem being addressed; (3) probability that the proposed project will accomplish its objectives; and (4) the extent to which other CDC programs will derive benefit from the project. Results from this panel review are converted into a comprehensive ranking, which is provided to the Director of CDC. Final funding decisions are made at this time.

Finally, staff within the Office of Program Planning and Evaluation work closely with program staff to ensure development of a clear statement of work for selected projects. Prior to initiation of procurements, a final ad hoc review of the project statement of work is completed.

ATSDR Evaluation Program

The Agency for Toxic Substances and Disease Registry (ATSDR) receives its funds from Environmental Protection Agency Superfund appropriations, rather than from public health appropriations; therefore, ATSDR does not receive a 1-percent evaluation set-aside. Nevertheless, ATSDR is responding to the changes mandated in its program planning and evaluation efforts by the National Performance Review and the Government Performance and Results Act. To meet those requirements, ATSDR staff members modify the agency's planning process, incorporating implementation strategies and outcome performance measures.

Prominent issues addressed in the new planning system emphasize ATSDR's priority of improving the health of people affected by hazardous substances polluting the environment. Using exposure assessments and demographic data to identify people at risk and, more directly, assessing and addressing the concerns of ATSDR customers represent some of the improvements. The new planning system provides the basis for measuring ATSDR performance and making systematic improvements as part of its internal evaluation activities.

Summary of Fiscal Year 1996 CDC Evaluations

A total of 17 evaluation projects were completed in FY 1996, 6 of which are reported in appendix A. Ten of these projects were funded through the 1-percent evaluation set-aside, and 7 were funded with program dollars. These studies are of four types: data policy and surveillance studies, program evaluations, evaluations of educational interventions, and assessments of specialized aspects of current or proposed programs.

Data Policy and Surveillance Studies

Three studies sought to assess the implications of policy decisions on various existing or proposed data and surveillance systems. One of these projects was part of an effort to develop a comprehensive monitoring system to track the impact of health reform and provide feedback to policymakers. Although such a system was expected to monitor the impact of the 1994 Health Security Act, the report suggests that a system could be designed to inform health policy, regardless of the ultimate shape and timing of health reform. A second study was undertaken to assist in developing an appropriate accountability system for the Vaccine for Children (VFC) program that is both rigorous and practical for use in clinical settings. As part of this study, data collection procedures for proposed systems were tested to determine their practicality and usefulness for collecting vaccine usage data. The study also provided information about the acceptability of any accountability system to health care providers, since a system that is perceived as overly burdensome could discourage participation in the VFC program. Another study evaluated whether and to what extent use of CDC WONDER/PC by State and local public health practitioners positively affects the practice of public health, along a variety of dimensions (e.g., efficiency, scope, quality, and timeliness of work).

Program Evaluations

Four program evaluation studies were completed in FY 1996. These studies included evaluations of CDC's Prevention Effectiveness Activity, the National Center for Injury Prevention and Control's injury grant program, and the National Nosocomial Infections Surveillance Program. In addition, a case study of the Henry J. Kaiser Family Foundation's Community Health Promotion Program was completed.

Evaluations of Educational Interventions

Three of the four evaluations of educational interventions completed in FY 1996 focused on provider educational issues. One of these projects addressed health care providers' use and perceptions about vaccine package inserts, in an effort to improve the utility of inserts in communicating the nature and extent of dangers imposed by vaccines. A second immunization educational intervention developed and field tested a quality assurance tool designed to facilitate provider self-assessment, emphasize the importance of following prevention guidelines, and collect data about provider practices. A third study of this type assessed the effectiveness of the domestic violence module of an innovative physician training program that is being implemented at the University of California-Los Angeles Medical School. The final education-related evaluation study examined a variety of questions related to the distribution, awareness, use, and perceptions of the Chronic Disease Prevention (CDP) File. The CDP File consists of the Health Promotion and Education Database, the Comprehensive School Health Database, the Cancer Prevention and Control Database, and the Chronic Disease Prevention Directory.

Assessments of Specialized Aspects of Current or Proposed Programs

Five studies relating to discrete aspects of current or proposed programs were completed in FY 1996. Two of these projects were developmental and resulted in training materials. One activity concluded with the provision of an evaluation training course and materials for CDC Planning and Evaluation Officers. The other project identified key factors in the successful development and implementation of CDC-funded youth violence prevention projects and compiled the findings into a document to help individuals, groups, or communities implement youth violence prevention and intervention projects. Another study examined the impact of mandated managed care on health care. Based on eight case study organizations, this evaluation found (1) little prevention activity; (2) no relationship, or a poor one, with the local health department in half of the organizations; and (3) a dispersion of activities among community organizations that challenges the traditional roles of State and local health departments.

The two final studies of this type were related to immunization. One examined issues surrounding consent for adolescent immunization. As adolescent vaccination expands, particularly hepatitis B vaccination, issues related to need for parental consent for receipt of indicated vaccines become increasingly important. Similarly, operational issues surrounding the provision of immunization services is critically important. The final study evaluated how audits conducted in the State of Georgia from 1986 to 1994 affected immunization coverage levels in the preschool population and determined medical and management policies and practices that influence immunization rates.

CDC Evaluations in Progress

CDC's evaluation studies, in progress during FY 1996, consisted of four types: evaluations of data policy and surveillance studies; program evaluations; development of performance measures; and assessments of specialized aspects of proposed or existing programs.

Evaluations of Data Policy and Surveillance Systems

Ten studies are under way that relate to data management or reporting systems. These studies include evaluations of the use of data transmitted through CDC's National Electronic Telecommunications System for Surveillance (NETSS) and the Public Health Laboratory Information System (PHLIS), as well as an assessment of the National Immunization Survey (NIS). Evaluations of specific issues critical to data quality are also under way. These studies include evaluations of racial and ethnic identification data, redesign of the National Health Interview Survey (NHIS), and an examination of the effectiveness of CDC surveillance for drug-resistant pneumococcal infections.

In the case of the study of NETSS and PHLIS, determinations about how surveillance data transmitted through each of these systems are used at local, State, and Federal levels will be made. Similarly, the centerpiece of the NIS study is to design an evaluation system that will focus on three main questions: How well does the NIS meet the program needs of the National Immunization Program for estimates of immunization coverage rates for population groups of special interest? Is there a way to simplify the analytic methods currently used on the survey without jeopardizing the statistical integrity of the survey? Is there a way to make the survey data collection less costly without jeopardizing the statistical integrity of the survey?

Both the evaluation of racial and ethnic identification data and the redesign of the NHIS represent efforts to understand and refine survey operational issues. Finally, the study evaluating CDC's current surveillance system for drug-resistant Streptococcus pneumoniae will evaluate this system's hospital-based sentinel surveillance system for sensitivity and representativeness. Active population-based surveillance for invasive pneumococcal infections in two geographically distinct areas will be conducted. Surveillance areas for this evaluation are either adjacent to a sentinel surveillance hospital or in a community with one or more hospitals sufficiently large to serve as a comparison for the rest of the community.

Program Evaluations

Eight program evaluations are currently in progress. Included in this number are evaluations of the National Institute for Occupational Safety and Health (NIOSH) Health Hazard Evaluation Program, the Field Epidemiology Training Program, the National Laboratory and Training Network, and the Prevention Centers Program.

The study of the Health Hazard Evaluation (HHE) Program is a process evaluation. As such, the overall purpose of the study is to develop a tool and procedure by which the NIOSH can conduct an ongoing evaluation of the effectiveness of the HHE Program, which responds to 400 to 500 requests for on-site health hazard evaluations each year from employers, employees, employee representatives, or other Federal, State, or local agencies.

By contrast, two of the other studies mentioned are outcome evaluations. The evaluation of the Field Epidemiology Training Program (FETP) will ascertain whether the program has achieved its objectives, which are to train public health professionals in applied epidemiologic skills, to promote the sustainability of autonomous FETP's, and to develop a global network of national programs.

The evaluation of the National Laboratory Training Network (NLTN) will assess the degree to which the NLTN achieves its goals and mission. Study questions fall into five general categories: offerings related to needs of laboratories and their staff; quality of the training provided; impact of the training provided; quality of outreach and marketing; and barriers to training.

Finally, the evaluation of the Prevention Centers Program will assess to what extent CDC-supported research is providing the public health community with workable strategies to address major public health problems. Prevention research issues related to innovation, relevancy, dissemination, application, and quality of research are the focus of this project.

Development of Performance Measures

Four ongoing projects involve the development of performance measurement systems. Two of these studies relate directly to the Government Performance and Results Act of 1993 (GPRA). A third study will develop indicators by which the fiscal and programmatic impact of HIV-prevention community planning may be measured. The final study of this type entails the development of an evaluation protocol for State-based diabetes control cooperative agreement programs.

Assessments of Specialized Aspects of Proposed or Existing Programs

Fourteen specialized studies relating to discrete aspects of current or proposed programs are in progress. These studies include an evaluation of tuberculosis (TB) Outreach Worker Activities, as well as a community context study seeking to understand minors' access to tobacco products. Two studies related to violence prevention are also included in this category. One of these studies relates to suicide in Native American communities, and the other considers various implementation aspects of youth violence-prevention programs.

Each of the studies mentioned has at its core a desire to understand particular human behaviors and motivations. A clear understanding of these issues must be present before wholehearted program and fiscal commitments are made. For example, the Advisory Committee for the Elimination of TB recommended increased funding for TB outreach and workers conducting the outreach. However, prior to committing resources to this strategy, it is incumbent upon CDC to have a clear understanding of how outreach work is currently conducted and what outreach workers do in their everyday work. It is through the evaluation of TB Outreach Workers that the National Center for HIV, Sexually Transmitted Disease, and TB Prevention will be provided with baseline data about how outreach workers function.

Similarly, a community context study is also being conducted that will refine CDC's understanding of the relationships between public policies prohibiting minors' access to tobacco, implementation and enforcement of such policies, tobacco vendor perceptions, actions that may influence the sale of tobacco products to minors and the use of tobacco products by minors.

Finally, two evaluations addressing violence-prevention issues are also under way. The first of these studies is being conducted jointly by CDC and the Indian Health Service. This study will evaluate surveillance systems, process indicators, and outcomes of four multifaceted suicide-prevention programs. The focus of the second violence-prevention study involves evaluating the effectiveness of a selected program for health care providers and battered womens' advocates. Specifically, the ability of this program to successfully diagnose, manage, refer, and otherwise assist female victims of intimate partner violence will be assessed.

New Directions for CDC Evaluation

CDC's evaluation program will continue to support studies aimed at improving public health programs. This focus on improvement is consistent with the congressional intent of GPRA. Efforts to develop and implement performance indicators are currently under way at CDC. Similarly, as programs have been developing and implementing performance indicators, projects that assess the effectiveness and efficacy of such indicators have been initiated.

CDC's evaluation priorities in upcoming years will be based on five cross-cutting strategies that address the programmatic needs of its 11 Centers, Institutes, and Offices.

Strengthen Core Public Health Functions

A number of key activities at CDC center around the agency's commitment to strengthening the Nation's core public health functions. Tangible results of this commitment are reflected by CDC's work with State and local health departments and other partners throughout the country to obtain the information necessary for monitoring and evaluating health, conducting epidemiologic and laboratory studies, developing new technologies, and providing training and technical assistance to ensure that States and communities can protect the health of their citizens. Ongoing and proposed evaluation studies seek to answer how well CDC is fulfilling this commitment.

Enrich Our Capacity to Respond to Urgent Threats to Health

Epidemiologic investigations and laboratory work done by CDC and State and local health departments enable CDC to address urgent threats to public health in a timely and effective manner. Similarly, in a world increasingly threatened by emerging infections, CDC's leadership role in this area is critical. A number of studies are under way that address emerging infections, including Group B Streptococcus, perinatal HIV, and drug-resistant pneumococcal infections.

Develop Nationwide Prevention Strategies

One of CDC's key roles is to translate knowledge about effective methods of preventing disease and injuries into nationwide strategies that reach people in communities throughout the country. Several studies under way seek to assess the extent to which guidelines developed by CDC affect disease prevention activities throughout the Nation.

Promote Women's Health

Recognizing that women's health issues have not always received the attention they have warranted, CDC has identified promotion of women's health as one of its top priorities. Although a number of projects are funded by CDC's Office on Women's Health, there are several ongoing evaluation studies that address important women's health issues. One such study is an evaluation of guidelines for preventing perinatal HIV infection by measuring how recommendations are translated into routine clinical practice. Successful implementation of recommendations depend on the ability to maximize the opportunities for HIV-infected women to learn their HIV infection status, to be offered and receive preventive therapy, and to gain access to health and social services for themselves and their infants.

Invest in Our Nation's Youth

CDC's commitment to investing in the youth of our Nation is exemplified by school health education programs. However, CDC's investment is not limited to school-based intervention programs. For example, one study is designed to assess the effectiveness of various teen pregnancy prevention interventions. This study will result in a systematic methodology to identify and evaluate community intervention components that show promise based on behavioral science theory, available evaluation information, and the consensus of leaders in the field.