Performance Improvement 1998. Agency for Toxic Substances and Disease Registry


MISSION: To prevent exposure and adverse human health effects and diminished quality of life associated with exposure to hazardous substances from waste sites, unplanned releases, and other sources of pollution present in the environment.

Evaluation Program

The Agency for Toxic Substances and Disease Registry (ATSDR) was created as a Federal agency by the Comprehensive Environmental Response, Compensation, and Liability Act (CERCLA), more commonly known as Superfund legislation. ATSDR was created to implement the health-related sections of CERCLA and other laws that protect the public from hazardous waste and environmental spills of hazardous substances. ATSDR relies heavily on strong working relationships with tribal, State, and local organizations; private, nonprofit, and community-based organizations; and other Federal agencies to implement critical environmental health programs. As part of the public health assessment process, ATSDR recommends actions that can be taken to reduce or mitigate the risks of adverse health effects from exposures to toxic substances to the Environmental Protection Agency (EPA) and State and local governments.

The ATSDR evaluation program is coordinated with the agencywide strategic planning process, which was started in March 1996 to implement requirements of the Government Performance and Results Act (GPRA). ATSDR's strategic goals and its annual performance plan are the result of an interactive process that reflects a long-term commitment by Agency staff to develop stronger relationships among external clients and stakeholders, to assess products and services using relevant data, and to improve our processes and systems for more efficient accomplishment of ATSDR's mission. The following strategic questions are important to the Agency:

FACE="WP MathA">‚ How should ATSDR evaluate the public health needs of its stakeholders and involve those stakeholders in the Agency's program planning, implementation, and evaluation?

  • How should ATSDR's roles in assessment, science, health promotion, and service be implemented to meet the needs of its stakeholders and to achieve its mission?
  • How does ATSDR ensure that it has the authorities and resources necessary to address the public health needs of its stakeholders?
  • How should ATSDR evaluate the public health impact of its products and services?

Effectively focusing ATSDR's programs on these issues enables the Agency to understand stakeholders' needs, address its mandates, and evaluate its programs so that ATSDR operations, management, products, and services can be improved.

ATSDR receives its funds from the Hazardous Substance Trust Fund/Superfund appropriations rather than Public Health Service appropriations; therefore, ATSDR does not receive a 1-percent evaluation set-aside. Nevertheless, the new planning system provides the basis for measuring ATSDR performance and for making systematic improvements a part of its internal evaluation activities.

An example of such "in-house" evaluation activities through ATSDR resources is the Evaluation Working Group. This group was formed to develop a long-term strategy for site-specific evaluations to better assess the outcomes and impact of ATSDR activities at hazardous waste sites. The Methyl Parathion Evaluation Workshop was established to allow coordination between and continue work being accomplished by ATSDR, the EPA, and State and local agencies when responding to the illegal spraying of homes and businesses with methyl parathion, an agricultural pesticide. The workshop's intended purpose is to discuss approaches and reach a consensus on the best way to document the activities in the methyl parathion program.

Summary of Fiscal Year 1997 Evaluations

In FY 1997, ATSDR (1) developed an agencywide inventory of evaluation activities, (2) conducted an assessment of its site-specific evaluation needs, and (3) developed a guidance document for State cooperative agreement evaluation activities. These activities and information enabled ATSDR to assess why certain public health actions or decisions may or may not be working effectively. These activities also allowed for the identification of program strengths and weaknesses and suggestions to improve the efficiency and effectiveness of public health actions.

A number of States collaborated with ATSDR to develop a guideline for assessing the impacts and outcomes of site-specific health education activities. The documents described in the following paragraphs are the results of this collaborative effort. These tools will allow cooperative agreement participants to assess the overall improvement in public health as a result of site-specific health education activities and demonstrate the cooperative agreement program's effectiveness.

Site-Specific Outcome Measurement Guidelines for ATSDR Cooperative Agreement States (6853.1) provides guidelines for assessing the impacts and outcomes of site-specific health education activities. Evaluation of State Capacity Building--Program Announcement #607, published in the Federal Register, was also developed collaboratively by ATSDR staff members and State partners to assist in the overall evaluation of the effectiveness of the capacity building effort occurring in participating States. The evaluation also includes an assessment of ATSDR activities by State cooperative agreement personnel.

Next, ATSDR developed the document, Did We Make a Difference? ATSDR's Framework for Community-Focused Evaluation (6853) to provide guidance and information on how to conduct evaluations of ATSDR's site-specific programs and activities.

Evaluations in Progress

ATSDR has a number of ongoing evaluation activities. In addition to conducting traditional evaluation studies, ATSDR's commitment to viewing evaluation as an integral part of program planning is exemplified in the following activity now in progress.

Hazardous Substances Emergency Events Surveillance (6854). This evaluation of ATSDR's Hazardous Substances Emergency Events Surveillance (HSEES) program is based on the ability to measure the sensitivity and reliability of the system. Sensitivity is measured by the ability of the system to receive notification of events involving hazardous substances meeting the HSEES case definition. Reliability is measured by the ability of the system to receive uniformly recorded data from the 13 State health department partners.

To address the sensitivity of HSEES, ATSDR provides to each participating State health department a list of events involving hazardous substance releases that occurred in their State during the previous quarter and that were reported to one of two national data bases--the Department of Transportation's Hazardous Materials Information System or the EPA's Emergency Response Notification System. This activity provides the State partners with a mechanism to detect whether there were (1) incidents reported to one of the national data bases that were not reported to HSEES, (2) incidents reported to HSEES but not reported to one of the national data bases, or (3) incidents reported to both HSEES and a national data base.

To address the reliability of HSEES, ATSDR mails to participating State health departments each quarter a report describing a case study involving the release or threatened release of one or more hazardous substances. State health department partners review the case study and complete a hard-copy data collection form, which is returned to ATSDR. Then ATSDR compares the responses on these forms to each other and to ATSDR's preferred responses.