Performance Improvement 1998. Centers for Disease Control and Prevention


MISSION: To promote health and quality of life by preventing and controlling disease, injury, and disability.

Evaluation Program

The Centers for Disease Control and Prevention (CDC) conducts evaluation studies designed to provide information essential for CDC's 11 strategies to achieve its mission:

  • Conduct public health research, including epidemiology, laboratory, behavior, and social sciences.
  • Develop and implement ongoing evaluation of scientific research.
  • Assure that scientific information is communicated effectively to the general public.
  • Develop and implement public health information systems for monitoring and promoting the health of the Nation.
  • Deploy multidisciplinary teams to detect and investigate health threats.
  • Develop and implement a system for establishing CDC scientific and programmatic priorities.
  • Routinely evaluate the effectiveness and cost of CDC programs.
  • Demonstrate the value of investment in health prevention.
  • Collaborate with diverse partners.
  • Design, implement, and evaluate prevention programs based on community needs.
  • Prepare the public health leaders (present and future) through training in management and public health science.

CDC places high priority on evaluations to answer policy, program, and strategic planning questions related to the goals and objectives of Healthy People 2000. Performance improvement studies, such as those focusing upon the development of indicators consistent with the Government Performance and Results Act (GPRA), are of particular interest and import to the Agency. With the support of 1-percent evaluation funds, the GPRA planning process began at CDC in FY 1995 and is continuing. CDC's commitment to performance measurement is exemplified by this year's requirement that each project proposal be linked to one of CDC's strategic goals. These goals were identified through the GPRA strategic planning process.

On an annual basis, the Director of CDC provides guidance to the various Center, Institute, and Office Directors on 1-percent evaluation activities. This memorandum generally includes information about (1) types of studies to be carried out with 1-percent evaluation funds, (2) the format for submitting proposals, and (3) a timeline for selection. Each proposal undergoes multiple levels of review. Initial review is conducted by staff within the Office of Program Planning and Evaluation. Subsequent reviews are completed by the Office of the Assistant Secretary for Planning and Evaluation. Study authors are provided with comments, questions, and recommendations made by reviewers. In addition to providing their responses, authors are given the opportunity to revise their proposals at this time.

A panel of CDC evaluators, scientists, and program managers is convened to review and rank proposals. Review criteria include (1) relevance to prevention effectiveness, (2) relative importance of the public health problem being addressed, (3) probability that the proposed project will accomplish its objectives, and (4) extent to which other CDC programs will benefit from this project. Results from the review panel are presented to the Director of CDC for final funding decisions.

Finally, staff within the Office of Program Planning and Evaluation work closely with program staff to ensure development of a clear statement of work for selected projects. Prior to initiation of procurements, a final ad hoc review of the project's statement of work is completed.

Summary of Fiscal Year 1997 Evaluations

A total of 22 evaluation projects were completed in FY 1997. These studies are of four types: program evaluations, data policy and surveillance studies, development of performance measures, and assessments of specialized aspects of proposed or existing programs.

Program Evaluations

Six program evaluations were completed in FY 1997. These included an evaluation of the programmatic impact of CDC's 1-Percent Evaluation Program (6581 and 6581.1), an evaluation of suicide interventions in three Native American communities (5504), an evaluation design for the Business Responds to AIDS (BRTA) Program (6277), and an evaluation of the National Laboratory Training Network (NLTN) (5507). These studies represented three types of evaluations: informational, process, and outcome. For example, one of the studies looked at the design of BRTA, which was initially undertaken to help businesses create and implement comprehensive HIV/AIDS workplace programs, but also included questions that could help track trends in worksite policies, screening programs, and health promotion activities. In addition, one evaluability assessment that focused on CDC's Traumatic Brain Injury/Spinal Cord Injury program (6338) was completed.

Data Policy/Surveillance Studies

Four data policy/surveillance studies were completed in FY 1997. Two of the studies addressed reporting issues. For example, one study was conducted to ascertain whether the codes for International Classification of Disease (ICD)-10 should be modified to allow for greater specificity for morbidity use (5503). A second study in this reporting category evaluated data collected on birth records completed by multiracial and Hispanic women to see how they interpret the race question on the birth certificate (5918). Finally, two other studies addressed the area of surveillance, looking at the effectiveness of CDC surveillance for drug-resistant Streptococcus pneumoniae and the feasibility of developing a public health surveillance plan to assess current community public health. (6580, 6580.1)

Development of Performance Measures

In keeping with the congressional intent of the Government Performance and Results Act (GPRA) of 1993, one study was conducted in FY 1997 to support development of strategic planning at CDC (5928). This report offers a record of CDC's experience in its early implementation of GPRA by documenting the process, summarizing key findings, and making recommendations for strategies and approaches for the ongoing implementation of GPRA at CDC.

Assessments of Specialized Aspects of Proposed or Existing Programs

Eleven studies relating to discrete aspects of current or proposed programs were completed in FY 1997. Two of these projects were assessments of the Sexually Transmitted Disease (STD) prevention programs supported by the CDC (6579, 5348). One project was a comparative case study of local-level syphilis prevention efforts, while the other assessed the feasibility of parallel STD surveillance systems currently operated by CDC. Additionally, two studies were conducted to assess the effects of HIV prevention activities: (1) through HIV Community Planning to learn how community planning is confirming, enhancing, and changing HIV prevention programs (5927); and (2) through the Prevention Marketing Initiative, which is aimed to influence behaviors that contribute to the sexual transmission of HIV among young people below 25 years of age (6337). Another study looked at CDC's efforts to link health promotion and disease prevention research to public health practice (6535).

Three studies focused on issues related to women. The first of these studies identified and investigated factors that place women at risk for an abusive relationship and, in turn, how women can protect themselves from further violence during and after the relationship (6515). The second study focused on presenting successful strategies used by grantees of the National Breast and Cervical Cancer Early Detection Program (NBCCEDP) to reach women in need of mammography screening, especially low-income, underserved, and uninsured women (6273, 6273.1). The information will be shared with grantees in order to describe how various programs and interventions have reached marginalized populations. This information will also be useful for private sector and other organizations that are working to provide prevention services to hard-to-reach groups.

Two additional studies measuring the effectiveness of particular programs and activities were completed during FY 1997. These studies involved an evaluation of the National Immunization Survey (6477) and the Fatality Assessment and Control Evaluation (FACE) Program (5506.1). Finally, another activity focused on the first phase of a national study designed to profile the relationship between youths' access to tobacco and the contextual variables that define communities' policies, attitudes, and activities regarding youth and tobacco (6474).

Evaluations in Progress

CDC's evaluation studies in progress during FY 1997 consisted of four types: program evaluations, data policy and surveillance studies, development of performance measures, and assessments of specialized aspects of proposed or existing programs.

Program Evaluations

Nine program evaluations are currently in progress. Included are evaluations of the C. Everett Koop Community Health Center (CHIC) (6709), diabetes control programs (6584), the Field Epidemiology Training Program (FETP) (5923, 5923.1), evaluation of teen pregnancy interventions (6274), lead poisoning prevention training (6702), sexually transmitted disease (STD) partner notification (6711), impact of the Clinical Laboratory Improvement Amendments (CLIA) (6335), and CDC's Prevention Centers Program (5919). The study of the C. Everett Koop Community Health Center is a process evaluation that will look at the effectiveness of the center to serve as a national physician-based model. This will be accomplished by assessing (1) the effectiveness of the CHIC products and services from the users' perspective; (2) the use of the products and services, as well as barriers to their use; (3) the effectiveness of collaborative arrangements with nonprofit organizations, corporations, libraries, government agencies, and voluntary health agencies; and (4) the effectiveness of the products and services, such as quality of resources.

By contrast, the remaining studies in this category are outcome evaluations. The evaluation of CDC-supported, State-based diabetes control programs seeks to develop and implement an evaluation strategy to improve capacity for uniform assessment of States' progress in reducing the burden of diabetes on an evolving health system.

Similarly, the evaluation of CDC's Field Epidemiology Training Program (FETP) will ascertain whether the program has achieved its objectives, which are to train public health professionals in applied epidemiological skills, to promote the sustainability of autonomous FETPs, and to develop a global network of national programs.

The evaluation of teen pregnancy interventions will assess and test the adolescent pregnancy prevention initiatives in 13 cities within the United States. This project will involve the development and testing of efficient, systematic, and durable approaches to translating evaluation findings and program experiences of the 13 communities into changes in program interventions in demonstration and replication communities.

The last of the outcome evaluations involves the development of a systematic evaluation of notification to sexual partners of individuals who are diagnosed with sexually transmitted diseases. This project seeks to ascertain whether the substantial investment of State, Federal, and local health care dollars is decreasing disease prevalence or incidence when partners are notified.

Evaluation of Data Policy and Surveillance Systems

Ten studies are under way that relate to data management or reporting systems. Seven studies include evaluations of the U.S. Standard Certificates (6699), medical certification process for death certificates (4230), pregnancy-related violence (6712), Assisted Reproductive Technology (ART) embryo lab procedures (6716), use of data from Immunization Information Systems (IIS) (6713), evaluation of data uses of the National Electronic Telecommunications System for Surveillance (NETSS) and the Public Health Laboratory Information System (PHLIS) (6332), and the Urban Health Systems Sentinel Network (6276). In addition, three surveys are being conducted to evaluate guidelines for preventing perinatal HIV infection (6279), assess immunization knowledge and practice of primary care providers in the U.S. (5930.1), and inform the public about skin cancer (6710). The Skin Cancer National Survey examines the knowledge, attitudes, and behaviors of parents regarding skin cancer and the protection of their children from the sun. Results from this survey will enable CDC to design and distribute pertinent health messages about sun exposure.

The overall objective of the U.S. Standard Certificates evaluation is to examine the birth, death, and fetal-death information currently obtained from State vital registration programs to determine if relevant and high-quality data are being collected for State and national decisionmaking. The assessment will also consider what changes should be made to the standard certificates to improve their utility.

The other two evaluations are examining the use of survey instruments in data collection. In the area of pregnancy-related violence, a national population-based survey will be conducted of administrators and clinical providers in family planning programs. Information gained in the survey will be used to develop recommendations for the development of clinical guidelines. Similarly, since CDC is responsible for developing a model certification program for ART embryo laboratories in the United States, a survey will be conducted on the laboratories currently using ART to provide an enumeration of those procedures and practices.

Finally, at this point in the development cycle of the Immunization Information Systems (IIS), an assessment is being conducted on how the IIS data are used. The objectives of this project are to (1) assess how IIS data are used to assess public clinic and private provider performance and to identify structural and functional barriers to using the data; (2) assess how IIS data are used to assess community-based immunization coverage; (3) assess how IIS data are used to monitor the impact of changing recommendations in immunization schedule and the health care delivery system; and (4) promote the development of immunization registries.

Development of Performance Measures

Four ongoing projects involve the development of performance measurement systems. One of these projects, Development and Implementation of Strategic Planning at CDC, seeks to gather and analyze information that supports a comprehensive, unified, and data-driven strategic and performance planning framework for the CDC (6275). Three other projects focus on the development of indicators that will be used to assess HIV prevention efforts in the community, management and operational aspects of HIV community planning, and the coordination of multiple HIV prevention programs (5518, 6278, 6707).

Assessment of Specialized Aspects of Proposed or Existing Programs

Fifteen specialized studies relating to discrete aspects of current or proposed programs are in progress. These wide-ranging studies include evaluations of the development of a public health surveillance plan (5925), a feasibility study of Pap screening in STD clinics (6705), dental treatment and health status differences attributable to water fluoridation (6334), and the effectiveness of Group B streptococcal disease prevention guidelines (6333), still other studies assess the public health impact of the National Institute for Occupational Safety and Health (NIOSH) health hazard evaluation program (5922), tuberculosis outreach worker activities (5502), elder care groups (6708), small business interventions (6336), the Morbidity and Mortality Weekly Report (MMWR) recommendations for the use of bike helmets (6700), coordinated tobacco control efforts (6714), CDC's technical assistance to local health departments on violence prevention (5929), the efficacy of different types of vancomycin-resistant enterococci (6703), and HIV-related projects (6701, 6704, 6715, and 6706).

With the goal of complete eradication of tuberculosis by the year 2000, one of the studies focuses on an evaluation of tuberculosis outreach worker activities. Specifically, the project will (1) identify how these workers accomplish their tasks, (2) assess the quality of outreach workers' interactions with clients and supervisors, and (3) evaluate the amount of time spent on various research activities.

Many of the studies being conducted hope to provide the public with better information for all age groups. For instance, the elder care focus group study was undertaken in order to obtain data that will be used to produce an elder care health and safety handbook in the United States. One study focuses on the safety of our youth, as well as adults, with a desire to develop updated recommendations and guidelines for the use of bicycle helmets. In addition, an evaluation of tobacco control efforts will continue by looking at the impact and the relative cost effectiveness of tobacco control programs.

Several HIV prevention-related projects are in progress. Projects are being conducted to develop community indicators for HIV, to administer PCP prophylaxis for women and persons in underserved communities, to evaluate the current HIV technical assistance (TA) network, and to determine how the next iteration of TA could be improved. Finally, a study to determine the feasibility of developing a demonstration project to assess identification and screening of women at high risk of HIV infection is in progress. The community indicators study will identify (1) cultural and social-structural characteristics of communities that are associated with HIV/AIDS risk behavior and prevention efforts; (2) theories, assumptions, or hypotheses that explain the causal role of community characteristics; and (3) measures or indicators that could be used to assess changes in those characteristics that support sustained individual behavior change and community prevention efforts. This information will be used to support community-level research on HIV/AIDS prevention.

The goal of evaluating the CDC HIV Technical Assistance Network is to provide CDC staff who are responsible for planning the next iteration of a TA network and stakeholders of the current CDC-supported TA network with valuable information that can be used to develop and strengthen the existing TA model.