Performance Improvement 1995: Evaluation Activities of the Public Health Service is the first annual report on evaluation activities of the U.S. Public Health Service (PHS). As a report to Congress, it summarizes the findings of PHS evaluations completed in fiscal year (FY) 1994. In general, the report is intended for three audiences: decisionmakers who need information on program results across the broad spectrum of public health activities; program managers who need information on how they can make improvements in program operations and outcomes; and the public health community, which can benefit by applying the knowledge and lessons learned from PHS evaluations.
Evaluation plays an important role in the execution of the Public Health Service mission. Every year PHS programs are engaged in efforts to determine whether programs and services reach intended populations or communities
and achieve desired results. The information obtained is applied to how those programs and services can be improved. The theme of this report is performance improvement, which is the focus of Secretary Donna Shalala's Continuous Improvement Program (CIP), involving numerous changes and initiatives throughout the Department of Health and Human Services to increase the effectiveness and efficiency of public health programs. Of the approximately $14 billion in the FY 1994 budget for program activities, PHS Agencies used almost $27 million to conduct evaluations useful for understanding the outcomes and improving the performance of PHS programs. In FY 1994, PHS Agencies produced 71 evaluation reports and supported more than 180 evaluation projects in progress.
Chapter I provides a description of the PHS Evaluation Program. The range of evaluation activities supported by PHS Agencies includes outcome, impact, or process evaluations; policy assessments; cost-benefit or cost-effectiveness analyses; survey data analyses; management studies; and evaluation syntheses. Evaluation methodology projects, such as evaluation feasibility and instrument development studies, are also performed. Other project support activities include evaluation technical assistance, report dissemination, training, and conferences. In this overview, the policies and management of the PHS Evaluation Program are described, including annual project planning procedures, funding levels from FY 1990 to FY 1994, quality assurance systems, dissemination mechanisms, procedures for application of evaluation results, and future directions for PHS evaluation activities.
Chapter II highlights 13 FY 1994 PHS evaluations selected from recently completed studies covering six broad topic areas.
Access to Health Care for Special Populations
Making a Difference: Interim Status Report of the McKinney Research Demonstration Program for Homeless Mentally Ill Adults. The findings of this demonstration project indicate that provision of coordinated multidisciplinary services can significantly reduce homelessness and improve the mental health status of participants. Data were collected from participants along five dimensions (psychiatric symptomatology, substance abuse, quality of life, physical health, and residential stability) before and after the various interventions were offered at the six study sites. The data indicate that homeless people who are mentally ill will use services if they are accessible and targeted to their needs, that these services will enable people to find permanent housing, and that formerly homeless people with mental illness can be valuable staff resources.
Report to Congress on the Indian Health Service With Regard to Health Status and Health Care Needs of American Indians in California in Response to Public Law 100-713 analyzes data from vital statistics, databases, and patient records from tribal clinics to assess the health status and access to health care services of American Indians from federally and non--federally recognized tribes in California. Indians from non-federally recognized tribes were found to have a poorer health status than Indians from federally recognized tribes, and the health status of both groups was substantially lower than that of other populations in the State.
Adolescent Health Problems
Developing Effective Health Communication Strategies for High-Risk Youth Outside of School reports on a focus group study that questioned 160 high-risk youths aged 10 to 18 years about their health practices (tobacco use, substance use, unprotected sex, and violence). Although the youths were fairly knowledgeable about health practices, they did not incorporate this awareness into their behavior. They were willing, however, to listen and follow the advice of trusted, credible adults in alternative settings.
Evaluating Educational Outcomes of School Health Programs is a followup study to a large-scale, randomized, controlled field trial that assessed the effectiveness of a Life Skills Training course. Fifteen class periods were devoted to teaching social resistance and other personal and coping skills to seventh-grade students in New York, with booster classes offered in the eighth and ninth grades. Findings indicated that the probability of students' smoking, drinking immoderately, or using marijuana was as much as 40 percent lower for students who received the intervention and the probability of using multiple drugs was as much as 60 percent lower.
Evaluating the Effects of School-Based Intervention Programs To Prevent Teenager Drug Use and Abuse offers a blueprint for evaluations of health programs for youths and for assessing the effect these programs have on academic performance. Most studies reviewed concentrated on health outcomes. A few studies, however, examined educational outcomes, demonstrating improvements in basic academic skills, reductions in tardiness and absenteeism, and reductions in drug use. Two designs for evaluating school health interventions are proposed: analysis of existing data collected from previous evaluations, and large-scale multischool demonstrations using experimental and control group designs and random assignment.
Indoor Allergens: Assessing and Controlling Adverse Health Effects, performed by the National Academy of Sciences, is an extensive review of the literature assessing the relationship between indoor allergens and allergic diseases. Findings indicate that African Americans and children are at greatest risk for developing complications associated with indoor allergens; that children under 18 years old account for nearly half of all emergency room visits for asthma; and that asthma mortality rates for African Americans are two to three times greater than the rates for whites. Simple steps such as regular washing of bedding, keeping carpeting to a minimum, and use of special filters can significantly reduce the presence of indoor allergens.
HIV Service Networks in Four Rural Areas examines how HIV services are organized and delivered in rural areas with low and high AIDS prevalence and provides a portrait of HIV/AIDS epidemiology, HIV-related service networks, barriers and gaps in service, and funding mechanisms. A typology classifies areas based on degree of rurality, AIDS prevalence, and the epidemiological and demographic characteristics of the infected populations. Diagnosis and treatment of HIV infection are found to be significantly hampered by stigma and lack of physician knowledge.
Outreach/Risk Reduction Strategies for Changing HIV-Related Risk Behaviors Among Injection Drug Users looks at efforts to reduce risky behaviors of injection drug users (IDUs). Interventions at 28 sites were studied. Outcome measures looked at needle risk behavior, sexual risk behaviors, and total frequency of drug injection. Data were collected on participant demographics, and on behaviors and knowledge before and after the interventions. Following interventions, IDUs reported significant increases in use of new needles, bleach for cleaning injection equipment, and condoms. Changes in needle-related behaviors were more dramatic than changes in sexual behaviors.
Adverse Events Associated With Childhood Vaccines: Evidence Bearing on Causality reports the findings of an Institute of Medicine expert panel that reviewed published books, articles, and abstracts and listened to public testimony to determine whether a causal relationship exists between specific vaccines and adverse effects in children. Of the 49 conditions in which adverse effects from childhood vaccines were suspected, evidence in 12 cases substantiated a causal relationship; evidence in 4 cases rejected a causal relationship; and evidence in the remaining 33 cases was equivocal. The committee concluded that the available data often were insufficient to establish or reject causality, and the risk of dying or developing serious neurologic or immunologic complications appeared
to be very low.
A Study of the Economic Underpinning of Vaccine Supply examined issues related to the economics of production and distribution of vaccines. The study investigated (1) alternative models for purchase and distribution; (2) the effect of scientific advances on research, development, and purchase of vaccines; (3) the application of economic theory to the vaccine market; (4) comparisons of distribution systems managed by manufacturers and State agencies; and (5) the implications of purchasing vaccines from foreign firms. Findings indicate that supplying vaccines at low prices to physicians, encouraging parents to have their children vaccinated, and having effective and convenient distribution networks help to increase vaccination rates.
Public Health Infrastructure
Advanced Practice Nursing Education: Strategies for the Allocation of the Proposed Graduate Nursing Education Account. The rapidly changing health care delivery system is expected to cause a shortage of advanced practice nurses (APNs) in the near future. The study analyzed policy options on APN student characteristics, supply, overall education costs, and financing. The advantages, disadvantages, and costs of each option for ensuring a stable source of funds to support APN education were discussed.
Evaluation of Centers for Disease Control and Prevention (CDC) and Agency for Toxic Substances and Disease Registry (ATSDR) Training Activities. The study, using focus groups and site visits, examined the training needs of local health departments. It was found that local health departments were deficient in identifying their training needs and needed training in advocacy, evaluation, and public information. Anticipating increasingly limited resources at CDC and local agencies, recommendations emphasized the use of new training technology such as interactive software and satellite communications.
Evaluation of the Morbidity and Mortality Weekly Report (MMWR) Series. A mail/telephone survey of MMWR readers revealed that the CDC publication is read with great regularity, that each issue is passed along to several readers, and that most people have been reading the report for more than 5 years. Primary care physicians use the information to update their knowledge about diagnosis, treatment, and disease outbreaks. The MMWR was credited for its accuracy, relevance, and concise reporting. Suggestions for change include improved electronic access and larger page size.
Chapter III provides an overview of the evaluation activities of the eight PHS Agencies--presenting information on the evaluation program, a summary of evaluations completed in FY 1994 and evaluations in progress, and a discussion of future directions for Agency evaluations.
Agency for Health Care Policy and Research
The goals of the Agency for Health Care Policy and Research (AHCPR) evaluation program are to assess the Agency's effectiveness in meeting major and long-term priorities and goals; obtain information quickly to respond to critical Agency, PHS, and departmental concerns; and conduct internal evaluations to improve the efficiency of key program areas. Several evaluation mechanisms are used to achieve these goals: special evaluation studies; peer review of grant applications; the User Liaison Program, which provides information on the value of AHCPR-supported research to State policymakers; and other efforts such as focus groups to provide feedback on AHCPR products. During FY 1994, six evaluations were completed by AHCPR on two topics: the process of developing clinical practice guidelines and the evaluation of health care delivery. AHCPR currently supports 11 projects to facilitate improvements in health care quality and continues its commitment to improving the quality of AHCPR-supported clinical practice guidelines. For the future, AHCPR's programs must generate the information and tools needed to improve health care delivery and its outcomes and translate research findings into forms of information that actively assist consumers, practitioners, payers, and others in making effective health care decisions.
Centers for Disease Control and Prevention and Agency for Toxic Substances and Disease Registry
The Centers for Disease Control (CDC) places a high priority on evaluations seeking to answer policy, program, and strategic planning questions related to the mission of the Agency. Evaluations typically focus on programs to monitor the health of populations and communities, investigate health or disease problems, develop public health policies, implement prevention programs, promote healthy behaviors, foster safe environments, and provide public health leadership and training. The Agency for Toxic Substances and Disease Registry (ATSDR) evaluation program focuses largely on internal studies of its mission of environmental protection established by the 1986 Comprehensive Environmental Response, Compensation, and Liability Act (the Superfund). CDC completed 12 evaluations in FY 1994 in the areas of training and information dissemination, surveillance, program effectiveness, prevention, and costs of disease. Two evaluations are highlighted in chapter II: an evaluation of CDC and ATSDR training activities and a survey of readers of the MMWR. CDC has a total of 32 evaluations in progress in four areas: surveillance and data collection studies; program evaluations; community/intervention effectiveness studies; and evaluation methodology studies. In the future, CDC will be initiating projects designed to provide data for performance indicators and to assess the effectiveness and efficacy of such indicators.
Food and Drug Administration
Evaluation at the Food and Drug Administration (FDA) is largely a line management responsibility rather than one for specialized evaluation staffs. Projects are focused in the areas of performance management, customer participation, and more rigorous rulemaking. In FY 1994, the FDA conducted evaluations in three areas: implementation of the Prescription Drug User Fee Act of 1992 (PDUFA); the negotiation process toward medical device user fees; and the harmonization of international regulatory requirements--
"regulatory benchmarking"--in which specific components of the regulatory process in different countries are compared and assessed by program managers. Specific evaluations are currently in progress at the FDA on PDUFA management and performance of the Mammography Quality Standards Act of 1992. Also FDA has ongoing evaluation efforts to develop performance measures for additional user fee programs and to develop a customer-sensitive dimension to the commencement of negotiated or consensual rulemaking. Management changes in government are creating systemic changes in the FDA's evaluation function: FDA evaluation efforts are now largely driven by the mandates of the Government Performance and Results Act (GPRA) of 1993 and its corollaries.
Health Resources and Services Administration
The objectives of the Health Resources and Services Administration (HRSA) evaluation program are to improve program management and policy development and to provide information that will enhance strategic planning, budget decisions, and legislative planning. High priority is given to studies that assess program effectiveness or outcomes, or improve capacity for measuring performance. In FY 1994, HRSA completed evaluations in the following program areas: effectiveness of community health centers; linkage of mental health and primary care services; HIV service networks in rural areas; implementation of the Ryan White Comprehensive AIDS Resources Emergency (CARE) Act of 1990; rural health and telecommunications; populations served under the Hill-Burton Uncompensated Services Program; beneficiaries of nursing education projects; training for preventive medicine specialists; and strategic planning for the Bureau of Health Professions. HRSA has more than 40 projects under way. Subjects of the major program evaluations include the Healthy Start Initiative (intended to reduce infant mortality), the community and migrant health centers, the National Practitioner Data Bank, the Healthy Schools/Healthy Communities Program, the Injury Prevention Implementation Incentive Grants Program, and various health services funded under the Ryan White CARE Act. The top priority in the future for HRSA evaluations will be development of performance measures and data systems in conjunction with implementation of the GPRA. Future HRSA evaluations will focus on the Agency's mission of expanding access to care for the underserved, and such crosscutting priority areas as academic/community partnerships in health professions education, community infrastructure building, managed care, expanded roles for States, information technology, and HIV/AIDS.
Indian Health Service
The Indian Health Service (IHS) evaluation program serves the Agency's program and policy objectives, developed in consultation with tribal communities. It provides valid and reliable information to help the Director promote the IHS vision and guide its implementation. In FY 1994, the IHS completed nine evaluations on topics of service delivery, health status, and management. Three evaluations are having a major impact on program planning: an assessment of maternal and child health data in the Navajo area; an assessment of health status and access to care for Native Americans in California; and long-term health care projections for alcohol-related hospitalizations. The IHS has 21 evaluations in progress. Two of these underscore the IHS commitment to the prevention of child abuse and family violence. One report examines the extent of child abuse and neglect among American Indian tribes and the ways in which the IHS responds to these problems. The other is a case study of family violence in four American Indian communities. More than 100 unstructured interviews were conducted at the four sites to probe the nature and extent of family violence--spousal abuse, child abuse and neglect, child sexual abuse, and elder abuse. In the future, the IHS will initiate evaluations in three program areas: mental health services for urban Indians, regional treatment centers for substance abuse disorders, and health services for elderly American Indians.
National Institutes of Health
Evaluation is an integral part of the role of the National Institutes of Health (NIH) in supporting biomedical research, training, and public education. The NIH peer review system is a major part of its overall evaluation strategy: research proposals from scientists around the nation are subjected to a rigorous assessment
by fellow scientists before they are funded. National advisory councils, boards of scientific counselors, and consensus development conferences perform regular research evaluation functions. Four of the eight major evaluations completed by NIH in FY 1994 are highlighted in chapter II. The topics were adverse reactions to vaccines, indoor allergens, drug abuse prevention strategies, and interventions for high-risk behavior and HIV infection. Currently, NIH has 24 evaluations in progress, ranging from small- to large-scale assessments, from evaluation feasibility studies to full-blown evaluations. Future plans for NIH evaluations include an examination of new ways to carry out more efficient peer review of grant applications; a survey of public understanding about biomedical science and how Americans get their information about health; and development of new measures of NIH's internal performance, required by the GPRA.
Substance Abuse and Mental Health Services Administration
The Substance Abuse and Mental Health Services Administration (SAMHSA) is committed to evaluating its programs to assess the effectiveness of treatment and prevention approaches and systems of care; accountability for Federal funds; and the achievement of SAMHSA's program and policy objectives. SAMHSA's evaluations of demonstration programs generate new knowledge to lead the field in developing policies that improve services. During FY 1994, SAMHSA completed six evaluations of programs targeted to four population groups identified as being in greatest need of substance abuse and mental health services: pregnant and postpartum women and their infants, children with serious emotional disturbance, high-risk youth, and the homeless mentally ill. Currently, SAMHSA has 10 major evaluations under way in the following areas: program accountability, evaluation of demonstrations, reinforcing behavioral health, and commitment to customer service. SAMHSA's future evaluation activities will respond to emerging trends such as managed care and health care reform. Evaluation activities will reinforce the critical role of behavioral health in general health care.
Office of the Assistant Secretary for Health
The primary role of the Office of the Assistant Secretary for Health (OASH) is coordination and development of evaluations across the entire Public Health Service, often identifying potential program areas that could benefit from a collaborative evaluation. In FY 1994, OASH completed 17 evaluations in the areas of health care reform, immunizations, adolescent and school health, nutrition, primary care, emergency preparedness, international health, and health services delivery. Currently, OASH has 14 program and policy evaluations under way in two areas: health care reform and population-based public health services. Two issues of health care reform are being examined: (1) the role of Federal, State, and local health information systems in achieving public health objectives, and (2) the impact of health care reform on the health care workforce and the role of academic health centers. Other major evaluations include an examination of the clinical preventive services program, Put Prevention into Practice, and an assessment of the cost-effectiveness of clinical preventive services. In FY 1995, OASH will support evaluation activities in the PHS Agencies to examine the impact of managed care and health information systems on public health objectives.
A complete inventory of the 71 PHS evaluation projects completed in 1994 is provided in appendix A, and PHS Agency evaluation projects in progress are listed in appendix B.
Review criteria used by the special panel of senior editorial advisors for assessing evaluations can be found in appendix C.