Performance Improvement 1995

Foreword

Continuous improvement in government is a central theme of the Clinton Administration. As we move toward a balanced Federal budget, funds available for all Departments will be increasingly constrained. All programs must become more efficient and must demonstrate their effectiveness in improving the lives of all Americans.

The Department of Health and Human Services (DHHS) is committed to examining and redirecting its programs to ensure that its customers--the American people--are being well served. In the spring of 1993, shortly after Vice President Gore initiated the National Performance Review, the Continuous Improvement Program (CIP) was established in DHHS. The CIP calls on all DHHS employees to work together as agents of change to create a more customer-oriented, cost-efficient, and effective government. It is important to ensure that decisions for change are based on sound information about programs and policies. A solid evaluation program can provide the information needed to guide constructive change, including programmatic, budgetary, and policy decisions.

The U.S. Public Health Service (PHS) recognizes the importance of evaluation in improving the health and quality of life for all Americans. PHS supports and encourages evaluation efforts that--

• determine whether programs and services achieve their desired effect and how to improve them;
• measure the success of programs in reaching and helping target populations or communities;
• communicate knowledge about program successes and other lessons learned to the public health community; and
• develop or refine effective evaluation tools.

PHS Evaluation Program studies have had a major impact on planning, budgeting, and legislative development. For example, a 1989 National Academy of Sciences evaluation of the training and supply of biomedical and behavioral scientists helped the National Institutes of Health (NIH) determine the number of positions to be supported by National Research Service Awards (NRSAs) and justify NIH training budget requests to meet future needs for biomedical and behavioral scientists. In 1993, the Centers for Disease Control and Prevention (CDC) evaluated the impact of an influenza vaccination program for the elderly, documenting the cost-effectiveness of a vaccination promotion campaign for reducing incidence of secondary acute respiratory bacterial and viral infections. The results influenced the Department's decision to make influenza vaccine reimbursable by Medicare as a cost-effective clinical preventive service.

In addition, the PHS Evaluation Program has played a role in supporting development of evaluation tools for use by the larger public health community. In 1991, the National Research Council's Committee on AIDS Research and the Behavioral, Social, and Statistical Sciences produced a series of reports on strategies for evaluating CDC's AIDS prevention programs. These evaluation methodology reports first helped provide the CDC with a blueprint for designing more effective health communication strategies for AIDS prevention. The reports also made these evaluation tools available to public health practitioners, enabling them to assess the effectiveness of AIDS prevention interventions.

In keeping with the PHS commitment to produce high-quality, useful evaluations, we are pleased to present Performance Improvement 1995: Evaluation Activities of the Public Health Service. This report is the first in a series of annual reports documenting evaluation efforts across all PHS Agencies and Offices. The report has three audiences: decisionmakers who need information on program results across the broad spectrum of public health activities; program managers who want to know how other public health programs are operating and how performance can be improved; and the community of researchers, advocates, and practitioners who will use the program information and the evaluation tools.

The report contains information on projects completed in fiscal year (FY) 1994 and their applications. For example,

• Research on adverse events associated with childhood vaccines is being used to educate parents about vaccine-associated risks and to guide policies related to childhood immunization and victim compensation.
• An Institute of Medicine report on the impact of nonindustrial indoor allergens spurred the creation of a new research program and numerous recommendations for patient education, treatment, and methods to improve indoor environments.
• A school health program evaluation produced designs that school systems can use to evaluate school health interventions and their impact on student academic performance and health.

This report samples the diverse array of evaluation studies conducted by PHS in 1994 and provides information on evaluations in progress. The report presents a comprehensive picture of the PHS Evaluation Program--its past, present, and expected future activities.

Future studies will place even greater emphasis on outcome or impact evaluations, increased methodological and scientific rigor, and projects that are crosscutting and provide widely applicable results. Projects also must meet information needs outside the PHS. They must contribute to our national understanding of how public health and preventive approaches improve the health of all Americans, reduce the risks to health, and enhance the accessibility and availability of care.

Program Performance 1995 is organized into three chapters. Chapter I describes the PHS Evaluation Program--its activities, funding, planning and management, and future directions. Chapter II highlights the results from 13 FY 1994 evaluations selected for their potential application by the public health community. Chapter III presents the evaluation activities of the seven PHS Agencies and the Office of the Assistant Secretary for Health (OASH), including information on their evaluation programs, evaluations completed in FY 1994, evaluations in progress, and future directions. A complete inventory of the 71 PHS evaluation projects completed in FY 1994 is provided in appendix A, and PHS Agency projects in progress are listed in appendix B. Review criteria used by the special panel of senior editorial advisors for assessing program evaluations can be found in appendix C.

We hope that you will find this report useful and informative.

Donna E. Shalala
Secretary
U.S. Department of Health and Human Services

Philip R. Lee, M.D.
Assistant Secretary for Health
U.S. Public Health Service

Senior Editorial Advisors

The following individuals provided editorial guidance in the development of Performance Improvement 1995:

David Blumenthal, M.D., M.P.P.
Chief, Health Policy Research and Development Unit
Massachusetts General Hospital
Medical Practices Evaluation Center
Boston, Massachusetts

Joel Cantor, Ph.D.
Director of Evaluation
Robert Wood Johnson Foundation
Princeton, New Jersey

Eleanor Chelimsky, Ph.D.
President
American Evaluation Association
Washington, D.C.

Gordon H. DeFriese, Ph.D.
Professor of Social Medicine, Epidemiology and Health Policy and Administration
Cecil G. Sheps Center for Health Services Research
The University of North Carolina at Chapel Hill
Chapel Hill, North Carolina

Molly Engle, Ph.D.
School of Medicine
University of Alabama-Birmingham
Birmingham, Alabama

Karen Glanz, Ph.D., M.P.H.
Cancer Research Center of Hawaii
University of Hawaii
Honolulu, Hawaii

Lawrence W. Green, Dr.P.H.
Director, Institute of Health Promotion Research
Professor, Department of Health Care and Epidemiology
University of British Columbia
Vancouver, B.C., Canada

Margaret A. Hamburg, M.D.
Commissioner
New York City Health Department
New York, New York

Holly Korda, Ph.D.
Health Services Policy and Research
Cambridge, Massachusetts

John Kralewski, Ph.D.
Director, Institute for Health Services Research
School of Public Health
University of Minnesota
Minneapolis, Minnesota

Mark Krushat, Ph.D.
Office of the Inspector General
Department of Health and Human Services
Washington, D.C.

Kenneth McLeroy, Ph.D.
Health Promotion Sciences Department
University of Oklahoma
Oklahoma City, Oklahoma

Ricardo Millett, Ph.D.
Director of Evaluation
W.K. Kellogg Foundation
Battle Creek, Michigan

Joseph P. Newhouse, Ph.D.
John D. and Catherine T. MacArthur
Professor of Health Policy and Management
Director, Division of Health Policy Research and Education
Harvard University
Boston, Massachusetts

Cheryl Perry, Ph.D.
Division of Epidemiology
University of Minnesota
Minneapolis, Minnesota

Ira Raskin, Ph.D.
Health Policy Consultant
EEI
Alexandria, Virginia

Debra Rog, Ph.D.
Center for Mental Health Research
Vanderbilt University
Washington, D.C.

Mary Ann Scheirer, Ph.D.
Evaluation Consultant
Washington, D.C.

Lee Sechrest, Ph.S.
Department of Psychology
University of Arizona
Tucson, Arizona

In addition to reviewing the report, Drs. Chelimsky, Engle, Glanz, Korda, Kralewski, Krushat, McLeroy, Raskin, Rog, Scheirer, and Sechrest served on a panel charged with identifying the projects that would be highlighted in this report.

Acknowledgments

Performance Improvement 1995: Evaluation Activities of the Public Health Service describes the continuous efforts of the PHS Agencies to examine public health programs for the efficiency of their operations and their effectiveness in achieving their objectives. The planning, development, and coordination of those evaluations is largely the responsibility of the following PHS planning and evaluation offices:

Agency for Health Care Policy and Research
Office of Program Development
Phyllis Zucker, Director

Centers for Disease Control and Prevention
Office of Program Planning and Evaluation
Martha Katz, Director

Food and Drug Administration
Office of Planning and Evaluation
Paul L. Coppinger, Associate Commissioner

Health Resources and Services Administration
Office of Planning, Evaluation, and Legislation
Ronald H. Carlson, Associate Administrator

Indian Health Service
Office of Planning, Evaluation, and Legislation
Luana Reyes, Director

National Institutes of Health
Office of Science Policy and Technology Transfer
Lana R. Skirboll, Director

Office of the Assistant Secretary for Health
Office of Disease Prevention and Health
Promotion and Health Planning and Evaluation
Susanne A. Stoiber, Acting Deputy Assistant Secretary for Health

Substance Abuse and Mental Health Services
Administration
Office of Policy and Program Coordination
Frank Sullivan, Director
Office of Applied Studies
Daniel Melnick, Director

Preparation of this report and the appendixes was coordinated by staff in the Office of Disease Prevention and Health Promotion and Health Planning and Evaluation, under the supervision of J. Michael McGinnis, M.D., Assistant Surgeon General. Staff responsible include James Harrell; Paul Johnson, Ph.D.; Janice Radak; Melanie Timberlake; and Valerie Welsh. Development and production of the report was supported under contract by the services of Muriel Levin, Ed.D.; the staff of EEI; and Miriam Davis, Ph.D., special consultant.

The following persons from PHS Agencies contributed to the writing, collection of information on evaluation activities, and reviews of the report:

Agency for Health Care Policy and Research
Irma Arispe, Ph.D.

Centers for Disease Control and Prevention
Wilma Johnson
Nancy Cheal

Food and Drug Administration
Tim Hegarty
Mary Gamunev

Health Resources and Services Administration
Anabel Crane
Cassandra Lyles

Indian Health Service
Linda Arviso-Miller
Leo Nolan

National Institutes of Health
John Uzzell
David Chananie, Ph.D.
Anita Harris

Office of the Assistant Secretary for Health
Andy Lefton
Barbara Secka
Kirby Weldon

Substance Abuse and Mental Health Services
Administration
Anna Marsh, Ph.D.
Andrew Freeman

Executive Summary

Performance Improvement 1995: Evaluation Activities of the Public Health Service is the first annual report on evaluation activities of the U.S. Public Health Service (PHS). As a report to Congress, it summarizes the findings of PHS evaluations completed in fiscal year (FY) 1994. In general, the report is intended for three audiences: decisionmakers who need information on program results across the broad spectrum of public health activities; program managers who need information on how they can make improvements in program operations and outcomes; and the public health community, which can benefit by applying the knowledge and lessons learned from PHS evaluations.

Evaluation plays an important role in the execution of the Public Health Service mission. Every year PHS programs are engaged in efforts to determine whether programs and services reach intended populations or communities
and achieve desired results. The information obtained is applied to how those programs and services can be improved. The theme of this report is performance improvement, which is the focus of Secretary Donna Shalala's Continuous Improvement Program (CIP), involving numerous changes and initiatives throughout the Department of Health and Human Services to increase the effectiveness and efficiency of public health programs. Of the approximately $14 billion in the FY 1994 budget for program activities, PHS Agencies used almost $27 million to conduct evaluations useful for understanding the outcomes and improving the performance of PHS programs. In FY 1994, PHS Agencies produced 71 evaluation reports and supported more than 180 evaluation projects in progress.

Chapter I

Chapter I provides a description of the PHS Evaluation Program. The range of evaluation activities supported by PHS Agencies includes outcome, impact, or process evaluations; policy assessments; cost-benefit or cost-effectiveness analyses; survey data analyses; management studies; and evaluation syntheses. Evaluation methodology projects, such as evaluation feasibility and instrument development studies, are also performed. Other project support activities include evaluation technical assistance, report dissemination, training, and conferences. In this overview, the policies and management of the PHS Evaluation Program are described, including annual project planning procedures, funding levels from FY 1990 to FY 1994, quality assurance systems, dissemination mechanisms, procedures for application of evaluation results, and future directions for PHS evaluation activities.

Chapter II

Chapter II highlights 13 FY 1994 PHS evaluations selected from recently completed studies covering six broad topic areas.

Access to Health Care for Special Populations

Making a Difference: Interim Status Report of the McKinney Research Demonstration Program for Homeless Mentally Ill Adults. The findings of this demonstration project indicate that provision of coordinated multidisciplinary services can significantly reduce homelessness and improve the mental health status of participants. Data were collected from participants along five dimensions (psychiatric symptomatology, substance abuse, quality of life, physical health, and residential stability) before and after the various interventions were offered at the six study sites. The data indicate that homeless people who are mentally ill will use services if they are accessible and targeted to their needs, that these services will enable people to find permanent housing, and that formerly homeless people with mental illness can be valuable staff resources.

Report to Congress on the Indian Health Service With Regard to Health Status and Health Care Needs of American Indians in California in Response to Public Law 100-713 analyzes data from vital statistics, databases, and patient records from tribal clinics to assess the health status and access to health care services of American Indians from federally and non--federally recognized tribes in California. Indians from non-federally recognized tribes were found to have a poorer health status than Indians from federally recognized tribes, and the health status of both groups was substantially lower than that of other populations in the State.

Adolescent Health Problems

Developing Effective Health Communication Strategies for High-Risk Youth Outside of School reports on a focus group study that questioned 160 high-risk youths aged 10 to 18 years about their health practices (tobacco use, substance use, unprotected sex, and violence). Although the youths were fairly knowledgeable about health practices, they did not incorporate this awareness into their behavior. They were willing, however, to listen and follow the advice of trusted, credible adults in alternative settings.

Evaluating Educational Outcomes of School Health Programs is a followup study to a large-scale, randomized, controlled field trial that assessed the effectiveness of a Life Skills Training course. Fifteen class periods were devoted to teaching social resistance and other personal and coping skills to seventh-grade students in New York, with booster classes offered in the eighth and ninth grades. Findings indicated that the probability of students' smoking, drinking immoderately, or using marijuana was as much as 40 percent lower for students who received the intervention and the probability of using multiple drugs was as much as 60 percent lower.

Evaluating the Effects of School-Based Intervention Programs To Prevent Teenager Drug Use and Abuse offers a blueprint for evaluations of health programs for youths and for assessing the effect these programs have on academic performance. Most studies reviewed concentrated on health outcomes. A few studies, however, examined educational outcomes, demonstrating improvements in basic academic skills, reductions in tardiness and absenteeism, and reductions in drug use. Two designs for evaluating school health interventions are proposed: analysis of existing data collected from previous evaluations, and large-scale multischool demonstrations using experimental and control group designs and random assignment.

Environmental Health

Indoor Allergens: Assessing and Controlling Adverse Health Effects, performed by the National Academy of Sciences, is an extensive review of the literature assessing the relationship between indoor allergens and allergic diseases. Findings indicate that African Americans and children are at greatest risk for developing complications associated with indoor allergens; that children under 18 years old account for nearly half of all emergency room visits for asthma; and that asthma mortality rates for African Americans are two to three times greater than the rates for whites. Simple steps such as regular washing of bedding, keeping carpeting to a minimum, and use of special filters can significantly reduce the presence of indoor allergens.

HIV/AIDS

HIV Service Networks in Four Rural Areas examines how HIV services are organized and delivered in rural areas with low and high AIDS prevalence and provides a portrait of HIV/AIDS epidemiology, HIV-related service networks, barriers and gaps in service, and funding mechanisms. A typology classifies areas based on degree of rurality, AIDS prevalence, and the epidemiological and demographic characteristics of the infected populations. Diagnosis and treatment of HIV infection are found to be significantly hampered by stigma and lack of physician knowledge.

Outreach/Risk Reduction Strategies for Changing HIV-Related Risk Behaviors Among Injection Drug Users looks at efforts to reduce risky behaviors of injection drug users (IDUs). Interventions at 28 sites were studied. Outcome measures looked at needle risk behavior, sexual risk behaviors, and total frequency of drug injection. Data were collected on participant demographics, and on behaviors and knowledge before and after the interventions. Following interventions, IDUs reported significant increases in use of new needles, bleach for cleaning injection equipment, and condoms. Changes in needle-related behaviors were more dramatic than changes in sexual behaviors.

Immunization

Adverse Events Associated With Childhood Vaccines: Evidence Bearing on Causality reports the findings of an Institute of Medicine expert panel that reviewed published books, articles, and abstracts and listened to public testimony to determine whether a causal relationship exists between specific vaccines and adverse effects in children. Of the 49 conditions in which adverse effects from childhood vaccines were suspected, evidence in 12 cases substantiated a causal relationship; evidence in 4 cases rejected a causal relationship; and evidence in the remaining 33 cases was equivocal. The committee concluded that the available data often were insufficient to establish or reject causality, and the risk of dying or developing serious neurologic or immunologic complications appeared
to be very low.

A Study of the Economic Underpinning of Vaccine Supply examined issues related to the economics of production and distribution of vaccines. The study investigated (1) alternative models for purchase and distribution; (2) the effect of scientific advances on research, development, and purchase of vaccines; (3) the application of economic theory to the vaccine market; (4) comparisons of distribution systems managed by manufacturers and State agencies; and (5) the implications of purchasing vaccines from foreign firms. Findings indicate that supplying vaccines at low prices to physicians, encouraging parents to have their children vaccinated, and having effective and convenient distribution networks help to increase vaccination rates.

Public Health Infrastructure

Advanced Practice Nursing Education: Strategies for the Allocation of the Proposed Graduate Nursing Education Account. The rapidly changing health care delivery system is expected to cause a shortage of advanced practice nurses (APNs) in the near future. The study analyzed policy options on APN student characteristics, supply, overall education costs, and financing. The advantages, disadvantages, and costs of each option for ensuring a stable source of funds to support APN education were discussed.

Evaluation of Centers for Disease Control and Prevention (CDC) and Agency for Toxic Substances and Disease Registry (ATSDR) Training Activities. The study, using focus groups and site visits, examined the training needs of local health departments. It was found that local health departments were deficient in identifying their training needs and needed training in advocacy, evaluation, and public information. Anticipating increasingly limited resources at CDC and local agencies, recommendations emphasized the use of new training technology such as interactive software and satellite communications.

Evaluation of the Morbidity and Mortality Weekly Report (MMWR) Series. A mail/telephone survey of MMWR readers revealed that the CDC publication is read with great regularity, that each issue is passed along to several readers, and that most people have been reading the report for more than 5 years. Primary care physicians use the information to update their knowledge about diagnosis, treatment, and disease outbreaks. The MMWR was credited for its accuracy, relevance, and concise reporting. Suggestions for change include improved electronic access and larger page size.

Chapter III

Chapter III provides an overview of the evaluation activities of the eight PHS Agencies--presenting information on the evaluation program, a summary of evaluations completed in FY 1994 and evaluations in progress, and a discussion of future directions for Agency evaluations.

Agency for Health Care Policy and Research

The goals of the Agency for Health Care Policy and Research (AHCPR) evaluation program are to assess the Agency's effectiveness in meeting major and long-term priorities and goals; obtain information quickly to respond to critical Agency, PHS, and departmental concerns; and conduct internal evaluations to improve the efficiency of key program areas. Several evaluation mechanisms are used to achieve these goals: special evaluation studies; peer review of grant applications; the User Liaison Program, which provides information on the value of AHCPR-supported research to State policymakers; and other efforts such as focus groups to provide feedback on AHCPR products. During FY 1994, six evaluations were completed by AHCPR on two topics: the process of developing clinical practice guidelines and the evaluation of health care delivery. AHCPR currently supports 11 projects to facilitate improvements in health care quality and continues its commitment to improving the quality of AHCPR-supported clinical practice guidelines. For the future, AHCPR's programs must generate the information and tools needed to improve health care delivery and its outcomes and translate research findings into forms of information that actively assist consumers, practitioners, payers, and others in making effective health care decisions.

Centers for Disease Control and Prevention and Agency for Toxic Substances and Disease Registry

The Centers for Disease Control (CDC) places a high priority on evaluations seeking to answer policy, program, and strategic planning questions related to the mission of the Agency. Evaluations typically focus on programs to monitor the health of populations and communities, investigate health or disease problems, develop public health policies, implement prevention programs, promote healthy behaviors, foster safe environments, and provide public health leadership and training. The Agency for Toxic Substances and Disease Registry (ATSDR) evaluation program focuses largely on internal studies of its mission of environmental protection established by the 1986 Comprehensive Environmental Response, Compensation, and Liability Act (the Superfund). CDC completed 12 evaluations in FY 1994 in the areas of training and information dissemination, surveillance, program effectiveness, prevention, and costs of disease. Two evaluations are highlighted in chapter II: an evaluation of CDC and ATSDR training activities and a survey of readers of the MMWR. CDC has a total of 32 evaluations in progress in four areas: surveillance and data collection studies; program evaluations; community/intervention effectiveness studies; and evaluation methodology studies. In the future, CDC will be initiating projects designed to provide data for performance indicators and to assess the effectiveness and efficacy of such indicators.

Food and Drug Administration

Evaluation at the Food and Drug Administration (FDA) is largely a line management responsibility rather than one for specialized evaluation staffs. Projects are focused in the areas of performance management, customer participation, and more rigorous rulemaking. In FY 1994, the FDA conducted evaluations in three areas: implementation of the Prescription Drug User Fee Act of 1992 (PDUFA); the negotiation process toward medical device user fees; and the harmonization of international regulatory requirements--
"regulatory benchmarking"--in which specific components of the regulatory process in different countries are compared and assessed by program managers. Specific evaluations are currently in progress at the FDA on PDUFA management and performance of the Mammography Quality Standards Act of 1992. Also FDA has ongoing evaluation efforts to develop performance measures for additional user fee programs and to develop a customer-sensitive dimension to the commencement of negotiated or consensual rulemaking. Management changes in government are creating systemic changes in the FDA's evaluation function: FDA evaluation efforts are now largely driven by the mandates of the Government Performance and Results Act (GPRA) of 1993 and its corollaries.

Health Resources and Services Administration

The objectives of the Health Resources and Services Administration (HRSA) evaluation program are to improve program management and policy development and to provide information that will enhance strategic planning, budget decisions, and legislative planning. High priority is given to studies that assess program effectiveness or outcomes, or improve capacity for measuring performance. In FY 1994, HRSA completed evaluations in the following program areas: effectiveness of community health centers; linkage of mental health and primary care services; HIV service networks in rural areas; implementation of the Ryan White Comprehensive AIDS Resources Emergency (CARE) Act of 1990; rural health and telecommunications; populations served under the Hill-Burton Uncompensated Services Program; beneficiaries of nursing education projects; training for preventive medicine specialists; and strategic planning for the Bureau of Health Professions. HRSA has more than 40 projects under way. Subjects of the major program evaluations include the Healthy Start Initiative (intended to reduce infant mortality), the community and migrant health centers, the National Practitioner Data Bank, the Healthy Schools/Healthy Communities Program, the Injury Prevention Implementation Incentive Grants Program, and various health services funded under the Ryan White CARE Act. The top priority in the future for HRSA evaluations will be development of performance measures and data systems in conjunction with implementation of the GPRA. Future HRSA evaluations will focus on the Agency's mission of expanding access to care for the underserved, and such crosscutting priority areas as academic/community partnerships in health professions education, community infrastructure building, managed care, expanded roles for States, information technology, and HIV/AIDS.

Indian Health Service

The Indian Health Service (IHS) evaluation program serves the Agency's program and policy objectives, developed in consultation with tribal communities. It provides valid and reliable information to help the Director promote the IHS vision and guide its implementation. In FY 1994, the IHS completed nine evaluations on topics of service delivery, health status, and management. Three evaluations are having a major impact on program planning: an assessment of maternal and child health data in the Navajo area; an assessment of health status and access to care for Native Americans in California; and long-term health care projections for alcohol-related hospitalizations. The IHS has 21 evaluations in progress. Two of these underscore the IHS commitment to the prevention of child abuse and family violence. One report examines the extent of child abuse and neglect among American Indian tribes and the ways in which the IHS responds to these problems. The other is a case study of family violence in four American Indian communities. More than 100 unstructured interviews were conducted at the four sites to probe the nature and extent of family violence--spousal abuse, child abuse and neglect, child sexual abuse, and elder abuse. In the future, the IHS will initiate evaluations in three program areas: mental health services for urban Indians, regional treatment centers for substance abuse disorders, and health services for elderly American Indians.

National Institutes of Health

Evaluation is an integral part of the role of the National Institutes of Health (NIH) in supporting biomedical research, training, and public education. The NIH peer review system is a major part of its overall evaluation strategy: research proposals from scientists around the nation are subjected to a rigorous assessment
by fellow scientists before they are funded. National advisory councils, boards of scientific counselors, and consensus development conferences perform regular research evaluation functions. Four of the eight major evaluations completed by NIH in FY 1994 are highlighted in chapter II. The topics were adverse reactions to vaccines, indoor allergens, drug abuse prevention strategies, and interventions for high-risk behavior and HIV infection. Currently, NIH has 24 evaluations in progress, ranging from small- to large-scale assessments, from evaluation feasibility studies to full-blown evaluations. Future plans for NIH evaluations include an examination of new ways to carry out more efficient peer review of grant applications; a survey of public understanding about biomedical science and how Americans get their information about health; and development of new measures of NIH's internal performance, required by the GPRA.

Substance Abuse and Mental Health Services Administration

The Substance Abuse and Mental Health Services Administration (SAMHSA) is committed to evaluating its programs to assess the effectiveness of treatment and prevention approaches and systems of care; accountability for Federal funds; and the achievement of SAMHSA's program and policy objectives. SAMHSA's evaluations of demonstration programs generate new knowledge to lead the field in developing policies that improve services. During FY 1994, SAMHSA completed six evaluations of programs targeted to four population groups identified as being in greatest need of substance abuse and mental health services: pregnant and postpartum women and their infants, children with serious emotional disturbance, high-risk youth, and the homeless mentally ill. Currently, SAMHSA has 10 major evaluations under way in the following areas: program accountability, evaluation of demonstrations, reinforcing behavioral health, and commitment to customer service. SAMHSA's future evaluation activities will respond to emerging trends such as managed care and health care reform. Evaluation activities will reinforce the critical role of behavioral health in general health care.

Office of the Assistant Secretary for Health

The primary role of the Office of the Assistant Secretary for Health (OASH) is coordination and development of evaluations across the entire Public Health Service, often identifying potential program areas that could benefit from a collaborative evaluation. In FY 1994, OASH completed 17 evaluations in the areas of health care reform, immunizations, adolescent and school health, nutrition, primary care, emergency preparedness, international health, and health services delivery. Currently, OASH has 14 program and policy evaluations under way in two areas: health care reform and population-based public health services. Two issues of health care reform are being examined: (1) the role of Federal, State, and local health information systems in achieving public health objectives, and (2) the impact of health care reform on the health care workforce and the role of academic health centers. Other major evaluations include an examination of the clinical preventive services program, Put Prevention into Practice, and an assessment of the cost-effectiveness of clinical preventive services. In FY 1995, OASH will support evaluation activities in the PHS Agencies to examine the impact of managed care and health information systems on public health objectives.

A complete inventory of the 71 PHS evaluation projects completed in 1994 is provided in appendix A, and PHS Agency evaluation projects in progress are listed in appendix B.

Review criteria used by the special panel of senior editorial advisors for assessing evaluations can be found in appendix C.

Chapter I. The Evaluation Program in the Public Health Service

The Evaluation Program in the Public Health Service

The mission of the U.S. Public Health Service (PHS) is to protect and improve the mental and physical health of the American people and to close the gaps in the health status of disadvantaged populations through a variety of activities aimed at health policy development, research, public health practice, personal health services, regulation, infrastructure development, information dissemination, education, and training. Accomplishing this mission is the task of the eight Agencies that make up PHS: the Agency for Health Care Policy and Research (AHCPR), the Agency for Toxic Substances and Disease Registry (ATSDR), the Centers for Disease Control and Prevention (CDC), the Food and Drug Administration (FDA), the Health Resources and Services Administration (HRSA), the Indian Health Service (IHS), the National Institutes of Health (NIH), and the Substance Abuse and Mental Health Services Administration (SAMHSA). The Office of the Assistant Secretary for Health (OASH) is charged with program, policy, budget, and legislative coordination for the activities of these Agencies.

The PHS Evaluation Program plays an integral role in carrying out the PHS mission by reviewing various aspects of program performance of the PHS Agencies and OASH and by identifying means of improving that performance. The three major goals of the Evaluation Program are--

• To provide information on PHS programs that helps high-level government officials and Members of Congress make decisions related to policy, budget, and strategic planning;

• To help PHS managers examine the effectiveness and efficiency of program operations and thus improve performance; and

• To disseminate PHS evaluations--study results and methodological tools--that are useful to the larger public health community of State and local health officials, researchers, advocates, and practitioners.

These goals parallel the major audiences for PHS evaluations: (1) government decisionmakers, (2) PHS program managers, and (3) the larger public health community. For the first two groups, PHS uses ongoing program evaluation as both a philosophy and a tool to support informed decisionmaking that affects the health of the Nation. The PHS Agencies and OASH assess and provide technically reliable information on the extent to which these efforts achieve their desired results. This information is the primary tool that PHS uses to improve program performance on an ongoing basis.

For nongovernmental stakeholders, PHS has an important obligation to foster the development of (1) new knowledge about the effectiveness of public health programs and interventions and (2) evaluation tools for use by the larger public health community. The 1988 Institute of Medicine (IOM) report, "The Future of Public Health," stressed the Federal Government's important responsibility to support knowledge development and dissemination through data gathering, research, and information exchange. Evaluations conducted by PHS play a significant role in fulfilling this responsibility.

This chapter describes the PHS Evaluation Program. It provides an overview of the kinds of evaluation activities supported by the program and then describes one of the funding sources that PHS Agencies and OASH use to support these evaluations. It details PHS evaluation policies and management, including planning procedures, quality assurance, dissemination mechanisms, and use of results. The chapter concludes with a discussion of future directions for the program.

PHS Evaluation Activities

The evaluation activities sponsored by PHS and described in this report assess program performance (efficiency, effectiveness, responsiveness); analyze results based on those assessments; and use the resulting information in policymaking and program management. These activities are diverse and include the full spectrum of evaluation methodologies developed over the last quarter century. The classification of PHS evaluation activities presented in figure 1 summarizes that diversity.

Figure 1

Range of PHS Evaluation Activities

A. Evaluation projects

1. Outcome evaluations: assessing the immediate or intermediate effects of a program with respect to the stated goals or objectives.
2. Impact evaluations: assessing the broader results, intended or unintended, of a program on populations or institutions involved.
3. Implementation or process evaluations: assessing the nature of program inputs and outputs and their relationship to stated goals and objectives.
4. Policy assessments: examining health policies with respect to their development, implementation, or impact on public health or program activity.
5. Cost-benefit or cost-effectiveness analyses: developing methodology and its application to assess the relationship of program results to program costs (direct and indirect), often in comparison with alternative programs.
6. Survey data analyses: evaluating the results of PHS programs or policies by conducting or analyzing data obtained from surveys.
7. Management studies: examining the effectiveness or efficiency of the administration or operation of PHS programs and Offices.
8. Evaluation syntheses: integrating the results from multiple independent evaluation studies within a defined program or policy area in a fashion that improves the accessibility and application of those results.

B. Methodology projects

1. Evaluation feasibility studies: assessing the clarity and importance of program goals and objectives, the consensus of program stakeholders on the potential utility of evaluation information, and the availability of relevant performance data before committing to a full-scale program evaluation.
2. Evaluation design projects: procuring assistance in the development of an evaluation design, measurement tools, or analytic models in preparation for fully implementing an evaluation.
3. Instrument development projects: developing evaluation instruments (design, measurement, or analytic) for a specific PHS program or for general use by the public health community.

C. Evaluation support activities

1. Evaluation technical assistance: providing assistance to PHS program managers or Office directors on any aspect of evaluation planning; project design, implementation, or analysis; or use of results.
2. Evaluation dissemination: identifying target audiences and mechanisms to inform program constituencies and evaluation stakeholders about evaluation results.
3. Evaluation training/conferences: maintaining the professional skills and expertise of evaluation staff through training and promoting the dissemination of PHS evaluations through conference symposiums.

PHS evaluation projects typically fall into a combination of these categories. For example, comprehensive PHS evaluations generally examine both process and outcome or impact. Knowing only whether goals and objectives are achieved is insufficient without also knowing how well the program was implemented and whether its goals and objectives were appropriate in the first place. Similarly, evaluation feasibility and design activities generally represent the crucial first phase of major PHS process and outcome/impact evaluations.

Evaluation Funding

Funding for the PHS Evaluation Program is largely provided by a 1 percent evaluation set-aside legislative authority in Section 241 of the Public Health Service Act (42 U.S.C. \xa4 238j). This authority was established in 1970 when Congress amended the Act to permit the Secretary of Health and Human Services to use up to 1 percent of appropriated funds to evaluate authorized programs.[1] Amounts available from the 1 percent set-aside authority are determined annually and are subject to certain exclusions, such as program management and block grant funds, and deductions for health surveys and special evaluation projects. The remaining set-aside funds are distributed among the PHS Agencies and OASH for use under guidelines developed annually by the Assistant Secretary for Health.

Table 1 shows the set-aside evaluation funds used over the past 4 years by the PHS Agencies and OASH.[2] In addition to the set-aside, PHS Agencies support evaluation activities with appropriated program funds. Since other funds are used for demonstrations, health services research, or program developments that meet the general description of program evaluation, the total PHS funds used to evaluate its programs exceed the amount indicated in table 1.

Table 1

PHS Use of the Evaluation Set-Aside, Fiscal Years 1990-1994 (in thousands)

FISCAL YEAR	FY 1990	FY 1991	FY 1992	FY 1993	FY 1994
PHS Agency
ADAMHA/SAMHSA	$3,125	$3,763	$4,730	$2,906	$1,834
AHCPR	154	683	550	364	451
CDC	1,800	1,550	1,700	1,573	2,000
HRSA	3,432	5,119	3,645	6,753	5,359
NIH	2,849	2,234	4,500	3,854	6,500
OASH	3,000	3,000	3,252	3,500	9,366
Total Use	$14,360	$16,349	$18,377	$18,950	$25,510

 

Although the $25.5 million in FY 1994 set-aside funds used for evaluation amounted to only about one-tenth of 1 percent of the total PHS Agency appropriations for that year ($21.5 billion) and two-tenths of 1 percent of the funds applicable to the set-aside program, the PHS Evaluation Program provides a vital resource for program monitoring and improvement.

Evaluation Policy and Management

The policies and management of the PHS Evaluation Program, carried out on a regular basis by the PHS Agencies and coordinated by OASH, involve five basic functions:

1. Evaluation planning to establish priorities and coordinate development of new evaluations;
2. Quality assurance to maintain the standards of evaluation projects;
3. Project management and tracking;
4. Dissemination of evaluation results to program managers and other appropriate audiences; and
5. Application of these results for program improvement.

These functions are described below in general terms. Additional information on individual Agency and OASH evaluation programs is included in chapter III.

1. Evaluation Planning

Evaluation planning is conducted annually in concert with program planning, legislative development, and budgeting cycles. Before the start of each fiscal year, evaluation guidance is issued by the Assistant Secretary for Health (ASH) to signal the priority PHS program and policy areas for evaluation. Typically, they include new programs, those undergoing major change, those that are candidates for reauthorization, and those for which key budget decisions are anticipated. In addition, emphasis has been given to evaluations that support strategic planning goals and objectives; evaluation syntheses that address program areas currently under review by both Congress and the Department; and evaluations that cut across PHS Agencies and have broad implications for program or policy change. In FY 1995, a new element was introduced into the PHS evaluation planning process: a panel of experts from universities and private research centers offered consultation before the development of the PHS guidance for FY 1996.

In response to the ASH guidance, each PHS Agency submits a plan of its evaluation strategies and proposed projects for the immediate and subsequent fiscal years. Sharing information allows the Agencies to learn about evaluation proposals being considered throughout PHS, promotes PHS-wide coordination, and avoids duplicative efforts. The evaluation planning process also facilitates coordination with Agencies outside PHS, such as the Departments of Agriculture, Education, and Justice.

2. Quality Assurance Most evaluation projects are developed at the program level, and the initial review is conducted by a committee of Agency-level policy and planning staff members. Before a project is approved, it is reviewed for technical quality, generally by a second committee of staff who are skilled in evaluation methodology. Technical review committees follow a set of criteria for quality evaluation practice established by each Agency. Some PHS Agencies also have external evaluation review committees composed of evaluation researchers and policy experts from universities and research centers. More details on the quality assurance procedures for the various Agencies and OASH are presented in chapter III.

3. Project Management and Tracking A computer database, the PHS Evaluation Management System (PHS-EMS), has been operating since 1992 for managing information on evaluations and policy research studies. Information is continuously entered into the database, based on reports from PHS Agencies, and it is used to produce reports that track the progress of evaluations under way or to locate information about recently completed evaluations. In FY 1996, a component will be added to allow Agency evaluation staff remote computer access to the database. This database system is also coordinated with the HHS Policy Information Center (PIC)--the departmental evaluation database and library maintained by the Office of the Assistant Secretary for Planning and Evaluation. Updated records on PHS evaluations and the final reports on completed evaluations are regularly transferred to the PIC. As an information database and library resource, the PIC contains nearly 6,000 completed, ongoing, and planned evaluation and policy research studies conducted by the Department, including key studies completed outside the Department by the General Accounting Office (GAO) and private foundations.

4. Dissemination Typically, the results of PHS evaluations are disseminated through targeted distribution of final reports, articles in refereed journals, and presentations at professional meetings and conferences. Although the major responsibility for disseminating results lies with the PHS Agencies, departmentwide efforts are under way to expand dissemination to the public health community. Abstracts of all studies maintained in the PIC database are now accessible to the public health community through the Department's World Wide Web server (http://www.hhs.gov) on the Internet. The preparation of this report is another effort to expand dissemination of results; the report will be distributed widely to State and local public health agencies, schools of public health and other university health programs, and private foundations. Finally, a number of evaluation syntheses are being developed to enhance the accessibility of results from multiple evaluations that address similar issues.

5. Application PHS evaluations are generally used directly by program managers to improve program operations and efficiency. In addition, in accordance with a 1993 GAO recommendation (see Publication No. GAO/PEMD-93-13), more evaluation focus will be directed to outcome/impact evaluations that can be applied by Congress and others for program planning, budgeting, and legislative action. This shift is also consistent with implementation of the Government Performance and Results Act (GPRA) and the National Performance Review (NPR), which seek to improve the efficiency and effectiveness of Federal Government programs and the services provided to customers of those programs and directly to citizens.

NPR stresses the importance of consumer satisfaction surveys (two of which are described in chapter II) and performance measures as tools for improving Government programs. Performance measures have been a cornerstone of the total quality management (TQM) movement in business and industry, where production outputs and quality outcomes can often be readily defined. Likewise, NPR has challenged PHS program managers to examine their outputs and outcomes and to conduct studies that would establish valid and reliable indicators of program performance.

While NPR has greatly influenced Federal Agencies to begin developing performance measures, it is GPRA that will link performance measures to the Federal Government's annual planning and budgeting process. The ultimate goal of GPRA is to move Federal Agencies toward performance budgeting. Performance budgets will provide Agencies with information on the direct relationship between program spending and expected results, and the anticipated effects of varying spending levels on results. As for the PHS Evaluation Program, the goal of having performance measures in place by FY 1999 will require a substantial investment of evaluation resources and will consequently change the nature and use of these evaluations.

Future Directions in Evaluation

In addition to NPR and GPRA, PHS must be responsive to public health issues of the future. Public health has an opportunity to change the American health care system, shifting its focus toward protecting and improving health. This opportunity to reinvent public health is the theme of the PHS Strategic Plan for FY 1995. The plan centers on the following three goals, which PHS, working with the public health community, will pursue:

• Help the State health agencies--and their component communities--shift from a treatment-oriented to a health-oriented health care delivery system;

• Become partners in the managed care revolution to ensure that vulnerable populations have access to cost-effective preventive services; and

• Take a leadership role in the National Information Infrastructure (NII) from the sides of both consumer health information and population-based or integrated national, State, local, and private health care information systems.

As the following pages detail, PHS leadership envisions these three goals as guiding future priorities of the Evaluation Program.

1. Shift from personal care to population-based services The PHS FY 1995 Strategic Plan focuses on strengthening the public health infrastructure of the Federal Government, tribal governments, States, and communities to identify and address high-priority health problems for their populations and to do so in a way that focuses on the outcomes of their activities. PHS intends to provide the States with more flexible funding by consolidating several categorical programs into Performance Partnership Grants (PPGs). Partnerships are proposed in six program areas--mental health, substance abuse, immunizations, infectious disease, chronic disease, and prevention. These grants would be based on the premise that giving the States greater flexibility to develop and manage programs in exchange for increased accountability for results will encourage more effective use of Federal funds to address national public health objectives, such as those developed for Healthy People 2000. The success of PPGs with the States depends on knowing what the grant funds can achieve (goals), whether the results over time can be known (measures), and how performance can be rewarded (incentives). Program goals, to be specified in legislation, would lay out national objectives for each grant. Performance measures would quantify national and State progress toward the goals. The measures must be results oriented, measuring success based on program outputs and health outcomes developed mutually by PHS Agencies and the States. Performance incentives would be an integral part of the individual contracts between the States and the relevant PHS Agency, including specific performance targets and timeframes for achieving them. The incentives might, for example, be to extend greater flexibility to States that meet their performance targets.

The PHS Evaluation Program will play an important role in each performance partnership, from the development of the performance measures to the collection and analysis of indicator data. PHS Agencies will work with the States to select the measures, design data collection strategies, and develop analytic models for interpreting results relative to performance targets. The States will be able to use grant funds to develop integrated information systems for managing programs and measuring performance.

2. Public health partnership with managed care The managed care revolution has the potential to establish the foundation for a full partnership between public health and the personal care system on behalf of the health of defined populations. With payments based on annual premiums, managed care creates financial incentives that reward health plans for having healthy clients. Potentially, this incentive could encourage health plans to make sure that their enrollees take advantage of cost-effective preventive services. It also encourages plans to work with and support the efforts of public health agencies and community organizations, which can prevent unnecessary disease and health care costs through communitywide interventions. Realizing the benefits of organized systems of care, however, requires a strong system of performance monitoring to ensure that health plans are living up to their commitments and taking responsibility for improving the health of enrolled populations.

In the next few years, PHS Agencies will examine the impact of managed care on public health systems. First, Medicaid Waiver (Section 1115) demonstrations in five States will be evaluated to determine the extent to which those eligible have participated and the effects of the program on the care participants receive. Particular attention will be given to access and quality issues for special populations, such as American Indians and Native Alaskans, children with special health care needs, people in underserved areas, substance abusers, and the chronically mentally ill. The effects on costs to providers, especially the extent to which cost neutrality is achieved, will also be studied.

Many academic health centers (AHCs) have been particularly affected by the managed care revolution. PHS will analyze the changes in the capacity of AHCs resulting from the fiscal impact of competitive health insurance and Medicaid managed care plans. The impact of these organizational changes on medical education, biomedical research, advances in medical technology, access to tertiary care, and the provision of services to indigent populations will be assessed.

In the future, PHS will also support health services research in the areas of quality measurement and consumer choice. These studies will provide valuable preliminary information to managed care organizations and other providers as they develop their quality measurement tools. PHS evaluations are already under way at AHCPR to examine clinical practice guidelines and assess how well they can be incorporated into managed care operations. AHCPR will also assess alternative educational methods for disseminating the guidelines and review criteria for changing the practice behavior of providers.

3. Leadership in the National Information Infrastructure The National Information Infrastructure (NII) refers to the web of electronically accessible information resources available to Americans. The NII holds potential for rapid and vastly enhanced interactive information flow. Multi-user list server technologies will allow infor-mation that is sent to a single location to go to all entities on the list. Electronic bulletin boards and multiple-recipient electronic mail (e-mail) are revolutionizing the way information is dis-seminated and the ability to have followup communication and dialogue. More and more communities or their critical components are becoming part of local and wide-area health information networks, often called community health information networks. These networks can ease transmission of and access to consumer health information.

The Federal Government has played a major role in the production and dissemination of health information to consumers, but its role in evaluating that information is limited. It monitors effective dissemination, including who receives the information, how it is used (particularly by specific population segments), and what behavior changes are associated with it. For example, research has been conducted to determine how target consumers become aware of, receive, accept, and use disseminated information. The test of effective use is the extent to which target audiences become more informed, make decisions, or change behavior patterns and attain better health as a result of using the disseminated information. The Federal Government also helps determine whether consumer health information improves patient satisfaction, care, outcome, and overall quality of life. PHS, particularly, should be measuring the effect of consumer health information on such traditional aspects of public health as maternal and child health, childhood immunization, prevention and control of communicable diseases, and environmental health problems. The issue of maintaining or validating the quality and integrity of health information is also a serious concern in a digital information marketplace. The Government's role in ensuring that the content of such health information (either publicly or privately generated) remains scientifically sound needs to be defined.

In addition to evaluating the consumer health information dimension of NII, the PHS Evaluation Program will examine the role of population-based health information systems in achieving public health objectives. A strong consensus about the features of a public health/personal health care information network is emerging among major stakeholders. At the Federal level, most legislative bills envision a public/private partnership in which the private sector owns and operates the bulk of the system and the Federal Government establishes the overall policy framework, including national uniform standards for reporting and electronic exchange of data; unique identifiers for individuals, health plans, and providers; and strong privacy and security protections. This type of NII health information system could simplify administration and enhance the usefulness of information in several ways. It could create a standardized clinical vocabulary and coding structure for health information. It could ensure a secure environment for the transmission and exchange of health information. It could allow information routinely collected in health care service delivery and payment to be used for other health-related purposes, such as promoting access and quality of care, achieving public health objectives, and advancing medical research.

As a major goal of the PHS FY 1995 Strategic Plan, the PHS Agencies will develop integrated information systems that protect privacy; reduce reporting burdens; and provide participants in both the public and personal health care systems with the information they need to define health problems, assess performance, ensure quality, support decisionmaking, and take timely action to protect and improve health. To support this goal, the PHS Evaluation Program will examine PHS programs and policies that help State and local communities develop their information systems serving both public health and medical providers. One current OASH project is developing indicators that communities can use to assess performance of the health care system and foster a closer collaboration between the public health and personal health care systems to achieve public health goals.

An assessment of national, State, and local health data systems is also essential to identify optimal processes for developing performance measures for the public and personal health care systems, in the framework of Federal-State partnerships designed to achieve public health objectives. PHS will be reviewing and evaluating past efforts, current issues, and future opportunities for partnerships with the States to conduct health surveys that collect data for planning, policy development, research, and other purposes. Evaluation information is needed on the successes and failures of past PHS data collection efforts with the States to aid discussions about strategies for future partnerships in this area.

Chapter II. Highlights of Selected Evaluations From the Previous Year

Highlights of Selected Evaluations From the Previous Year

In this evaluation report, the Public Health Service (PHS) highlights evaluations of general interest to the public health community and illustrates the diversity of PHS evaluations completed in fiscal year (FY) 1994. Presented in this chapter are summaries of 13 evaluation projects selected by a special panel of the Senior Editorial Advisors, based on criteria identified in appendix C and applied to approximately 40 reports nominated by PHS Agencies. These criteria are--

• Is the report important? Does it address a significant issue or problem for which evaluation would help to confirm or to change program direction, or to measure program impact? Are the findings likely to be useful and generalizable?
• Is the report methodologically sound? Are its concepts, designs, data collection, and analyses conducted and reported in a competent manner?
• Is the report faithful to the data? Do the conclusions and recommendations logically follow from the data and analyses, and are they relevant to the questions asked?

The 13 evaluations selected by the panel are summarized here in the following categories:

• Access to Health Care for Special Populations
• Adolescent Health Problems
• Environmental Health
• HIV/AIDS
• Immunization
• Public Health Infrastructure

Each summary includes a brief abstract; a description of the study, including its purpose, background, methods, findings, and use of results; the names of any publications that resulted; and the name and address of the person to contact for additional information.

Access to Health Care for Special Populations

Making a Difference: Interim Status Report of the McKinney Research Demonstration Program for Homeless Mentally Ill Adults

Highlights

This report synthesizes the findings from five separate demonstration projects that evaluated the effect of offering case management and housing to the homeless mentally ill. Early findings indicate general success at reducing homelessness and at improving mental health of study participants: homeless adults with severe mental illness were willing to use accessible services targeted to their needs. The factors contributing to the success of the project include mental health treatment, receipt of entitlement income, and reliance on an interdisciplinary team of staff, including some with mental illness who were formerly homeless. These findings offer strategies for the design and management of programs for the homeless mentally ill.

Purpose

The goal of the 1987 Stuart McKinney Act was to create programs to reduce homelessness. Among numerous provisions, the Act authorized demonstration grants to examine new approaches to relieving homelessness. This project was designed to explore new ways of reducing homelessness and improving the mental health of homeless people by offering distinct combinations of case management and housing to address one of the most challenging problems facing our cities.

Background

An estimated 600,000 people are homeless in the United States, based on 1987 estimates. One-third, or 200,000, are homeless adults with severe mental illness. Besides suffering from schizophrenia, depression, or other mental disorders, they commonly are afflicted by substance abuse, tuberculosis, and HIV. This constellation of mental and physical conditions is believed to place the homeless mentally ill beyond the reach of the conventional social service and health care system.

Launched by the National Institute of Mental Health in 1990, the McKinney Research Demonstration Program for Homeless Mentally Ill Adults was funded for 3 years with a total of $16.8 million. The program was transferred to the Center for Mental Health Services in the newly created Substance Abuse and Mental Health Services Administration as a result of the ADAMHA Reorganization Act of 1992.

Methods

A total of 896 homeless adults with severe mental illnesses were studied at five sites--Boston, Baltimore, San Diego, and two sites in New York City. Fifty-seven percent of the participants had psychotic disorders (e.g., schizophrenia) and 32 percent had affective disorders (e.g., depression). More than one-third were living in shelters, slightly less than one-third were living on the street, and the remainder were living in community settings, hospitals, or jails. Across all sites, the underlying goals were to offer a combination of housing and social services that would train participants in daily living, link them to needed social and medical services, help them receive entitlements for which they are eligible, and assist them in securing housing. Study sites varied in intensity, duration, or approach. The demographic portrait that emerged was sufficiently similar to the general homeless population that the final results are expected to have widespread applicability to thousands of communities throughout the Nation.

Interventions tested varied across sites according to the level of case management, the type of housing, and the degree of client engagement with the social service or health care system at the outset of the study. The comparison group of homeless mentally ill at most sites received traditional services. Study participants were randomly assigned to the new services or to traditional services.

Five broad outcome measures unified the analysis across study sites--psychiatric symptomatology, substance abuse, quality of life, physical health, and residential stability. Participants were interviewed at several stages (at baseline and at followups at 6 months, 9 or 12 months, and 18 or 24 months) to determine their progress on these measures. They were given fairly uniform questions along with a standard battery of widely accepted measures of mental health status. Some of the interview data were corroborated by case manager reports. While investigators are performing separate analyses at each site, additional analyses are being undertaken across sites. The effectiveness of the interventions in reducing costly hospitalizations and other health expenditures was examined at two study sites.

Findings

Results are still preliminary, covering the 6- to 12-month followup to the intervention. Early findings reveal that each site's intervention was successful at reducing homelessness and improving the mental health of study participants. These preliminary findings, expected to be sustained when the study is concluded, should also hold true when the findings from all sites are combined.

These general findings were amplified by many discrete, yet interrelated, findings. Most important, it was found that the homeless who are mentally ill were willing to use accessible services targeted to their needs, repudiating the belief that these people are beyond help. When they did take advantage of mental health treatment, there was a decrease in psychiatric symptoms and in the use of costly inpatient hospitalization. In one of the projects, inpatient days were halved.

Another finding was that targeted services decreased homelessness, making permanent housing an attainable goal--as long as there were appropriate levels of support. A critical period for intervention was during the transition from the shelter to community-based housing. The receipt of entitlement income was another vital element to realizing study goals. At one of the sites, the intervention yielded a two- to threefold increase in the number of participants being supported by Supplemental Security Income and Social Security Disability Income.

Formerly homeless people with mental illnesses also were found to be a valuable staff resource: they staffed four of the five sites, helping to identify and engage participants, sensitizing professional staff to their needs, and acting as role models.

Substance abuse, which was widespread among participants when the study began (47 to 78 percent), exacerbated homelessness. Participants' substance abuse was felt by clinical staff to be more significant than their mental illness in preventing them from finding or keeping housing.

Use of Results

Emerging policy recommendations thus far integrate service systems at all levels for the homeless mentally ill, bringing together mental health, substance abuse treatment, social services, and the criminal justice system; emphasize substance abuse treatment as an integral part of mental health services; offer clients a greater range of housing options; and provide preventive health care and education to reduce morbidity and mortality from severe illnesses experienced by this population.

Publications

More than 30 publications had emanated from the study at the time of this report, in such journals as Community Mental Health Journal, Psychiatric Services, Hospital and Community Psychiatry, Psychosocial Rehabilitation, American Journal of Psychiatry, Evaluation Bulletin, and Schizophrenia Bulletin.

Agency sponsor:

Substance Abuse and Mental Health Services Administration

Federal contact:

Roger Straw, Ph.D.
Center for Mental Health Services
Substance Abuse and Mental Health
Services Administration
Parklawn Building, Room 11C26
5600 Fishers Lane
Rockville, MD 20857
(301) 443-3606 Fax: (301) 443-0541

Principal investigators:

Anthony Lehman, M.D., The Baltimore Project, Baltimore, MD

Stephen Goldfinger, M.D., The Boston Project, Boston, MA

David Shern, Ph.D., The New York City Street Outreach Project, New York, NY

Elie Valencia, J.D., The New York City Critical Time Project, New York, NY

Richard Hough, Ph.D., The San Diego Project, San Diego, CA

Report to Congress on the Indian Health Service With Regard to Health Status and Health Care Needs of American Indians in California in Response to Public Law 100-713

Highlights

This study documented the health status and access to health care services of American Indians in California, especially those in tribes that are not federally recognized. Through analysis of vital statistics and other databases, comparisons were made between California Indians who are members of federally recognized tribes and those who are not. On the basis of these and other comparisons, American Indians in non-federally recognized tribes in California generally were found to have poorer health status than those in federally recognized tribes. The health status of both groups is inferior to that of other populations, underscoring the importance of maintaining and expanding coverage to the entire American Indian population of California.

Purpose

The Indian Health Service (IHS) provides comprehensive health care to American Indians and Alaska Natives throughout the United States. By Federal law, care is given to members of federally recognized tribes, except in California and a few other States, where care is given to all Indians, not just those who are members of federally recognized tribes.

This study analyzes and interprets numerous data sources to enhance understanding of the health status of American Indians in California, including members of federally recognized and non-federally recognized tribes.

Background

In 1990, an estimated 242,000 American Indians resided in California. Approximately 100,000 are registered at California Indian tribal health clinics. According to IHS records, about 26,000, or one-fourth, are not members of a federally recognized Indian tribe of California. If they seek IHS-funded care in the State, they must rely on tribally operated facilities that operate under contract to the IHS. Throughout the State, there are a total of 21 tribally operated rural clinics, 7 Indian urban health programs, and 13 freestanding Indian alcoholism programs. Even though there are no tribally operated inpatient services, IHS provides for inpatient care through contracts with local providers.

In most States, IHS-funded health care is limited by law to members of federally recognized tribes. But in California, as well as some other States, there has been a longstanding tradition of providing care to all Indians, not just those in federally recognized tribes. This practice was codified in 1988 with Federal legislation, the Indian Health Care Amendments of 1988.

At the time this report was undertaken, information about IHS-funded care provided to California Indians was limited because participation in IHS reporting systems had only recently begun. In addition, the information systems of many non-Indian health care facilities either did not identify Indian clients or did not identify them correctly. Little was known about whether membership in a federally recognized tribe had an impact on health status.

Methods

The study conducted by the University of California, San Francisco, compared the health status of California Indians--including members of non-federally recognized tribes--with that of all Californians. On some measures, comparisons were made with Indians throughout the United States and/or with the U.S. population as a whole. The study relied on numerous existing sources of data, such as vital statistics, hospital discharge records, infectious disease case reports, and client and financial data (especially from California's Medicaid system). Information was also gathered from selected State and Federal health and welfare programs.

All persons identified as American Indian or Alaska Native were included in these analyses. Because federally recognized status could not be identified from vital statistics, inferential measures were used to classify California counties according to whether they were Indian or non-Indian counties. The Indian counties were further subdivided into those whose residents were primarily members of a federally recognized tribe and those whose residents were primarily not members of federally recognized tribes. Thus, residents of "non-federally recognized counties" were, for the purposes of the study, considered to be California Indians who were primarily members of non-federally recognized tribes.

In addition to using existing data sources, researchers collected information on a sample of 348 California Indians age 18 and older who were not members of federally recognized tribes. This information was collected by tribal health programs from patient registration files in tribal clinics. The information included current sources of health care, resources for payment, and availability and accessibility of alternatives to IHS-funded care.

Findings

Based on multiple measures, the health status of California Indians in non-federally recognized tribes was determined to be no better and, in some ways, worse than that of California Indians in federally recognized tribes. And those in federally recognized tribes generally had poorer health status than Californians as a whole.

From 1986 through 1988, 10.9 percent of all California births were to teenagers (women under the age of 20). Yet, 16.7 percent of all California Indian births were to teenagers, and an even higher proportion, 20.7 percent, were to Indian teenagers in non-federally recognized counties in California. Births to teenagers were most common in areas where Indian clinics were least available.

Among California Indian babies born during this time, 6.4 percent had low birth weight, compared with a statewide figure of 5.2 percent for whites and Hispanics. Only 68.3 percent of pregnant California Indians received prenatal care in their first trimester, compared with 74.4 percent of all pregnant Californians. In non-federally recognized California counties, 9.1 percent of pregnant Indians received late (third trimester) or no prenatal care, versus 7.6 percent of those in federally recognized counties.

Alcohol and tobacco use were found to have a major impact on the health of California Indians, irrespective of tribal status. Chronic liver disease and cirrhosis caused a higher proportion of deaths among California Indians (6.7 percent) than among all Indians in the United States (4.3 percent) and the total population of California (1.9 percent). Overall, 33.6 percent of deaths among Indian women and 42.1 percent of deaths among Indian men were alcohol-related, compared with 4.3 percent for women and 8.4 percent for men of all races in California.

Also in the years 1986 through 1988, 41.7 percent of deaths among Indian women and 37.4 percent of deaths among Indian men were attributed to cigarette smoking, compared with 12.4 percent and 17.8 percent of all female and male deaths, respectively, among all races in California.

Premature mortality was also a serious problem among California Indians. Deaths in that population between 1986 and 1988 were more than twice as likely to occur before the age of 45 than statewide (28.4 percent versus 13.3 percent). Mortality under the age of 25 accounted for 11.2 percent of Indian deaths, compared with 5.3 percent of all deaths in California.

The survey of members of non-federally recognized tribes revealed that 60 percent identified local tribal health programs as their usual source of care. They experienced restricted access to other sources of health care, given that 33 percent reported no health coverage. Members of non-federally recognized tribes were found to have unmet needs for a variety of health services, such as dental care, diabetic and orthopedic supplies, and eyeglasses.

Use of Results

The results suggest that access to IHS-funded tribal health programs and clinics in California is essential for non-federally recognized California Indians. There is no evidence that resources outside the IHS are adequate to meet their health care needs. In addition, tribally operated health services provide a valued source of care for those who prefer Indian-specific and culturally competent services. Restrictions in eligibility or inadequate levels of funding could have serious health consequences.

Study findings have broad implications for health promotion and disease prevention among California Indians. Strategies are particularly important for reducing tobacco and alcohol consumption, lowering rates of heart disease, expanding cancer screening programs, preventing accidents, and increasing early prenatal care.

Publication

Highlights are available through the Monograph Series of the Institute for Health Policy Studies, University of California, San Francisco: American Indians in California: Health Status and Access to Health Care.

Agency sponsor:

Indian Health Service

Federal contact:

Leo J. Nolan
Division Director
Division of Program Evaluation and Policy Analysis
Indian Health Service
12300 Twinbrook Parkway, Suite 450
Rockville, MD 20857
(301) 443-4700 Fax: (301) 443-1522

Principal investigator:

Leila Beckwith
University of California, Los Angeles, CA

Adolescent Health Problems

Developing Effective Health Communication Strategies for High-Risk Youth Outside of School

Highlights

This study assessed the knowledge and attitudes of high-risk youth who regularly engage in practices such as tobacco use, substance use, unprotected sex, and violence. High-risk youth were found to be fairly knowledgeable about healthy practices, yet they do not incorporate this awareness into practice. Their willingness to listen to trusted, credible adults in alternative settings was among the findings that can be incorporated into the delivery of targeted health promotion programs and the design of future research projects.

Purpose

High-risk youth, who often do not attend school, are less likely than their counterparts to be reached by health promotion programs and services provided through schools and other youth-oriented community organizations. This project was undertaken to learn more about the attitudes and beliefs of these young people in order to foster the development of more effective health promotion programs and services.

Background

Many serious health problems affect young people. For example, in 1987, approximately 2.5 million teenagers were infected with a sexually transmitted disease. In that same year, an estimated 2.5 million teenage girls became pregnant unintentionally. Many young adults with AIDS were likely infected with HIV during adolescence (because of the lag time between infection and onset of AIDS). Youth at particular risk are those ages 10 to 18 who regularly engage in a cluster of risky practices: smoking and drinking, using drugs, having unprotected sex, and/or being victims or perpetrators of violence. These risky behaviors, which begin as experimentation and may become ingrained habits, compromise the possibility of a long and healthy life.

Prevention strategies must reach and influence young people before they adopt risky behaviors, but conventional health education programs may not reach high-risk youth who do not attend school regularly or participate in community activities.

Methods

This focus group study, conducted by S.W. Morris & Company, Inc., questioned 160 high-risk youth ages 10 to 18 about their health practices. From December 1990 through March 1991, 24 focus groups were conducted with young people of various races and ethnic groups. Each focus group was composed of individuals of the same age, gender, and race/ethnicity. Community-based organizations recruited participants and provided facilities for the focus groups. The moderator followed a standard format seeking answers to questions about health as a priority; knowledge and attitudes toward smoking, AIDS, pregnancy, alcohol and other drugs, and violence; barriers to the adoption of positive health practices; and opportunities for programs to address risky behaviors.

To the extent possible, each group was led by a moderator of the same gender and race/ethnicity as the participants. To ensure confidentiality and encourage the trust of participants, moderators audiotaped and kept handwritten notes of the focus group discussions. African-American and Mexican-American participants were recruited from the inner city. Another group of Mexican Americans came from a suburban alternative school. Other African Americans, Mexican Americans, and American Indians were drawn from inner-city community-based and religious organizations and a tribal community center. White participants were identified through a rural site with countywide acces-sibility.

After the focus groups were completed, two meetings were conducted with staff from national youth organizations and local youth programs. The staff recommended that discussion groups be conducted with the parents and caregivers of high-risk youth. In December 1992, eight mixed-gender adult discussion groups were conducted at the same sites as the youth focus groups.

Focus group methodology has several limitations. Because population sampling procedures are not used, results are not generalizable. Moreover, Mexican-American youth were the only Hispanic participants, and no Asian/Pacific Islander Americans were included in this study. White participants were drawn from a rural area, while all other groups were recruited from suburban and inner-city communities. Estimates of health-related behaviors were based on participant self-reports. The findings of the study represent an analysis of moderators' observations, transcripts, and other information derived from the study. As such, the findings are qualitative, not quantitative, in nature.

Findings

Approximately two-thirds of the focus group participants resided in households receiving public assistance. Nearly one-half lived with a single parent. The majority were truants or school drop-in/dropouts. About one-third reported substance abuse or arrest records. Adult supervisors estimated that over 40 percent were involved in gangs.

Participants were fairly knowledgeable about many current health issues, such as smoking, unprotected sexual activity, and alcohol and other drug use. They were not as knowledgeable about HIV infection and AIDS. Their knowledge resulted from high exposure to health information from many channels, especially from parents, families, and friends. There was little reported reliance on mass media as a source of health information.

Despite awareness of the consequences, many participants reported engaging in behaviors that pose health risks. In this population, risk, violence, and death were seen as a normal part of life, and there was a lack of faith in the future. Therefore, potential future benefit offered little motivation to discontinue a risky practice experienced as enjoyable. Of 12 life priorities, participants selected "being loved" and "having a family" as the top two. "Being healthy" ranked eighth, suggesting a relative lack of emphasis.

Participants reported having much unstructured, unsupervised time. Many expressed a desire to talk with a nonjudgmental adult they could trust, someone who understands what they are experiencing in life, but they did not have such a person at home or at school. Adult focus group participants identified numerous influences on youth, including friend/peer pressure, media and popular culture, and racism/prejudice. Many parents found it difficult to initiate discussions about risky behaviors with their children. Many parents and caregivers reported practicing risky health behaviors themselves. They recognized that their actions conveyed a mixed message when they tried to discourage youth from adopting the same behaviors. While acknowledging responsibility for the young people in their care, these adults also felt that communities should do more for at-risk youth.

Use of Results

The focus group findings reinforced the tenet that knowledge alone is insufficient to change health practices. Participants' emphasis on linkages between health problems (e.g., substance abuse and AIDS) underscored the need for a multiple risk factor approach in the design of outreach programs. Any approach must also incorporate the provision of specific services to help at-risk youth develop the skills to adopt healthier lifestyles. Outreach programs must also take advantage of trusted, credible adults who can help youth make difficult changes.

Findings suggest that health information, programs, and services might be effectively provided in alternative settings that high-risk youth view as "safe havens," that is, those outside mainstream institutions. Nevertheless, a family outreach component should be included, since at-risk youth identified family as an important and trusted influence. Parents need services and support to become better role models and to enhance their communication skills.

Music may have potential as a communication channel to high-risk youth. High-risk youth were suspicious about messages from celebrities, who they thought were interested only in money or exposure. They expressed concern about subtle racial and ethnic stereotyping in health messages, even those intended to be culturally relevant. Therefore, the report recommended careful message crafting, testing, and targeting. Because at-risk youth cannot envision their own future, they cannot be motivated to avoid or discontinue risky health behaviors on the basis of future benefits. Health communications need to impart the belief that there will be a future.

Publication

Not yet published.

Agency sponsors:

U.S. Department of Education, U.S. Department of Health and Human Services, U.S. Department of Justice, and U.S. Department of Transportation

Federal contact:

Mary Jo Deering, Ph.D.
Office of Disease Prevention and Health Promotion
Office of the Assistant Secretary for Health
Switzer Building, Room 2132
330 C Street SW
Washington, DC 20201
(202) 205-5968 Fax: (202) 205-9478

Principal investigator:

Ruth Karimi
S.W. Morris & Company, Inc., Bethesda, MD

Evaluating Educational Outcomes of School Health Programs

Highlights

This study addressed the role of school health programs in improving academic performance. It described the available evaluation literature, analyzed the benefits and limitations of alternative research designs, and offered two major approaches to the design of an evaluation. With this guidance, schools can justify not only the health importance but also the academic importance of their health programs.

Purpose

This study was undertaken to provide guidance to educators in evaluating the effects of health programs on students' school performance. School performance includes grades and other indicators of educational achievement, attendance and dropout rates, and educational expectations. The three specific goals were (1) to review the state-of-the-art of school health program evaluations; (2) to describe the kinds of measures used and their feasibility for assessing educational achievement following participation in health programs; and (3) to provide two research options for evaluating the relationship between school health programs and educational performance.

Background

There are eight general types of school health programs: health education, health services, efforts to promote a healthy school environment, school food service, physical education and fitness, integrated school and community health promotion, school counseling, and health promotion for school faculty and staff. Both the national education agenda, Goals 2000, and the national health agenda, Healthy People 2000, explicitly recognize the relationship between education and health. Although it is generally accepted that good health is necessary for effective learning, a lack of well-designed evaluations has resulted in little empirical evidence on the link between school health interventions and educational outcomes. The absence of evidence has led to retrenchment and the shifting of resources from health programs to academic programs. To maintain present school health programs and to encourage new programs, strong empirical evidence is needed on how these programs affect students' academic performance. This study offers educators the tools to collect this empirical evidence.

Methods

This study, conducted by Mathematica Policy Research, Inc., provides a framework for assessing educational outcomes of school health interventions through the following methods: (1) a literature review of both the current knowledge about the effectiveness of school health interventions and the important design issues for evaluating health and educational outcomes of school health programs; and (2) consultation with an advisory panel of researchers and policymakers on the types of health interventions likely to affect school performance, the relevant educational outcomes, data sources for evaluation, and the types of evaluation designs that are feasible in school settings. "Intervention" was defined as a specific plan, program, program component, or strategy aimed at changing students' health attitudes, behavior, or health status and, subsequently, their school performance.

Findings

In review of the evaluations of eight types of school health programs, programs varied in effectiveness for student health outcomes. Gains were moderate to large in students' health knowledge and smaller for changes in attitudes, behaviors, and health status. In addition, programs generally required ongoing intervention to sustain positive behavioral effects. The health interventions that have been most thoroughly evaluated are nutrition programs, physical activity programs, and drug abuse prevention programs.

Fewer studies examined the educational outcomes of health programs. Of these studies, a handful demonstrate improvements in basic academic skills, reductions in tardiness and absenteeism, and reductions in use of drugs.

The study provided a list of the data requirements for measuring student short-term educational performance and descriptive information about students, schools, communities, and school health interventions. Issues and problems about data collection, availability of data from student surveys, and obtaining and using these data are also thoroughly reviewed. Some problems related to data collection are (1) obtaining the release of sensitive data on student behaviors; (2) gaining cooperation of school officials whose schools are not receiving health intervention funds; (3) obtaining informed consent from students and parents; and (4) designing a format to gather data. Gaps in school records regarding health interventions, student demographics, school characteristics, and teachers' evaluation of student behaviors make assessments more difficult.

The study proposes two major alternative designs for evaluating school health interventions: (1) analysis of existing data collected from previous school health program evaluations, including nationally representative samples of students, and (2) large-scale multischool demonstrations using experimental and comparison-group designs and random assignment. The study describes the benefits and limitations of each approach and the most appropriate type of statistical analysis.

Use of Results

This report is expected to provide educators and school health researchers with a blueprint for a well-designed evaluation of health programs and their impact on academic performance. Evaluations of this kind can provide justification for continuing or enhancing school health programs for academic and health reasons.

Publication

In press under the title Evaluating Educational Outcomes of School Health Programs.

Agency sponsor:

U.S. Department of Health and Human Services, Office of Disease Prevention and Health Promotion

Federal contact:

James Harrell
Office of Disease Prevention and Health Promotion
Office of the Assistant Secretary for Health
Switzer Building, Room 2132
330 C Street SW
Washington, DC 20201
(202) 401-6295 Fax: (202) 690-7054

Principal investigator:

Barbara Devaney
Mathematica Policy Research, Inc., Princeton, NJ

Evaluating the Effects of School-Based Intervention Programs To Prevent Teenager Drug Use and Abuse

Highlights

This study provided followup to a large-scale, randomized, controlled field trial of school-based training in substance abuse prevention. It examined the effectiveness of Life Skills Training (LST), an approach that teaches social resistance and other personal and coping skills. The study provides the first evidence that school-based intervention programs conducted by regular classroom teachers can produce durable reductions in tobacco, alcohol, and illicit drug use. These salutary effects were sustained 6 years after the introduction of the program, providing impetus for the widespread adoption of similar programs.

Purpose

This followup study evaluated the long-term effectiveness of a New York school-based substance abuse prevention program started 6 years earlier. It surveyed high school seniors who had been trained in the seventh grade in a substance abuse prevention program called Life Skills Training. This approach teaches social resistance skills and general personal and social skills, that is, ways to improve self-esteem, decisionmaking, and problemsolving. The study also analyzed what types of adolescents were at risk for drug use.

Background

Substance use patterns established in adolescence have serious consequences for the Nation's health. In 1993, 76 percent of high school seniors used alcohol and 31 percent used illicit and/or illegal drugs. In addition, of those seniors surveyed, 29.9 percent had smoked cigarettes within the last 30 days and 10.7 percent had used smokeless tobacco within the last 30 days. Providing students with factual information about drugs and drug abuse was the most common prevention approach up to 1985, but it did not appear to be effective in changing behaviors.

The rationale for this study was to evaluate newer approaches aimed at increasing awareness of the social influences that encourage drug use and improve social resistance, personal, and coping skills. In 1985, the National Institute on Drug Abuse (NIDA) funded a study in 56 schools in New York State to determine the long-term effectiveness of a substance abuse prevention program for students in seventh grade, with booster sessions in the eighth and ninth grades. Earlier followups (when students were in the ninth grade) revealed that LST was effective at reducing the use of gateway drugs: tobacco, alcohol, and marijuana. The use of tobacco, for example, declined by over 50 percent.

The followup study described here examined whether the training continued to be effective in the senior year of high school and whether its effectiveness extended to illicit drugs in addition to marijuana. The 6 years of followup data also enabled researchers to analyze what types of students were more prone to drug use.

The New York State study was also begun to overcome a dearth of information about the effectiveness of substance abuse prevention programs. Little evidence had been accumulated on (1) the durability of program effects more than a year afterward; (2) program effects on multiple drug use and/or use of illicit drugs other than marijuana; or (3) programs using techniques of random assignment, with the school as the unit of assignment and analysis.

Methods

The initial program, offered in 1985, consisted of 12 curriculum units taught in 15 class periods in the seventh grade, with booster classes in the eighth and ninth grades. The intervention was based on an LST curriculum that taught cognitive-behavioral skills for increasing awareness of the causes and effects of substance use, resisting peer and advertising pressure, making decisions, managing anxiety, communicating effectively, asserting one's rights, and developing personal relationships and other psychological skills. The curriculum was randomly assigned to schools under two conditions: (1) teachers received training materials, training, and staff support;
or (2) teachers received only packaged training materials. A third group of schools, the control group, did not receive any substance abuse training.

The original study, conducted by Cornell University Medical College, used a true experimental randomized design and analyzed outcomes as well as measures of curriculum implementation. To ensure comparability of results across different schools, the schools were selected on the basis of similar demographic variables such as gender, age, and ethnicity. The sample was approximately 52 percent male and 91 percent white.

Followup data presented in this part of the study were collected on a cohort of 3,597 participants (60 percent of the original 5,954 students) at the end of the 12th grade. Separate analyses were performed on a restricted sample of 2,752 students from 50 schools who received at least 60 percent of the prevention program to determine the effectiveness of the two preventive interventions when implemented with an acceptable degree of fidelity. Two questionnaires were used: the first was exactly the same as an earlier followup, and the second was a brief, anonymous survey of illicit drug use. Data on illicit drug use were collected anonymously from 454 individuals using a mailed survey.

Findings

The probability of smoking, drinking immoderately, or using marijuana was significantly lower--as much as 40 percent lower--for students who received the prevention program than for those in the control group. The strongest prevention effects were found among students who received at least 60 percent of the intervention and for students whose teachers received the most staff support.

The probability of using multiple drugs was significantly lower--by as much as 60 percent--for weekly use. Long-term prevention effects were found 6 years after initial assessment for tobacco, alcohol, and marijuana use, and after 6 \xba for illicit drug use. Booster sessions were found to contribute to the reduction in substance use.

The students least likely to engage in drug use had better knowledge of drug use prevalence, better skills for refusing drug offers, fewer expectations about the benefits of drug use, and increased life skills and psychological adjustment, among other factors.

Use of Results

The results of this long-term followup provide empirical support for the efficacy of a school-based prevention strategy based on LST in preventing drug use and abuse. Because the study population was primarily white, research is needed to determine the effectiveness of this type of program for inner-city minority students.

To produce durable prevention effects, the study revealed that school-based drug abuse prevention programs must (1) increase knowledge of drug use prevalence rates, and improve skills for refusing drug offers as well as general life skills; (2) decrease expectations and positive attitudes and beliefs about the benefits of drug use; (3) be properly and completely implemented; and (4) include booster sessions.

Publication

Botvin, Gilbert, et al. 1995. Long-term followup results of a randomized drug abuse prevention trial in a white middle-class population. Journal of the American Medical Association, 273(14): 1106-1112.

Agency sponsor:

National Institutes of Health

Federal contact:

Larry A. Seitz, Ph.D.
Division of Epidemiology and Prevention Research
National Institute on Drug Abuse
National Institutes of Health
5600 Fishers Lane, Room 9A-53
Rockville, MD 20857
(301) 443-1514 Fax: (301) 443-2636

Principal investigator:

Gilbert J. Botvin, Ph.D.
Cornell University Medical College, Ithaca, NY

Environmental Health

Indoor Allergens: Assessing and Controlling Adverse Health Effects

Highlights

This study assessed the relationship between indoor allergens and allergic diseases such as asthma and hay fever. It found the indoor environment to have many allergens that cause allergic disease. African Americans and children were found to be at greatest risk of adverse effects. By recommending relatively simple control strategies, the study provided practical guidance to patients and professionals, many of whom have little or no information about the adverse health effects of indoor allergens. It also proposed an agenda for further research.

Purpose

An expert committee of the Institute of Medicine (IOM) at the National Academy of Sciences was charged with providing an independent, comprehensive, state-of-the-science evaluation of the impact of nonindustrial indoor allergens on the development of adverse health effects in humans (e.g., asthma and other allergic conditions). Its report was designed to assess the extent of the problem in the United States, to define the populations commonly affected, and to identify specific indoor airborne agents responsible for inducing asthma and/or other allergic reactions.

Background

More than 50 million Americans--about 20 percent of the population--will develop an allergy-related disease at some point in their lives. Asthma alone afflicts an estimated 10 to 20 million Americans. Asthma is a major reason for trips to the emergency room and for work and school absenteeism. The economic burden of asthma in 1990 was estimated at $6.2 billion, a 39 percent increase since 1985. This information, combined with steady increases in the occurrence and severity of asthma and other allergic diseases, raises concerns about the potential adverse health effects of indoor air and the extent to which the indoor environment can be controlled.

Indoor allergens may be responsible for a large proportion of allergic diseases, including asthma, hay fever, chronic bronchitis, "sick building" syndrome, and allergic skin reactions. The increased incidence, prevalence, and severity of these conditions may be the result of Americans' spending more than 90 percent of the day inside buildings that are increasingly airtight and that frequently contain numerous sources of allergens, including dust mites, fungi, house pets, rodents, cockroaches, and certain chemicals.

On the basis of these concerns, several Federal agencies requested that IOM evaluate the public health significance of indoor allergens. The Committee on the Health Effects of Indoor Allergens was created by the IOM Board on Health Promotion and Disease Prevention, in collaboration with the U.S. Environmental Protection Agency, the National Institutes of Health[1], and the Agency for Toxic Substances and Disease Registry. This multidisciplinary committee included nationally renowned allergists, immunologists, biomedical engineers, epidemiologists, psychologists, physicians, and other health professionals.

Methods

The 16-member committee conducted an extensive review of published information, including population studies, medical case reports, basic research studies, and engineering and architectural documents, as they related to the design and operation of heating, ventilation, and air-conditioning systems. The committee also evaluated the methods used to (1) diagnose asthma and other allergic diseases, (2) measure levels of indoor allergens, and (3) reduce the concentrations of these allergens. Where possible, committee members used quantitative risk assessment methods to identify relationships between specific indoor allergens and allergic reactions.

Findings

The committee's review of research revealed African Americans and children (particularly those under age 5) to be at greatest risk for developing complications associated with exposure to indoor allergens. Children under 18 years old account for nearly 50 percent of all emergency room visits for asthma. Asthma mortality rates for African Americans are two to three times greater than rates for whites.

The committee found that allergens produced by these organisms cause allergic disease: house dust mites, fungi and other microorganisms, cats, dogs, cockroaches, and rodents. Dust mites--microscopic organisms that live primarily in carpeting and upholstery--appear to be critical for the development of allergic asthma in children. Allergens shed by house pets cause hay fever and asthma and can persist for a long time after a pet has left the premises.

The diagnosis of allergic disease is based primarily on a medical history linking exposure to symptoms. Laboratory studies, including skin tests or in vitro tests for allergic antibodies (immunoglobulin E [IgE]) or other classes of antibodies, are important for confirming a diagnosis of allergy. Pulmonary function tests are helpful for following the course of disease. Peak flow meters, because they are portable and can be used by patients, are particularly useful for the diagnosis and management of asthma.

Research suggests steps to markedly reduce exposure to indoor allergens, thus lowering the health risks and medical needs and costs of asthma/allergy patients. Relatively simple measures, such as washing bedding regularly at high temperatures, keeping carpeting to a minimum, and using special air filters, can significantly improve indoor environmental conditions. Well-designed and maintained heating, vacuuming, and air-conditioning equipment will eliminate most indoor allergens.

Panel members found that physician and other health care providers had little or no information about the adverse health effects of indoor allergens. Awareness was similarly low among patients, the general public, and those involved in the design and maintenance of residential and commercial buildings.

Use of Results

The committee's report contained numerous recommendations for patient education and treatment, research, and environmental controls.

One of the recommendations for those diagnosed with asthma is to determine whether they are among the 50 percent of cases whose asthma can be attributed to allergic factors. The committee recommends allergy testing of asthmatics who require more than occasional treatment. If these patients are found to be allergic to one or more indoor allergens, the committee recommends that they be given specific, practical information about how to reduce their exposure to indoor allergens.

Control of indoor allergens also requires engineering control strategies to improve the use of heating, vacuuming, and air-conditioning equipment; to consider the serious problem of carpeting as a reservoir of indoor allergens; and to define moisture control in naturally and mechanically ventilated buildings.

The research agenda developed by the committee represents a variety of approaches to understand and control indoor allergens. Research recommendations range from improvement of testing methods and identification of new allergens to developing more allergen-free products and determining the effects of long-term allergen avoidance on the quality of life.

The report has stimulated the development of a research program announcement, "Environmental Agents and Asthma," prepared by NIEHS in collaboration with NIAID and NHLBI. An announcement of this kind is one vehicle NIH uses to channel research on specific topics of public health importance. Within the Healthy People 2000 Environmental Health Subcommittee, the report has served as a springboard for extensive discussions on effective ways to alter indoor environmental conditions and on additional research needs.

Publication

Pope, A.M., R. Patterson, and E. Burge, eds. Indoor Allergens: Assessing and Controlling Adverse Health Effects. Washington, DC: National Academy Press, 1993.

Agency sponsor:

National Institutes of Health

Federal contact:

Marshall Plaut, M.D.
Asthma and Allergy Branch
Division of Immunology, Allergy, and Transplantation
National Institute of Allergy and Infectious Diseases
National Institutes of Health
Solar Building, Room 4A23
6003 Executive Boulevard
Rockville, MD 20892
(301) 496-8973 Fax: (301) 402-2571

Principal investigator:

Andrew Pope, Ph.D.
National Academy of Sciences, Washington, DC

HIV/AIDS

HIV Service Networks in Four Rural Areas

Highlights

This series of case studies was designed to shed light on how HIV services are organized and delivered in rural areas with high and low AIDS prevalence. For each area, extensive descriptions are provided about the epidemiology of HIV/AIDS, the gaps in services, and the financing of services. Stigma and lack of practitioner knowledge about the treatment and management of HIV infection are found to hamper diagnosis and treatment. Because of the diversity of rural areas, a classification system was designed to help distinguish between rural areas with different prevalence rates and infected populations. The classification scheme can be used in HIV planning and technical assistance. This study provides a portrait of rural HIV network development patterns that can inform public policy.

Purpose

This study was undertaken to help States enhance the quality, availability, and organization of services for rural residents with HIV. It was also intended to provide information to the Federal Government and Congress to use in designing and funding HIV programs in rural areas. The study examined (1) how different rural areas plan, organize, and deliver HIV-related services; (2) what gaps and barriers exist to providing care; and (3) how Title II grant funds can be used more effectively in developing rural HIV-related service networks. Title II of the Ryan White Comprehensive AIDS Resources Emergency (CARE) Act of 1990 authorizes formula grants to the States to provide outpatient medical and support services to people with HIV and AIDS. In many States these grants are the only source of funding for services in lower incidence areas.

Background

Although about 20 percent of Americans reside in rural areas, they account for only about 5 percent of total U.S. AIDS cases. However, the number of new cases in rural areas is growing faster than in urban areas. Little is known about the epidemiology of AIDS in rural areas or how these areas have organized to plan for and deliver HIV-related services. To obtain preliminary answers to these questions, the Bureau of Health Resources Development in the Health Resources and Services Administration (HRSA) funded a study of HIV-related service delivery in four rural areas.

According to the Centers for Disease Control and Prevention (CDC) classification system, AIDS cases in nonmetropolitan areas are classified as "rural." Rural areas exhibit great variation with respect to population density, proximity to urban areas, the prevalence of AIDS, and the populations most affected by HIV. There is great heterogeneity in AIDS prevalence (cases per 100,000 population) in rural areas.

Methods

At regional meetings, representatives of the South Atlantic and Mountain Census Divisions discussed the characteristics and needs of people with HIV/AIDS, the organization and delivery of HIV-related services, and the financing of HIV/AIDS care in rural areas of their States. They also helped develop an AIDS-specific typology for categorizing different rural environments, formulating study questions, obtaining information to design case studies, and developing the discussion guide for site visits. Information from the literature review also was used to develop the discussion guide. During the site visits, information was gathered on community attitudes toward HIV/AIDS, HIV-related service gaps and barriers, service utilization patterns, outreach activities, and funding mechanisms.

Case studies were used to investigate rural areas with dramatically different AIDS prevalence rates. The South Atlantic Census Division was selected for its high AIDS prevalence and high access to HIV-related services. The Mountain Census Division was selected for its low AIDS prevalence and low access to services. In each division, two sites were identified: the Mountain Division was represented by sites in southeast Idaho and southeast New Mexico, and the South Atlantic Division was represented by the Edisto Health District of South Carolina and the Treasure Coast area of Florida.

Each case study included (1) a literature review of the characteristics of rural residents with HIV/AIDS and rural HIV-related service delivery systems; (2) a review of HRSA grantees' applications, HIV service plans, and rural HIV service delivery models; (3) site visits, including interviews of State and local officials, service providers, people with HIV, community leaders, and others; and (4) a review of data from State and local programs, reports, and Census Bureau information.

Findings

The report provided extensive descriptions of the epidemiology of HIV/AIDS, HIV-related service networks, barriers and gaps in service, and funding mechanisms in the four rural settings. To distinguish among rural areas, a typology was developed to classify areas based on degree of rurality, prevalence of AIDS, and the epidemiological and demographic characteristics of the infected populations. The typology can be used to select study sites for epidemiological and health services research on rural AIDS and to analyze variations in the development and organization of HIV service networks. It also identifies key attributes of rural environments that can influence program implementation and the transferability of service delivery models.

HIV testing and counseling were provided in all settings, primarily in health departments. Reporting by name and partner notification varied considerably. Primary care in the low-HIV-prevalence and sparsely populated Mountain Census Division was usually provided by private practice physicians. In the South Atlantic Census Division sites, which had higher rates of HIV infection and a more concentrated population, most HIV-related services were provided through public health clinics and community health centers.

Stigma against HIV/AIDS patients, men who have sex with men, intravenous drug users, and persons with low-income status, as well as lack of practitioner knowledge about the treatment of HIV infection, appeared to limit the number of private practitioners willing to diagnose and treat AIDS patients in rural areas. In Idaho and New Mexico, stigma was compounded by a shortage of primary care physicians.

Gaps in services varied with the type of rural setting. Service gaps found in most rural settings were in primary medical care, dental care, mental health counseling, substance abuse services, professional training on the treatment and management of HIV infection, transportation, and housing. Hospital-based secondary and tertiary care were available at all sites, but accessibility was limited by lack of transportation, a shortage of well-trained staff, and patients' inability to pay. Implementing case management services was difficult in some rural areas because of widely dispersed clientele, lack of experienced providers, and an inadequate reimbursement system for rural case management. In the Treasure Coast area of Florida, where the number of AIDS cases is growing rapidly, the main gap was the limited capacity of the public health system to care for all patients and to intervene at an early stage of infection.

Financing mechanisms also varied. Funds for patient care came from Federal grants, State legislatures, local governments, private insurance, Medicaid, and Medicare. In Florida, most of the State HIV funds were going to urban areas. However, the Ryan White CARE Act, Title II
formula grants benefited rural areas by giving priority to reported AIDS cases in the most recent years rather than using cumulative totals. Some programs, such as the Edisto HIV clinic, were completely dependent on Federal funds. Medicaid was the major financing mechanism for low-income residents. Although the coverage was fairly comprehensive, the criteria for eligibility and benefits varied by State, and many low-income persons did not qualify.

Planning functions, using Title II funds, were assumed by some State governments and, in two areas, by regional HIV care consortia. Regional consortia in the Mountain States had problems recruiting members because of the large distances between communities. The increase in rural AIDS cases suggests that more funding is needed to build a systematic body of knowledge about HIV infection in rural populations and to plan and develop rural HIV-related service delivery systems.

Use of Results

The case studies identified two distinct approaches to providing medical care to rural residents with HIV: (1) an HIV clinic, located in a rural health department or community health center, that develops cooperative working agreements with physicians in private practice, and (2) care provided by primary care physicians who are linked with a medical school or rural-based regional care facility for training and backup consultation. Such regionalized approaches to HIV service delivery make it easier for rural residents to access services and appear to be more cost effective than urban-based outreach programs. The case studies also documented the need to place more emphasis on HIV training and education in rural areas in order to create more positive environments for the development of medical and support services. Targeted Federal funding, through the Ryan White CARE Act and other programs,
can help rural areas plan for the development of HIV services and experiment with new ways of delivering HIV care.

The study's findings and recommendations are being used to design a national study of HIV care in rural areas that will be funded by HRSA through an interagency agreement with the Agency for Health Care Policy and Research (AHCPR). The rural study will be integrated into the design of a larger HIV Cost and Services Utilization Study that AHCPR is funding through a cooperative agreement with the RAND Corporation. A probability sample of 500 rural residents with HIV will be interviewed at two time points about their use of health and support services, barriers to care, unmet service needs, and satisfaction with service quality. Case study findings also are being used to formulate HRSA policies on rural HIV care and to provide technical assistance to States with large rural populations.

Publication

Not yet published.

Agency sponsor:

Bureau of Health Resources Development, Health Resources and Services Administration

Federal contact:

Katherine Marconi, Ph.D.
Office of Science and Epidemiology
Bureau of Health Resources Development
Health Resources and Services Administration
Room 7A07 Parklawn Building
5600 Fishers Lane
Rockville, MD 20857
(301) 443-6560 Fax: (301) 443-2511

Principal investigator:

David E. Berry, Dr.P.H.
University of Nevada, Las Vegas, NV

Outreach/Risk Reduction Strategies for Changing HIV-Related Risk Behaviors Among Injection Drug Users

Highlights

This study assesses the effectiveness of innovative community-based outreach intervention/prevention programs for reducing HIV-related risk behaviors of injection drug users (IDUs) and their sexual partners. Under the sponsorship of the National AIDS Demonstration Research (NADR) Project, the outreach programs offer information about HIV infection, strategies for reducing the risk, counseling, support groups, and training in cognitive and social skills. Significant increases are found in the use of new rather than reused needles, bleach for cleaning injection equipment, and condoms--all behaviors that reduce HIV risk. Study findings are so noteworthy that a permanent program has been created.

Purpose

This study sought to assess the effectiveness of community-based outreach programs to curtail high-risk behaviors leading to HIV infection. The outreach programs were designed and sponsored by the National AIDS Demonstration Research Project, a demonstration program striving to reduce the risk of HIV infection among intravenous drug users and their sexual partners.

Background

As of 1995, about 0.4 percent of the U.S. population are estimated to be infected with HIV. Innovative strategies are needed to reach elusive populations and motivate them to change their high-risk behaviors. Educational campaigns and the mass media have increased awareness about the dangers of HIV transmission through shared injection equipment. While some IDUs have changed behaviors, many continue to share their injection equipment and engage in high-risk sexual behaviors, which places them, their sexual partners, and their infants at risk. Knowledge of effective interventions was so limited that many public officials doubted that IDUs would participate in AIDS prevention programs and were reluctant to spend public funds to reach them.

In 1987, the Community Research Branch of the National Institute on Drug Abuse (NIDA) initiated the NADR Project as a demonstration program. The program was restricted to IDUs who were not currently in drug treatment programs and to their sexual partners. Its purpose was to design and implement innovative outreach intervention/prevention strategies and to determine program effectiveness through a built-in evaluation component.

The NADR Project was discontinued in 1992, but it was a model for the inauguration of a formal program of cooperative agreements to communities to help them curb risky behaviors of IDUs that lead to HIV infection.

Methods

Interventions at each of the 28 sites studied by NOVA Research Company varied somewhat but shared key elements. Community outreach workers, often former IDUs, contacted and recruited IDUs and their sexual partners to participate. Participants were offered voluntary HIV testing and received counseling and education that was culturally relevant. The counseling was delivered and reinforced by a credible source who also provided skills training and/or material resources.

The program collected and analyzed data on 13,475 IDUs and 1,637 sexual partners of IDUs. Investigators used a before/after design, collecting information at baseline and comparing it to that obtained at 6-month followup. Standardized baseline and followup assessment instruments were used to collect comparable data across all sites. Clients were assigned--either randomly or through systematic sampling--to a standard or enhanced intervention at most sites. The standard intervention generally included education and referral for needed community services. The enhanced intervention included a variety of more intensive services, such as couples counseling, development of peer networks, and social skills training.

Three summary risk scales were developed for use as outcome measures: needle risk behavior, sex risk behaviors, and total frequency of drug injection. Specific behaviors such as cleaning needles with bleach or using condoms were used for assessing the impact of the intervention.

Analyses focused on the participants' demographic characteristics, their behaviors and knowledge before and after interventions, the relationship of the type and intensity of intervention modalities to HIV-related risk behaviors, and other risk factors related to the level of risk. Analysis indicated that participants reliably reported demographic factors, injection drug use, and use of alcohol and marijuana. The reliability of reported needle use and sexual behaviors was more difficult to measure.

Findings

IDUs reported significant decreases in HIV-related risk behaviors after interventions. Significant increases were observed in the use of new needles, bleach for cleaning injection equipment, and condoms. Changes in needle-related behaviors of IDUs were more dramatic than changes in sexual behaviors.

The enhanced interventions were slightly more effective than standard interventions in reducing risk. However, this difference was not uniform across all sites and was related to the duration, format, and content of the enhanced intervention at each site. Specific components that were important to reducing risk were providing condoms and/or bleach and demonstrating the proper use of both. Around 33 percent of IDUs entered drug treatment during the followup period, thus reducing their risk for HIV and drug-related illnesses. At followup, subjects who were long-time IDUs, Hispanic, previously in drug treatment, unemployed, and heroin injectors reported greater use of injected drugs than did other IDUs.

The analysis at 17 sites revealed increased use of condoms, reduced number of sex partners, and reduced use of noninjected drugs. Average sex risk scale scores were not significantly different between the standard and enhanced interventions.

While both IDUs and their sexual partners began with high levels of AIDS-related knowledge, the project further enhanced this knowledge. AIDS knowledge levels were significantly higher among white than among African-American or Hispanic IDUs and sex partners. Females scored higher than males on sex risk knowledge, general risk knowledge, and total knowledge. No significant differences were revealed in knowledge scale scores between subjects in the standard and enhanced interventions.

Use of Results

This evaluation provides evidence that IDUs and their sexual partners can successfully take advantage of education and counseling to reduce the risk of becoming infected with HIV. The interventions were found to be so effective that they have been incorporated into formal programs administered by NIDA.

The study contained numerous suggestions for further research. Since the interventions varied across sites and had different impacts on subgroups, greater use of process measures was recommended for future research to document intervention exposure and to relate exposure to changes in behavior. Further investigation is needed into the efficacy of specific components dealing with bleach and condoms over more costly enhanced interventions. Methods to improve response rates at followup need further development. Long-term analysis of the data is needed to assess lasting impact on risk reduction.

Publication

NIDA, the NIDA/NADR research consortium, and individual project sites are disseminating the results through articles in refereed journals, training manuals and videotapes, and national database tapes for secondary data analysis.

Agency sponsor:

National Institutes of Health

Federal contact:

Richard Needle, Ph.D., M.P.H.
Community Research Branch
Division of Epidemiology and Prevention Research
National Institute on Drug Abuse
National Institutes of Health
5600 Fishers Lane, Room 9A-42
Rockville, MD 20857
(301) 443-6720 Fax: (301) 443-2636

Principal investigator:

NOVA Research Company, Bethesda, MD

Immunization

Adverse Events Associated With Childhood Vaccines: Evidence Bearing on Causality

Highlights

An expert panel was convened to evaluate whether certain childhood vaccines cause adverse events. Vaccines against tetanus, diphtheria, measles, mumps, polio, Haemophilus influenzae type b (Hib), and hepatitis B were reviewed. By weighing the available evidence, researchers determined which adverse events can be caused by each vaccine, which cannot, and which are undetermined. The findings are being incorporated into information given to all parents and into proposals to compensate victims.

Purpose

An expert assessment was sought of the scientific and medical literature on the health risks to children from the administration of vaccines or vaccine components against tetanus, diphtheria, measles, mumps, polio, Haemophilus influenzae type b, and hepatitis B.

Background Concerns about the safety and potential risks of vaccination have been raised by the media, the public, and the medical community. In response to these concerns, Congress passed the National Childhood Vaccine Injury Compensation Act of 1986 (P.L. 99-660). One component of the law requires the Department of Health and Human Services, with the help of the Institute of Medicine (IOM), to investigate the adverse effects of childhood vaccines. This mandate led to the development and publication of two evaluation reports, one of which was completed earlier and focused on adverse effects from pertussis and rubella vaccines. The second report focuses on adverse events to the following childhood vaccines: tetanus, diphtheria, measles, mumps, polio, Hib, and hepatitis B. The report was prepared by the Vaccine Safety Committee constituted by the IOM. The report did not examine risk-benefit comparisons between vaccines nor did it make recommendations about immunization schedules, vaccine purity, or vaccine production techniques.

Methods During an 18-month period, the multidisciplinary 14-member committee reviewed more than 7,000 abstracts, more than 2,000 published books and articles, and public testimony to determine whether the available scientific and medical evidence supported or rejected a causal relationship between specific vaccines and adverse health effects in children. Members of the expert panel included immunologists, pediatricians, internists, neurologists, virologists, microbiologists, epidemiologists, and public health specialists.

The reviewers used both qualitative and quantitative statistical methods in their evaluation of four types of evidence: (1) biological data; (2) individual case reports, series of case reports, and uncontrolled observational (population) studies; (3) controlled observational studies; and (4) controlled clinical trials. The adverse health effects evaluated included neurologic disorders, immunologic reactions, and death. Information from all sources of evidence was weighed and results were categorized as follows: no bearing on a causal relation, inadequate to accept or reject a causal relation, favors rejection of a causal relation, favors acceptance of a causal relation, or establishes a causal relation.

Findings The committee studied a total of 49 conditions in which childhood vaccines were suspected of leading to adverse health events; most incidents were reported in uncontrolled studies and case reports. Of the 49 cases, the evidence for 12 cases favored or established a causal relation; the evidence for 4 cases rejected a causal relation; and the evidence for the remaining 33 cases was equivocal. The cause-and-effect relationships identified by the committee are detailed in figure 2.

Figure 2

Summary of the Potential Adverse Health Effects of Certain Childhood Vaccines [a]

Evidence establishes a causal relationship between--

• Diphtheria and tetanus toxoids and anaphylaxis (severe allergic reaction)
• Hepatitis B vaccine and anaphylaxis
• Measles vaccine and death from infection by the measles virus in the vaccine
• Measles-mumps-rubella vaccine and thrombocytopenia (a blood disorder) and anaphylaxis
• Oral polio vaccine and polio; oral polio vaccine and death from infection by the polio virus in the vaccine

Evidence favors a causal relationship between--

• Diphtheria and tetanus toxoids and various neurologic conditions, including Guillain-Barré syndrome
• Measles vaccine and anaphylaxis
• Oral polio vaccine and Guillain-Barré syndrome
• Unconjugated polysaccharide polyribosylribitol phosphate (PRP) Hib vaccine [b] and early-onset Hib disease in children 18 months or older

Evidence fails to support a causal relationship between--

• Diphtheria and tetanus toxoids and encephalopathy (abnormal brain development), infantile spasms, and death from sudden infant death syndrome (SIDS)
• Conjugate Hib vaccines and early-onset Hib disease

a. Based on evidence from 49 suspected cases.
b. Hib, Haemophilus influenzae type b.

The committee drew two general conclusions from these data: (1) available data were often insufficient to establish or reject causality, and (2) despite the limited data, the risk of dying or developing serious neurologic or immunologic conditions because of childhood vaccination appears to be very low.

The committee recommended active surveillance systems to follow representative samples of new vaccine recipients. It also recommended the technique of meta-analysis to combine the statistical power of future clinical and epidemiological studies of vaccines licensed or under development.

Use of Results The findings of this report have already been incorporated into proposals to revise the list of injuries presumed to have been caused by selected vaccines. This list, called the Vaccine Injury Table, identifies conditions for which automatic compensation is provided to victims and families under the National Childhood Vaccine Injury Compensation Act. The findings have already been used by the Centers for Disease Control to revise the information that practitioners are required to give parents before vaccination. Finally, changes are expected in the Food and Drug Administration's regulation of vaccine labeling.

The report also stimulated the creation of a new forum, the Vaccine Safety Forum, convened by the IOM at the request of the National Vaccine Program Office. The forum members--parents, manufacturers, government officials, and vaccine experts--are meeting periodically to discuss vaccine safety.

The report identified several research needs in assessing and quantifying the potential health risks of childhood vaccination, including (1) increased surveillance of reports of arthritis and neurologic disorders, such as demyelinating disease, following hepatitis B vaccination; (2) improved followup on reports of death and serious adverse events that may be associated with vaccination; (3) increased use of large databases to supplement other reporting systems; and (4) establishment of disease registries for rare medical conditions that may be triggered or exacerbated by vaccination.

Publication Stratton, K.R., C.J. Howe, and R.B. Johnston, Jr., eds. Adverse Effects Associated With Childhood Vaccines: Evidence Bearing on Causality. Washington, DC: National Academy Press, 1994.

Agency sponsor: National Institutes of Health

Federal contact: Regina Rabinovich, M.D.
Division of Microbiology and Infectious Diseases
National Institute of Allergy and Infectious Diseases
National Institutes of Health
Solar Building, Room 3A07
6003 Executive Boulevard
Rockville, MD 20892
(301) 496-5893 Fax: (301) 480-4528

Committee chair: Richard B. Johnson, M.D.
The March of Dimes Birth Defects Foundation, White Plains, NY

A Study of the Economic Underpinning of Vaccine Supply

Highlights

This study addressed issues related to the economics of production and distribution of vaccines including models of vaccine purchases, effects of scientific advances on vaccine development and distribution, and the merits of alternative distribution systems. It found that States that supply vaccines at low prices to physician offices and encourage parents to have their children vaccinated have somewhat higher rates of immunization. These findings have been applied by the Centers for Disease Control and Prevention (CDC) in negotiating lower prices from manufacturers and have broad utility for policy and legislation.

Purpose Healthy People 2000 targets having more than 90 percent of the Nation's children vaccinated for all recommended diseases by their second birthday and eliminating domestic cases of measles, polio, and rubella. Meeting these targets requires a continuous supply of safe and effective vaccines at stable prices.

In response to concerns about the safety and stability of the U.S. vaccine supply, the National Vaccine Program Office and the CDC investigated the vaccine environment in the United States, focusing on the economics of vaccine supply and purchase and its implications for public health policy.

Background Vaccinating children against infectious diseases is one of the most effective means of preventing disease. Vaccinations for nine diseases are recommended for healthy children. While the United States does an excellent job of vaccinating school-age children, only about 37 to 56 percent of 2-year-olds are fully immunized against preventable childhood diseases, well short of the goal of at least 90 percent. Currently, only two domestic firms and two State laboratories manufacture and sell vaccines for nine childhood diseases. Three of the recommended childhood vaccines--OPV, HBV, and MMR[2]--are each produced by only one domestic firm. The shelf life of a vaccine is short (10 to 18 months), and vaccine production time can range from 7 to 13 months. As a result, the loss of even one manufacturer can result in a vaccine shortage. When shortages occur, vaccine prices escalate.

Methods This study consisted of several policy papers that analyze the economic and commercial aspects of ensuring an adequate supply and distribution of vaccines. Specifically, the study investigated (1) alternative models for the purchase and distribution of vaccines; (2) the effect of scientific advances on research, development, and purchase of vaccines; (3) the application of economic theory to the vaccine market; (4) comparisons of vaccine distribution systems administered by vaccine manufacturers and State agencies; and (5) the implications of purchasing vaccines from foreign firms. The papers developed for the project were presented at the Annual Meeting of the American Academy of Pediatrics held in Washington, D.C., in November 1993.

Findings In 1992, all doses of DTP[3], oral polio, and MMR needed to vaccinate a 2-year-old cost about $95 in the private sector, a 12-fold increase since 1977. Vaccine suppliers attribute the price increases to increases in their costs, but others believe the strong market power of U.S. vaccine manufacturers is at least partly responsible.

In 1986, Congress passed the National Childhood Vaccine Injury Compensation Act, which provided no-fault compensation to persons injured by DTP, MMR, or oral polio vaccines, with the compensation paid from excise taxes collected on the covered vaccines. The year this legislation was enacted, the price per dose more than doubled. In 1992, for example, the DTP excise tax of $4.56 accounted for nearly one-half of the total price per dose.

Sizable fixed costs associated with entry in the vaccine production industry were found to arise in two areas: technological and regulatory. Costs per unit are higher when vaccines are produced in small batches and are minimized if all vaccine is produced at one facility. Beyond some limits imposed by the batch production nature of vaccine technology, it can be costly or nearly impossible to make more vaccine available. Restrictions on entry into the market (in terms of required size and time of entry) may permit existing firms to earn above normal returns. However, production by a single firm raises issues of economic efficiency and market stability.

Recently, there has been substantial and renewed interest in the development of new vaccines or techniques to manufacture substitutes for existing vaccines. This change is the result of advances in biotechnology and higher prices and presumably higher profits involved in the production of current vaccines. Alternative but substitutable products may introduce a modest amount of competition. If the alternative products turn out to be medically superior, the vaccine industry would likely return to a configuration of one or two firms producing virtually all of the output.

In 1983, to meet temporary interruptions in vaccine supply, CDC received funds from Congress for a vaccine stockpiling program. The stockpile includes up to a 6-month rotating inventory of the recommended childhood vaccines. CDC has used the stockpile at least five times since 1983. Congress did not reappropriate funds for the stockpile in 1992 or 1993. The stockpiling program provides relatively inexpensive insurance against supply interruptions but may require more stable funding.

Although there may be a role for public ownership of a standby bottling and filling facility, public financing of a more elaborate standby production facility appears inadvisable. Such a facility would be inflexible and costly and would inevitably have to compete directly with private manufacturers of vaccine.

About one-half the childhood vaccine supply in the United States is purchased by the private sector, one-fourth by the Federal Government, and one-fourth by State governments. The contract prices negotiated by CDC, the largest single purchaser, are about one-half the price paid by physicians who purchase vaccines directly from the manufacturers. CDC could exploit its growing market power to obtain better prices for vaccines. Bulk purchases could also lower the costs of establishing a domestic distribution network.

Private market supply to the government is likely to work more efficiently and with greater assurance when the government's acquisition behavior is predictable and trustworthy. Another alternative is to have the government as the sole source of vaccine, with the entire supply purchased by the government under the usual (or some modification of the usual) government competitive bidding process.

The role of foreign suppliers in meeting either the supply assurance or pricing goals in the near term is likely to be limited. Thus far, entry into the U.S. market by established foreign vaccine producers holding U.S. licenses has had little discernible effect on prices, although the poor quality of available data makes this conclusion highly provisional. Moreover, the potential role of foreign suppliers in meeting domestic interruptions in U.S. vaccine supplies is severely constrained by licensure considerations. These considerations are also likely to make emergency procurement of vaccines from international public health agencies or their vaccine suppliers infeasible.

States that supply vaccine at low prices to physician offices, whether from their own production or by bulk purchase, appear to achieve somewhat higher rates of immunization. Low acquisition cost for the private physician is not the only aid to high rates of vaccination; efficient and convenient distribution, as well as State activities to encourage parents to have their children vaccinated, are also effective.

Use of Results The study has provided the Public Health Service (PHS) with fundamental knowledge about vaccine pricing over the past 20 years. This knowledge has placed PHS in a better position to negotiate contracts with vaccine manufacturers, a role that has been enhanced as a result of 1993 legislation giving CDC the authority to negotiate prices for States participating in the CDC State Immunization Grant and Vaccines for Children programs. These two programs were designed to eliminate price as a barrier to immunization for selected groups of usually low-income children.

As a result of this evaluation, the National Vaccine Program Office has commissioned further studies to examine particular vaccine manufacturers and the costs incurred in vaccine development and production. Other recommendations concerning bidding practices and bulk purchase are also under consideration.

Publication The National Vaccine Program Office and CDC plan to publish the manuscript in 1995 under the working title "A Study of the Economic Underpinning of Vaccine Supply."

Agency sponsor: National Vaccine Program Office

Federal contact: Stephen Sepe
National Vaccine Program Office
Centers for Disease Control and Prevention
1600 Clifton Road NE, MS D-24
Atlanta, GA 30333
(404) 639-4450 Fax: (404) 639-6036

Principal investigator: Craig Thornton, Ph.D.
Mathematica Policy Research, Inc., Princeton, NJ

Public Health Infrastructure

Advanced Practice Nursing Education: Strategies for the Allocation of the Proposed Graduate Nursing Education Account

Highlights

Options were explored for paying for the education of advanced practice nurses (APNs) to increase their supply. Through policy analysis and original cost estimates, the study proposed four options for structuring and allocating a proposed account, put forward by the Administration's proposed Health Security Act, that would have established Federal support for graduate nurse training programs. Despite the failure of Congress to act on health care reform, the need for additional advanced practice nurses continues to grow, and this study has helped clarify the problem.

Purpose This study was undertaken to inform policy-makers about the growing need to finance the education of advanced practice nurses. In light of expected shortages in the supply of APNs, the study created and analyzed four options and their estimated costs for ensuring a stable source of funds.

Background APNs are trained at the master's level, while basic nurse training typically consists of 2 to 4 years of undergraduate education. Many States permit APNs to provide primary care and to refer patients to more specialized care, if necessary. Compared with physicians, APNs are less costly to educate, receive lower salaries, and more frequently practice in medically underserved areas. These features make APNs an attractive investment to policymakers interested in expanding access to primary care services.

There are four types of APNs: nurse practitioners, who are generally trained in primary care and have the authority in some States to write prescriptions; certified nurse-midwives, who deliver babies and provide primary care to women; certified registered nurse anesthetists; and clinical nurse specialists, who provide mostly hospital-based care in almost all medical specialties.

Shortages are projected in the supply of APNs, who, along with primary care physicians, are expected to be in greater demand because of the growth of managed care. The Administration's Health Security Act and other legislative proposals introduced in Congress in 1994 recognized the importance of providing funds for training nurse practitioners, nurse-midwives, and other types of APNs. This evaluation was undertaken specifically to assist in the implementation of a special account proposed by the Health Security Act to finance the education of APNs. The funds would have been drawn from a proposed tax on health care premiums and the account was devised as a capped entitlement. Bipartisan interest remains in cultivating the growth of APNs, despite the failure of global health care reform measures in the previous Congress.

Methods This study by the University of Pennsylvania provided an analysis of policy options and developed estimates for each. The policy analysis, compiled from published sources, contained data on APN student characteristics, supply, overall education costs, and financing. The analysis contrasted APN students with medical residents, since the proposed Health Security Act modeled the funding for APN education after that for graduate medical education. The impact of previous legislation was also documented. The study provided four alternative options
for structuring and distributing the proposed account. Each option contained a cost estimate based on published data on tuition, fees, stipends, and program costs.

Findings The available evidence suggests a scarcity in the supply of APNs under health care reform scenarios. For example, the University of Pennsylvania Division of Nursing estimated an additional 31,000 nurse practitioners would be needed by 2000, an increase of about 60 percent beyond the current supply. In 1986, a study by the Congressional Budget Office estimated that there were an estimated four job openings for every graduating certified nurse-midwife and nurse practitioner. The study documented that about 75 percent of APN students are enrolled part-time and pointed out that full-time status would hasten the flow of graduates.

The study highlighted important differences in the structure of educational programs for APNs and medical residents, which complicates the development of parallel financing programs. The most profound difference is that residents receive a stipend for their services and are not required to pay tuition. Students preparing to become APNs are charged tuition (averaging $4,316 per year, but ranging from $180 to $42,000). Their stipends are modest and vary from $10,000 to $12,000 per year. Unlike medical residents, APN students commonly enroll part-time and depend on some tuition reimbursement from their employers.

Past health professions training legislation has had a measurable impact on recruitment. After Federal support was initiated in the 1960s through Title VIII of the Public Health Service Act, the number of master's-level nursing students increased. When the funds began to decline in the 1970s, an increasing number of students enrolled part-time. The number of part-time students rose from about 2,000 in 1972 to 20,000 in 1992. However, the number of full-time students remained stable during this period at about 5,000.

The study examined four options for allocating the funds to targeted subsets of students. The four options, which students they cover, what charges they cover, and what they cost are presented in table 2. Advantages and disadvantages of the options are discussed in the study.

Table 2

Four Options for Allocating Funds in a Proposed Account Financing the Training of Advanced Practice Nurses

---

Number of Option Students Total Cost Option Pays

---

Full-time students only 7,204 $208,303,040 Tuition vs. Fees Stipend All full-time equivalents a 12,507 $348,294,936 Program costs b

---

Average tuition for full-time students 7,204 $208,303,040 Tuition vs. Fees Stipend Actual tuition for full-time students 7,204 $202,383,924 Program costs

---

Students in clinical programs only 5,381 $150,880,516 Tuition vs. Fees Stipend All full-time students c 7,204 $202,383,924 Program costs

---

Full-time students in 1-year programs 7,204 $190,391,300 Clinical stipend vs. Full-time students in 2-year programs 7,204 $236,512,800 Clinical faculty salary support

---

a. Full-time equivalents are estimated by the following formula: full-time students + .38 (part-time students). b. Program costs are used to pay college faculty salaries and indirect medical expenditures for hospitals. c. Students in clinical, administrative, and teaching programs.

---

The first option--restricting the fund to full-time students--had the advantages of administrative ease and producing graduates at a faster pace, especially those in nurse practitioner and nurse-midwife programs. These APN students are more likely than the other types to be enrolled full-time. The disadvantage was that it did not help those enrolled part-time, who constitute the majority of students. The second option was to offer students reimbursement based on average, rather than actual, tuition. While this option is increasingly favored by policymakers because it averages a wide range of costs (from $200 to $1,600 per year), it would have penalized so many private institutions that the negligible difference in total costs made this option unattractive. A third option estimated the cost of covering the two-thirds of APN students entering clinical practice, rather than teaching and administration. This option was found to have many disadvantages, including significant disruptions in how nursing programs determine program costs. The final option--limiting the account to the clinical portion of training through support strictly for clinical stipends and for clinical faculty salaries--was considered least attractive.

Use of Results The study results provide critical information to policymakers as they continue to consider options for ensuring the supply of advanced practice nurses as alternative primary caregivers.

Publication Aiken, L.H., M.E. Gwyther, and E.M. Whelan. "Federal Support of Graduate Nursing Education: Rationale and Policy Options." Health Affairs.

Agency sponsor: Department of Health and Human Services

Federal contact: Marcy Gross
Senior Associate
Office of the Assistant Secretary for Health
Department of Health and Human Services
Room 740G, Hubert H. Humphrey Building
200 Independence Avenue SW
Washington, DC 20201
(202) 690-5824 Fax: (202) 690-5432

Principal investigator: Linda Aiken, Ph.D, R.N., F.A.N.
University of Pennsylvania School of Nursing, Philadelphia, PA

Evaluation of Centers for Disease Control and Prevention (CDC) and Agency for Toxic Substances and Disease Registry (ATSDR) Training Activities

Highlights

Local health departments are facing fiscal upheavals. This study assessed their current and emerging training needs through focus groups and site visits. It also sought to determine whether these needs match the training activities offered by CDC and ATSDR. Local health programs were found to need training in advocacy, evaluation, and public information. CDC, which has limited training resources, was found to have no periodic mechanisms for staying in touch with State and local training needs. Recommendations, including those for new training technology, that is, interactive software and satellite communications, have prompted CDC and ATSDR to develop training activities that dovetail with local needs.

Purpose In light of the dramatic changes occurring throughout the public health system, this evaluation had two goals: to determine the current and emerging needs of local health departments and to determine whether CDC and ATSDR training activities not only meet these needs but also anticipate future needs. Training produced and delivered by CDC[4], is intended to improve the performance of the public health workforce at the State and local level. Training activities provided through workshops, conferences, regional training centers, and other vehicles cover a range of scientific, technical, and managerial topics.

Background The public health system, especially at the local level, is endangered by shrinking budgets and shortsightedness about the value of public health, according to a widely respected 1988 report, The Future of Public Health, by the Institute of Medicine. The three major functions of public health--assurance, assessment, and policy development--are undergoing change. While most resources are currently devoted to delivering services--assurance activities--more emphasis is expected to be needed on assessment (i.e., investigation and surveillance of health hazards) and policy development.

Reforms are expected to induce shifts in the role of public health agencies and professionals at all levels. Public health professionals represent a variety of academic disciplines and backgrounds. They are often called on to use not only expertise in disease prevention, disability, and injury, but also broader scientific, technical, and managerial skills. In the future, local public health departments are likely to require a wider range and different emphasis of skills among workers. As key providers of training for public health departments, CDC needs current information on training needs, target audiences, topical coverage, and the adequacy and appropriateness of existing training methods.

Methods Macro Systems International conducted site visits at nine selected local health departments and, using focus groups with health department staff, identified training needs. A uniform discussion guide was used at each site to uncover training needs, training activities, and approaches that might enhance training.

Interviews were also conducted with principal management officials at each of CDC's Centers, Institutes, and Offices (CIOs). In addition to conducting interviews, researchers developed an updated training inventory and incorporated it into an on-line electronic training catalog. At the time of the study, this catalog was available to CDC staff and a limited number of public health departments.

Qualitative and quantitative analyses were performed on the data from all focus groups and interviews.

Findings Training plays an important role in the mission of all CIOs, yet the nature and content of training varied widely. In general, training appropriately emphasized assessment, but its additional emphasis on assurance contrasts with the expectation that public health is moving away from this activity.

Local health departments were found to be deficient at accurately assessing their training needs. When needs assessments were done, they tended to capture information only on current employees, not potential employees.

CDC's training in advocacy, evaluation, and public information--the areas of need identified by local health departments--tended to be irregularly offered. Advocacy is in the realm of policy development and involves building constituencies and identifying resources in the community. Training needs also exist in such topical areas as mental health, substance abuse, cancer, cardiovascular disease, food-borne illness, and lead poisoning.

CDC has limited resources to meet training needs in spite of overwhelming demands. In general, CIOs favor standup training but are unfamiliar with other potential media and approaches, such as self-study curricula, video conferences, interactive software, and satellite downlinks. Despite a preference for standup training, the majority of CIOs have taken advantage of the benefits of distance learning by offering courses through the Public Health Training Network.

Because CDC training activities rarely reach the local level, local health departments are often unaware of training opportunities. Local health departments are aware of "The Future of Public Health" framework and endorse it as a strategy. They recognize that their future lies in expanded assessment activities and policy development and that their assurance capabilities are being downsized.

The study also found a lack of periodic mechanisms for CDC to stay abreast of the training needs of local health departments.

Use of Results The study and its recommendations updated CDC's on-line training catalog and stimulated its nationwide availability. CDC also is hastening the development of a computer-based job aid that will enable CDC to be aware of the training needs of local health departments.

Other recommendations called for more resources for training activities, particularly in policy development and public information. Efficiencies could be achieved by integrating the interests of several CIOs and ensuring that all health departments have access to information about training. Curricular content and evaluation techniques need updating and improvement. More efforts should be made to assess needs and explore alternative media and advanced technology.

Publication Not yet published. Final report is available through the HHS Policy Information Center in the Office of the Assistant Secretary for Planning and Evaluation.

Agency sponsor: Centers for Disease Control and Prevention

Federal contact: Dennis McDowell
Public Health Practice Program Office
Centers for Disease Control and Prevention
1600 Clifton Road NE, MS E-20
Atlanta, GA 30333
(404) 639-3707 Fax: (404) 639-3982

Principal investigator: Thomas Chapel
Macro Systems International, Atlanta, GA

Evaluation of the Morbidity and Mortality Weekly Report Series

Highlights

Readers from diverse professional backgrounds were found to value the Morbidity and Mortality Weekly Report (MMWR) for its accuracy, relevance, and concise reporting format. The number of readers is likely double the 450,000 on mailing lists. Recommendations may result in some refinements to content and format.

Purpose The Morbidity and Mortality Weekly Report, published every week by CDC, is the agency's major vehicle for communicating timely public health information. As the mission of CDC has expanded beyond concern for infectious diseases to chronic disease and environmental health, this study sought to evaluate the utility of the MMWR in meeting the needs of all of its public health constituencies.

Background The MMWR has been published weekly since 1961 as an extension of CDC's surveillance activity. More recently, the MMWR has been the first source of information on such public health issues as outbreaks and etiology of AIDS, legionella, and hantavirus, as well as studies of toxic shock syndrome. The MMWR's goal is to document events of public health interest, to disseminate information to those who need it, and to communicate with CDC constituents. MMWR articles have the potential not only of informing, but also of directly helping to prevent disease.

In recent years, the scope of the MMWR has broadened--as the mission of CDC has expanded--to include chronic diseases, environmental and occupational health, and injuries. Having dedicated several staff positions to the publication and recognizing its high visibility, CDC was eager to understand the evolving readership needs of the MMWR.

The MMWR is distributed to more than 450,000 people. Primary distribution (17,000) is through the CDC mailing list; distribution also is through reprinting in the Journal of the American Medical Association (JAMA) and subscriptions to the Massachusetts Medical Society.

Methods Battelle conducted a mail/telephone readership survey with 375 individuals on the MMWR mailing list, 375 subscribers to the MMWR through the Massachusetts Medical Society, and 250 primary care physicians who subscribe to JAMA, which publishes MMWR excerpts in a section called "Notes from CDC: Leads from the MMWR." The survey sought to identify reading habits, readership characteristics, use of MMWR information, feedback on publication content and format, and interest in electronic access.

Findings The MMWR is read with great regularity: two-thirds of respondents had read three or more MMWR issues in the months prior to the survey, and most readers had read the MMWR for 5 years or more. Because the MMWR is often distributed to institutions, it is frequently passed along to additional readers. The actual readership is estimated at more than twice the number of issues distributed. The professional backgrounds of MMWR readers are diverse, and the two dissemination modes evaluated (the CDC mailing list and those who subscribe through the Massachusetts Medical Society) complement each other in terms of professional distribution.

Primary care providers most frequently use information in the MMWR to update their information about diagnosis, treatment, and disease outbreaks. The MMWR is also widely used in teaching, public education, planning, and advocacy. The rate of MMWR citations in the scientific research literature is rising. The number of citations of MMWR articles in scientific journals increased more than threefold from 1981 to 1983 and from 1987 to 1989.

Readers value the content of the MMWR and cite its accuracy, relevance, and concise reporting format, but they have suggestions for changes. Some readers would like a larger page size and others are interested in greater electronic access to the MMWR. Currently, they can access the MMWR if they are connected to the Public Health Network. A majority of the respondents, however, were not equipped with the necessary computer hardware and software at the time of the survey.

The MMWR is generally achieving its major objectives to provide for the information needs of CDC's constituents: it is publishing news of public health importance; distributing time--critical health information to those who need to know; publishing quickly and accurately; assisting and strengthening public health surveillance; and promoting efforts of State and local health agencies.

Use of Results The study recommended that the MMWR maintain the breadth of its coverage, possibly enlarge the format, increase outreach efforts, and improve electronic access.

CDC found this study highly useful in shaping the future of the MMWR. It plans to form an internal ad hoc group, consisting of CDC program managers and independent public health officials, that will make recommendations about the MMWR's content and format.

Publication Not yet published. Final report is available through the CDC Policy Information Center.

Agency sponsor: Centers for Disease Control and Prevention

Federal contact: Richard A. Goodman, M.D., M.P.H.
Epidemiology Program Office
Centers for Disease Control and Prevention
Building 1, Room 5104
1600 Clifton Road NE
Atlanta, GA 30333
(404) 639-3636 Fax: (404) 639-3950

Principal investigator: James Hersey
Battelle, Arlington, VA

Chapter III. PHS Agency Evaluation Activities

PHS Agency Evaluation Activities

The eight Agencies of the Public Health Service (PHS) and the Office of the Assistant Secretary for Health (OASH) maintain their own evaluation programs--including the functions of evaluation planning and policy review; quality assurance through technical review; project coordination and management; dissemination of reports; and utilization of results. This chapter provides an agency-by-agency summary of the evaluation activities. Each section begins with an overview of the Agency's evaluation program, describing its philosophy, policies, and procedures. Next are summaries of the major evaluations completed in fiscal year (FY) 1994 and the evaluations currently in progress. Finally, there is a discussion of new directions for the Agency's evaluations, including priorities for future program and policy evaluation projects. Abstracts and contact persons for all evaluations completed in FY 1994 are presented at appendix A, organized by Agency. Appendix B lists all evaluations in progress, by Agency. Review criteria used by the special panel of senior editorial advisors for assessing evaluations can be found in appendix C.

Agency for Health Care Policy and Research

MISSION: To generate and disseminate information that improves the health care system.

AHCPR Evaluation Program The Agency for Health Care Policy and Research (AHCPR) has designed a portfolio that responds to three types of evaluation needs:

• The need for comprehensive information to assess the Agency's effectiveness in meeting its major and long-term priorities and goals.

• The need for information from fast-track or quick-turnaround projects to respond to critical Agency, PHS, and departmental concerns.

• The need for information from internal evaluations to improve the efficiency of key program areas.

To address these evaluation needs, an evaluation component is built into each AHCPR project. The evaluation mechanisms, which vary depending on the project, include special evaluation studies; peer review of grant applications and subsequent review by the National Advisory Council for Health Care Policy, Research, and Evaluation; the User Liaison Program (which provides information on the value of our research to State policymakers); and other evaluation efforts such as focus groups to provide feedback on AHCPR products and to provide baseline information to assist in the design of future evaluation projects.

AHCPR received delegated authority to review evaluation projects in 1992 and, as part of delegated review, established a formal structure to review proposed evaluation projects that coincides with the review of concepts for all other Agency projects. Thus, review of proposed evaluation contracts is integrated into the Agency's formal planning and budget process.

AHCPR has implemented a two-tier process for reviewing evaluation projects eligible for 1 percent set-aside funds. The process begins with an executive evaluation review to assess the policy relevance and relative priority of proposals submitted. This review is conducted by the Administrator and senior staff. The second, or technical, review assesses policy-relevant project proposals for feasibility, soundness of design, costs, potential importance of the findings, and relation to ongoing evaluation activity. This second review is conducted by the Task Force on AHCPR Evaluation Projects, a group consisting of one individual with evaluation expertise from each Office and Center.

Summary of FY 1994 Evaluations During fiscal year (FY) 1994, AHCPR completed six evaluations on two major topics: examining the process of developing clinical practice guidelines, and monitoring and evaluating health care delivery.

One of the statutory responsibilities of AHCPR is to arrange for the development, review, and revision of clinical practice guidelines. AHCPR-supported guidelines may be used by physicians, educators, and health care practitioners to help determine how diseases, disorders, and other health conditions can most effectively and appropriately be prevented, diagnosed, treated, and managed. The development of guidelines relates to AHCPR's strategic plan goal to determine what works best in clinical practice and to translate that knowledge into clinical decisionmaking. The first three AHCPR-supported evaluation studies completed in FY 1994 focused on strengthening the methodology underlying the guidelines by examining the strengths and limitations of current methods, and by identifying improved approaches and how they might be implemented.

The first project examined the methods used to conduct peer and pilot reviews for six of the guidelines. Peer and pilot review methods are used before a guideline is released to ensure its methodological soundness and scientific accuracy as well as to obtain information on its clarity, organization, and format. The study culminated in several recommendations for improving the quality of peer and pilot review.

Two other studies focused on methods for incorporating treatment cost analyses into the guideline development process. One examined the methodologies and data sources used in cost studies for eight of the guidelines and examined some of the difficulties in making cost estimates about various guideline treatment options. The other focused on identifying databases that could be used to provide information to generate cost estimates. This study found that no single database can be used to estimate the costs of recommendations in clinical practice guidelines. It is necessary to assemble data from several sources to estimate costs because claims databases sometimes lack information on certain services, and there are significant questions about the accuracy of data in various databases.

Information from these three studies has been used to inform two AHCPR reports to Congress on methods for developing guidelines and methods for including cost information in the guidelines. The findings have also been used to formulate a more uniform approach to guideline development based on what works best from the various approaches used to date.

The second area of emphasis among the AHCPR evaluations completed in FY 1994 relates to the Agency's strategic plan goal to monitor and evaluate health care delivery. Critical to fulfilling this goal is research that can be useful to policymakers by providing a greater understanding of the effects of emerging delivery systems and changes in the health care market structure. One such project focused on methods and issues related to the implementation and integration of advanced clinical information systems. The report presented a design for studying integrated health care information systems, including issues such as study feasibility, site selection, research design, and a data collection plan.

Another study examined inappropriate extension of hospitalization and the role of discharge planning, and identified research needs and potential existing data sources. Major findings include the following: 5 percent of extended hospital stays are not medically necessary; the cost of these stays was conservatively estimated at $420 million in 1987 dollars. Reasons for extended hospital stay include lack of sufficient nursing home beds and other alternate placement settings, delays in qualifying patients for posthospital benefits, and delays in appointing conservators for incompetent patients. Discharge planning appears to have only a small impact on extended hospital stays.

The final evaluation completed in FY 1994 assessed the availability of research on the cost-effectiveness of managed care health plans. The study found managed care plans had lower hospital utilization, greater use of less costly alternatives to expensive procedures and tests, greater use of preventive measures, comparable quality of care, and somewhat lower enrollee satisfaction generally, but greater satisfaction with cost. The study recommended additional research on managed care performance, including more comprehensive research projects to determine what works in managed care, rather than just whether managed care works.

Evaluations in Progress AHCPR currently supports seven evaluations that focus on its mission to generate and disseminate information that improves the health care system. AHCPR's ongoing projects provide information that will facilitate improvements in health care quality by advancing the science of quality measurement. In addition, AHCPR continues its commitment to improving the quality of AHCPR-supported clinical practice guidelines.

Two prominent FY 1995 projects relate to improving the science of quality measurement and improvement. The first, the Measurement Typology Project, was designed to collect and summarize information on clinical quality measures. These are technical quality measures used to estimate the extent to which health care providers deliver services that are appropriate for each patient's condition; whether services are provided safely, competently, and in an appropriate timeframe; and what the outcomes are from the services provided. The project summarizes 40 clinical performance measure sets--which contain a total of 1,287 performance measures--and provides a framework for evaluating the usefulness of various measures. The next phase of this project will expand and refine the typology, which ultimately will provide the foundation for an ongoing national resource of validated clinical quality measures that will assist in quality measurement and improvement.

The second project, to be completed in FY 1995, was designed to focus on consumer information needs. It developed a model consumer survey to examine how consumers perceive the quality of health care they receive, including their attitudes toward their access to care, use of specific health services, and health outcomes. This project and the Measurement Typology Project are complementary efforts to facilitate evaluation of quality from the perspectives of clinical quality assessment and consumer satisfaction.

AHCPR is also continuing its commitment to improving its clinical practice guidelines. Three projects focus on methods for improving the efficiency and effectiveness of guideline development, and three large-scale targeted evaluation projects will be completed during FY 1995. These projects will provide information on the usefulness of the guidelines for quality measurement and improvement.

An evaluation design study conducted by the George Washington University Center for Health Policy is examining methods for studying the process of guideline development. A follow-on study being conducted by the American Institutes for Research (AIR) is applying these evaluation design principles to examine factors that influence providers' views of the quality of various guidelines. Some key findings of the George Washington University study compare AHCPR's guidelines to those sponsored by other organizations. AHCPR's guidelines are found to address a broader audience (including a wider range of clinicians and clinical settings and a wider range of nonclinicians, including patients, families, payers, regulators, and purchasers); to have very detailed and comprehensive written policies and procedures for their development; to have more specialized expert resources on which to draw; and to use multiple methods for external review.

These differences have important implications for evaluating the credibility, usability, and potential impact of the AHCPR-supported guidelines on provider practice. Recommendations from this study, as well as those from the AIR follow-on project, are expected to be used to improve guideline development. AHCPR is also undertaking a project with the Institute of Medicine to examine optimal methods for selecting guideline topics.

In addition to projects to improve the guideline development process, AHCPR has sponsored a number of targeted evaluation projects. Through the use of evaluation contracts, cooperative agreements, and intramural research projects, AHCPR has supported evaluative efforts for guidelines on acute postoperative pain; urinary incontinence; benign prostatic hyperplasia; cataracts in adults; prediction and prevention of pressure ulcers; and depression in primary care.

Two targeted evaluation projects, to be completed in FY 1995, will provide important information on the feasibility of using the guidelines for quality measurement and improvement, as well as for insight into the difficulties of implementing guidelines. The first project develops, implements, and evaluates quality and utilization review criteria and educational outreach based on AHCPR-supported clinical practice guidelines. The project uses five Medicare Peer Review Organizations (PROs) to develop criteria based on three guidelines (urinary incontinence, acute postoperative pain, and benign prostatic hyperplasia [BPH]); develop and test training materials to use the criteria for case review; pilot test the criteria to assess intra- and inter-rater reliability; apply guideline-based review criteria; and compare guideline-based review with the review systems currently used by PROs. The PROs are playing an integral role in developing, implementing, and evaluating alternative educational outreach strategies based on the BPH guideline.

The second project is building on AHCPR's existing efforts to evaluate clinical practice guidelines by translating AHCPR-supported guidelines into medical review criteria for use in a variety of care settings including hospitals, health maintenance organizations (HMOs), ambulatory clinics, and physicians' offices. This project, like the project described in the previous paragraph, will assess the usefulness of alternative educational interventions for disseminating the guidelines and review criteria and for changing provider practice behavior. One component of the project focuses on cataract in adults, and the other on pressure ulcers in adults. The criteria for this project will be developed with the Department of Veterans Affairs (VA) and will be tested in a number of VA facilities. In a follow-on project, AHCPR and the VA will test alternative methods for disseminating the guideline and using the criteria as tools to assess the effect of guidelines on practice patterns.

New Directions for Evaluation AHCPR is in the process of developing a new strategic plan that reexamines current activities in relation to a rapidly changing health care marketplace. This new direction recognizes that AHCPR's programs must generate the information and tools needed to improve health care delivery and its outcomes. In addition to generating new knowledge, AHCPR's portfolio will focus on translating research findings into forms of information that actively assist consumers, practitioners, payers, and others in making effective health care decisions.

AHCPR is also responding to new challenges to government to become more efficient, flexible, creative, and effective. The Agency recognizes the need for evaluation activities to be linked to the planning process and to yield information that can inform policymaking, budget planning, and program management.

In the coming years AHCPR will continue to support research in the area of quality measurement and improvement. A Request for Applications, titled Consumer Assessments of Health Plans Study (CAHPS), was announced in the NIH Guide in May 1995. The proposed study will build on the consumer survey project described above. The project will demonstrate and evaluate the use of consumer surveys to assess health plans. AHCPR will also build on work in the area of clinical performance measurement by expanding the Measurement Typology Project to incorporate guideline-based clinical performance measures into the measures inventory.

Centers for Disease Control and Prevention and Agency for Toxic Substances and Disease Registry

MISSION: To promote health and quality of life by preventing and controlling disease, injury, and disability.

and

Agency for Toxic Substances and Disease Registry MISSION: To prevent exposure and adverse human health effects and diminished quality of life associated with exposure to hazardous substances from waste sites, unplanned releases, and other sources of pollution in the environment.

CDC Evaluation Program The Centers for Disease Control and Prevention (CDC) place a high priority on evaluations seeking to answer policy, program, and strategic planning questions. Evaluation studies are developed and selected on the basis of eight strategies to achieve its mission. These strategies are to--

• Monitor health
• Detect and investigate health problems
• Conduct research to enhance prevention
• Develop and advocate sound public health policies
• Implement prevention methods
• Promote healthy behaviors
• Foster safe and healthful environments
• Provide leadership and training

The CDC Director provides annual guidance to the various Center, Institute, and Office (CIO) Directors on 1 percent set-aside evaluation activities. This memorandum generally includes information about the types of studies to be carried out with 1 percent evaluation funds. Each proposal undergoes multiple levels of review. Initial review is conducted by the Office of Program Planning and Evaluation. Subsequent reviews are completed by CDC analysts in the Office of the Assistant Secretary for Health (OASH) and the Office of the Assistant Secretary for Planning and Evaluation (OASPE). Study authors are provided with comments, questions, and recommendations made by reviewers. In addition to providing their responses, authors are given the opportunity to revise their proposals at this time.

A panel of CDC evaluators, scientists, and program managers are convened to review and rank proposals. Review criteria include (1) relevance to prevention effectiveness; (2) relative importance of the public health problem being addressed; (3) probability that the proposed project will accomplish its objectives; and (4) extent to which other CDC programs will benefit from the project. Results from this panel review are converted into a comprehensive ranking that is provided to the Director of CDC. Final funding decisions are made at this time.

Finally, staff in the Office of Program Planning and Evaluation work closely with program staff to ensure development of a clear statement of work for selected projects. Before initiation of procurements, a final ad hoc review of the project statement of work is completed.

ATSDR Evaluation Program Agency for Toxic Substances and Disease Registry (ATSDR) receives its funds from Environmental Protection Agency/Superfund appropriations rather than Public Health appropriations; therefore, ATSDR does not receive a 1 percent evalu-ation set-aside. Nevertheless, the Agency is responding to the changes mandated in its program planning and evaluation efforts by the National Performance Review and the Government Performance and Results Act (GPRA) of 1993. To meet those requirements, ATSDR staff members modified the Agency's planning process, incorporating implementation strategies and outcome/performance measures.

Prominent issues addressed in the new planning system emphasize ATSDR's commitment to improve the health of people affected by hazardous substances polluting the environment. Improvements include using exposure assessments and demographic data to identify people at risk and, more directly, assessing/addressing the concerns of customers. The new planning system provides the basis for measuring ATSDR performance and making systematic improvements as part of its internal evaluation activities.

Summary of FY 1994 CDC Evaluations CDC completed 12 evaluations in fiscal year (FY) 1994. These evaluations covered training and information dissemination, surveillance, program effectiveness, prevention, and costs of disease.

Training and information dissemination was the focus of several evaluations, two of which were highlighted in chapter II. The first was an evaluation of CDC and ATSDR training activities that assessed the training needs of State and local health departments and inventoried CDC's current training activities. It provided an example of using an evaluation to document current practice to help generate a new agenda for program action. The other highlighted evaluation was a survey of readers of the Morbidity and Mortality Weekly Report, a CDC publication of interest to public health professionals around the Nation. The survey found that this publication generally met the needs of its readers and was valued for its accuracy, relevance, and concise reporting format. Study recommendations are expected to help fine-tune this publication in response to reader suggestions.

Information dissemination activities also were addressed in an evaluation sponsored by the Office on Smoking and Health (OSH). Recommendations for key management and operational aspects of information dissemination in OSH were made based on interviews with key officials in CDC and outside the organization. The evaluation also made suggestions for strategic planning in view of OSH's evolving leadership role in the tobacco control community.

With respect to surveillance, several evaluations focused on gathering statistics for policy analysis and decisionmaking. One evaluation addressed the resiliency of the Model State Vital Statistics Act and Regulations to accommodate changes in social customs and technology of registering vital events and statistics. The results are being used to advise States about the need for revisions in the collection of vital statistics. Two evaluations are concerned with medical records in relation to national survey data. Both studies are intended to determine whether medical records confirm survey respondents' reports about selected conditions and impairments. In the first study, selected data elements of the 1988 National Maternal and Infant Health Survey (NMIHS) were compared to records maintained by the States. This evaluation assessed the quality and completeness of the information reported, identified discrepancies, and examined the nature and frequency of discrepancies. The second study evaluated the accuracy of diagnostic reporting in the National Health Interview Survey. Information collected from users of health care services was compared with infor-mation collected from the source of the health care services.

A third category of studies sought to evaluate health programs to determine their effectiveness. One study assessed the effect of burgeoning patient loads on sexually transmitted disease (STD) clinics, the impact of changing funding levels, and the increase in more elaborate patient testing. The study examined the effectiveness of services and identified factors that contributed to overburdening the delivery system. A similar type of study assessed the effectiveness and identified exemplary practices of State-based diabetes control programs (DCPs) in providing services with the potential for reducing diabetes-related mortality and morbidity. The outcomes examined were (1) the number of people reached by the programs; (2) the improved coverage provided by clinics; (3) the level of integration into ongoing medical service delivery; and (4) the effect of leveraging resources for diabetes programs. Findings showed that DCPs have had a measurable effect on diabetes services. Recommendations for program expansion were provided as part of this evaluation.

"Assessing Prevention Effectiveness: A Collaborative Effort With Selected Health Maintenance Organizations" is the first part of a two-phase study. A key component of this study includes the development of a framework and process to assess prevention effectiveness in health maintenance organizations (HMOs), types of services provided, and their potential to work with CDC. Data collected in Phase I provide the information required to move to Phase II of the study.

The purpose of the evaluation study by the Division of Cancer Prevention and Control is to collect data pertinent to program-related decisions. The evaluation focuses on components of a comprehensive breast and cervical cancer early detection and control program. In addition to these components (public and provider education, quality assurance, surveillance, screening, and followup), the combined effect of components is assessed.

The final category of evaluations sponsored by CDC developed methods to estimate direct medical costs for various diseases. One study estimated direct medical costs of chronic hepatitis B, concentrating on acute care costs. Costs from both Medicaid and private sector data were collected. Another study estimates the direct medical costs of congenital syphilis, using 1990 figures. The estimate included medical care costs for the first year, special education costs required by children with congenital syphilis, and lifetime custodial care costs, all of which were categorized by severity.

CDC Evaluations in Progress CDC has a total of 32 evaluations in progress. They fall into four general categories: surveillance/data collection studies; program evaluations; community/intervention effectiveness studies; and evaluation methodology studies. Performance improvement is a major focus of each of these studies.

Surveillance/data collection is the focus of the largest number of evaluations. For example, a study of the effectiveness of CDC surveillance for drug-resistant pneumococcal infections addresses drug resistance, which was identified as a major challenge in CDC's Emerging Infections Plan. This project will evaluate the validity of antimicrobial resistance data collected from sentinel hospitals, using CDC's sentinel hospital surveillance program. The hospital surveillance program, located in 13 hospitals in 12 States, is designed to determine the magnitude of drug-resistant pneumococcal disease and to provide clinicians with the ability to select optimal regimens of empiric therapy. By using population-based surveillance for invasive pneumococcal disease in two geographically distinct areas, this project will evaluate the extent to which CDC's sentinel surveillance program is capturing drug-resistant pneumococcal disease.

Another study that falls within the category of surveillance/data collection entails evaluation of STDs in the United States. The objectives of this study are to (1) determine the accuracy of CDC's surveillance data on STDs by comparing them with independently collected data from a survey of providers; and (2) to determine, also from a survey of providers, the extent of adherence to CDC's diagnostic and treatment guidelines for STDs and to identify ways of increasing compliance with those guidelines.

Program evaluations, also an important focus of numerous current evaluations, are being undertaken for grant programs, including the Lead Poisoning Prevention Program and the Injury Prevention and Control Program. Other studies in this group involve evaluations of the National Laboratory Training Network, the San Juan Laboratory's Dengue Hemorrhagic Fever Program, and the Fatality Assessment and Control Program.

Community-based interventions are the subject of several other evaluations. Four studies address the prevention of violence. These include projects focused on youth violence prevention, suicide in Native American communities, domestic violence medical education programs, and support systems for battered women.

Evaluation methodology is the focus of several ongoing projects. For example, one project is developing a comprehensive evaluation strategy that can be incorporated into planning, budget, and legislation for the National Center for Chronic Disease Prevention and Health Promotion.

New Directions for CDC Evaluation Evaluation studies focusing on program performance and effectiveness will continue to be of primary importance to CDC. As CDC moves toward a comprehensive performance monitoring system, focused studies in this area will be of utmost importance. Evaluations will be conducted to provide data for decisionmaking regarding the need for broader program implementation. Similarly, as programs develop and implement performance indicators, projects designed to provide data for performance measurement and to assess the effectiveness and efficacy of such indicators will be initiated.

Food and Drug Administration

MISSION: To protect and promote public health through food, drug, medical device, and cosmetic regulation.

FDA Evaluation Program Systemic changes in the government management environment are strongly influencing the setting, conduct, and use of evaluation activities in the Food and Drug Administration (FDA). Three forces--all related to the mandates of the Government Performance and Results Act (GPRA) of 1993--are reshaping the evaluation function in the FDA.

1. Performance management. The reorientation of all government managers toward performance management has shifted the responsibility for program evaluation from specialized staff offices and contractor studies to day-to-day line managers. Picking appropriate program goals, establishing valid measures toward those goals, and collecting management information to record measured progress are now integral parts of the government manager's responsibility. Relearning the role of management with regard to these shifted responsibilities is a key priority. The old order--evaluation as a proprietary domain of staff offices--is a bygone luxury to FDA programs in the era of streamlined government.

2. Customer participation. Most of FDA's management measures of its own performance are also measures of the performance of the regulated industries. Thus, while the beneficiary of FDA's performance is ultimately the general public, the Agency operates in such a manner that it supplies industry with an essential component of commercial success. FDA's approval of a new drug, for example, not only satisfies a legal requirement but also assures the public of the safety and efficacy of the drug. Collaboration between the FDA and its regulated customers regarding the design and coordination of the joint responsibilities to ensure effective, high-quality products has been a revolutionary concept, but it is becoming the norm under the customer-conscious GPRA directives.

3. Increasing rigor of the rulemaking process. The FDA establishes standards of safety and efficacy through rules published in the Federal Register. Many safety and efficacy standards are also performance standards that industry is obliged to meet. Examples are the Good Manufacturing Practices regulations. Today, virtually every FDA final rule of significant magnitude includes elements found in classic program evaluations: a critique of the existing system, alternatives for better performance, performance and cost tradeoffs, reactions and suggestions of customers, and conclusions, with an action timetable for implementation. New directives from the Administration as well as possible legislative action by Congress will make the rulemaking process of the future even more like the classic evaluation process.

In sum, FDA's evaluation efforts are driven by the mandates of GPRA and its corollaries, are carried out by line managers rather than specialized evaluation staffs, and are focused in the areas of performance management, customer participation, and more rigorous rulemaking. Because it receives its funds from Agriculture appropriations rather than Public Health appropriations, the FDA does not manage a 1 percent evaluation set-aside.

Summary of FY 1994 Evaluations Fiscal year 1994 was an important test period for the new evaluation paradigm. The following examples show how evaluation has been integrated into the line manager responsibilities and interlocked with the affected customer.

Implementation of the Prescription Drug User Fee Act (PDUFA). In FY 1994, the more than 1,000 full-time employees involved in FDA's drug and biologic review process completed their first full year using the performance management goals jointly established by the FDA and industry. Line managers developed measures for evaluating progress toward these performance measures, reported the results to Congress and the several hundred industry customers of this program, and used the resulting management information to self-manage the program toward higher FY 1995 performance goals. Line managers have also focused on streamlining, achieving other review efficiencies, and reducing the regulatory burden while maintaining the high quality and standards associated with FDA decisionmaking.

The negotiation process toward medical device user fees. During FY 1994, the FDA and representatives of the medical device industry engaged in a series of meetings that constituted a joint evaluation of each other's performance in the device development process. Current performance of both parties was assessed quantitatively; new alternatives were modeled, critiqued, and cost-audited by independent accountants; and the specifications for a new program based on shared performance expectations were translated into a framework for multiyear performance goals. While the culminating legislative step was not realized in 1994, the joint effort stands as a model of a customer-oriented, performance-based program evaluation.

Harmonization of international regulatory requirements. While benchmarking has long been an important evaluation technique in the private sector, it has not been widely used in the Federal sector--and especially in regulatory agencies--because of the perceived uniqueness in function and process of the Government Agencies. At the international level, of course, benchmarking--especially for regulators--is not only possible but highly desirable. Global forums such as the International Committee on Harmonization invite such benchmarking. Their efforts are maturing and reaching the point at which specific components of the regulatory process in different countries have been compared and assessed by the managers of regulatory programs. The initial successes suggest that regulatory benchmarking in the context of international forums will be an increasingly important way for regulators to evaluate and improve their performance.

Evaluations in Progress The following evaluation objectives will influence line managers during FY 1995:

Performance management. Managers of all FDA programs are evaluating their performance measures in light of GPRA standards. Broad-based training of managers to enable them to evaluate the performance design of their responsibilities is under way. The first phase of this multiyear process toward a new standard of performance measure will end with the formulation of FDA's FY 1997 budget.

Additional user fee programs. FDA managers will finalize measures for two performance--oriented user fee programs identified in the President's budget. One is a culmination of efforts by FDA managers and industry representatives for a medical device user fee act; the other involves developing with import brokers shared expectations and performance goals for a user fee-supported electronic processing system for imports.

Commencement of negotiated rulemaking. In FY 1995, the FDA will add a new customer-sensitive dimension to its increasingly rigorous rulemaking function by beginning to implement the President's directive to promote negotiated, consensual rulemaking. This marriage of customer participation and rigorous rulemaking will receive serious attention from FDA managers.

Two major projects are currently under way:

PDUFA management. During FY 1995--the middle year of the 5-year user fee program--the FDA will use the performance data from this GPRA-style program to generate the resource alignment needed to achieve the high performance goals for FY 1997--the final year of the current user fee legislation.

Mammography Quality Standards Act of 1992 (MQSA) assessment. The FDA's second major user fee program, MQSA, is also a performance-oriented, GPRA-style program. Although it is similar in design to PDUFA, much of the data on its performance are more external to the agency. The FDA is therefore using a contract study to gather data from affected facility sites to gauge the effects of the standards on the availability of mammography facilities.

New Directions for Evaluation Changes in government management are creating systemic changes in the FDA's evaluation function. A new paradigm driven by line managers' performance responsibilities, by the imperatives for involving customers, and by directives for analytically rigorous rulemaking has replaced the traditional practice of evaluation studies guided by specialized evaluation staffs or third party consultants.

Health Resources and Services Administration

MISSION: To improve health by assuring quality health care to underserved, vulnerable populations and by promoting a primary care and public health workforce.

HRSA Evaluation Program The objectives of the Health Resources and Services Administration (HRSA) evaluation program are to improve program management and policy development and to provide information that will enhance strategic planning, budget decisions, and legislative planning. Consequently, a high priority is given to studies that will (1) assess program effectiveness or outcomes, or (2) enhance the Agency's capacity to measure performance by identifying meaningful performance indicators, creating or refining databases, and developing study designs or approaches. Over the past several years, HRSA has made considerable progress in expanding such capacity.

The evaluation program in HRSA is designed to ensure that the four Bureaus and the Office of Rural Health Policy (ORHP) identify their particular information needs within a framework established by the Administrator each spring through a memorandum on evaluation planning guidance. Other staff offices may also propose studies. After study proposals are reviewed for clarity by staff of the Office of Planning, Evaluation, and Legislation (OPEL), proposals are reviewed for technical merit by a committee of senior evaluators representing each Bureau, ORHP, the Agency for Health Care Policy and Research, the National Center for Health Statistics (NCHS), the Office of the Assistant Secretary for Health, and the Office of the Assistant Secretary for Planning and Evaluation. Authors receive written comments on methodological or other design issues. Proposals are then reviewed by a committee of the Bureau Directors and other senior HRSA executives, whose role is to consider the proposals for relevance to important policy, budgetary, or legislative issues; potential to answer questions about program effectiveness or impact; and degree of attention to crosscutting topics. Proposals and committee recommendations are submitted to the Administrator for decision. OPEL staff work closely with program staff to develop a strong scope of work and other required documentation.

HRSA places major emphasis on the dissemination of information about findings and utilization of completed studies. Developing articles for publication in the professional literature and presentation at professional meetings is strongly encouraged, and draft manuscripts are typically called for as a contractor product. Briefings on completed studies of broad interest are scheduled periodically for the Administrator and other senior officials. An annual report on completed evaluation studies describes study purposes and findings, and tells how study results have been disseminated and used. Another annual publication provides brief summaries that describe the purpose and proposed methodology for studies initiated in the preceding fiscal year.

Summary of FY 1994 Evaluations Two significant studies were completed that examined aspects of the community and migrant health centers. "Evaluation Design: Evaluation of the Effectiveness of Community Health Centers" developed a study methodology to examine the efficacy and effectiveness of the centers. This work has provided a way to assess the impact these providers have on the health status of their users. "Linkage Programs: Survey of Mental Health Services" examined a joint National Institute on Drug Abuse/HRSA demonstration program designed to test community-based approaches for linking substance abuse and primary care services. The study found that the projects had been successful in creating supportive environments for the assessment, referral, and delivery of mental health services. Findings from this study will help other providers of primary care services establish the necessary alliances and treatment networks to reduce delivery system fragmentation and develop a continuum of mental health services.

"HIV Service Networks in Four Rural Areas," highlighted in chapter II, is a series of case studies undertaken to help States identify ways to improve the quality, availability, and organization of services for rural residents with HIV. The case studies described two distinct approaches to providing medical care to rural residents with HIV: (1) cooperative working relationships between HIV clinics and physicians in private practice, and (2) linkages between physicians and a medical school or rural-based regional care facility for training and backup consultation. Findings are being used to formulate HRSA policies on HIV care in rural areas and to provide technical assistance to States with large rural populations. The findings are also being used to design a rural component, funded by HRSA, for a large national study of AIDS services sponsored by the Agency for Health Care Policy and Research.

Three other AIDS-related studies were completed by HRSA in FY 1994. Two of these address the priority area of community infrastructure building. "Implementation of Title I of the Ryan White CARE Act of 1990" established a preimplementation baseline in Baltimore, Maryland, and Oakland, California, covering the organization and funding of HIV-related services, the functioning of planning councils, and the perceptions of HIV patients regarding service delivery. The Henry J. Kaiser Family Foundation subsequently funded followup surveys on these issues. "The Participation of People With HIV in Title I HIV Health Services Planning Councils" studied planning councils in Atlanta, Philadelphia, San Diego, and Seattle, and identified factors supporting the initial and sustained involvement of HIV-positive individuals in these councils. It also identified alternative methods employed by the councils to gain input from and provide feedback to HIV-positive populations in the absence of their direct participation on the councils. "Evaluation of Methods for Estimating Unit Costs of HIV Health and Support Services" developed common definitions for service, cost, and expenditure categories, and identified potential indicators of cost savings attributable to coordinated, comprehensive, community-based care systems.

"The Demographic and Treatment Characteristics of the Hill-Burton Population" obtained data on a sample of patients receiving services under the Hill-Burton Uncompensated Services Program and compared these with data on the general population. The study found that, compared with the total U.S. population, the Hill-Burton population was more likely to be female, young adult, and unmarried, with a very low household income.

Three studies dealt with health professions programs. "Survey of Beneficiaries of Nursing Education Projects" concluded that Federal support of these projects has enhanced the supply of advanced practice nurses. "The Training and Practice of Preventive Medicine Specialists: An Underutilized Resource" studied the area and, based on its findings, recommended incorporating preventive medicine into the medical school curriculum. In light of the increased attention being given to improved management practices, "Evaluation of Bureau of Health Professions Strategic Directions--Phase I" developed a set of outcome indicators and identified potential data sources and gaps relating to programs under Titles VII and VIII of the Public Health Service Act.

Evaluations in Progress The 41 studies that were ongoing as of September 1994 include four types: (1) measurement of program outcomes, effectiveness, or impact; (2) development of performance indicators, and/or data systems or databases; (3) design and/or testing of study approaches; and (4) provision of other types of information for program management or policy development.

The largest study of program performance and outcomes is the National Evaluation of the Healthy Start initiative. Begun in September 1993, this 5-year prospective study includes both process and outcomes components. The process evaluation is assessing the development and implementation of comprehensive, coordinated systems of perinatal care in the 15 initially funded Healthy Start sites. The outcomes component concerns changes in the health status of pregnant women and infants across the sites. Client-specific data from a standard data set will be used to examine the relationship among clients' sociodemographic characteristics, project-specific service use, and maternal and infant health outcomes. Secondary data, including linked birth and death records and Medicaid claims data, will be used to compare outcomes of clients residing in Healthy Start project areas with those of women living in matched comparison areas. The study also will employ a series of more qualitative methodologies to gain information about the unique features of each of the 15 projects. Four questions are being addressed. Did the Healthy Start initiative succeed? If so, why? If not, why not? And, what would be required for a similar intervention to succeed in another setting?

Two other studies are assessing the effectiveness of the community and migrant health centers programs, using national samples of grantees and including both quantitative and qualitative techniques. Additional studies concern effectiveness and outcomes of the National Practitioner Data Bank, the Healthy Schools/Healthy Communities program, and the Injury Prevention Implementation Incentive Grants program. One further study examines the effects of Ryan White CARE Act Title I funding on services for active and recovering HIV-infected drug users.

Among studies of the second type are projects to assess data systems for the Ryan White CARE Act Titles I and II programs, and to develop national integrated requirements estimates for physician assistants, nurse practitioners, certified nurse-midwives, and physicians.

The Agency also has several ongoing studies to design and test study approaches. One effort is intended to design a method to evaluate the validity of current estimates of the reported mortality rate among infants of Mexican descent residing in the United States and examine factors contributing to the reported low rate. The study includes nontraditional sources of data concerning births that occur outside the confines of the infant tracking systems. Another study is developing a self-assessment tool for use by HIV health planning councils in Title I cities and by care consortia in Title II states. Grantees and their planning councils and care consortia have expressed interest in methodologies and resources that will help them know what they are accomplishing. The tool will be sensitive to the limited resources available for planning and evaluation, providing a cost-effective method for evaluating performance. An additional study is developing a design for assessing variations in the procurement rates of Organ Procurement Organizations (OPOs). Identification of the factors associated with low procurement rates should help increase these rates.

Finally, several studies are intended to provide other types of information for program management or policy development. One exploratory project on the nature of telemedicine systems serving rural populations is providing baseline information in connection with the new HRSA Rural Telemedicine Grant program. An additional study is examining the impact of certain Medicaid waivers on Federally Qualified Health Centers. Another study is examining the relationship of medical school characteristics to graduates who choose primary care and ultimately provide care to the underserved. Still another study is exploring the impact of the Ryan White CARE Act on strategies for serving African Americans. One of the study products will be a report providing detailed examples of effective and ineffective processes and strategies for enhancing access for African Americans. The report will also identify performance indicators being used and approaches for gathering data to measure performance. Finally, a 2-year study by the Institute of Medicine is defining the place of primary care in the health care environment of the future and developing normative goals for primary care.

New Directions for Evaluation The major priority of HRSA evaluations in FY 1995 and 1996 will be studies to continue the development of meaningful measures and data systems to facilitate performance assessment. During FY 1995, HRSA is conducting a project to review the status of performance indicators for all line organizations and major programs and determine the degree to which data sources are currently available for applying the appropriate indicators. The impetus for this effort is the implementation of the GPRA, which will begin with the FY 1997 budget.

In FY 1995 and 1996, HRSA's other evaluation studies will continue to reflect the Agency's mission of expanding access to care for the underserved and such crosscutting priority areas as academic/community partnerships in health professions education, community infrastructure building, managed care, expanded roles for States, information technology, and HIV/AIDS. HRSA also expects to develop multiyear evaluation goals for selected program areas in connection with new authorizing legislation and to initiate or continue studies of effectiveness, outcomes, and/or impact in selected program areas.

Indian Health Service

MISSION: To provide a comprehensive health services delivery system for American Indians and Alaska Natives, allowing for maximum tribal involvement in developing and managing programs to meet their health needs.

IHS Evaluation Program The evaluation program of the Indian Health Service (IHS) serves the Agency's programmatic and policy objectives. These objectives, developed in consultation with tribal communities, were most recently articulated in a "vision statement" of the IHS Director in November 1994. The statement calls for the IHS to continue to be the best primary care rural health system in the world. This goal is predicated, in part, on additional program emphasis in areas in which demographic and disease indicators point to a strong need for service delivery: elderly care, child abuse, women's health care, and substance abuse. Substance abuse, for example, is considered to be one of the most pressing problems facing American Indians and Alaska Natives[1] because of its toll in morbidity, mortality, and social turmoil.

The IHS evaluation program provides valid and reliable information to help the Director promote this vision for the Agency, and the evaluation program is also expected to help guide its implementation. The program traditionally has addressed three general areas--service delivery, health status, and management. The areas are interrelated, especially as the IHS faces times of fiscal austerity. If the IHS is to improve service delivery and maintain quality care, it must increase efficiency, effectiveness, and accountability. Management practices are continually being reviewed to ensure that IHS programs operate efficiently to ensure quality patient care and preventive care--the cornerstones of the IHS.

The IHS conducts technical and policy reviews of proposed evaluations--much like other PHS evaluation programs--but the program also has several distinguishing features. The most noteworthy feature is the active participation of tribes in the development and execution of evaluation studies. American Indians are recognized as primary stakeholders whose needs and concerns must be addressed, especially if the IHS is to help tribes develop and provide health care to their communities. For example, the IHS sponsors roundtable discussions with tribal leaders and Indian health program officials to elicit advice and consultation about directions for evaluations and policies. This approach to evaluation--a responsiveness to stakeholders and an orientation toward a pluralism of cultural values--is referred to by professional evaluators as a "fourth-generation" evaluation. Earlier types of evaluations, which began after World War II, were more technical, descriptive, and marked by the judgments and values of the evaluator.

Another distinguishing feature is that most evaluations are performed by contracting firms owned and operated by American Indians. This policy is an outgrowth of regulations and guidelines intended to increase the participation of American Indians and tribes in program management and service delivery.

Summary of FY 1994 Evaluations The IHS completed nine evaluations in FY 1994, focusing on service delivery, health status, and management. These areas are pivotal to the Director's vision of ensuring the delivery of high-quality primary care. Three of these evaluations deserve special notice because they address areas that are slated for renewed programmatic emphasis in the coming years.

One evaluation assessed the availability, utilization, and quality of data sets describing the health status of women and children in the Navajo area. Maternal and child health (MCH) data sets play a vital role in monitoring the health status of American Indian populations, shaping programs and assessing their effectiveness. The first phase of the project sought, through interviews and site visits, to determine whether IHS service unit staff were able to fulfill existing data requirements use the data that were generated. The project found staff to be inundated with data requests that did not completely meet their needs for the care of patients and for the promotion of maternal and child health. The second phase examined the quality of a specific automated data set on childbirth outcomes by comparing it with original chart entries. The comparisons were favorable, thus reinforcing the quality of the automated data set. The findings and recommendations of the first phase of the report are leading to a better understanding of MCH data needs and to refinements in the types of MCH data required of service area staff, along with improved vehicles for obtaining them.

Another evaluation, highlighted in chapter II, analyzed the health status of American Indians in California. It documented the health status of, and access to health care services for, those in both federally recognized and non-federally recognized tribes. These and other comparisons showed that the overall health status of both groups was poorer than that of the California population as a whole. The findings underscore the significance of maintaining and expanding coverage to the entire American Indian population of California.

A third evaluation developed long-term health care projections for alcohol-related hospitalizations. The projections--which were based on a new method called "long-term projection methodology"--sought to determine the impact of changes in hospitalizations as a result of advances and innovations in treatment, tech-nology, and behavior. The technique forecast a 20 percent decline in alcohol-related hospitalizations. While these projections were encouraging, they were insufficient to meet Healthy People 2000 objectives. This finding is one of many that have led to a renewed IHS emphasis on prevention of substance abuse. This study will be used specifically to guide evaluations such as that planned on regional adolescent alcoholism treatment centers (described below under "New Directions for Evaluation") and will provide tribes with information relevant to alcoholism program planning, policy development, and program evaluation. For example, the study informs IHS providers that the only age-gender group showing increases in alcohol-related hospitalizations is American Indian women over the age of 45.

Evaluations in Progress The IHS currently is sponsoring 21 evaluations covering service delivery, management, and prevention. Two of these evaluations underscore the IHS commitment to the development of new programs to prevent child abuse and family violence. Violence and abuse are serious problems that erode the quality of family life and can result in death, serious injury, or long-term health and behavioral problems. Family violence and child abuse are difficult to study because of underreporting and shifting definitions of what is acceptable behavior. The diversity of cultures, languages, and customs of American Indian tribes also heightens the difficulty of determining the extent of these problems and evaluating what works to prevent them.

One report, which is expected to be completed in FY 1995, is an assessment of the extent of child abuse and neglect among American Indian tribes and the ways in which the IHS responds to these problems. The evaluation also develops a model program to combat child abuse and neglect. Preliminary results from survey data indicate about 34 percent of American Indian children are at risk of becoming victims, and substance abuse commonly plays a role. The intervention program under development is being modeled after a successful program developed in 1985 by the State of Hawaii. The program provides home visitor services to new mothers, ensures continuity of medical care for children, and links families with essential services.

The other evaluation report is a case study of family violence in four distinct American Indian communities. Researchers conducted more than 100 unstructured interviews at the four sites to probe the nature and extent of family violence, which was defined as spousal abuse, child abuse and neglect, child sexual abuse, or elder abuse. They found little consensus across the four study sites about which type of family violence represents the gravest problem. These and other study findings are being incorporated into a model to guide the development of prevention programs. Eventually, this model will be available for use by an individual, family, group, or tribe to develop programs to prevent or reduce family violence in American Indian communities.

New Directions for Evaluation The IHS foresees the need for evaluations in the following areas: mental health services for urban Indians, regional treatment centers (RTCs) for substance abuse disorders, and health services for elderly American Indians. Urban American Indians, who account for approximately 56 percent of the American Indian population according to the 1990 Census, have important mental health needs. The urban American Indian usually lives in poverty and has little or no support system, in contrast to the American Indian living on a reservation among others who live at the same economic level and rely on the security of extended family and housing assistance. Indians who come to the cities encounter a physical environment, social organization, interpersonal behavior, attitudes, values, and sometimes even language that are foreign to their experience. They may suffer from feelings of isolation, depression, desperation, and anxiety, and may have problems with self-esteem. The traditional support of the extended family may be totally lacking. In 1976, Congress passed the Indian Health Care Improvement Act (P.L. 94-437), which was landmark legislation for all Indian health concerns but particularly for urban populations. Title V of the legislation specifically authorized health outreach and referral and the delivery of services to American Indian people in urban areas. Before this, health care delivery was not permitted off the reservation. The urban health program was expanded in 1988 in the Indian Health Care Amendments (P.L. 100-713). These amendments clarified the types of primary care and outreach services that could be provided, paving the way for the provision of mental health services. Mental health services are considered ripe for evaluation because the IHS supports extensive services in some urban areas and few, if any, services in other urban areas. This disparity needs to be evaluated.

Nine RTCs offer residential substance abuse treatment to Indian youth, most of them aged 12 to 19, in recognition of the fact that young people are especially vulnerable to long-term patterns of alcohol and drug abuse. Although alcoholism is decreasing among older American Indians, it remains stable in adolescents. Alcohol abuse among American Indian youth usually begins during adolescence or earlier. The RTC programs were legislated into existence by Congress and are an important part of the continuum of care for American Indian youth. Programs that began in the late 1980s and were accredited by the Joint Commission on Accreditation of Health Care Organizations are now deemed ready for evaluation because they have been in operation for several years and they appear to play an important role in reducing substance abuse. Evaluations are needed to document the impact of these programs, including outcome measures; to determine the level of agreement between referring and discharge diagnoses of individuals admitted to RTCs; to examine the characteristics and histories of youth and staff at the RTCs; and to determine the impact these centers have had in the local areas. The results can be used to redefine the operation of existing RTCs and guide the design of future centers.

Longer life expectancy among American Indians is posing new challenges for IHS service delivery. Demographics showing that life expectancy has increased from 61 years in 1972-74 to 71.6 years in 1986-88 have led to the creation and expansion of special services geared to the elderly. Evaluation of health services for elderly persons who are ambulatory, homebound, or institutionalized is considered critical if the IHS is to effectively meet the needs of this growing population. Evaluations should address the increasing need for long-term care. The emerging nature of the problem gives the IHS an opportunity to define and develop the most appropriate services for American Indian elders before the need overwhelms American Indian communities and leaves many without services. Improving the quality of life for elderly American Indians is thought to be a critical need in the next few decades.

National Institutes of Health

MISSION: To discover and disseminate new knowledge leading to improved health for all Americans.

NIH Evaluation Program Evaluation is an integral part of the role of the National Institutes of Health (NIH) in the support of biomedical research, training, and public education. Evaluation studies are undertaken to ensure that NIH meets its specific goals to--

• support biomedical and behavioral research of the highest quality;

• inform health researchers, health care providers, industry, and the public of advances and opportunities to improve health;

• support the training and continued availability of biomedical and behavioral scientists;

• support the facilities and equipment needed to sustain scientific progress; and

• manage its resources effectively and efficiently.

A distinguishing feature of the NIH Evaluation Program is the variety of evaluation instruments it employs. The most familiar instrument is a formal evaluation study that examines whether a program has successfully met its objectives. But NIH supports a host of evaluation strategies that go beyond traditional program evaluations. The NIH peer review system is one type of evaluation strategy: research proposals from scientists around the Nation are subjected to a rigorous assessment by fellow scientists, and only the most meritorious proposals receive funding. Other evaluation strategies include national advisory councils, boards of scientific counselors, consensus development conferences, and committees. These groups are charged with assessing a body of research to establish priorities, developing long-range goals and strategies, addressing emerging issues, identifying significant opportunities, assessing needs for new programs and activities, and recommending expansion, realignment, or continuation of ongoing programs.

The reason for the diversity of evaluation instruments lies in the nature of research. Research--especially basic research--depends on pursuing the unknown. The results of a research program and the generation of new knowledge usually cannot be anticipated. Consequently, research does not readily lend itself to the most common type of evaluation, an outcomes evaluation. Programs that provide services or promulgate regulations are most suited to outcomes evaluation because they are intended to achieve explicit, preconceived objectives. NIH attempts to evaluate its research programs with methods suited to the serendipitous nature of the research enterprise.

Another distinguishing feature of the NIH Evaluation Program is its use of the 1 percent set-aside evaluation fund strictly for programs that transcend individual NIH Institutes. The focus on NIH-wide evaluations is a self-imposed policy. NIH relies on its component Institutes, Centers, and Divisions (ICDs) to generate requests for funding of NIH-wide projects from the 1 percent set-aside, in addition to those that are centrally directed. The ICDs also conduct individual evaluations supported by their own program funds.

In June 1991, the Office of the Assistant Secretary for Health (OASH) authorized NIH to approve all set-aside funded evaluations, whatever the budget, while it maintained an ex officio presence in the review process. A two-tiered system is used to review project requests for 1 percent set-aside funding. One tier is the Evaluation Policy Oversight Committee (EPOC), and the other is the Technical Merit Review Committee (TMRC). The EPOC includes representatives from the Office of the Director, NIH, and ICD representatives at the level of Director, Deputy Director of an ICD, or Associate Director of the Office of the Director. EPOC conducts policy- level concept reviews of proposals for NIH-wide evaluation studies that use set-aside funds, establishes the overall NIH set-aside budget, and oversees the process. EPOC recommendations are approved by the Director, NIH, or designee before the initiation of any study. The TMRC is responsible for the technical review of the submissions and for recommending to the EPOC whether a project fits within departmental guidelines for the set-aside fund.

Evaluations and evaluation priorities pertaining to individual ICDs are shaped by ICD Directors and Deputy Directors. The results help ICDs and the Director, NIH, establish priorities, develop long-range goals and strategies, and review programs in terms of scientific excellence, relevance, cost, and uniqueness.

Summary of FY 1994 Evaluations The eight evaluations completed in FY 1994 addressed almost all elements of NIH's mission, from research to public education. Four of the eight evaluations are highlighted in chapter II. The evaluations summarized below illustrate the diversity of the NIH Evaluation Program through a commitment to evaluating a body of research that informs public policy, evaluating the Nation's need for scientific manpower, and evaluating public health information.

One of these four evaluations addressed the safety of selected childhood vaccines. This study was mandated by Congress under Section 313 of Public Law 99-660 to yield essential information that would help the Public Health Service draft recommendations about the use of, and compensation for adverse reactions to, vaccines against tetanus, diphtheria, measles, mumps, polio, Haemophilus influenzae type b, and hepatitis B. The study entailed an expert review by a committee of the Institute of Medicine. The committee examined all relevant medical and scientific literature on the potentially serious risks associated with currently licensed childhood vaccines. Its findings about each vaccine have been incorporated into brochures given to parents before children are vaccinated and into proposals to revise the list of compensable injuries presumed to have been caused by certain vaccines.

Another evaluation was on national needs for biomedical and behavioral research personnel. This evaluation addressed the Nation's future need for biomedical and behavioral research scientists and the contribution of NIH training grants called National Research Service Awards (NRSAs). The study was the 10th in a continuing series of reports to NIH and the U.S. Congress on this topic.

The National Research Service Award Act of 1974 consolidated all previous training authorities into the NRSA program. The Act authorized both predoctoral and postdoctoral support to individuals and to institutions. To implement the Act, NIH set up individual fellowships and grants to institutions for training predoctoral and postdoctoral students. Close to $400 million is spent annually on these training grants. A National Research Council expert committee, under contract to NIH, found that although the NRSA program is relatively small in terms of the total number of trainees, it is enormously powerful in its ability to change research emphasis and to attract the highest quality individuals to research careers. It is viewed as a prestigious, highly competitive program, and it is clear that initiatives introduced through the NRSA program can have a powerful impact on intended new research directions or constituencies. A final evaluation was performed to determine the feasibility of assessing NIH-supported research to increase condom use. This project was a feasibility study, or evaluability assessment, to determine if an outcomes evaluation could be performed in a second phase and if it would be useful to do so. The evaluation was motivated by public health efforts to prevent the spread of AIDS through the use of condoms. Its objectives were to assess the findings of condom use research efforts, guide the development of future program areas, and suggest methodological guidelines to facilitate the evaluation of future condom research programs.

The evaluation identified and inventoried the universe of condom use research studies supported by NIH grants. A reproducible methodology that combined automated database searching of NIH grants with human judgment was established. The methodology identified more than 500 relevant studies. A sample of 76 studies was examined in detail to identify how well demographic characteristics were defined, which sampling methodologies were employed, whether and what type of comparison group was used, and so on. The final report was widely disseminated. The evaluation was valuable because it generated useful methodological tools for meta-analysis and because it led to a decision not to pursue a larger scale evaluation. What emerged was a research agenda that the ICDs (seven had participated in the technical advisory group for the evaluation) could use to address the vital questions related to condom use most closely related to their institutional missions.

Evaluations in Progress NIH has 24 evaluations in progress. They range from small- to large-scale assessments, from evaluability studies to full-blown evaluations. One study builds on a longstanding role in ensuring the training and continued availability of superior biomedical and behavioral scientists through the NRSAs. The first study objective is to conduct an evaluation design study, that is, develop a detailed plan for a comprehensive evaluation of career outcomes of the predoctoral and postdoctoral trainees and fellows and the NRSA programs in which they have participated. The second objective is to develop an approach to characterize the nature and quality of the training actually experienced by present and former trainees and fellows and to differentiate between a good training program and simply good trainees. No baseline data are available on trainees, and program versus selection effects have not been studied. The third objective is to develop an approach to tap the perceptions of NIH staff, present and former NRSA trainees and fellows, and university administrators about the nature and impact of the training program. This study is expected to be completed in 1995.

A second example is a study of the Physician Data Query (PDQ), a comprehensive cancer database intended primarily for cancer health professionals. The database contains state-of-the-art treatment summaries and information on supportive care, screening, prevention, and experimental drug therapies. The objectives of the study are to survey PDQ database users to determine who is using the database and how the information is used, and to assess user satisfaction with the information and the method of retrieval, for example, CD-ROM, on-line, or hard copy.

The National Cancer Institute is directing a study to identify ways to increase target audiences. The study will produce a written summary of the activities and analyses of the project, data tapes and documentation of all questionnaire responses, and a computerized system of data collection, tabulation, and analysis. It will also include suggestions for improvements to the PDQ database to ensure that it meets user needs.

The third evaluation examines research resources available to primate researchers throughout the Nation. Through special grants, NIH supports seven Regional Primate Research Centers, a unique national network of nonhuman primate research and resource laboratories established in the early 1960s. The centers are geographically located throughout the United States, and each is closely affiliated with an academic institution. Center activities include the conduct of biomedical and behavioral research; research resource support to investigators funded by other sources; the maintenance of more than 18,000 nonhuman primates; the establishment of breeding programs to meet the centers' research requirements; conservation and preservation programs; expert professional and technical support to investigators; and training of pre- and postdoctoral professionals in primatology research. When the nonhuman primate is the most appropriate species to study, the centers provide a cost-effective response to the need for national repositories of nonhuman primates, scientific expertise, and specialized facilities and equipment.

The evaluation is assessing all aspects of the Regional Primate Research Centers program, such as its effectiveness in meeting current program objectives and guidelines, scientific distribution and emphasis of research programs, future planning decisions, and collaborations with and access by non-center investigators. Other issues to be addressed are compliance with policies and guidelines, financial management, host relationship, grant award process, peer review, status in the scientific community, and reporting and dissemination of program results. The centers are being compared with several selected non-center institutions with significant NIH-supported nonhuman primate research. The results of the study will be used to improve and refine the centers program.

New Directions for Evaluation In peer review, for example, NIH has been examining streamlining review of grant applications to make more effective use of reviewers' time. Streamlining has been tried on an experimental basis and is now being implemented fully. The results will be closely watched. The Government Performance Results Act (GPRA) of 1993 requires NIH to develop a strategic plan, an annual performance plan, and performance indicators by September 1997. This effort is currently a central focus of NIH evaluation activities. NIH is exploring a variety of indicators for potential areas in the strategic plan such as information dissemination, technology transfer, and its reinvention program. In addition, priority will be given to funding 1 percent set-aside project proposals submitted by the ICDs and the NIH Office of the Director that relate to the GPRA. Also, the EPOC will reexamine and redefine NIH evaluation priorities for the future.

Substance Abuse and Mental Health Services Administration

MISSION: To improve the quality and availability of prevention, treatment, and rehabilitation services for substance abuse and mental illness.

SAMHSA Evaluation Program The Substance Abuse and Mental Health Services Administration (SAMHSA) is committed to evaluating its overall programs and individual grant projects to assess--

• the effectiveness of treatment and prevention approaches and systems of care;

• the accountability for Federal funds; and

• the achievement of SAMHSA's programmatic and policy objectives.

To the greatest extent appropriate and feasible, SAMHSA will encourage the use of comparable data elements and instruments across its evaluations in order to work toward a comprehensive evaluation system and to minimize respondent burden.

SAMHSA conducts grant programs under a variety of legislative authorities. These authorities can generally be grouped into two types: services and demonstrations. The evaluation required for a particular grant program depends on the type and purpose of the program. SAMHSA evaluates its service programs to provide information to program managers about the accountability of Federal funds. Its evaluations of demonstration programs generate new knowledge to lead the field in developing policies that improve services. Evaluation results from the two types of programs are used to improve the performance of SAMHSA and the field of substance abuse and mental health services. Program and evaluation staff work together to identify the questions or goals each grant program should address and to propose appropriate evaluation strategies.

The two types of grant programs represent two facets of SAMHSA's mission: service delivery and knowledge development. SAMHSA's leadership in the field depends on the successful interaction of these two facets of its mission. SAMHSA uses evaluation to identify effective approaches to prevention and treatment and uses service delivery funds to provide incentives to the field to implement effective approaches. Major emphases of SAMHSA's mission are the development, identification, and dissemination of effective strategies and systems for treatment and prevention.

SAMHSA is now implementing a new, integrated model of evaluation and planning. Strategic planning will identify priorities, such as managed care, that drive the development of grant programs and evaluations. In compliance with the Government Performance and Results Act (GPRA), SAMHSA is attempting to improve performance by identifying performance goals and indicators as part of its strategic planning process. The formulation of programmatic and evaluation priorities will include consultation with the SAMHSA and center advisory councils and with other experts in the fields of evaluation and service delivery. Early and continuous coordination of program planning and evaluation design will result in the articulation of evaluable programmatic objectives. Evaluations will show how well the overall grant programs have achieved their objectives, and SAMHSA will translate these results into information that can be used for program and policy development. The strategic planning and policy development processes will then use these results to refine SAMHSA's priorities and objectives.

This evaluation policy will help SAMHSA achieve its goal of continually informing policy and program development with knowledge culled from past performance. In this way SAMHSA can best serve its customers by enhancing the quality of public substance abuse and mental health services.

In compliance with the Public Health Service (PHS) guidelines for the technical review of evaluations, SAMHSA has a standing review committee of PHS staff who are evaluation specialists from Agency for Health Care Policy and Research, Centers for Disease Control and Prevention, Health Resources and Services Administration, National Institutes of Health, and SAMHSA. Representatives of the Office of the Assistant Secretary for Health and the Office of the Assistant Secretary for Planning and Evaluation serve as ex officio members, and the SAMHSA evaluation coordinator serves as committee chair.

Evaluation project proposals are generally prepared by SAMHSA program staff in the various centers. The standing committee reviews each proposal on the following criteria: clarity of evaluation objectives; appropriateness and feasibility of the specifications for evaluation design and methods; appropriateness of the plans for dissemination of results; and use of previous relevant evaluations and existing program data systems.

Each proposal must clearly state the relationship of the evaluation to SAMHSA's overall policy priorities and evaluation program. The committee generally reviews proposals for broad, comprehensive evaluations of SAMHSA program activities, such as the cross-site evaluations of demonstration grant programs. It does not review the evaluation proposals of individual grantees.

Summary of FY 1994 Evaluations During FY 1994, SAMHSA completed six evaluations. The six studies reflect some of the population groups SAMHSA has identified as being in greatest need of substance abuse and mental health services. These include pregnant and postpartum women and their infants, children with serious emotional disturbance, high-risk youth, and the homeless mentally ill.

The evaluation of a demonstration program serving pregnant and postpartum women and their infants examined the results of improved coordination, availability, and accessibility of health and substance abuse-related services. A substantial number of the women served by these programs reduced their substance use, and their babies were generally healthy. SAMHSA will use these findings to encourage improvements in the performance of service programs for substance-abusing pregnant and postpartum women and their infants.

SAMHSA completed the design for an evaluation of a comprehensive community services program for children with serious emotional disturbance. The purpose of the evaluation is to collect information on client and systems outcomes. The findings should help both policymakers and service providers improve the performance and cost-effectiveness of mental health services available to children.

Another evaluation conducted this year examined the cost-effectiveness of a drug treatment enrichment program in curbing drug use among students receiving training at Job Corps Centers. Job Corps Centers are residential employment and training programs for high-risk youth. Students participating in the enrichment program showed a significant decrease in drug use compared with a control group. The findings of this study are useful in verifying the effectiveness of substance abuse treatment and in informing the field about appropriate substance abuse services for high-risk youth.

An evaluation of a demonstration program for homeless mentally ill adults is highlighted in chapter II of this report. This study examined the results of improved linkage, integration, and availability of housing and social and medical services. The program was successful in reducing homelessness and improving the mental health of study participants. These results are encouraging communities to better integrate services for the homeless mentally ill.

An evaluation of demonstration projects for community partnerships to prevent substance abuse has produced information on the successful implementation of substance abuse prevention approaches that employ a comprehensive, coordinated, communitywide strategy. Evaluation results of the impact of the partnership program are forthcoming. The Second Report to Congress on Alcohol and Drug Abuse Prevention presents preliminary results of the National Structured Evaluation (NSE). The NSE is a systematic assessment of substance abuse prevention approaches implemented across the country. Final results of this study, to be presented in a third and final report to Congress, will have important implications for improving the performance of substance abuse prevention programs.

Evaluations in Progress SAMHSA currently has 10 major evaluations under way in the following general areas: program accountability, evaluation of demonstrations, reinforcing behavioral health, and commitment to customer service. Each type is described below, with some examples.

Program accountability evaluations are undertaken to inform program management and help managers refine program operations. This is the primary type of evaluation conducted on SAMHSA's service grant programs. For example, the children's mental health service program has an ongoing evaluation. The evaluation will yield continuous information on program implementation and on outcomes for children and families served. This information will be used for reports to Congress, feedback to grantees, program development, and performance improvement.

Evaluations of demonstrations are designed to generate new knowledge for policy development. The primary purpose of SAMHSA's demonstration programs is to generate new knowledge to lead the field in the development of policies that improve services. For example, evaluation results on substance abuse prevention for high-risk youth will allow policymakers to draw inferences about the effectiveness of certain interventions for this population. The Job Corps evaluation will continue to assess an enriched substance abuse treatment program for adolescents. An evaluation of the program for access to community care and effective services and supports (ACCESS) will provide information for the design of ongoing service programs at the Federal, State, and local levels.

Other SAMHSA evaluations strive to reinforce behavioral health as a cornerstone of public health. In the context of managed care and health care reform, SAMHSA's interagency collaborations promote the integration of services and reinforce the role of alcohol, drug, and mental health services as a critical component of general health care. Illustrations of such activities include collaboration with the Administration on Children and Families to evaluate a substance abuse prevention program for pregnant and postpartum women and infants.

The National Treatment Improvement Evaluation Study (NTIES) was designed as one of a family of studies of substance abuse treatment, including two studies conducted by the National Institutes of Health (NIH). This collaboration will enhance SAMHSA's ability to draw inferences about treatment effectiveness. SAMHSA will continue working with the Department of Housing and Urban Development to design a method by which therapeutic communities can be incorporated into public housing areas. An HIV/AIDS mental health services demonstration program is jointly funded by NIH and HRSA.

Because of its commitment to the recipients of alcohol, drug abuse, and mental health services, SAMHSA uses evaluation to guide the field in improving the integration of services and their responsiveness to the needs of clients. For example, the evaluation of the community partnership program will determine the part-nership's ability to achieve integration and responsiveness at the user level. Primary purposes of the community coalitions demonstration program are to increase the efficiency of services at the community level and to enhance the service delivery system's responsiveness to its users. The evaluation of the program will assess the effectiveness of this approach.

SAMHSA has promulgated a comprehensive treatment model that includes a broad array of treatment and ancillary services targeted for individuals whose substance abuse has severely affected many domains of their lives. The services include housing, child care, transportation, social services assistance, legal aid, and mental and physical health screenings and treatment, as appropriate. Every SAMHSA grant announcement for substance abuse treatment will require that grantees implement the comprehensive treatment model. Evaluations will assess the effectiveness of the model.

New Directions for Evaluation SAMHSA is designing its evaluation activities to guide programmatic and policy decisions. The evaluation activities will complement the SAMHSA and PHS strategic plans and will respond to emerging trends such as managed care and health care reform. For example, in FY 1995 SAMHSA will initiate a program of State contracts to evaluate managed care and treatment outcome. The program will provide support to States to conduct short- and long-term studies at the State and provider levels of the effects of managed care on substance abuse treatment access, cost, and outcomes. SAMHSA will support States' use of evaluation methodologies including State financial, provider, and client databases.

These evaluation activities will reinforce the critical role of behavioral health in general health care. Results of the evaluations will ensure accountability and generate new knowledge. SAMHSA will use these results to improve integration of services and responsiveness to the needs of individuals suffering from or at risk for alcohol, drug abuse, and mental health problems.

Office of the Assistant Secretary for Health

MISSION: To provide public health advice to the Secretary of Health and Human Services and executive direction to the PHS Agencies through coordination and oversight of their programs.

OASH Evaluation Program The primary role of the Office of the Assistant Secretary for Health (OASH) is to coordinate and develop evaluations across the entire Public Health Service. The Assistant Secretary for Health (ASH) issues guidance each fiscal year to the PHS Agencies for preparation of their plans outlining evaluation priorities and projects proposed for implementation in that fiscal year (as mentioned in chapter I). OASH reviews the Agency plans to identify gaps in evaluation data needed for program or policy development. OASH also identifies potential program areas that could benefit from a collaborative evaluation.

In addition to its PHS-wide coordination function, OASH uses the 1 percent set-aside funds to initiate its own evaluations or policy studies, usually in consultation with relevant Agency program offices. The projects conducted by OASH are generally short, quick-turnaround policy studies to support initiatives of the ASH. Occasionally, OASH will initiate or participate in large-scale evaluations, particularly those that cut across PHS program areas.

The OASH review and approval of evaluation projects is performed by the Evaluation Policy Review Committee (EPRC), which makes recommendations to the ASH on OASH evaluation priorities for each fiscal year; reviews project proposals for relevance to PHS strategic planning objectives; and provides advice to the ASH in determining the OASH use of 1-percent evaluation funds and allocation of those funds to approved projects. Among the criteria used by the EPRC to review proposed projects are relevance to ASH priorities for policy, legislative, budget, program development, or crosscutting PHS issues; relevance to departmental and PHS strategic planning objectives and related evaluation strategies; coordination with other PHS and departmental offices; and relevance to the Healthy People 2000 objectives. OASH evaluation staff review all EPRC-approved projects for technical quality and feasibility, providing feedback to OASH program offices on project proposals and statement of work for contract procurements. In addition, the staff serves as a resource group for individual technical assistance and consultation to project officers in developing evaluation proposals.

Summary of FY 1994 Evaluations In FY 1994, OASH completed 16 evaluations in diverse areas of public health: health care reform, immunizations, adolescent and school health, nutrition, primary care, emergency preparedness, international health, and other smaller regional studies of health services delivery. Most of these evaluations focused on policy analysis to support initiatives in health care reform. Despite the failure of health care reform legislation, these studies are relevant because health care reform proposals are still being considered by Congress and because market-based reforms are occurring in the absence of legislation.

One study on the returns on investment in public health examined the contributions of public health programs in reducing the incidence of preventable illnesses and injuries. This work summarized the effectiveness of public health strategies from a cost-benefit analysis perspective to underscore the importance of the public health infrastructure in realizing the two primary goals of health care reform: to expand access to care and to control costs.

Two related studies examined the implications of health care reform on the health care workforce. The study on a proposed Graduate Nursing Education Account, which is highlighted in chapter II, documented the projected scarcity of advance practice nurses such as nurse practitioners, certified nurse-midwives, certified registered anesthetists, and clinical nurse specialists. It analyzed several options for channeling funds to support advance practice nursing education to meet future needs. Another study examined the supply, training, and distribution of primary care providers as part of health care reform efforts. It examined the rationale for new proposals to control the supply of new physicians, proposals for an all-payer pool for graduate medical education, and estimates of the number of advanced practice nurses that would be needed as substitutes in the case of a reduction in the number of physicians.

In recent years, OASH has given priority to evaluations of PHS immunization policies and programs. One study, also highlighted in chapter II, examined the economic underpinnings of the vaccine supply. A major finding was that States that supply vaccines at low prices to physician offices and encourage parents to have their children vaccinated have higher rates of immunization. In another study, OASH developed a design to assess the functional capabilities of the Vaccine Adverse Event Reporting System (VAERS). This design project will help PHS officials evaluate VAERS to determine whether it is providing reliable and accurate information, and how health care providers, manufacturers, and consumers perceive the effectiveness of the system.

Support for nutrition policy studies has been another OASH evaluation priority. In FY 1994, OASH completed a project with the American Institute of Nutrition to examine state-of-the-art techniques for measuring nutritional status and to make recommendations for a core set of nutritional status indicators for low-income populations. Development of these indicators will be important for the development of national and State nutrition monitoring systems, particularly in expanding the survey coverage of low-income populations.

Adolescent health, including school heath education programs, is another OASH program evaluation priority area. Two projects in this area, both described in chapter II, were completed in FY 1994. The first study sought ways to evaluate the academic benefits of school health programs. It developed a general framework, with alternative research designs, for assessing the effects of school health interventions on students' school performance. The second study was a focus group study of the knowledge, attitudes, behaviors, beliefs, and environments of youth engaged in high-risk behaviors. The results of this study will be used to develop appropriate health messages to youth about high-risk behavior.

Evaluations in Progress OASH has 14 program and policy evaluations under way in two major areas: health care reform and population-based services. In addition, OASH is supporting several smaller projects investigating crosscutting PHS issues and several evaluations specific to OASH program offices. Some of the evaluations that are likely to be completed in FY 1995 are described below.

With respect to health care reform, OASH is continuing to examine two major issues: the role of Federal, State, and local health information systems in achieving public health objectives, and the impact of health care reform on the health care workforce and the role of academic health centers (AHCs).

Health information systems at all levels of government are the focus of three projects. One is developing sets of indicators that communities can use to assess performance and to foster closer collaboration between public health and personal health care systems and employers in achieving specific public health goals. Another is reviewing current State laws, policies, practices, and environments governing public health reporting systems, health care information systems, health data confidentiality, and access and linkage policies. This project will identify aspects of existing laws, policies, or practices that create barriers to standardization, integration, data linkage, and access relating to health information systems within and across States. Third, OASH is supporting the National Committee on Vital and Health Statistics in a project to conduct analyses for the development of uniform health data sets for enrollment and encounter information from health care providers in reformed health care systems. Data derived from the uniform data sets will provide a foundation for monitoring and evaluating public health programs as well as systems of care in the framework of national health information networks.

Health care reform's impact on the health care workforce and on the AHCs, where many health professionals are trained, is the subject of two projects. First, OASH initiated a project to study how the growth of competitive health care financing and service delivery systems based on managed care affects the financial support available to AHCs and graduate medical education. The project focuses on AHCs in three cities: San Diego, Minneapolis-St. Paul, and Washington, D.C. Another project is examining the extent to which teaching hospitals associated with the AHCs are reorganizing and/or reducing services and other activities in areas in which competitive health plans hold a substantial share of the market. The project will assess the impact of such changes on hospital training programs for physicians.

In addition to these analytic studies, OASH will support a series of roundtable discussions among leading national experts in government, academic, clinical, and research settings regarding the role of AHCs in an era of health care reform. The purpose of these discussions is to exchange views and reach a national consensus on questions of the potential impact of health care reform on AHCs. Many questions are being raised about their mission, financial capability, and contribution to patient care and public health; how they can best participate in a national shift from training specialist physicians to training primary care practitioners; how health care reform affects clinical research; and what strategies may best protect the quality of clinical research and its contributions to public health.

The OASH evaluation program has also given priority to evaluations in the area of clinical preventive services. The Office of Disease Prevention and Health Promotion (ODPHP) will soon complete the design for the "Put Prevention Into Practice" initiative, a package of resource materials for health personnel in clinical settings to promote more effective use of clinical preventive services for their patients. The design will be implemented in late FY 1995. The results of this evaluation will help national primary care provider organizations determine the optimal use of these materials in their respective clinical settings.

ODPHP is also supporting a major effort to improve methodologies for evaluating the cost-effectiveness of clinical preventive services. A group of specialists in cost-effectiveness analysis, both inside and outside government, is working to develop guidelines to apply cost-effectiveness methods to clinical preventive services. The group will address current controversies about these methods and propose alternatives for improving the comparability of cost-effectiveness analyses.

As mentioned above, OASH often initiates evaluations of programs or policy issues that cut across the PHS Agencies. Several OASH projects in progress illustrate collaborative evaluations. Since 1993, OASH has worked with the Food and Drug Administration (FDA), the National Institutes of Health (NIH), and the Substance Abuse and Mental Health Services Administration to evaluate the departmental methadone regulations, a project expected to be completed in FY 1995. The study is examining the impact of Federal regulations on the provision of methadone treatment services and on the development of new anti-addictive medications. The final report will contain recommendations for improving the current methadone regulations.

In another project, OASH is working with the Centers for Disease Control and Prevention, the FDA, and NIH to study HIV transmission to hemophiliacs through blood products. In the early period of the AIDS epidemic, before the invention of rapid laboratory tests for identifying HIV infection, blood transfusion services for hemophiliacs operated without effective HIV screening or production methods to eliminate the virus. The Secretary of the Department of Health and Human Services, Donna Shalala, launched an investigation in 1993 to get a more complete understanding of the events that occurred in those early years. The results, expected in FY 1995, will be helpful for ensuring the safety of the Nation's blood supply against future challenges.

OASH also uses evaluation funds to support projects that measure the effectiveness of program activities managed by the various OASH Offices. For example, the Office of Minority Health presently is evaluating its HIV/AIDS Education/Prevention Grant Program. The grants program is providing assistance to community-based organizations in developing innovative approaches for reaching persons living in minority communities who are at risk for HIV infection. The evaluation is now analyzing data on disease prevention intervention models that have a likelihood of successful replication in other minority communities.

Another example of OASH evaluation activity is the efforts of the OASH Office of Research Integrity (ORI) to examine the consequences of being accused of research misconduct, as well as the consequences for the whistleblower. The results of these two projects will be useful to ORI for monitoring institutional compliance with PHS regulations on research misconduct, planning educational activities, and making improvements in PHS policies and procedures for responding to allegations of research misconduct.

New Directions for Evaluation The PHS FY 1995 Strategic Plan has a goal to strengthen the public health infrastructure. In recognition of the danger faced by health departments as a result of years of shrinking budgets, the PHS Agencies and OASH are striving to augment the capability of the Federal Government, tribal governments, States, and communities to identify and address high-priority health problems for their populations. One approach is to expand the public health knowledge base by supporting an appropriate balance of focused and multidisciplinary research in the areas of basic biological science, clinical medicine, public health practice, behavioral and social sciences, epidemiology, health systems and services, nutrition, and occupational and environmental health. In coming years, OASH will support the development and implementation of measures that will assess the capacity of State and local public health agencies to perform population-based functions and the extent to which States meet PHS program goals.

Managed care is changing the way health care is paid for and delivered and shifting responsibilities for care. This has serious implications not only for vulnerable populations but also for public health "safety net" providers, academic health centers, and public health agencies. This year, OASH will participate with the PHS Agencies and the Health Care Financing Administration to examine the impact of managed care on public health systems, primarily through demonstrations of the Medicaid Waiver (Section 1115) in five States. The Medicaid Waiver allows States to experiment with alternative approaches to improving the access and quality of health care services for the uninsured and reducing the costs of care. The analysis is expected to focus on the number of eligible participants, the costs to providers, the quality and types of care, and the access for special populations--such as American Indians and Native Alaskans, children with special health care needs, people in underserved areas, substance abusers, and the chronically mentally ill.

Appendix A. Abstracts of Completed PHS Evaluations Fiscal Year 1994

Abstracts of Completed PHS Evaluations, Fiscal Year 1994

The abstracts below provide brief descriptions of the 71 evaluation reports completed by Public Health Service Agencies and the Office of the Assistant Secretary for Health (OASH) during fiscal year 1994. The abstracts are listed by alphabetical order of the Agencies, followed by OASH. The sponsoring Agency, report title, abstract, and Federal contact person(s) are listed for each evaluation report. For more information on any of the studies, please see the contact person(s) listed at the end of the abstract. It may be possible to obtain copies of some evaluations in two ways:

Policy Information Center Final reports for most of the studies listed in this appendix have been submitted to the Department of Health and Human Services (DHHS) Policy Information Center (PIC), a centralized source of information on in-process, completed, and ongoing health and human services evaluations; short-term evaluative research; and policy-oriented projects. The PIC maintains a resource database containing information on more than 6,000 completed and ongoing studies sponsored by DHHS, other Federal Agencies, and private sector entities.

The PIC identification number appears after the Federal contact person(s) information. The PIC provides executive summaries (where available) at no cost. For more information about using PIC resources, please contact--

Policy Information Center
Office of the Assistant Secretary for Planning and Evaluation
Department of Health and Human Services
Room 438F, Hubert H. Humphrey Building
200 Independence Avenue SW
Washington, DC 20201
(202) 690-6445

National Technical Information Service The National Technical Information Service (NTIS) is an information clearinghouse run by the Department of Commerce. It provides reports in both paper and microform formats for a fee. If a final report has been registered with this service, the NTIS accession number follows the Federal contact person(s) information. For more information about ordering copies of Government reports from NTIS, please contact--

National Technical Information Service
Department of Commerce
5285 Port Royal Road
Springfield, VA 22161
(703) 487-4650

Agency for Health Care Policy and Research

Listing of Abstract Titles

Analysis of Peer and Pilot Review Data Clinical Practice Guidelines

Evaluation of Cost Analyses Conducted on Clinical Practice Guidelines: Final Report

Identification of Data Sources Appropriate for Use in Conducting Cost Analysis Required for the Development of Clinical Practice Guidelines: Final Report

Overcoming Barriers to Integration and Implementation of Clinical Information Management Systems

Report on Inappropriate Extension of Hospitalization

Research on the Cost Effectiveness of Managed Care Health Plans: A Literature Review

TITLE: Analysis of Peer and Pilot Review Data Clinical Practice Guidelines

ABSTRACT NUMBER: 001

ABSTRACT: The Agency for Health Care Policy and Research (AHCPR), through its Office of the Forum for Quality and Effectiveness in Health Care (Office of the Forum), is mandated by Congress to develop clinical practice guidelines. This development process includes a review of draft guidelines by independent health practitioners and other relevant professionals to provide information about their quality and potential usefulness (peer review). Additional information can be obtained about a guideline's value if its recommendations are assessed in patient care situations by practicing clinicians (pilot review). The purpose of this project was to review and analyze the results of peer and pilot testing for six AHCPR-sponsored guidelines and to develop recommendations for future peer review and pilot testing methodologies. This contract culminated in a report that offered several recommendations for improving peer and pilot review procedures, including limiting the number of peer and pilot reviewers to 50; increasing the number of primary care, consumer advocate, and nonphysician reviewers; improving the yield of completed reviews by contacting potential reviewers in person; providing improved instructions to reviewers; providing longer timeframes for organizational reviews; and drafting a rationale for changes recommended for a guideline when review comments are evaluated and saving this draft as documentation of the panel's deliberations.

FEDERAL CONTACT OFFICE: Office of Program Development

PERSON: Irma Arispe

PHONE NUMBER: 301/594-1455

PIC NUMBER: 5651

PERFORMER ORGANIZATION: Harvard University School of Public Health, Boston, MA

P.I.: R. Heather Palmer, M.D.

TITLE: Evaluation of Cost Analyses Conducted on Clinical Practice Guidelines: Final Report

ABSTRACT NUMBER: 002

ABSTRACT: This study examined the methodologies and data sources used in cost studies for eight clinical practice guidelines (urinary incontinence, pressure ulcers, cataract, depression, low back pain, otitis media, benign prostatic hyperplasia, and sickle cell anemia). It was found that the primary distinction among the studies was whether they estimated treatment costs under the guidelines. Analysts in the studies that did not estimate treatment costs felt it would require too many assumptions to do so. They were concerned about the assumptions that must be made to estimate treatment costs and made the following recommendations: reduce the number of assumptions necessary to estimate treatment costs and restrict the empirical analysis to the direct medical costs of treatment; compare treatment patterns and costs before and after guideline implementation; use a variety of data sources--such as administrative claims, clinical trials, and medical record abstractions; use episodes of care as the unit of analysis, to provide a link between the clinical and economic dimensions of the guidelines; use a substantive expert as analyst, so that he or she is able to start on the description of current practice immediately without waiting for guidance from the panel; and emphasize the importance of the cost studies to the panels since the panels are typically made up of medical professionals who may place greater emphasis on clinical practice and outcomes and less emphasis on cost.

FEDERAL CONTACT OFFICE: Office of Program Development

PERSON: Irma Arispe

PHONE NUMBER: 301/594-1455

PIC NUMBER: 5652

NTIS ACCESSION NUMBER: PB 95-193694

PERFORMER ORGANIZATION: SysteMetrics, Cambridge, MA

P.I.: William Crown, Ph.D.

TITLE: Identification of Data Sources Appropriate for Use in Conducting Cost Analysis Required for the Development of Clinical Practice Guidelines: Final Report

ABSTRACT NUMBER: 003

ABSTRACT: Databases that provide cost information relating to clinical guidelines have not been established. However, several databases have been identified as worthy of investigation because of their potential utility in estimating costs of guideline components. The purpose of this study was to investigate the content of these databases, evaluate their usefulness, develop a cost-measurement methodology using these existing databases, and describe how the methodology could be used for selected guidelines. It was determined that no one database can be used to estimate the costs of clinical guidelines. Since often there are no data on certain services in claims databases or there are significant questions about the accuracy of available data, data must be assembled from several sources. Per-patient costs for current and guideline patterns need to be multiplied by data on the prevalence of the condition to arrive at aggregate cost for the United States. In this calculation, differences in per-patient costs are projected to understand overall impact.

FEDERAL CONTACT OFFICE: Office of Program Development

PERSON: Irma Arispe

PHONE NUMBER: 301/594-1455

PIC NUMBER: 5653

NTIS ACCESSION NUMBER: PB 95-193652

PERFORMER ORGANIZATION: Center for Health Policy Studies, Columbia, MD

P.I.: Benjamin Dugger, Sc.D.

TITLE: Overcoming Barriers to Integration and Implementation of Clinical Information Management Systems

ABSTRACT NUMBER: 004

ABSTRACT: The purpose of the feasibility study was twofold: (1) to determine the practicality of studying the barriers to implementation and integration of advanced clinical information systems and (2) to specify the type of research design for an implementation study to investigate these issues. Despite the promise of clinical information systems, there have been few concerted research efforts to understand the barriers that impede dissemination or the benefits that accrue from implementation. The report presents the design for a major, and feasible, research study of integrated health care information systems. With written letters of cooperation from 24 institutions, the project would study 8 to 16 sites (depending on funding level) representing 4 to 8 major vendor products and would focus on both outcomes of implementation and barriers to adoption of system features. The report provides details on study feasibility, site selection, research design, and a data collection plan.

FEDERAL CONTACT OFFICE: Office of Program Development

PERSON: Irma Arispe

PHONE NUMBER: 301/594-1455

PIC NUMBER: 5039

NTIS ACCESSION NUMBER: PB 95-193371

PERFORMER ORGANIZATION: ABT Associates, Cambridge, MA

P.I.: Catherine Joseph

TITLE: Report on Inappropriate Extension of Hospitalization

ABSTRACT NUMBER: 005

ABSTRACT: The purpose of this contract was to perform a literature review and interview experts about inappropriate hospital-stay extension and the role of discharge planning, and to produce a summary document identifying research needs and potential existing data sources. The project was undertaken to address a Senate Appropriations Committee request for information on the extent to which inappropriate hospital duration occurs, the reasons for such inappropriate extension of hospitalization, and the extent to which hospital discharge planning can reduce inappropriate continuation of hospital stays. The project also describes best practices in hospital discharge planning. Key findings of the project are these: 5 percent or more of total hospital days are likely to be accounted for by extended stays that are not medically necessary, the cost of which is conservatively estimated at $420 million in 1987 dollars; the reasons for extended hospital stays include lack of sufficient nursing home beds and other alternative placement settings, paperwork delays in qualifying patients for posthospital benefits, and delays in appointing conservators for incompetent patients. Discharge planning appears to have only a small impact on extended hospital stays, but it may be having greater impact on appropriateness of placement and the likelihood of readmission. The problem is pervasive, despite incentives for early discharge. A solution is more likely to be a systemic change coming from outside the hospital; or there may be some level beyond which reduction of such stays is infeasible.

FEDERAL CONTACT OFFICE: Office of Program Development

PERSON: Irma Arispe

PHONE NUMBER: 301/594-1455

PIC NUMBER: 5655

NTIS ACCESSION NUMBER: PB 95-193710

PERFORMER ORGANIZATION: Research Triangle Institute, Washington, DC

P.I.: James S. Lubalin, Ph.D.

TITLE: Research on the Cost Effectiveness of Managed Care Health Plans: A Literature Review

ABSTRACT NUMBER: 006

ABSTRACT: The purpose of this project was to identify, compile, categorize, and critically analyze published and unpublished literature on the cost-effectiveness of managed health care plans, to draw conclusions about the cost-effectiveness of various approaches to these plans and indicate important remaining policy-relevant gaps in knowledge. Managed care plans were compared with indemnity (fee-for-service) plans with regard to utilization and cost performance. It was found that managed care plans had lower hospital utilization, greater use of less costly alternatives to expensive procedures and tests, greater use of preventive measures, comparable quality of care, and somewhat lower enrollee satisfaction generally but much greater satisfaction with cost. The study also concluded that policymakers need much more evidence on managed care plan performance. It recommends funding larger, more comprehensive research projects to determine what works in managed care, rather than just whether managed care works.

FEDERAL CONTACT OFFICE: Office of Program Development

PERSON: Irma Arispe

PHONE NUMBER: 301/594-1455

PIC NUMBER: 5654

PERFORMER ORGANIZATION: Center for Health Policy Research, George Washington University, Washington, DC

P.I.: Michele Solloway, Ph.D.

Centers for Disease Control and Prevention and Agency for Toxic Substances and Disease Registry

Agency for Toxic Substances and Disease Registry

Listing of Abstract Titles

Assessing Prevention Effectiveness: A Collaborative Effort With Selected Health Maintenance Organizations (Phase I)

Assessment of Information Dissemination Activities--Office on Smoking and Health

Congenital Syphilis: Estimated Direct Medical Care Cost for the United States--1990

Development of an Evaluation Plan for DCPC's State-Based Comprehensive Breast and Cervical Cancer Early Detection and Control Program

Estimating Expenditures for Chronic Hepatitis B

Evaluation of CDC and ATSDR Training Activities

Evaluation of National Health Interview Survey Diagnostic Reporting

Evaluation of Sexually Transmitted Disease Clinic Flow and Utilization

Evaluation of the Morbidity and Mortality Weekly Report Series

Lessons From Implementing State-Based Diabetes Control Programs: An Evaluation

Report of the Working Group To Revise the Model State Vital Statistics Act and Regulations

Vital Records Evaluation Using 1988 National Maternal and Infant Health Survey Data

TITLE: Assessing Prevention Effectiveness: A Collaborative Effort With Selected Health Maintenance Organizations (Phase I)

ABSTRACT NUMBER: 007

ABSTRACT: This project, the first of a two-phase study, reviewed existing health maintenance organizations (HMOs) to develop a framework and the process to assess prevention effectiveness; the characteristics, organization, and types of preventive services offered; and the potential for working in collaboration with CDC. Based on essential criteria developed to guide the process of site selection, a series of site visits was conducted. A data collection instrument and semistructured interview was conducted during each of the site visits. Strengths and weaknesses of HMO data collection systems were identified and recommendations were provided to CDC on which five HMOs would most appropriately meet the needs of CDC for phase two of the study.

FEDERAL CONTACT OFFICE: Office of Program Planning and Evaluation

PERSONS: Wilma Johnson, Nancy Cheal

PHONE NUMBER: 404/639-3453

PIC NUMBER: 4111

PERFORMER ORGANIZATION: Battelle, Arlington, VA

P.I.: Bryan R. Luce, Ph.D.

TITLE: Assessment of Information Dissemination Activities--Office on Smoking and Health

ABSTRACT NUMBER: 008

ABSTRACT: The purpose of this study was to develop recommendations for key management and operational aspects of the information dissemination function in the Office of Smoking and Health (OSH), with particular emphasis on the response system. Although the study focuses on information dissemination, its broader context is OSH's evolving leadership role within the tobacco control community. The project consisted of internal and external interviews. The external interviews were conducted with individuals representing Federal Agencies that might be using information approaches relevant to OSH. In addition to information dissemination models, reports of site visits, and telephone interviews, the final report contains recommendations to OSH on organizational considerations and strategic planning.

FEDERAL CONTACT OFFICE: Office of Program Planning and Evaluation

PERSONS: Wilma Johnson, Nancy Cheal

PHONE NUMBER: 404/639-3453

PIC NUMBER: 4684

PERFORMER ORGANIZATION: Macro International, Atlanta, GA

P.I.: Katie Baer, M.P.H.

TITLE: Congenital Syphilis: Estimated Direct Medical Care Cost for the United States--1990

ABSTRACT NUMBER: 009

ABSTRACT: The purpose of this study was to estimate the 1990 direct medical costs of congenital syphilis in the United States. Three types of costs were included in the study: medical care incurred during the first year of life; special education for infants who experience impairment due to congenital syphilis; and lifetime custodial care for infants who experience profound effects of congenital syphilis. After classifying cases by severity, researchers created a computer model that integrated incidence data with varying assumptions (such as case severity distribution, price level, etc.). Based on the analysis, it is estimated that the present value of costs for medical care, special education, and lifetime custodial care for the 1990 incident congenital syphilis cohort was in the range of $100-$200 million.

FEDERAL CONTACT OFFICE: Office of Program Planning and Evaluation

PERSONS: Wilma Johnson, Nancy Cheal

PHONE NUMBER: 404/639-3453

PIC NUMBER: 5301

PERFORMER ORGANIZATION: Battelle, Washington, DC

P.I.: Bryan R. Luce, Ph.D.

TITLE: Development of an Evaluation Plan for DCPC's State-Based Comprehensive Breast and Cervical Cancer Early Detection and Control Program

ABSTRACT NUMBER: 010

ABSTRACT: The primary purpose of this evaluation was to collect data pertinent to program-related decisions. The evaluation focused on State health agency breast and cervical cancer program components (public education, provider education, quality assurance, surveillance, screening, and followup) as well as the combined effect of program components. Short-term, intermediate, and long-term program and health outcomes were assessed. Assessment of program development and implementation was considered along with traditional evaluations of program efficiency and effectiveness.

FEDERAL CONTACT OFFICE: Office of Program Planning and Evaluation

PERSONS: Wilma Johnson, Nancy Cheal

PHONE NUMBER: 404/639-3453

PIC NUMBER: 4744

PERFORMER ORGANIZATION: Research Triangle Institute, Research Triangle Park, NC

P.I.: Thomas J. Cook, Ph.D.

TITLE: Estimating Expenditures for Chronic Hepatitis B

ABSTRACT NUMBER: 011

ABSTRACT: The purpose of this effort was to provide CDC with estimates of the direct medical cost of hepatitis B, particularly the acute care costs of services rendered to individuals identified as having or having had an episode of hepatitis B. These estimates were developed using recent paid claims histories from reimbursement files for persons recorded as having been treated for one or more specific diagnoses associated with hepatitis B. Data collection proceeded on a dual track: both a private market source and Medicaid data were obtained. The Medicaid data were procured from the repository from the Tape-to-Tape data development effort conducted for Health Care Financing Administration (HCFA). States that participated in this project supplied their machine-readable files to an HCFA contractor, and those files were translated into a single common format. The States from which Medicaid data were obtained for this study are California, Georgia, and Michigan. Private market data were obtained from United Health Care (UHC), an organization that directs and/or maintains data for managed care plans in a number of States. The review of these data and methods was successful in that generally consistent estimates were computed from each of the four jurisdictions. Further, there were no bizarre variations with respect to age, race, or sex. Unanticipated results and glitches in some areas might be resolved with further examination.

FEDERAL CONTACT OFFICE: Office of Program Planning and Evaluation

PERSONS: Wilma Johnson, Nancy Cheal

PHONE NUMBER: 404/639-3453

PIC NUMBER: 5802

PERFORMER ORGANIZATION: Battelle, Arlington, VA

P.I.: Ruth E. Brown, M.S.

TITLE: Evaluation of CDC and ATSDR Training Activities

ABSTRACT NUMBER: 012

ABSTRACT: As local health departments face fiscal upheavals, this study assessed their current and emerging training needs through focus groups and site visits. Training needs were compared with a newly developed inventory of training activities offered by CDC. Local health programs were found to need training in advocacy, evaluation, and public information. They were found to be deficient at training needs assessment, a formal way of identifying and monitoring the skills employees need to do their jobs better or to attain their career goals. CDC, which itself has limited training resources, is found to have no periodic mechanisms for staying in touch with State and local training needs. Recommendations, including those for new training technology (i.e., interactive software and satellite communications), are galvanizing CDC and ATSDR to develop training activities that dovetail with local needs.

FEDERAL CONTACT OFFICE: Public Health Practice Program Office

PERSON: Dennis McDowell

PHONE NUMBER: 404/639-3707

PIC NUMBER: 4745

PERFORMER ORGANIZATION: Macro Systems International, Atlanta, GA

P.I.: Thomas Chapel

TITLE: Evaluation of National Health Interview Survey Diagnostic Reporting

ABSTRACT NUMBER: 013

ABSTRACT: The purpose of this project was to evaluate diagnostic reporting in the National Health Interview Survey, which addressed major current health issues through the collection and analysis of national data on the civilian noninstitutionalized population of the United States. The survey provides national data on the incidence of acute conditions and injuries, the prevalence of chronic conditions and impairments, the extent of disability, and the use of health care services. The basic design of the evaluation study involved a comparison of information obtained from respondents to the basic health questionnaire, minimally modified, with reports obtained from the respondents' sources of health care. The primary objectives were to determine whether medical record documentation confirmed respondents' reports about selected conditions and impairments, to determine whether conditions mentioned in the record were mentioned by respondents, to assess the reporting and timing of doctor visits and hospitalizations, and to evaluate the validity of proxy reports to items in the questionnaire. The condition-related objectives are addressed in this final report; utilization data are addressed in an auxiliary report.

FEDERAL CONTACT OFFICE: Office of Program Planning and Evaluation

PERSONS: Wilma Johnson, Nancy Cheal

PHONE NUMBER: 404/639-3453

PIC NUMBER: 3199

PERFORMER ORGANIZATION: National Center for Health Statistics, Hyattsville, MD

P.I.: Deborah M. Winn

TITLE: Evaluation of Sexually Transmitted Disease Clinic Flow and Utilization

ABSTRACT NUMBER: 014

ABSTRACT: This study sought to study overburdening among sexually transmitted disease (STD) clinics in metropolitan areas with high STD morbidity. It assessed the impact of challenges to STD service delivery that developed in the late 1980s and early 1990s, including increasing incidence rates for STDs, addition of HIV services, retrenchment of funding, and more elaborate patient testing. These factors led to an inability in many clinics to see all clients who present for STD testing and/or treatment on a given day. The study sought to examine the effectiveness of STD clinics in providing services to those who seek care in the clinic and to clarify the factors that lead to overburdening.

FEDERAL CONTACT OFFICE: Office of Program Planning and Evaluation

PERSONS: Wilma Johnson, Nancy Cheal

PHONE NUMBER: 404/639-3453

PIC NUMBER: 4164

PERFORMER ORGANIZATION: Battelle, Arlington, VA

P.I.: Martha L. Hare, Ph.D.

TITLE: Evaluation of the Morbidity and Mortality Weekly Report Series

ABSTRACT NUMBER: 015

ABSTRACT: The Morbidity and Mortality Weekly Report (MMWR) is a principal means by which CDC communicates key information on a timely basis to the public health community. In recent years, the scope of the MMWR has broadened, reflecting the larger mission of CDC. This study sought to learn how well the MMWR is communicating information to this broader public health audience while continuing to convey timely information about outbreaks and infectious diseases. A mail/-telephone readership survey was given to 375 individuals on the MMWR mailing list, 375 subscribers to the MMWR through the Massachusetts Medical Society, and 250 primary care physicians who subscribe to the Journal of the American Medical Association (in which articles from the MMWR are reprinted). The study characterized selected audiences for the MMWR series, examined the public health impact of information published in the MMWR, and identified opportunities for improving the dissemination of public health information contained in the MMWR series. The study found that the MMWR was valued for its accuracy, relevance, and concise reporting format. Readers were from diverse professional backgrounds. The number of readers is likely double the 450,000 on mailing lists. Recommendations may result in some refinements to content and format.

FEDERAL CONTACT OFFICE: Epidemiology Program Office

PERSON: Richard A. Goodman, M.D., M.P.H.

PHONE NUMBER: 404/639-3636

PIC NUMBER: 5357

PERFORMER ORGANIZATION: Battelle, Arlington, VA

P.I.: James O. Hersey, Ph.D.

TITLE: Lessons From Implementing State-Based Diabetes Control Programs: An Evaluation

ABSTRACT NUMBER: 016

ABSTRACT: The purpose of this project was to assess the effectiveness and identify exemplary practices of State-based diabetes control programs (DCPs) in providing services with the potential of reducing diabetes-related mortality and morbidity. Intensive site visits to 9 of the 27 DCPs were conducted. The study analyzed archival data and interviews with DCP and non-DCP personnel in State and local programs. The study looked for exemplary practices associated with four outcomes: program reach--the number of persons screened, counseled, referred, and treated as the result of DCP activities; system change in terms of improved coverage for diabetes services and increased numbers of clinics and providers offering diabetes services; integration of diabetes services into ongoing medical service delivery; and leveraging resources for and institutionalization of diabetes programs. Findings indicated that DCPs have had measurable impact on diabetes services. Specific program accomplishments are cited, as are recommendations for program expansion and refinement.

FEDERAL CONTACT OFFICE: Office of Program Planning and Evaluation

PERSONS: Wilma Johnson, Nancy Cheal

PHONE NUMBER: 404/639-3453

PIC NUMBER: 5299

PERFORMER ORGANIZATION: Battelle, Arlington, VA

P.I.: James O. Hersey, Ph.D.

TITLE: Report of the Working Group To Revise the Model State Vital Statistics Act and Regulations

ABSTRACT NUMBER: 017

ABSTRACT: The purpose of this project was to determine what revisions were needed in the Model State Vital Statistics Act and Regulations to accommodate changes in social customs and technology relative to the registration of vital events and the analysis of vital statistics data. A group of consultants representing State vital registration and statistics executives advised the contractor as to what changes were necessary.

FEDERAL CONTACT OFFICE: Office of Program Planning and Evaluation

PERSONS: Wilma Johnson, Nancy Cheal

PHONE NUMBER: 404/639-3453

PIC NUMBER: 4065

PERFORMER ORGANIZATION: Pace Enterprises, Inc., Atlanta, GA

TITLE: Vital Records Evaluation Using 1988 National Maternal and Infant Health Survey Data

ABSTRACT NUMBER: 018

ABSTRACT: The purpose of this study was to compare selected data elements reported in the 1988 National Maternal and Infant Health Survey (NMIHS) with information contained in the original vital records, maintained by the States, covering the same set of birth events and/or fetal deaths. More than 3,200 live birth certificates and more than 4,500 fetal death reports were compared with data from the 1988 NMIHS. The quality and completeness of the information reported in the vital record was assessed, discrepancies with respect to the NMIHS identified, and the nature and frequency of the discrepancies examined. The four components of the study were (1) the check box items in the new birth and fetal death certificate; (2) underlying cause of fetal death; (3) industry and occupation; and (4) multiple causes of death. Of the check box items, medical risk factors, obstetric procedures, complications of labor and delivery, method of delivery, and newborn abnormalities were studied. Among the data sources, maternal alcohol and tobacco use and prenatal weight gain were compared.

FEDERAL CONTACT OFFICE: Office of Program Planning and Evaluation

PERSONS: Wilma Johnson, Nancy Cheal

PHONE NUMBER: 404/639-3453

PIC NUMBER: 4067

PERFORMER ORGANIZATION: JWK Inter-national Corporation, Annandale, VA

Food and Drug Administration

Listing of Abstract Titles

Evaluation of Influenza Virus Vaccine in High Risk Infection Categories

Increasing the Safety of the Blood Supply by Screening Donors More Effectively

TITLE: Evaluation of Influenza Virus Vaccine in High Risk Infection Categories

ABSTRACT NUMBER: 019

ABSTRACT: The objective of this study was to ensure the continued protection of the public from the influenza virus through ongoing investigation into the reactogenicity, immunogenicity, and protective efficiency of split product and whole virus influenza vaccines and other vaccines that are commercially available or being considered for licensure. Information from the study is helping scientific reviewers assess current vaccines, while providing guidance to developing vaccines for licensure in the Center for Biologics Evaluation and Research. The sera obtained during this study are being used to provide standard sets of sera so that results from different national and international reference laboratories can verify comparability of results. The studies in infants are reproduced by no other source and serve as an important reference resource.

FEDERAL CONTACT OFFICE: Office of Program Planning and Evaluation

PERSON: Tim Hagerty

PHONE NUMBER: 301/443-4076

PIC NUMBER: 4355

PERFORMER ORGANIZATION: Hackensack Medical Center, Hackensack, NJ

P.I.: Peter Gross

TITLE: Increasing the Safety of the Blood Supply by Screening Donors More Effectively

ABSTRACT NUMBER: 020

ABSTRACT: The goal of the study was to improve the blood donation process by integrating and simplifying all information (particularly HIV/AIDS-related information) presented to donors. The study evaluated mechanisms for blood donor education that address the needs of first-time donors and blood center staff. The study findings are helping the Center for Biologics Evaluation and Research staff screen out unsuitable donors and motivate healthy ones to donate more frequently. As a result of this study, a new donor screening process and a training program for health screeners were developed.

FEDERAL CONTACT OFFICE: Office of Program Planning and Evaluation

PERSON: Tim Hagerty

PHONE NUMBER: 301/443-4076

PIC NUMBER: 5685

PERFORMER ORGANIZATION: American Institutes for Research, Washington, DC

Health Resources and Services Administration

Listing of Abstract Titles

Evaluation Design: Evaluation of the Effectiveness of Community Health Centers

Linkage Programs: Survey of Mental Health Services

Perinatal Pilot-Test Summary and Implementation Report

HIV Service Networks in Four Rural Areas

Implementation of Title I of the Ryan White CARE Act of 1990

The Participation of People With HIV in Title I HIV Health Services Planning Councils

Evaluation of Methods for Estimating Unit Costs of HIV Health and Support Services

Rural Health Travels the Telecommunica-tions Highway--Reaching Rural

The Demographic and Treatment Characteristics of the Hill-Burton Population

Survey of Beneficiaries of Nursing Education Projects

The Training and Practice of Preventive Medicine Specialists: An Underutilized Resource

Evaluation of Bureau of Health Professions Strategic Directions--Phase I

TITLE: Evaluation Design: Evaluation of the Effectiveness of Community Health Centers

ABSTRACT NUMBER: 021

ABSTRACT: The purpose of this joint HRSA/OASH project was to develop a study design that would examine the efficiency and effectiveness of community and migrant health centers (C/MHCs); the centers' impact on the health status of their users; and ways in which the centers can improve their performance. Comparison of costs, utilization, and health outcomes for similar populations not using C/MHCs are part of the design. Issues of how and when various outcomes are reported, how data concerning them can be gathered, and whether the specified outcome measures are appropriate for evaluation were studied. The report calls for analysis of approximately 50 community health centers (CHCs) in 10 States. The report recommends that effectiveness and impact of the CHC model(s) be assessed by describing the characteristics of the CHC delivery model(s); measuring the impact of the model(s) on use of health services, health care expenditures, and indicators of health status; and identifying factors that contribute to the observed impact. The design formed the basis for an evaluation of the centers that began in September 1994.

FEDERAL CONTACT OFFICE: Division of Community and Migrant Health, Bureau of Primary Health Care

PERSON: Judy Rodgers

PHONE NUMBER: 301/594-4340

PIC NUMBER: 4918

PERFORMER ORGANIZATION: Lewin/VHI, Inc., Fairfax, VA

P.I.: Ann Zuvekas

TITLE: Linkage Programs: Survey of Mental Health Services

ABSTRACT NUMBER: 022

ABSTRACT: In 1989, HRSA's Bureau of Primary Health Care and the National Institute on Drug Abuse cosponsored the Integrated Primary Care and Substance Abuse Treatment Program, referred to as the "Linkage Program," a multiyear demonstration program supporting 21 community-based approaches for linking substance abuse and primary care services. This demonstration led to recognition of the need for mental health services in these linkage projects. A survey was designed to examine various delivery system features pertaining to availability, scope, and delivery of mental health services, and barriers to providing these services. The 15 projects that received fourth-year continuation grants (which included seven community health centers) were the focus of this study. The major finding was that most linkage projects evolved a supportive environment for assessment of mental health problems and a system of referral to and delivery of mental health services. However, projected need for mental health services outstripped delivery system capacity. Findings from this study will help other providers of primary care services establish the necessary alliances and treatment networks to reduce delivery system fragmentation and develop a continuum of inpatient and ambulatory mental health services.

FEDERAL CONTACT OFFICE: Division of Programs for Special Populations, Bureau of Primary Health Care

PERSON: Thomas M. Coughlin

PHONE NUMBER: 301/594-4450

PIC NUMBER: 3557.2

PERFORMER ORGANIZATION: MDS Associates, Inc., Wheaton, MD

P.I.: Marilyn Falik, Ph.D.

TITLE: Perinatal Pilot-Test Summary and Implementation Report

ABSTRACT NUMBER: 023

ABSTRACT: The purpose of this study was to develop and pilot test a patient-specific data collection tool for perinatal patients and infants receiving services in Community and Migrant Health Centers (C/MHCs) funded by HRSA's Bureau of Primary Health Care (BPHC) and to prepare a plan for implementing this tool. This study is part of an effort to develop and implement a BPHC-wide data and evaluation strategy to improve the availability of information about patients served by grantees. Two patient-specific forms were designed to capture information regarding specific process and outcomes measures. The first would collect data about pregnant women using prenatal and/or postpartum services in C/MHCs, and the second would collect data on infants age 12 months or younger whether or not their mothers used prenatal or postpartum services in a C/MHC. A pilot test was conducted in eight centers over 5 months, and feedback was collected from 22 additional centers via telephone interviews. The pilot test provided information about the data collection instruments with regard to availability of the requested data and completeness and reliability of the data collected, and identified the need to revise the data items, definitions, instructions, and formats. The study found that universal adoption of the forms would be burdensome and expensive. It recommends instead that the form be used in a study of 4,000 mother-child pairs at about 20 C/MHC sites over a 4-year period.

FEDERAL CONTACT OFFICE: Division of Programs for Special Populations, Bureau of Primary Health Care

PERSON: Thomas M. Coughlin

PHONE NUMBER: 301/594-4450

PIC NUMBER: 4208.1

PERFORMER ORGANIZATION: Lewin/VHI, Inc., Fairfax, VA

P.I.: Ann Zuvekas

TITLE: HIV Service Networks in Four Rural Areas

ABSTRACT NUMBER: 024

ABSTRACT: The purpose of this study was to identify and describe systems of HIV care in four rural areas. The study was designed to build understanding of how HIV services are being organized and delivered in different rural environments and to identify service gaps and barriers to care. The case studies identified two distinct approaches to providing medical care to rural residents with HIV: (1) an HIV clinic, located in a rural health department or community health center, that develops cooperative working agreements with physicians in private practice and (2) care provided by primary care physicians who are linked with a medical school or rural-based regional care facility for training and backup consultation. The case studies also documented the need to place more emphasis on HIV training and education in rural areas to create more positive environments for the development of medical and support services. The study's findings and recommendations are being used to design a national study of HIV care in rural areas that will be funded by HRSA. This rural study will be integrated with a large national study of HIV costs and service utilization that is being sponsored by the Agency for Health Care Policy and Research. Case study findings also are being used to formulate HRSA polices on rural HIV care and to provide technical assistance to States with large rural populations.

FEDERAL CONTACT OFFICE: Office of Science and Epidemiology, Bureau of Health Resources Development

PERSON: Katherine Marconi, Ph.D.

PHONE NUMBER: 301/443-6560

PIC NUMBER: 5642

PERFORMER ORGANIZATION: David Berry Company, Las Vegas, NV

P.I.: David Berry

TITLE: Implementation of Title I of the Ryan White CARE Act of 1990

ABSTRACT NUMBER: 025

ABSTRACT: This project measured the extent to which five policy objectives of Title I of the Ryan White CARE Act were achieved in Baltimore, MD, and Oakland, CA: (1) local control and definition of priority setting for the allocation of funds to community AIDS service organizations; (2) rapid disbursement of funds; (3) increased access to services by expanding existing services and adding new agencies; (4) improved coordination of HIV-related services; and (5) increased access to HIV services for underserved populations. To measure progress toward these objectives, surveys were conducted of all identified AIDS services providers and nonrandom samples of people with HIV in 1991 and 1993 in Baltimore and Oakland. Additionally, case studies of the Title I Planning Council processes in each city were completed. All policy objectives, with the exception of improved coordination among services, were met, and a series of recommendations was made for the future administration of this Act. The Henry J. Kaiser Family Foundation funded the 1993 surveys. (The report also presents planning council studies conducted in four eligible metropolitan areas that were not part of the HRSA baseline study.)

FEDERAL CONTACT OFFICE: Office of Science and Epidemiology, Bureau of Health Resources Development

PERSON: Moses B. Pounds

PHONE NUMBER: 301/443-6560

PIC NUMBER: 5639

PERFORMER ORGANIZATION: Johns Hopkins University, Baltimore, MD

P.I.: David D. Celentano, Sc.D.

TITLE: The Participation of People With HIV in Title I HIV Health Services Planning Councils

ABSTRACT NUMBER: 026

ABSTRACT: Ryan White CARE Act Title I Health Services Planning Councils are required by statute to include membership from "affected communities, including individuals with HIV disease." Experience has indicated, however, that eliciting and sustaining participation by these individuals is one of the major challenges facing councils. This study was conducted to identify factors supporting the initial and sustained involvement of HIV-positive individuals in the councils, and identify alternative methods employed by the councils to gain input from and provide feedback to HIV-positive populations in the absence of direct participation. In-depth discussions and focus groups were held in Atlanta, Philadelphia, San Diego, and Seattle, with a total of 113 participants, 90 of whom were HIV-positive. The study found that the councils have undergone an evolution with regard to how they recruit HIV-positive members and whom they recruit: all councils now express commitment to recruiting HIV-positive members of greater ethnic and gender diversity. All councils reported using numerous mechanisms for obtaining input from HIV-positive nonmembers, including public forums, client needs surveys, and participation in advocacy groups. Based on the study report, the Division of HIV Services has issued a policy statement on council membership.

FEDERAL CONTACT OFFICE: Division of HIV Services, Bureau of Health Resources Development

PERSON: Steven R. Young

PHONE NUMBER: 301/443-9091

PIC NUMBER: 5643

PERFORMER ORGANIZATION: Academy for Educational Development, Washington, DC

TITLE: Evaluation of Methods for Estimating Unit Costs of HIV Health and Support Services

ABSTRACT NUMBER: 027

ABSTRACT: The purpose of this study was to evaluate the procedures used to estimate costs of providing health and social services to people with HIV infection. This study compared two methods of costing Ryan White CARE Act services. The first method used an average unit cost approach: total program budgets are divided by total number of service units to yield an average unit cost. The results indicate a wide variation in costs, as a result of actual variations in the content, quality, and volume of services provided, as well as some reporting and methodological errors. The second methodology used a direct cost profile approach to construct the cost of one standard unit of service. This method was tested among a selected group of five Texas CARE Act providers using a questionnaire. This method proved difficult to administer and less accurate than the average unit cost method, although some cost estimates are provided. The study concluded that the average unit cost method is more intuitive and straightforward for CARE Act providers, primarily because most providers use program budgets. A manual for estimating the unit costs of HIV care is now available to Ryan White CARE Act grantees.

FEDERAL CONTACT OFFICE: Office of Science and Epidemiology, Bureau of Health Resources Development

PERSON: Christine Hager

PHONE NUMBER: 301/443-6560

PIC NUMBER: 5640

PERFORMER ORGANIZATION: Texas Department of Health, Austin, TX

P.I.: Susan Griffin

TITLE: Rural Health Travels the Telecommunications Highway--Reaching Rural

ABSTRACT NUMBER: 028

ABSTRACT: The Health Resources and Services Administration sponsored a workshop in November 1993, in Washington, DC, to consider the use of telecommunications technology for providing medical care to rural patients and for reducing the isolation of rural health professionals. The workshop was intended to address the application of telecommunications to clinical care (telemedicine) and to the education of health professionals (distance learning). The purpose of the workshop was to discuss policy issues that affect the development and appropriate use of telemedicine systems. The workshop included descriptions of operating telemedicine systems and the problems they have had to overcome. Workshop participants also provided advice for future telemedicine evaluations. A number of commissioned papers identified and assessed barriers and opportunities for applying this technology in rural areas. The workshop provided an opportunity to develop consensus in order to advance the effective use of this technology in rural areas. In addition, the workshop contributed to the development of the Rural Telemedicine Grant Program and of an exploratory evaluation of the program that began in September 1994. The workshop also strengthened collaborative relationships with staff of other Federal Agencies, such as HCFA and the Departments of Agriculture and Commerce, that are funding telemedicine projects.

FEDERAL CONTACT OFFICE: Office of Rural Health Policy

PERSON: Carole L. Minzter

PHONE NUMBER: 301/443-0835

PIC NUMBER: 5644

PERFORMER ORGANIZATION: Ryan-McGinn, Arlington, VA

P.I.: Teri Randall

TITLE: The Demographic and Treatment Characteristics of the Hill-Burton Population

ABSTRACT NUMBER: 029

ABSTRACT: The purpose of this study was to determine the demographic characteristics of the population receiving medical services under the Hill-Burton Uncompensated Services Program and the conditions for which treatment was required. These characteristics and conditions were compared with those of the general population. Data were obtained from both Hill-Burton records and patient medical records at each of the 50 facilities selected for participation in the study. Records for 4,835 patients were examined. Specific Diagnostic Related Groups and International Classification of Disease Codes were obtained for each record. The findings indicated that, compared with the total U.S. population, the study population was more likely to be female, young adult, and unmarried, with very low household income. The study population required services classified as injury, obstetrical, and ill-defined conditions more often than the general population. Patient records showed that 69.4 percent had no health insurance and 80.1 percent had no coverage for the services received.

FEDERAL CONTACT OFFICE: Office of Facilities Compliance, Bureau of Health Resources Development

PERSON: Jacob E. Tenenbaum

PHONE NUMBER: 301/443-4303

PIC NUMBER: 5641

PERFORMER ORGANIZATION: Arthur Testoff Company, New Carrollton, MD

P.I.: Arthur Testoff

TITLE: Survey of Beneficiaries of Nursing Education Projects

ABSTRACT NUMBER: 030

ABSTRACT: The purpose of this study was to evaluate the programs carried out under Title VIII of the Public Health Service Act and to prepare the biennial report to Congress mandated by Section 859(b) of the Act. The six programs evaluated were Nursing Special Projects Grants and Contracts, Nursing Education Opportunities for Individuals From Disadvantaged Backgrounds Program Grants, Grants for Advanced Nurse Education, Nurse Practitioner and Nurse-Midwifery Program Grants, Professional Nurse Traineeships, and Nurse Anesthetist Traineeships. Separate surveys of grantees and graduates were conducted. The overriding conclusion of this study is that the Title VIII programs are meeting the purposes they were designed to accomplish. Federal support has stimulated the development of projects that address significant nursing education and practice issues. Moreover, the funds provided for the development of educational programs for advanced practice nurses and for the support of students are creating a stable resource of advanced practice nurses.

FEDERAL CONTACT OFFICE: Division of Nursing, Bureau of Health Professions

PERSON: Evelyn B. Moses

PHONE NUMBER: 301/443-6315

PIC NUMBER: 4433

PERFORMER ORGANIZATION: Washington Consulting Group, Washington, DC

P.I.: Cyrus Baghelai

TITLE: The Training and Practice of Preventive Medicine Specialists: An Underutilized Resource

ABSTRACT NUMBER: 031

ABSTRACT: This project was designed to provide detailed information through case studies on the practice profile of residents in selected preventive medicine residency (PMR) programs funded by HRSA. The study examined the recruitment of residents during their training, the various mechanisms used to finance the training of residents, and the career options available to graduates of PMR programs. The study involved the appointment of an expert working committee whose members conducted site visits to 10 preventive medicine residency programs. The culmination of the project was a 3-day workshop held in Washington, DC, in August 1993, which brought together PMR program directors with the major constituencies for preventive medicine. Workshop recommendations included "marketing" preventive medicine to medical students, developing a national strategy for incorporating preventive medicine into the standard medical school curriculum, and reassessing the fourth year of the preventive residency program.

FEDERAL CONTACT OFFICE: Division of Associated, Dental, and Public Health Professions, Bureau of Health Professions

PERSON: Ronald B. Merrill

PHONE NUMBER: 301/443-6896

PIC NUMBER: 4911

PERFORMER ORGANIZATION: American College of Preventive Medicine, Washington, DC

P.I.: Hazel K. Keimowitz

TITLE: Evaluation of Bureau of Health Professions Strategic Directions--Phase I

ABSTRACT NUMBER: 032

ABSTRACT: This study is Phase I of an effort to develop a fully integrated, computerized data system to facilitate program effectiveness evaluation and data analysis of Titles VII and VIII programs administered by the Bureau of Health Professions (BHPr). The purpose of Phase I was to develop a set of outcome indicators to evaluate the effectiveness of programs in the context of BHPr's strategic directions and to identify potential data sources and gaps related to these indicators. Phase II will solicit external customer input on the proposed set of indicators developed in Phase I and identify strategies to address data gaps. A third phase may be needed to develop a consolidated data system for assessing progress toward achieving strategic program objectives bureauwide.

FEDERAL CONTACT OFFICE: Office of Research and Planning, Bureau of Health Professions

PERSON: Betty B. Hambleton

PHONE NUMBER: 301/443-1590

PIC NUMBER: 5497

PERFORMER ORGANIZATION: Lewin/VHI, Inc., Fairfax, VA

P.I.: Susanna Ginsburg

Indian Health Service

Listing of Abstract Titles

An Assessment of Emergency Medical Services

Evaluation of Home Care Services in the Aberdeen Area

Evaluation of Maternal and Child Health Data

Health Promotion and Aging: Improving the Understanding of Geriatric Patients Related to Medications and Their Proper Use

IHS Indirect Cost Study

Intervention Study To Influence Healthier Lifestyles Among Young Adolescent Navajo Indians

Long Term Health Projections for Alcohol Abuse-Related Hospitalizations

Report to Congress on the Indian Health Service With Regard to Health Status and Health Care Needs of American Indians in California in Response to Public Law 100-713

Use of an Alternate Model of Nursing Documentation To Increase Efficiency and Effectiveness of Patient Care Charting

TITLE: An Assessment of Emergency Medical Services

ABSTRACT NUMBER: 033

ABSTRACT: This project evaluated the Emergency Medical Service (EMS) program activities of the Indian Health Service (IHS) at national, area, and service unit levels. The project assessed how well the IHS EMS services are meeting national and applicable State standards. It identified strengths of the existing program as well as areas needing improvement. Because the same methodology has been and is being consistently applied in communities and States throughout the United States, the IHS assessment provided a useful comparison of the IHS program with those of the States. The project provided information on how EMS provided to reservation populations compared with EMS available to non-Indian rural communities. The end product is a report representing the consensus of review team members and contains detailed assessments in each of 10 specific areas. The report provides a basis on which to recommend improvements to the EMSs to meet the needs of the American Indian and Alaska Native communities.

FEDERAL CONTACT OFFICE: Office of Planning, Evaluation, and Legislation

PERSONS: Leo Nolan, Linda Arviso-Miller

PHONE NUMBER: 301/443-0416

PIC NUMBER: 5549

PERFORMER ORGANIZATION: Division of Medical Systems Research & Development, Tucson, AZ

P.I.: Peter G. Decker

TITLE: Evaluation of Home Care Services in the Aberdeen Area

ABSTRACT NUMBER: 034

ABSTRACT: This project explored issues related to home health care services in the Aberdeen [SD] Area as an alternative to hospital care and an adjunct to all other existing clinical services. The project evaluated needs as well as home care services provided by the IHS and others within the IHS system in the Aberdeen Area. The project relates to the range, quality, and appropriateness of curative and preventive services as well as broader financial and reimbursement issues. The purpose of the study was to explore the dimension of needs and the scope of existing community-based services to a specific population in the Aberdeen Area, specifically American Indians who currently receive or who would benefit from home care services. This included indivi-duals of all ages and with all types of health problems. With growing concern for cost containment and implementation of a prospective payment system, the health care industry has changed dramatically in the last 4 years. The health care literature is replete with analyses of the expansion in home care services and problems resulting from patients being discharged "quicker and sicker." These changes, coupled with technological advances, have increased the needs and demands for home care services and the types, complexity, and volume of services delivered nationally.

FEDERAL CONTACT OFFICE: Office of Planning, Evaluation, and Legislation

PERSONS: Leo Nolan, Linda Arviso-Miller

PHONE NUMBER: 301/443-0416

PIC NUMBER: 5000

PERFORMER ORGANIZATION: American Indian Health Care Association, St. Paul, MN

P.I.: Carol M. Baines

TITLE: Evaluation of Maternal and Child Health Data

ABSTRACT NUMBER: 035

ABSTRACT: The purpose of this evaluation, conducted in two parts, was to assess the availability, utilization, and quality of the Navajo Area Indian Health Service's data sets in maternal and child health (MCH). Part 1 was an assessment of the process of data collection and the perceived data needs. Interviews were conducted with service unit staff, affiliated tribal programs, IHS personnel specializing in data collection, Headquarters West, and IHS Headquarters in Rockville, among others. Findings included the following: the current MCH database does not meet the needs of clinical staff; researchers compiled a list of specific data requirements for a broad range of health parameters; there is a downward trend in multidisciplinary MCH teamwork; area offices emphasize management and finance rather than population-based health; there is no community--oriented health approach among the majority of health professions and service unit administrators; and infant and child health have become fragmented into specific pediatric initiatives. Part 2 was an assessment of the quality of an MCH data set through a comparison between computerized entries into delivery room logs and actual patient records. The congruity was high, with the exception of data on "planned pregnancies" and "anemia," supporting the validity and importance of continued integration of 15 or so most important variables into the Resource and Patient Management System. Findings are expected to increase the availability of the data to physicians and midwives nationally.

FEDERAL CONTACT OFFICE: Office of Planning, Evaluation, and Legislation

PERSONS: Leo Nolan, Linda Arviso-Miller

PHONE NUMBER: 301/443-0416

PIC NUMBER: 5544

PERFORMER ORGANIZATION: Maternal and Child Health Program, Billings, MT

P.I.: Diane Jeanotte, R.N.

TITLE: Health Promotion and Aging: Improving the Understanding of Geriatric Patients Related to Medications and Their Proper Use

ABSTRACT NUMBER: 036

ABSTRACT: This project surveyed Navajo patients' use of medications at the Chinle Service Unit and assessed the general level of patient understanding of medications. It also evaluated the effectiveness of using interactive video technology for non-English-speaking patients and those with limited English ability. Study recommendations will be helpful in assessing and improving Navajo geriatric health care. It was determined that the program for diabetic information was limited. The hardware video was bulky, ran too long, and was not practical for the major population with limited English abilities. There were no recommendations for use at other clinics in the IHS.

FEDERAL CONTACT OFFICE: Office of Planning, Evaluation, and Legislation

PERSONS: Leo Nolan, Linda Arviso-Miller

PHONE NUMBER: 301/443-0416

PIC NUMBER: 5006

PERFORMER ORGANIZATION: Headquarters East IHS, Rockville, MD

P.I.: Viola Dwight

TITLE: IHS Indirect Cost Study

ABSTRACT NUMBER: 037

ABSTRACT: This study determined the dollar amount of all IHS overhead costs, where and how money was spent, and how it was linked to the provision of direct health care. These goals were accomplished through interviews, research, and analysis of financial data. Interviews were conducted throughout the IHS organization. Interview information was compared with financial data to distinguish between overhead and direct health care. Functional analyses and cost center analyses were performed. Overhead costs for FY 1992 were calculated at 19.1 percent. Because some cost centers commingled direct and overhead costs, data obtained from individual cost centers were difficult to segregate and quantify. Cost accounting procedures were found not to be uniformly applied within IHS. Such inconsistency made it difficult to compare overhead costs among Area offices. The study proposed a refined cost center design that carefully distinguishes between overhead and direct care costs. It also offered recommendations, including timely reviews of cost center expenditures.

FEDERAL CONTACT OFFICE: Office of Planning, Evaluation, and Legislation

PERSONS: Leo Nolan, Linda Arviso-Miller

PHONE NUMBER: 301/443-0416

PIC NUMBER: 5552

PERFORMER ORGANIZATION: Native American Consultants, Inc., Washington, DC

P.I.: Joseph Tinneman

TITLE: Intervention Study To Influence Healthier Lifestyles Among Young Adolescent Navajo Indians

ABSTRACT NUMBER: 038

ABSTRACT: Approximately 1,200 seventh and eighth grade students from three junior high schools were studied for 2 school years, 1991 to 1993. Heights, weights, and a brief behavioral survey were recorded during the fall and spring of each year. At the intervention school, certificates for specific nutrition or fitness activities were completed by students and parents and collected by homeroom teachers. Homerooms competed for monetary prizes to be used for educational activities. The results for students at the intervention school were compared with those for students at the control school. A number of problems arose, some of which could be avoided in the future and some of which are intrinsic to this type of research. Teachers, administrators, parents, and students often had other commitments with higher priority, such as academic classes, exams, and athletics. Although students may have been willing to participate, some were reluctant to do something different from their peers, especially if their peers did not think it was "cool" to participate in the fitness-nutrition program. The number of students willing to participate was initially small and often inconsistent in followup responses. Distances between participating schools was great and made it difficult to achieve an impact on the students. Self-reported data, which are known from many studies to be fraught with problems, were not reliable enough to be generalizable.

FEDERAL CONTACT OFFICE: Office of Planning, Evaluation, and Legislation

PERSONS: Leo Nolan, Linda Arviso-Miller

PHONE NUMBER: 301/443-0416

PIC NUMBER: 5026

PERFORMER ORGANIZATION: Navajo Area IHS, Chinle, AZ

P.I.: Park W. Gloyd

TITLE: Long Term Health Projections for Alcohol Abuse-Related Hospitalizations

ABSTRACT NUMBER: 039

ABSTRACT: This study made long-term pro-jections for a specific disease and/or health category to determine (1) if current health and mortality patterns will continue without further scientific, treatment, or behavioral improvements and (2) the number of American Indians and Alaska Natives who will become ill or die from a specifically identified condition over the next 25 years. The study included an estimate of contributions expected from advances and innovations in current treatment and technology and changes in societal behavioral patterns, and evaluated how these contributions affected existing mortality rates and savings from eliminating medical costs and lost productivity. The study did reveal a decline in alcohol abuse; however, it was not as significant as it should have been. In fact, alcohol abuse may be worsening for women ages 55-64. Findings of the study include recommending long-term projection models that will give a truer picture of which intervention methodo-logies are most effective.

FEDERAL CONTACT OFFICE: Office of Planning, Evaluation, and Legislation

PERSONS: Leo Nolan, Linda Arviso-Miller

PHONE NUMBER: 301/443-0416

PIC NUMBER: 5034

PERFORMER ORGANIZATION: Support Services, Inc., Silver Spring, MD

P.I.: Walter Hillabrant, Ph.D.

TITLE: Report to Congress on the Indian Health Service With Regard to Health Status and Health Care Needs of American Indians in California in Response to Public Law 100-713

ABSTRACT NUMBER: 040

ABSTRACT: This study documented the health status and access to health care services of American Indians in California, especially in tribes that are not federally recognized. Through analysis of vital statistics and other databases, comparisons were made between California Indians who are members of federally recognized tribes and those who are not. On the basis of these and other comparisons, American Indians in non-federally recognized tribes in California generally had poorer health status than those in federally recognized tribes. The health status of both groups was inferior to that of other populations. For example, 6.4 percent of California Indian babies were low birthweight, whereas 5.2 percent of statewide births were low birthweight. These and related findings underscored the importance of maintaining and expanding coverage to the entire American Indian population of California. This study was coordinated with the California Area Office of the Bureau of Indian Affairs, the California Health Department, California State Indian organizations, and clinics.

FEDERAL CONTACT OFFICE: Division of Planning, Evaluation, and Legislation

PERSON: Leo Nolan

PHONE NUMBER: 301/443-4700

PIC NUMBER: 4955

PERFORMER ORGANIZATION: University of California, Los Angeles, CA

P.I.: Leila Beckwith

TITLE: Use of an Alternate Model of Nursing Documentation To Increase Efficiency and Effectiveness of Patient Care Charting

ABSTRACT NUMBER: 041

ABSTRACT: The purpose of the project was to evaluate current inpatient nursing documentation at three Billings [MT] Area hospitals and, at the option of the hospital, to evaluate a new documentation format. Two of the three hospitals chose to implement FOCUS charting, a new method that streamlines the process by requiring written notes only for abnormal patient characteristics. This method was developed by Creative Nursing Management of Minneapolis, Minnesota. The existing system relied on documentation of both normal and abnormal characteristics. The specific goals of FOCUS charting were to (1) improve the quality and content of nursing documentation; (2) facilitate inclusion of information required by certifying agencies; (3) decrease charting time; and (4) increase nursing satisfaction. The third site, Ft. Belknap, opted to keep the existing system because it is a small facility scheduled for conversion to an outpatient facility in the near future. The impact of current and new procedures was assessed via two surveys--a nursing satisfaction survey and a documentation time survey. Chart reviews were also performed at each site before and after implementation of the new documentation format. There was overall improvement with FOCUS charting, although patient education and discharge planning remained problematic. Regardless of the method, all service units successfully met documentation standards in an August 1994 survey by the Joint Commission on Accreditation of Health Care Organizations. The FOCUS chart method was permanently adopted by one of the hospitals.

FEDERAL CONTACT OFFICE: Office of Planning, Evaluation, and Legislation

PERSONS: Leo Nolan, Linda Arviso-Miller

PHONE NUMBER: 301/443-0416

PIC NUMBER: 5003

PERFORMER ORGANIZATION: IHS Billings Area Office, Billings, MT

P.I.: Jaloo Zelonis

National Institutes of Health

Listing of Abstract Titles

Adverse Events Associated With Childhood Vaccines--Evidence Bearing on Causality

Meeting the Nation's Needs for Biomedical and Behavioral Scientists

Evaluating the Effects of School-Based Intervention Programs To Prevent Teenager Drug Use and Abuse

Evaluation of NIH-Funded Condom Research Phase I: Evaluability Assessment

Indoor Allergens--Assessing and Controlling Adverse Health Effects

Outreach/Risk Reduction Strategies for Changing HIV-Related Risk Behaviors Among Injection Drug Users

Study To Assess the Current and Future Demand for Bone Marrow and the Impact on Size of the National Marrow Donor Registry

Survey of the Knowledge, Attitudes, and Practices of Physicians Related to Blood Use, Transfusion, and Donation

TITLE: Adverse Events Associated With Childhood Vaccines--Evidence Bearing on Causality

ABSTRACT NUMBER: 042

ABSTRACT: This study entailed a broad review of all relevant medical and scientific literature on the potentially serious risks associated with currently licensed childhood vaccines. This study was mandated by Congress under Section 313 of Public Law 99-660 to yield essential information that would help the Public Health Service draft recommendations on the use of certain mandated childhood vaccines and appropriate compensation for adverse reactions. The project was funded by the National Institutes of Health, the National Vaccine Program, the Centers for Disease Control and Prevention, and the Health Resources and Services Administration. The final report is a book, Adverse Events Associated with Childhood Vaccines--Evidence Bearing on Causality, published by the National Academy Press in 1993. It summarizes the relevant input from interested organizations and individuals and describes the nature and extent of the relationship, if any, between the specified adverse events and the vaccines or vaccine components. In addition, the book describes the available data on the circumstances under which administration of these vaccines increases risks of serious or chronic adverse events and on the definition of any known risk groups for these adverse events.

FEDERAL CONTACT OFFICE: Division of Microbiology and Infectious Disease, NIAID

PERSON: Regina Rabinovich, M.D.

PHONE NUMBER: 301/496-5893

PIC NUMBER: 4198.1

PERFORMER ORGANIZATION: Institute of Medicine, National Academy of Sciences, Washington, DC

P.I.: Sylvia Cunningham

TITLE: Meeting the Nation's Needs for Biomedical and Behavioral Scientists

ABSTRACT NUMBER: 043

ABSTRACT: This report examines the Nation's future need for biomedical and behavioral research scientists and the role the National Research Service Awards (NRSA) program can play in meeting them. The report is based on three activities: (1) a panel on estimation procedures was established to provide numerical analysis of educational and employment prospects and of national needs for biomedical and behavioral research personnel; (2) a public hearing was held to solicit the views of a broad constituency associated with NRSA programs; and (3) committee meetings were held to discuss and debate information and recommendations. The committee finds that the NRSA program, although small, is influential and prestigious. It is powerful in its ability to change research emphases and to attract the highest quality individuals to research careers. Furthermore, initiatives introduced through NRSA can have a powerful impact on new research emphases or constituencies. The report makes several recommendations, including the following: (1) raise the real value of stipends to a more competitive level; (2) maintain the annual number of predoctoral and postdoctoral awards in the basic biomedical sciences at 1993 levels; (3) increase the number of awards in the behavioral sciences; and (4) increase the awards in clinical sciences, oral health research, nursing research, and health services research.

FEDERAL CONTACT OFFICE: Office of the Director, NIH

PERSON: Walter Schaffer, Ph.D.

PHONE NUMBER: 301/435-2770

PIC NUMBER: 5567

PERFORMER ORGANIZATION: National Academy of Sciences, Washington, DC

P.I.: National Academy of Sciences

TITLE: Evaluating the Effects of School-Based Intervention Programs To Prevent Teenager Drug Use and Abuse

ABSTRACT NUMBER: 044

ABSTRACT: The purpose of this followup study was to assess the long-term effectiveness of a junior high school-based prevention program for reducing student drug use and abuse. The intervention was based on Life Skills Training, an approach that teaches social resistance and other personal and coping skills. In 1986, schools were randomly assigned to one of two approaches to teaching Life Skills Training or to a control group (no training). This followup study was conducted 6 years later to determine the long-term efficacy of the intervention. The probability of smoking, drinking immoderately, or using marijuana was significantly lower, by as much as 40 percent, for students who received the intervention programs than for those in the control group. The study provides the first evidence that school-based intervention programs conducted by regular classroom teachers can produce durable reductions in tobacco, alcohol, and illicit drug use. The results provided empirical support for the efficacy of a school-based prevention strategy based on a broad-spectrum, cognitive-behavioral approach in prevention of drug use and abuse. Long-term prevention of effects were found, providing impetus for the widespread adoption of similarly designed programs.

FEDERAL CONTACT OFFICE: Division of Epidemiology and Prevention Research, NIDA

PERSON: Larry A. Seitz, Ph.D.

PHONE NUMBER: 301/443-1514

PIC NUMBER: Unassigned

PERFORMER ORGANIZATION: Cornell University Medical College, New York, NY

P.I.: Gilbert J. Botvin, Ph.D.

TITLE: Evaluation of NIH-Funded Condom Research Phase I: Evaluability Assessment

ABSTRACT NUMBER: 045

ABSTRACT: The objectives of this evaluation were to (1) assess the findings of condom use research efforts; (2) guide the development of future program areas; and (3) suggest methodological guidelines to facilitate the evaluation of future condom use research programs. This initial phase was an evaluability assessment to identify and inventory the universe of condom research studies as well as to determine whether or not a formal evaluation of past NIH-funded behavioral research on condom use would be feasible and useful. The results of this evaluability assessment are to be used by the Technical Advisory Group to decide whether to proceed to subsequent phases of more intensive evaluation. The conclusions of this activity indicate that an expanded evaluation is feasible and could provide useful information to guide the development of future program areas and to facilitate the evaluation of future condom research programs. A critical feasibility issue for any research synthesis is the identification of the universe of relevant studies. The Phase I activity developed a reproducible methodology to create a sampling frame, established a sampling frame of more than 500 studies, and developed and applied definitions that permitted sorting studies by key research characteristics and abstracting relevant information about funded projects. The final report will be disseminated to NIH Institutes and committees as well as other organizations or individuals involved in condom use research.

FEDERAL CONTACT OFFICE: Office of Policy Analysis and Technology Transfer, NIAID

PERSON: Wendy Liffers

PHONE NUMBER: 301/496-6752

PIC NUMBER: 5494

PERFORMER ORGANIZATION: Research Triangle Institute, Research Triangle Park, NC

P.I.: John W. Tintoul

TITLE: Indoor Allergens--Assessing and Controlling Adverse Health Effects

ABSTRACT NUMBER: 046

ABSTRACT: The purpose of this study was to characterize what is known among a broad range of experts whose efforts are directed toward achieving cleaner indoor environments. The focuses of this study were to evaluate the indoor environment and to appraise the knowledge base commonly available to physicians with respect to human exposure to indoor allergens. A committee of experts was appointed to (1) examine and characterize what is known about the adverse effects on human health caused by reactions, including asthma, to indoor allergens; (2) characterize the magnitude of these problems nationally; (3) define the populations commonly affected; and (4) identify the specific causative agents. The study reviewed published epidemiological reports, estimated the economic impact of these diseases, and determined the risk factors. It examined methods used in patient diagnosis and measurement of indoor allergen exposures and evaluated methods of abatement and control of sources of allergens. The study resulted in a book titled Indoor Allergens: Assessing and Controlling Health Effects. The book pointed out that indoor allergen exposure is a major risk factor for asthma and that exposure to indoor allergens may be increasing, in part because of house construction methods that have made homes "tight" and in part because individuals are spending more time indoors. The study should raise public perception of the importance of indoor allergens in the pathogenesis of asthma and allergic diseases, and provide a framework for a research agenda to relieve the problem created by indoor allergens. The dissemination of the report should result in research proposals to pursue the suggestions in the report. In addition, NIAID intends to use the National Asthma Education Program to further disseminate the information in the report.

FEDERAL CONTACT OFFICE: Division of Immunology, Allergy, and Transplantation, NIAID

PERSON: Marshall Plaut, M.D.

PHONE NUMBER: 301/496-8973

PIC NUMBER: 4465

PERFORMER ORGANIZATION: Institute of Medicine, National Academy of Sciences, Washington, DC

P.I.: Andrew Pope, Ph.D.

TITLE: Outreach/Risk Reduction Strategies for Changing HIV-Related Risk Behaviors Among Injection Drug Users

ABSTRACT NUMBER: 047

ABSTRACT: This study evaluates the effectiveness of outreach programs for reducing HIV-related risk behaviors of injection drug users (IDUs) and their sexual partners. Under the sponsorship of the National AIDS Demonstration Research Project, the outreach programs offer information about HIV infection, strategies for reducing the risk, counseling, support groups, and training in cognitive and social skills. The programs operated at 28 sites, at which data were collected on 13,475 IDUs and 1,637 sexual partners of IDUs. Investigators used a before/after design, comparing data at 6-month followup with those at baseline by using the AIDS Initial Assessment (AIA) and AIDS Followup Assessment (AFA). Comparison of AIA and AFA data for IDUs and analysis of interview data revealed that the interventions yielded significant decreases in HIV-related risk behaviors, especially related to needle use. Significant increases were found in the use of new rather than reused needles, bleach for cleaning injection equipment, and condoms--all behaviors that reduce risk. This study provided evidence that IDUs and their sexual partners can be reached, can listen to educational strategies to reduce risk, and can change their HIV-related risk behaviors. Reduction in drug injection frequency and use of noninjected drugs was especially noteworthy. Study findings led to the creation of a permanent program.

FEDERAL CONTACT OFFICE: Division of Epidemiology and Prevention Research, NIDA

PERSON: Richard Needle, Ph.D., M.P.H.

PHONE NUMBER: 301/443-6720

PIC NUMBER: 4655.1

PERFORMER ORGANIZATION: NOVA Research Company, Bethesda, MD

TITLE: Study To Assess the Current and Future Demand for Bone Marrow and the Impact on Size of the National Marrow Donor Registry

ABSTRACT NUMBER: 048

ABSTRACT: The objective of the study was to estimate the current and future size of the National Marrow Donor Program (NMDP) Registry. The purpose of the NMDP was to develop and maintain a registry of volunteer unrelated bone marrow donors; to facilitate bone marrow transplants by serving as a coordinating and communications center for a network of donor, collection, and transplant centers in the United States and internationally; and to facilitate research into the efficacy of unrelated donor marrow transplants. Bone marrow transplantation is used to treat patients with chronic myelocytic leukemia, Franconi's anemia, and severe combined immune deficiency, as well as patients with acute leukemia and lymphoma who have relapsed following initial therapy. The scope of the program was enlarged in 1992 to increase the number of minority donors in the registry. Strategic options with financial, cost, and service level implications were developed that will be used in program management.

FEDERAL CONTACT OFFICE: Office of Program Planning and Evaluation, NHLBI

PERSON: Barbara Packard

PHONE NUMBER: 301/496-6331

PIC NUMBER: 5687

PERFORMER ORGANIZATION: Institute of Medicine, National Academy of Sciences, Washington, DC

P.I.: Michael A. Soto, Ph.D.

TITLE: Survey of the Knowledge, Attitudes, and Practices of Physicians Related to Blood Use, Transfusion, and Donation

ABSTRACT NUMBER: 049

ABSTRACT: NHLBI conducted a nationwide survey that will provide quantitative measures of the reported knowledge, attitudes, and practices related to autologous and homologous blood transfusion and donation that are held by two groups of practicing physicians--physicians in specialty groups who are heavy users of blood and blood products and physicians in specialty groups who are involved in primary patient care. These two groups of physicians represent nine physician specialties--six specialties that are heavy users of blood products (anesthesiology, general surgery, cardiovascular surgery, orthopedic surgery, obstetrics and gynecology, and neonatology) and three specialties of physicians involved in primary patient care (family practice, internal medicine, and general practice). Specifically, the survey measures how numerous advances in the use of blood and blood components and alternatives to homologous blood transfusion have affected the medical community. Survey results are being used to maximize the professional education component of the National Blood Resource Education Program and thereby to further ensure that blood and blood components are transfused only when therapeutically appropriate to ensure an adequate supply of safe blood and blood components to meet the Nation's needs.

FEDERAL CONTACT OFFICE: Office of Program Planning and Evaluation, NHLBI

PERSON: Barbara Packard

PHONE NUMBER: 301/496-6331

PIC NUMBER: 4008

PERFORMER ORGANIZATION: Maya Tech Corporation, Silver Spring, MD

Substance Abuse and Mental Health Services Administration

Listing of Abstract Titles

Evaluation Design for the Comprehensive Community Services Program for Children With Serious Emotional Disturbance

Evaluation of Demonstration Treatment Programs in Job Corps

Evaluation of OSAP Demonstration Projects for Community Partnerships

Evaluation of OSAP Demonstration Projects on Pregnant and Postpartum Women and Their Infants (PPWI)

Making a Difference: Interim Status Report of the McKinney Research Demonstration Programs for Homeless Mentally Ill Adults

Second Report to Congress on Alcohol and Drug Abuse Prevention, 1993

TITLE: Evaluation Design for the Comprehensive Community Services Program for Children With Serious Emotional Disturbance

ABSTRACT NUMBER: 050

ABSTRACT: This project developed an overall design and instrumentation package for the Community Mental Health Services Program for Children with Serious Emotional Disturbance. The design was formulated to ensure that information needed to support Federal, State, and local policymakers is readily available and that data collected are useful in improving systems of care for children. The product of this contract is an evaluation design, which will be implemented under another contract.

FEDERAL CONTACT OFFICE: Office of Demonstration Programs, Center for Mental Health Services

PERSON: Roger Straw, Ph.D.

PHONE NUMBER: 301/443-3606

PIC NUMBER: 5029

PERFORMER ORGANIZATION: Macro International, Calverton, MD

P.I.: Susan Zaro

TITLE: Evaluation of Demonstration Treatment Programs in Job Corps

ABSTRACT NUMBER: 051

ABSTRACT: This evaluation examines the cost-effectiveness of the Drug Treatment Enrichment Program in curbing drug use among students receiving training at Job Corps Centers. Job Corps Centers are residential employment and training programs for high-risk youth. The evaluation, which began in July 1992 and continues for 4 years, compares the course of the Drug Treatment Enrichment Program, which provides enhanced services to curtail drug use, with the standard drug prevention and treatment program used at Job Corps Centers. Student drug use is assessed at intake, during, and 12 months after training at four centers with the enrichment program and at four control centers with the standard program. The most recent update from this demonstration program, covering almost 3,000 students entering the centers over a 6-month period in 1993, is contained in the Second Annual Report (November 1993). The enrichment program was found to be more successful and also more costly than the standard program. While 63 percent of students entering the enrichment program used drugs in the month before the program, their drug use declined to 31 percent during the program. In contrast, while 69 percent of students entering the standard program reported drug use, their use declined to 45 percent. The total cost of the enrichment program at each center was 2.5 times that at the control center. The evaluation is still in progress, with more updates expected.

FEDERAL CONTACT OFFICE: Office of Scientific Analysis and Evaluation, Center for Substance Abuse Treatment

PERSON: Charlene Lewis

PHONE NUMBER: 301/443-7730

PIC NUMBER: 4523

PERFORMER ORGANIZATION: Caliber Associates, Fairfax, VA

P.I.: Patricia Devine

TITLE: Evaluation of OSAP Demonstration Projects for Community Partnerships

ABSTRACT NUMBER: 052

ABSTRACT: The National Evaluation of the Community Partnership Demonstration Program (CPDP) is mandated by Congress to document and describe CPDP's implementation and identify successful strategies and common inhibitors to forming effective partnerships and implementing intervention strategies. This is a two-tiered evaluation that includes a broad assessment of all 251 partnerships, using routine collection and analysis of uniform information and an intensive assessment of 36 selected partnerships. The process and outcome evaluation is in its third and final year of data collection, documentation, and evaluation of partnership form and function. Interim reports were prepared in 1992 and 1993.

FEDERAL CONTACT OFFICE: Office of Scientific Analysis and Review, Center for Substance Abuse Prevention

PERSON: Shakeh Kaftarian

PHONE NUMBER: 301/443-4783

PIC NUMBER: 4515

PERFORMER ORGANIZATION: ISA Associates, Alexandria, VA

P.I.: Janice Roehl

TITLE: Evaluation of OSAP Demonstration Projects on Pregnant and Postpartum Women and Their Infants (PPWI)

ABSTRACT NUMBER: 053

ABSTRACT: This evaluation assessed the implementation and effectiveness of the Pregnant and Postpartum Women and Their Infants (PPWI) Demonstration Program funded between 1989 and March 1991. The evaluation examined the process of implementing 90 PPWI grants: the coordination of program services within a large community environment and successes in overcoming obstacles to implementation. The outcome evaluation included 26 PPWI projects and addressed program effectiveness and the impact of the program on mother and child outcomes. Key findings of the evaluation included the following: The PPWI grantees were successful in improving the coordination, availability, and accessibility of health and alcohol and other drug-related services. At least one-third of the women served by these programs reduced their substance use. Babies born to PPWI clients were generally healthy, especially if their mothers took full advantage of prenatal services. Grantees made working agreements with an average of 13 other organizations for delivery of comprehensive services.

FEDERAL CONTACT OFFICE: Office of Scientific Analysis and Review, Center for Substance Abuse Prevention

PERSON: Soledad Sambrano

PHONE NUMBER: 301/443-9136

PIC NUMBER: 4517

PERFORMER ORGANIZATION: Macro International, Calverton, MD

P.I.: James Ross

TITLE: Making a Difference: Interim Status Report of the McKinney Research Demonstration Programs for Homeless Mentally Ill Adults

ABSTRACT NUMBER: 054

ABSTRACT: This evaluation is a multisite research demonstration project offering case management and housing to the homeless mentally ill. A total of 896 homeless adults with severe mental illness were randomly assigned to an intervention that combined housing and social services or to traditional services. The interventions varied across sites but were oriented to training participants in daily living, linking them to needed social and medical services, and assisting them in the receipt of entitlements and housing. Interim findings indicate that each of the five sites has been successful in reducing homelessness and improving the mental health of study participants: homeless adults with severe mental illness are willing to use accessible services targeted to their needs. The factors contributing to the success of the project include mental health treatment, receipt of entitlement income, and reliance on an interdisciplinary team of staff, including some formerly homeless people with mental illness. Participants' substance abuse was more significant than their mental illness in preventing them from finding or keeping housing. These interim findings are from followup at 6 to 12 months. Final results, from followup at 18 to 24 months, are expected to offer strategies for the design and management of programs for the homeless mentally ill.

FEDERAL CONTACT OFFICE: Center for Mental Health Services, SAMHSA

PERSON: Roger Straw, Ph.D.

PHONE NUMBER: 301/443-3606

PIC NUMBER: Unassigned

PERFORMER ORGANIZATION: Homeless Research Project

P.I.: Anthony Lehman, M.D., The Baltimore Project, Baltimore, MD

Stephen Goldfinger, M.D., The Boston Project, Boston, MA

David Stern, Ph.D., The New York City Street Outreach Project, New York, NY

Elie Valencia, J.D., The New York City Critical Time Project, New York, NY

Richard Hough, Ph.D., The San Diego Project, San Diego, CA

TITLE: Second Report to Congress on Alcohol and Drug Abuse Prevention, 1993

ABSTRACT NUMBER: 055

ABSTRACT: The National Structured Evaluation is the first systematic assessment of alcohol and other drug abuse prevention approaches that have been implemented across the country. The overall objective is to improve understanding of the elements of effective alcohol and other drug abuse prevention. It will address three evaluation questions. Which current prevention projects are effective and which factors contribute to their success? Can successful prevention efforts be replicated to other target populations? Which efforts are most effective with the general population, and which ones are most effective with special populations? The first report to Congress identifies drug abuse education and prevention efforts (including practices of Federal, State, and local governments) that focus on reducing alcohol and other drug abuse. The report lists programs that have evaluation components and reviews the reported findings and conclusions. The results show that many programs have not been evaluated; there are mixed results regarding strategies, outcomes, and impact on different target populations; and comprehensive community-based programs show the most promise. The report recommends evaluation technical assistance for program managers; increased program development research; support for comprehensive, community-based programs; and development of an evaluation information center.

FEDERAL CONTACT OFFICE: Office of Intergovernmental and External Affairs, Center for Substance Abuse Prevention

PERSON: Mel Segal

PHONE NUMBER: 301/443-5266

PIC NUMBER: 4313

PERFORMER ORGANIZATION: Conwal, Inc., Falls Church, VA

P.I.: Gay Hill, Ph.D.

Office of the Assistant Secretary for Health

Listing of Abstract Titles

Advanced Practice Nursing Education: Strategies for the Allocation of the Proposed Graduate Nursing Education Account

Assessment of AIDS Non-Acute Care AIDS Treatment Facilities

Developing Effective Health Communication Strategies for High-Risk Youth Outside of School

Evaluating Educational Outcomes of School Health Programs

Evaluation of NDMS 1992 Open Arms Exercise and Response to Florida Hurricane Andrew

Evaluation of the 1990 U.S.-Mexico Border Sister Cities Health Initiative: Final Report

An Evaluation of WomenÕs Reproductive Health Services Provided in Community/Migrant Health Centers in Region VIII

For a Healthy Nation: Returns on Investment in Public Health

Improving the Supply, Training, and Distribution of Primary Care Providers Under Federal Health Care Reform

Integration of Support Services Into Comprehensive Primary Care

Nutritional Status Indicators of Low-Income Populations

Patterns of Care for HIV/AIDS Patients

Region IX Hepatitis B Project Final Report

Report of the American Institute of Nutrition (AIN) Steering Committee on Healthy Weight

Study Design for an Evaluation Study of the Vaccine Adverse Event Reporting System (VAERS)

A Study of the Economic Underpinnings of Vaccine Supply

Supporting Statement for a Study To Evaluate the Effects of the Use of "Put Prevention Into Practice" Materials in Primary Health Care

TITLE: Advanced Practice Nursing Education: Strategies for the Allocation of the Proposed Graduate Nursing Education Account

ABSTRACT NUMBER: 056

ABSTRACT: This project examines issues related to the Administration's proposals to establish an all-payer pool to support graduate nurse training as part of health care reform. It provides background on the advanced practice nursing workforce and outlines possible approaches to allocation of the proposed graduate nurse education (GNE) fund. Trends in nursing supply and demand are addressed, including the number of nurses in practice by type of training, geographic location, and practice setting as well as statistics on the educational programs preparing them for practice. The discussion begins with the registered nurse workforce, as these practitioners form the pool eligible for advanced practice study, and moves to a consideration of each of the four areas of advanced practice nursing specialization (nurse practitioners, nurse-midwives, nurse anesthetists, and clinical nurse specialists). Next, trends in the costs of nursing education and sources of funding available to nursing programs and their students are examined. Finally, the author puts forth strategies pertaining to the allocation of GNE funds and gives details on cost simulations under various options.

FEDERAL CONTACT OFFICE: Office of the Assistant Secretary of Health, DHHS

PERSON: Marcy Gross

PHONE NUMBER: 202/690-5824

PIC NUMBER: 5668

PERFORMER ORGANIZATION: University of Pennsylvania School of Nursing, Philadelphia, PA

P.I.: Linda H. Aiken, Ph.D., R.N., F.A.N.

TITLE: Assessment of AIDS Non-Acute Care AIDS Treatment Facilities

ABSTRACT NUMBER: 057

ABSTRACT: The purpose of the project was to evaluate how the designs of three types of AIDS treatment facilities affected the services provided in the facilities. The question addressed was whether certain facility design features either contribute to or detract from the conduct of the treatment program(s) for which the facility is used. During the 4 fiscal years beginning with 1988, the Public Health Service made grants to assist in the construction or renovation of non-acute, intermediate, and long-term care facilities for patients with AIDS or other HIV-related conditions. Projects eligible for assistance were those for (1) renovation of existing traditional health care facilities such as hospitals and nursing homes, and other than traditional health care facilities, such as residential housing, and (2) construction of new health care facilities to provide comprehensive intermediate and/or long-term care for some or all of the various stages of illness an HIV-infected person may experience. Using a case study approach, a team (consisting of an AIDS treatment professional with experience in planning and implementing non-acute AIDS treatment programs and an architect with experience in the design of health care facilities) evaluated each of six facilities built in Region IX with grant assistance. The team gathered data from grant application files, direct observation of the facilities, and interviews with project sponsors. The final report describes significant aspects of the design, construction, and program use of each facility. Strengths and weaknesses of each facility are highlighted, with recommendations for facility designers. Intended users of project findings are designers of AIDS treatment facilities and treatment staff with responsibility for facility planning.

FEDERAL CONTACT OFFICE: Disease Prevention and Health Promotion, and Health Planning and Evaluation

PERSON: Paul Johnson, Ph.D.

PHONE NUMBER: 202/401-0734

PIC NUMBER: 5682

NTIS ACCESSION NUMBER: PB 95-194726

PERFORMER ORGANIZATION: ESS Architects, San Francisco, CA

P.I.: Abelardo Ruiz

TITLE: Developing Effective Health Communication Strategies for High-Risk Youth Outside of School

ABSTRACT NUMBER: 058

ABSTRACT: This focus group study generated information about the knowledge, attitudes, behaviors, beliefs, and environments of at-risk youth. Youth are considered to be at risk if they are ages 10 through 18 and regularly engage in tobacco use, substance use, unprotected sex, and/or violence. Target population data were collected through a literature review and focus groups. High-risk youth were found to be fairly knowledgeable about healthy practices, but they do not incorporate this awareness into practice. They were found to be willing to listen to trusted, credible adults in alternative settings. Project findings suggest strategies to (1) help service providers, intermediaries, families, and other adult role models to become effective health communicators; (2) focus on linked behaviors rather than categorical issues; and (3) integrate communications with programs and services. This project is the first interdepartmental communications research effort of the Public Health Service and the Departments of Justice, Education, and Transportation.

FEDERAL CONTACT OFFICE: Office of Disease Prevention and Health Promotion

PERSON: Mary Jo Deering, Ph.D.

PHONE NUMBER: 202/205-5968

PIC NUMBER: 4490

PERFORMER ORGANIZATION: S.W. Morris Company and Global Exchange

P.I.: Ruth Karimi

TITLE: Evaluating Educational Outcomes of School Health Programs

ABSTRACT NUMBER: 059

ABSTRACT: This evaluation design report presents a general framework for assessing the effects of school health intervention on students' school performance. As recognized by the Nation's health promotion and disease prevention and education agendas--Healthy People 2000 and Goals 2000--health and education are inextricably linked: good health is necessary for effective learning, and education is necessary for maintaining good health. However, robust empirical evidence of this link from well-designed evaluations does not exist. This study was undertaken to guide efforts to gather such empirical evidence. This report begins with an overview of the eight general categories of school health intervention: (1) health education, (2) health services, (3) healthy school environments, (4) school nutrition services, (5) physical education and fitness, (6) integrated school and community health promotion, (7) school counseling, and (8) health promotion for faculty and staff. It reviews the types of school performance measures and primary data sources for obtaining those measures. Measures include educational achievement (e.g., grade promotion patterns), student behaviors (e.g., attendance), and student attitudes toward school and themselves. The primary data sources are school records and student surveys. The report reviews the procedures needed to collect data from each source and the issues regarding the use of intermediate versus long-term educational outcomes. Two major design evaluation options are presented. The first option is to use data collected by national surveys by the National Center for Education Statistics. The second is to conduct multischool demonstrations, either as add-ons to existing evaluation of the health outcomes of school health programs or as new efforts to test alternative health interventions. With these options, schools gain the tools to justify not only the health importance but also the academic importance of their health programs.

FEDERAL CONTACT OFFICE: Office of Disease Prevention and Health Promotion

PERSON: James Harrell

PHONE NUMBER: 202/205-8611

PIC NUMBER: Unassigned

PERFORMER ORGANIZATION: Mathematica Policy Research, Princeton, NJ

P.I.: Craig Thornton, Ph.D.

TITLE: Evaluation of NDMS 1992 Open Arms Exercise and Response to Florida Hurricane Andrew

ABSTRACT NUMBER: 060

ABSTRACT: The National Disaster Medical System (NDMS) is a cooperative effort of public and private organizations and individuals to provide emergency medical treatment, casualty evacuation, and definitive medical care to victims of natural and manmade catastrophic disasters. This system is a cooperative effort among four Federal Departments and Agencies: the Department of Health and Human Services, the Department of Defense, the Department of Veterans Affairs, and the Federal Emergency Management Agency. The inadequate Federal response to Hurricane Andrew stimulated a number of evaluations of Federal catastrophic disaster programs and policies. These programs are usually activated by a request from the State when State resources are overwhelmed. The Office of Emergency Preparedness initiated an evaluation of NDMS emergency response, beginning with a conference to assess its performance in the Florida hurricane response. The results led to a full evaluation of the program. The results of the evaluation confirmed that NDMS is a valuable program, but it needs to be strengthened. Seven issue areas were identified, and other participants in the NDMS program were appointed to develop options and recommendations for these issue areas.

FEDERAL CONTACT OFFICE: Disease Prevention and Health Promotion, and Health Planning and Evaluation

PERSON: Paul Johnson, Ph.D.

PHONE NUMBER: 202/690-7945

PIC NUMBER: 5691

PERFORMER ORGANIZATION: Maximus, McLean, VA

TITLE: Evaluation of the 1990 U.S.-Mexico Border Sister Cities Health Initiative: Final Report

ABSTRACT NUMBER: 061

ABSTRACT: This report contains an evaluation of the activities initiated by the Office of International Health during 1990-91 to enhance the presence of the Public Health Service (PHS) in the U.S.-Mexico border region, with the objective of improving public health status. The activities included the U.S.-Mexico Border Health Group, Project Consenso, and the Sister Cities Health Initiative. The evaluation provided insights into these activities to guide future PHS efforts in bilateral health activities in the U.S.-Mexico border region. There are mutual concerns over the problems of air pollution, toxic waste dumping, sewage and water contamination, traffic congestion, and lack of services. The evaluators felt that Project Consenso was a success in that it set the stage for the continuation of a management process for the conduct of bilateral activities in health. It also pointed up the unique nature of the border communities and the need for U.S. and Mexican national presence in discussions regarding what are normally local public health issues. The strongest aspect of the development of the Sister Cities Health Initiative projects was the spirit of binational cooperation exhibited by all the participants. Ongoing dialog among the participants focusing on the role of local health authorities and technical assistance were key to project development and strongly suggest the model for future progress in the border region. The intent of the Sister Cities Health projects is to reinforce and strengthen local infrastructure, and it appears that they are doing just that. There was strong sentiment for more clearly defining the PAHO Field Office role in border activities. It was felt that the office could adopt a more consultative role with public and private health institutions on the border. The report contains recommendations made by participants interviewed to facilitate the continuing process of managing binational health issues in the border region.

FEDERAL CONTACT OFFICE: Disease Prevention and Health Promotion, and Health Planning and Evaluation

PERSON: Paul Johnson, Ph.D.

PHONE NUMBER: 202/690-7945

PIC NUMBER: 5628

PERFORMER ORGANIZATION: M.S. Mexico Border Health Association, El Paso, TX

P.I.: Ignacio G. Gosset, Ph.D.

TITLE: An Evaluation of Women's Reproductive Health Services Provided in Community/Migrant Health Centers in Region VIII

ABSTRACT NUMBER: 062

ABSTRACT: In 1993, a multidisciplinary interagency task force was formed to assess reproductive health services in various public health agency programs. The task force identified a gap in reproductive health care information in the community/migrant health care setting and embarked on an evaluation project to examine the regional public health information infrastructure. The evaluation study focused on a representative sample of women 20 to 44 years of age who received reproductive health care services from one of 36 study sites. A 220-item data collection instrument was designed to collect and analyze the frequency and extent of services provided. Findings were also compared across urban, rural, and frontier study populations. The demographic and utilization patterns revealed that the typical client was a married, Hispanic, employed female with a high school education, between 25 and 29 years of age. Preventive health screenings were the most highly documented health services delivered (blood pressure, weight, and pap smears). Prenatal records were consistently the most complete. Documentation for reproductive and other types of health promotion education and counseling appeared low when compared with other services. Services related to reporting of abuse and violence were the most poorly documented, followed by reporting of mental health problems. The evaluation study will provide regional programs with an information base to assess services for continuous improvement of quality of care.

FEDERAL CONTACT OFFICE: Disease Prevention and Health Promotion, and Health Planning and Evaluation

PERSON: Paul Johnson, Ph.D.

PHONE NUMBER: 202/690-7945

PIC NUMBER: 5670

NTIS ACCESSION NUMBER: PB 95-193686

PERFORMER ORGANIZATION: University of Colorado Health Sciences Center, Denver, CO

P.I.: Gene W. Marsh, Ph.D.

TITLE: For a Healthy Nation: Returns on Investment in Public Health

ABSTRACT NUMBER: 063

ABSTRACT: This study describes public health programs and the role they play at the national, State, and local levels in avoiding preventable injury and illness and decreasing demand for personal care services. It also suggests the need to increase the research base on the effectiveness of public health strategies. The population-based approaches described have provided the foundation for significant improvements in health and life expectancy, including declines in heart disease and stroke, infectious disease, and motor vehicle and workplace injuries. Other successes include the virtual elimination of polio since the mid-sixties, declines of nearly 50 percent in dental decay in children since 1974, and a 70 percent reduction in childhood blood lead levels since 1976. At the same time, persistent public health problems remain (e.g., outbreaks of new infectious diseases such as AIDS and familiar diseases such as tuberculosis). Tuberculosis provides an instructive example of a strong public health program that was effective in controlling a disease that resurged when the program was dismantled, resulting in 39,000 preventable cases and $421 million in avoidable direct medical costs between 1985 and 1992. This evaluation finds that appropriate investment in public health may lead to future savings on the medical care side. For example, conservative estimates of the impact of population-based strategies aimed at heart disease, stroke, occupational injuries, motor vehicle-related injuries, low birthweight, and gunshot wounds suggest that $69 billion in medical care spending for these conditions could be averted by 2000. This reduction will be achieved through reductions in cases of heart disease and stroke, estimated to reach 8.4 million and 3.3 million cases avoided, respectively, by 2015 as a result of public health prevention strategies.

FEDERAL CONTACT OFFICE: Disease Prevention and Health Promotion, and Health Planning and Evaluation

PERSON: Paul Johnson, Ph.D.

PHONE NUMBER: 202/690-7945

PIC NUMBER: 5680

PERFORMER ORGANIZATION: Battelle, Arlington, VA

P.I.: Bryan A Luce, Ph.D.

TITLE: Improving the Supply, Training, and Distribution of Primary Care Providers Under Federal Health Care Reform

ABSTRACT NUMBER: 064

ABSTRACT: This project provided an analysis of issues related to proposals for improving the supply, training, and distribution of primary care providers as part of health care reform. Many of the health care reform proposals considered by Congress included provisions that would have changed the mechanism for making decisions with respect to the number and types of new physicians. Information provided in this report includes a chart book on the supply, training, and distribution of physicians; technical papers providing background on the need for increased numbers of generalist physicians and the rationale for proposals to control the supply of new physicians; proposals for an all-payer pool for graduate medical education; estimates of the number of physicians in research training tracks at NIH and elsewhere; and preliminary estimates of the number of advanced practice nurses and physician assistants that would be needed to substitute for physicians if the number of physician residents were reduced.

FEDERAL CONTACT OFFICE: Disease Prevention and Health Promotion, and Health Planning and Evaluation

PERSON: Paul Johnson, Ph.D.

PHONE NUMBER: 202/690-7945

PIC NUMBER: 5666

PERFORMER ORGANIZATION: Center for Health Policy Research, George Washington University, Washington, DC

P.I.: Michele Solloway, Ph.D.

TITLE: Integration of Support Services Into Comprehensive Primary Care

ABSTRACT NUMBER: 065

ABSTRACT: Federally funded community and migrant health centers (C/MHCs) and other community resources, such as the Supplemental Food Program for Women, Infants, and Children (WIC) and mental health centers, are the principal resources for the prevention, detection, and treatment of a variety of health problems. The need to evaluate the integration of nutrition, mental health, social work, and dental services provided through the C/MHCs or their linkages with other community resources is evidenced by the compromised health status of underserved clients in this country. This project evaluated the integration of nutrition, mental health, social work, and dental services provided at C/MHCs with emphasis on special categorical populations, such as the HIV-infected, the homeless, substance abusers, women, and children. The final report includes analysis of the survey responses, reports of the site visits, and recommendations for techniques and strategies that can improve integration of services.

FEDERAL CONTACT OFFICE: Disease Prevention and Health Promotion, and Health Planning and Evaluation

PERSON: Paul Johnson, Ph.D.

PHONE NUMBER: 202/690-7945

PIC NUMBER: 5657

NTIS ACCESSION NUMBER: PB 95-193728

PERFORMER ORGANIZATION: Wilson Yoak Associates, Jefferson City, MI

P.I.: Lorna M. Wilson, R.N., M.P.H.

TITLE: Nutritional Status Indicators of Low-Income Populations

ABSTRACT NUMBER: 066

ABSTRACT: In 1988, the Public Health Service joined with the American Institute of Nutrition (AIN) in a cooperative agreement to examine state-of-the-art techniques for measuring nutritional status and to make recommendations for a core set of nutritional status indicators for low-income populations. A workshop, several symposia, and an AIN multidisciplinary task force on core indicators of nutritional status were planned and carried out. The collected reports resulting from these activities have been compiled in a final report. The task force drew on the symposia and workshop papers in its assessment, which provides an introduction to the concept of nutritional state and the components that must be considered in identifying core indicators for its assessment in difficult-to-sample populations. These components include food security, nutritional concerns, survey coverage, and sampling concerns. Core indicators of nutrition state are suggested in the context of the purposes for which the data are used.

FEDERAL CONTACT OFFICE: Disease Prevention and Health Promotion, and Health Planning and Evaluation

PERSON: Paul Johnson, Ph.D.

PHONE NUMBER: 202/690-7945

PIC NUMBER: 5658

NTIS ACCESSION NUMBER: PB 95-193736

PERFORMER ORGANIZATION: American Institute of Nutrition, Bethesda, MD

P.I.: Richard G. Allison, Ph.D.

TITLE: Patterns of Care for HIV/AIDS Patients

ABSTRACT NUMBER: 067

ABSTRACT: This study examined patterns of care for HIV/AIDS patients who received services in federally funded community health centers in New York and New Jersey. Site visits were made to a sample of six urban centers that represented different types by organizational setting (hospital-based, hospital-affiliated, and freestanding) and by health services delivery orientation (designated AIDS clinic or integrated care plan). The focus of the study was to describe the health services delivery models used by the centers or their approach to administration, organization, and operation of HIV-related health service programs and activities. The study found that despite differences in the organization of the centers, the administration and provision of services is basically the same. The report also presents findings on the types of services delivered, the role of case management, and increased demand for services by high-risk groups.

FEDERAL CONTACT OFFICE: Disease Prevention and Health Promotion, and Health Planning and Evaluation

PERSON: Paul Johnson, Ph.D.

PHONE NUMBER: 202/690-7945

PIC NUMBER: 4498

PERFORMER ORGANIZATION: Market Care, New York, NY

P.I.: George Z. Cestero

TITLE: Region IX Hepatitis B Project Final Report

ABSTRACT NUMBER: 068

ABSTRACT: The purpose of the project was to evaluate the effectiveness of hepatitis B health education/outreach program activities at six federally funded primary care centers in Region IX that target Asian and Pacific Islander patients. The carrier rate of this disease ranges from 10 to 15 percent in the Asian Pacific population in contrast to 0.3 percent in the U.S. population. The study objectives were to examine the role Southeast Asian refugee and Asian immigrant leaders play in the design and delivery of health education programs; develop criteria for measuring effective health education programs; identify effective hepatitis B health education programs; and identify the elements of effective programs. Three major barriers to the development of effective hepatitis B programs for Asian and Pacific Islander populations were identified: (1) lack of understanding of the importance of hepatitis B screening and vaccination by the patient population; (2) lack of reimbursement by third parties for hepatitis B screening and vaccines for adolescents and adults (currently, reimbursement is provided only for prenatal patients); and (3) limited availability of bilingual/bicultural interpreters trained in health education materials. The final report recommends that the following elements should be included in effective hepatitis B outreach and education programs: (1) staff who are sensitive to cultural beliefs and attitudes affecting hepatitis B; (2) reimbursement for hepatitis B screening and vaccines for all age groups; (3) translated health education materials; (4) integrated hepatitis B activities with standard patient care protocols; (5) use of ethnic media (TV, radio, newspapers, magazines); and (6) linkages with school-based health programs and other community-based outreach programs serving the high-risk Asian/Pacific Islander population.

FEDERAL CONTACT OFFICE: Disease Prevention and Health Promotion, and Health Planning and Evaluation

PERSON: Paul Johnson, Ph.D.

PHONE NUMBER: 202/690-7945

PIC NUMBER: 5681

NTIS ACCESSION NUMBER: PB 95-193389

PERFORMER ORGANIZATION: Association for Asian Pacific Community Health Organization, Oakland, CA

TITLE: Report of the American Institute of Nutrition (AIN) Steering Committee on Healthy Weight

ABSTRACT NUMBER: 069

ABSTRACT: As part of a cooperative effort between the Department of Health and Human Services' Public Health Service and the American Institute of Nutrition (AIN), AIN convened a 2-day workshop to review new data and perspectives pertinent to defining "healthy weight." AIN provided transcripts of the workshop and copies of submitted materials to PHS/OASH/ODPHP. A report from the AIN Steering Committee on Healthy Weight was published in the Journal of Nutrition (124:2240-2243, 1994). The report summarizes the Steering Committee's conclusions from the workshop, including areas of agreement, and identifies issues for which further data and discussions are needed. The report has been provided to the 1994-95 Dietary Guidelines Advisory Committee for use in its deliberations on the definition of healthy weight.

FEDERAL CONTACT OFFICE: Disease Prevention and Health Promotion, and Health Planning and Evaluation

PERSON: Paul Johnson, Ph.D.

PHONE NUMBER: 202/690-7945

PIC NUMBER: 5014

PERFORMER ORGANIZATION: American Institute of Nutrition, Bethesda, MD

P.I.: Richard G. Allison, Ph.D.

TITLE: Study Design for an Evaluation Study of the Vaccine Adverse Event Reporting System (VAERS)

ABSTRACT NUMBER: 070

ABSTRACT: This project provided an evaluation design to assess the functional capabilities (including reliability and data quality) of the Vaccine Adverse Event Reporting System (VAERS). It examined methods for improving reporting, followup, and analysis of VAERS data, and assessed the level of awareness about VAERS among health care providers and consumers. There are four major evaluation points: (1) the degree to which this system meets the program's needs and legislative requirements; (2) the reliability and accuracy of information submitted to it, including how well instructions are followed when completing forms; (3) management of secondary and/or followup information; and (4) the perception of VAERS among health care providers, manufacturers, and consumers.

FEDERAL CONTACT OFFICE: Disease Prevention and Health Promotion, and Health Planning and Evaluation

PERSON: Paul Johnson, Ph.D.

PHONE NUMBER: 202/690-7945

PIC NUMBER: 5013

PERFORMER ORGANIZATION: Research Triangle Institute, Washington, DC

P.I.: James S. Lubalin, Ph.D.

TITLE: A Study of the Economic Underpinnings of Vaccine Supply

ABSTRACT NUMBER: 071

ABSTRACT: This project studied the economic and commercial underpinnings of ensuring an adequate supply and distribution of vaccines. The project conducted a meta-analysis of related vaccine supply studies to identify alternative models for the purchase and distribution of vaccines; the effect of scientific advances on research, development, and purchase of vaccines; the application of economic theory to the vaccine market; comparisons of vaccine distribution systems administered by vaccine manufacturers and State agencies; and the implications of purchasing vaccines from foreign firms. The final report presented findings on price increases in vaccines since 1977; factors affecting production of supply; the economics of public versus private sector supply of vaccines; and the limited importance of foreign suppliers in meeting U.S. needs. The study found, for example, that States that supply vaccines at low prices to physician offices and encourage parents to have their children vaccinated have somewhat higher rates of immunization. These and many other findings have enhanced CDC's ability to negotiate lower prices from manufacturers and have broad utility for policy and legislation.

FEDERAL CONTACT OFFICE: PHS-CDC Immunization/Vaccine Stockpile, National Vaccine Program Office

PERSON: Chester Robinson, D.P.A.

PHONE NUMBER: 301/594-6350

PIC NUMBER: 5015

PERFORMER ORGANIZATION: Mathematica Policy Research, Princeton, NJ

P.I.: Craig Thornton, Ph.D.

TITLE: Supporting Statement for a Study To Evaluate the Effects of the Use of "Put Prevention Into Practice" Materials in Primary Health Care

ABSTRACT NUMBER: 072

ABSTRACT: This project developed an evaluation instrument to measure the impact of Put Prevention Into Practice (PPIP) materials on the delivery of preventive care services by primary care providers. Data generated will help PHS develop and refine the PPIP materials and provide direction to other Federal Agencies interested in using PPIP materials in their programs. This information will help primary care providers, national primary care provider organizations, and the academic community determine the appropriate use of these materials in clinical practice and education. These data will also provide policymakers involved with the reform of the health care delivery system with information on the utility of PPIP materials in increasing the delivery of clinical preventive services by primary care providers.

FEDERAL CONTACT OFFICE: Disease Prevention and Health Promotion, and Health Planning and Evaluation

PERSON: Paul Johnson, Ph.D.

PHONE NUMBER: 202/690-7945

PIC NUMBER: 5662

PERFORMER ORGANIZATION: Battelle, Arlington, VA

P.I.: James O. Hersey, Ph.D.

Appendix B. Inventory of PHS Evaluations in Progress

Inventory of PHS Evaluations in Progress

Agency for Health Care Policy and Research

Study Title
Expected Date of Completion

Design of a Survey to Monitor Consumers' Access to Care, Use of Health Services, Health Outcomes, and Patients' Satisfaction
3/27/95

Development of Quality/Utilization Review Criteria, Phases I and II, Cataract
3/31/95

Development of Quality/Utilization Review Criteria, Phases I and II, Pressure Ulcers
3/31/95

Assessment of the Nature and Extent of the Diffusion and Impact of MEDTEP Research Activities and Findings
9/15/95

Data Abstraction Pilot Test for Medicare Data Study
12/29/95

Developing Quality and Utilization Review Criteria: The American Medical Review Research Center (AMRRC) Project
12/29/95

Development of a Typology of Clinical Performance Measures for Quality Improvement
12/31/95

Centers for Disease Control and Prevention

Study Title
Expected Date of Completion

Evaluation of Subdomain Statistics and Small Area Estimates in the NCHS Integrated Survey Design
1/15/94

Evaluation of CDC and ATSDR Training Activities
2/7/94

Evaluation and Expansion of the Centers for Disease Control and Prevention (CDC) Evaluation Database
4/19/94

Evaluation of External Cause-of-Injury Codes
9/15/94

Evaluation of the Dengue Hemorrhagic Fever (DHF) Prevention and Control Program at the San Juan Laboratories, Puerto Rico
10/31/94

Evaluation of NCHS Data Systems for Producing Key Monitoring Indicators for Health Reform
11/26/94

Evaluation of the Content and Conduct of National Health and Nutrition Examination Surveys (NHANES)
11/30/94

Evaluation of the National Nosocomial Infections Surveillance System
11/30/94

Qualitative Evaluation of Hagase Cargo de Su Diabetes
11/30/94

Evaluation of the Activities of the Medical Examiner/Coroner Information Sharing Program (MECISP)
12/31/94

Evaluation of the State-Based Diabetes Control Cooperative Agreement Program
12/31/94

Evaluation of the Impact on Practice of the Guide for the Prevention and Treatment of Complications of Diabetes
1/15/95

Evaluation of the Lead Poisoning Prevention Program
2/28/95

Consultation on CDC Policy Issues--Formation of CDC Office of Women's Health
3/15/95

Evaluation of the Birth Defects Monitoring Program in Fetal Alcohol Syndrome Surveillance
3/30/95

Assessment of the Effectiveness of the National Laboratory Training Network
4/30/95

Evaluation for Development of a National Center for Chronic Disease Prevention and Health Promotion (NCCDPHP)
4/30/95

Process Evaluation of the Fatality Assessment and Control Program
7/14/95

Evaluation of ICD-10 for Morbidity Reporting Purposes in the United States
8/15/95

Evaluation of the CDC Injury Grant Program
9/15/95

Case Study Evaluation of the Henry J. Kaiser Family Foundation's Community Health Promotion Grants Program
9/30/95

Development and Implementation of an Evaluation Training Program
9/30/95

Evaluation of the Effectiveness of CDC Surveillance for Drug-Resistant Pneumococcal Infections
9/30/95

Evaluation of Prevention Effectiveness Methodologies
9/30/95

Assessing the Cost-Effectiveness and Cost Benefit of CDC-Funded Smoke Detector Programs
10/30/95

Initial MIS Development for the Division of Cancer Prevention and Control (DCPC)
10/30/95

Georgia State Clinic Assessments of Immunization Coverage: Impact of Management and Clinic Immunization Practices
11/30/95

Evaluation of the Medical Certification Process for Death Certificates
12/31/95

Evaluation of Suicide Prevention Interventions in Four Native American Communities
12/31/95

An Evaluation of CDC WONDER/PC Public Health Decision Making
1/31/96

Sexually Transmitted Disease (STD) Prevention in the United States
3/30/96

Evaluation of Tuberculosis Outreach Worker Activities
5/30/96

Food and Drug Administration

Study Title
Expected Date of Completion

Prescription Drug User Fee Act (PDUFA) Management
1/96

Mammography Quality Standards Act of 1992 (MQSA) Assessment
6/96

Health Resources and Services Administration

Study Title
Expected Date of Completion

Cost, Utilization Patterns, and Hospital Diagnoses of Medicaid Recipients Served and Not Served by Community Health Centers
11/30/94

An Organizational and Performance Assessment of Managed Care in Community and Migrant Health Centers
12/31/94

Health Maintenance Organization Primary Care Practitioner Staffing Patterns
12/31/94

Integration of Primary Health Care and HIV/AIDS-Related Services and Primary Health Care in Ryan White CARE Act Title III Funded Programs
1/31/95

Evaluation of Maternal and Child Health Bureau Injury Prevention Implementation Incentive Grants
2/28/95

Infant Mortality Among U.S. Women of Mexican Descent: Evaluating the Validity of Current Estimates
2/28/95

Evaluation of the Health Diary
3/31/95

Impact of Federally Qualified Health Center Implementation on Community and Migrant Health Center Revenues and Services
3/31/95

Development of a Model for Forecasting Health Services Utilization Under a Changing Health Care System
4/18/95

Current and Potential Provision of Mental Health Services in Community and Migrant Health Centers
4/30/95

Field Test and Assessment of Data Systems for Reporting Activities Under the Ryan White CARE Act Titles I and II Grant Programs--Part III
5/4/95

Evaluating How Needs of Women with HIV are Being Addressed in the Ryan White CARE Act Title I Planning Process
6/20/95

Assessment of the Impact of the National Practitioner Data Bank--Phase II
6/30/95

Developing a Database for the Study of Advanced Practice Nurses
6/30/95

Pediatric AIDS Evaluation--Phase II
6/30/95

Testing a Methodology to Assess the Health Careers Opportunity Program Postbaccalaureate Program--Phase II
6/30/95

Second National Primary Care Conference: Proceedings and Case Studies
7/25/95

Quality Assurance Procedures for the Uniform Reporting System for Titles I and II of the Ryan White CARE Act
7/30/95

Impact of the Ryan White CARE Act on Capacity Building in Community-Based Agencies Serving Hispanic Populations
8/7/95

Relationship of Medical School Characteristics to the Graduates Choosing Primary Care and Ultimately Providing Care to the Underserved
8/30/95

Study of Selected Aspects of State Medical Board Activities
8/30/95

Impact of the Ryan White CARE Act on Strategies for Serving African Americans
8/31/95

Evaluation of the Potential for Expanding the Capacity of Advanced Practice Nursing Education
9/8/95

Training and Educational Needs of the Public Health Workforce Under State Health Care Reform Programs
9/12/95

Development of National Integrated Requirements Estimates for Physician Assistants, Nurse Practitioners, Certified Nurse Midwives, and Physicians
9/28/95

An Evaluation of the Impact of Social Security Act Section 1115(a) Medicaid Waivers on Federally Qualified Health Centers
9/29/95

Lasting Impact of Emergency Medical Services for Children (EMSC) Funding
9/29/95

An Evaluation of the Health Care for the Homeless Program
9/30/95

Evaluation of Learning Needs Using Computerized Standard Education Needs Assessment Protocol for AIDS Education and Training Centers
9/30/95

Community and Migrant Health Centers' Immunization Efforts--A Joint Study with the Centers for Disease Control
10/31/95

Collaborative Models That Work: Examining the Relationship Between HIV/AIDS and TB Programs in Selected Title I Eligible Metropolitan Areas
11/28/95

Development of a Study Design for Evaluating Variations in OPO Organ Procurement Rates
12/8/95

The Future of Primary Care
12/31/95

Reasons African-American and White Waiting List Patients Are Unavailable for an Organ Offer
2/9/96

Changes in Provision of Services for Active/Recovering Drug Users with HIV in Ryan White CARE Act Title I Cities
3/20/96

Community Health Center User and Visit Survey
4/16/96

Rural Applications of Telemedicine
5/20/96

Development of a Self-Assessment Tool for HIV Health Services Planning Councils and HIV Care Consortia
9/30/96

An Evaluation of the Healthy Schools, Healthy Communities Program
3/27/97

Evaluation of Effectiveness and Impact of Community and Migrant Health Centers: Implementation Phase
9/29/97

National Evaluation of the Healthy Start Program
9/30/98

Indian Health Service

Study Title
Expected Date of Completion

Assessment of Prenatal Services Available and Delivered to Mothers of FAS/FAE-Affected Children in the Billings Area
1/30/95

National Model Adolescent Suicide Prevention Project
1/30/95

Navajo Adolescent Health Services--Needs Assessment and Service Evaluation
2/28/95

Contract Health Service Program--Viewpoint from the Patient Perspective
3/31/95

Evaluation of Diabetes Services Provided by Model Diabetes Program as Specified in PL 102-573
4/30/95

Tohono OÕodham Demonstration Project Evaluation
6/30/95

Benchmarking Home Health/Public Health Nursing Programs
9/30/95

Navajo Hypertension: Analysis and Directed Intervention
9/30/95

Outcome After Residential Treatment for 100 Alcohol-Dependent Navajo Males
9/30/95

Pharmacy Managed Care: A Cooperative Analysis of Tribal Programs, Costs, and Quality of Service
9/30/95

Point of Purchase Project
9/30/95

Prenatal Outreach Evaluation
9/30/95

Retrospective Data Analysis of Indian Women to Determine Health Status
9/30/95

Review and Evaluation of Pap Smear Histories and Pap Smears of Alaska Native Women Who Develop Cervical Cancer and High-Grade Neoplasia
9/30/95

Surveillance of High-Risk Youths for Alcohol/Substance Abuse
9/30/95

Evaluation of the Northern Plains Alcohol-Related Development Disabilities (ARDDs) Training Project
10/31/95

Navajo Youth Risk Behavior Study
12/20/95

Utilizing the Indian-Specific HRS to Evaluate Community HP/DP Programs
12/30/95

Conduct a Study to Determine the Impact of the Final Rule of Public Law 100-446, Eligibility for Health Care Services of the Indian Health Service (IHS)
12/31/95

Longitudinal Evaluation of IHS Alcohol/Substance Abuse Activities (Evaluation of Regional Treatment Centers)
6/30/96

Development and Implementation of a Strategic Planning Evaluation Model, Stages II and III
12/31/96

National Institutes of Health

Study Title
Expected Date of Completion

Improving the Non-Response in the Panel Study of Income Dynamics (PSID) to Enhance Inter- and Intragenerational Analyses
3/31/95

Joint NSF/NIH Survey of Public Understanding of Science
6/30/95

Development of Methodology to Evaluate the NHLBI Short-Term Training for Minority Students Program
9/30/95

Evaluation of the NHLBI Programs of Excellence in Molecular Biology
9/30/95

Evaluation of NIH-Supported Bioengineering Innovation
9/30/95

Workshop: Evaluation of Support for Bioengineering Research
9/30/95

An Evaluability Assessment of the National Research Service Act Awards
12/1/95

Improving Access to Biomedical Information: Evaluation of an Outreach Demonstration Program Targeting Minority Health Professions
12/30/95

Career Outcomes of a Matched Sample of Men and Women Ph.D.s in Science and Engineering
12/31/95

Comprehensive School Health Programs in Grades K-12
12/31/95

Measuring Social Inequalities in Health
12/31/95

Methodology to Assess the Impact of NHLBI Research
9/30/96

Vaccine Development: Establishing Priorities for the United States for the 21st Century
9/30/96

Evaluation of New Technology: Audit-Casi
12/96

Implementation Phase for an Evaluation of the Minority Access to Research Careers (MARC) Honors Undergraduate Research Training Program
12/96

Planning Phase for an Evaluation of the NIDR Center Program
12/15/96

Evaluation of a Knowledge-Based Expert System for Subject Indexing
12/31/96

Phase II: Survey to Assess the Impact of the NIH/NICHD Antenatal Steroids Consensus Development Conference (CDC)
12/31/96

Planning for Inventory and Analysis of Cancer Research Conducted and Supported by Federal Agencies
12/31/96

Survey of Academic Research Equipment
9/30/97

Genetic Testing Task Force
12/97

Development of a Database for NIH-Supported Rare Disease Research
12/31/97

Strategic Plan for Public Health Agencies
12/31/97

The Baltimore Project
12/15/99

Substance Abuse and Mental Health Services Administration

Study Title
Expected Date of Completion

Study of Implementation and Effectiveness of the Center for Substance Abuse and Prevention (CSAP) High-Risk Youth Demonstration Grants
9/25/95

Evaluation of Demonstration Treatment Programs in the Job Corps
9/29/95

Evaluation of the Faculty Development Program
9/30/95

National Treatment Improvement Evaluation Study (NTIES)
1/19/96

Evaluation of the Pregnant and Postpartum Women and Their Infants Demonstration Project
9/15/97

Outcome Evaluation for the Community Partnership Program
12/31/97

DC Courts Initiative Evaluation
12/31/99

Evaluation of the HHS Access to Community Care and Supportive Services (ACCESS) Initiative (Implementation Phase)
12/31/99

National Evaluation Data Coordination Center
12/31/99

Evaluation of High-Risk Youth Substance Abuse Prevention Initiatives Funded in 1994
4/2000

Office of the Assistant Secretary for Health

Study Title
Expected Date of Completion

Evaluation of the Office of Minority Health HIV/AIDS Education/Prevention Grant Program
12/30/94

Review of FIAU/FIAC Clinical Trials
1/31/95

Consequences of Whistleblowing for the Whistleblower in Misconduct in Science Cases
2/15/95

An Evaluation of Prevention and Screening Services Relating to Breast and Reproductive System Cancer Among Nine Community and Migrant Health Centers (C/MHCs) in California
3/30/95

Study of HIV Transmission to Hemophiliacs Through Blood Products
3/31/95

Formative Evaluation of an Automated Telecommunication System to Support Case Managers Providing Health and Human Services to High-Risk Families
6/30/95

Consequences of Being Accused of Research Misconduct
7/28/95

Development of Planning and Evaluation Criteria for Public Health and the National Information Infrastructure
9/30/95

Evaluation of Health Needs of Newly Arriving Refugees
9/30/95

Identifying State Legal Barriers to Integrated Health Information Systems
9/30/95

Support of Efforts to Improve the Methodology for Evaluating the Cost-Effectiveness of Clinical Preventive Services
2/28/96

Evaluation of Core Health Data Sets for Enrollment and Encounters
9/30/96

Public Health Performance Monitoring and Evaluation
12/1/96

Analysis of State and Local Public Health Policy, Programs, and Legislation
9/30/98

Appendix C. Review Criteria for Assessing Program Evaluations Used by the Special Panel of Senior Editorial Advisors

Review Criteria for Assessing Program Evaluations

Used by the Special Panel of Senior Editorial Advisors

OVERALL SIGNIFICANCE

The study addresses a significant issue of policy relevance; evaluation findings are likely to be useful.

CONCEPTUAL CRITERIA

Conceptual foundations

A literature review is included; the project is shown to be logically based on previous findings; the report uses theory and/or models; program assumptions are stated; the evaluation draws from previous evaluations (if any); there is linkage with, and description of, a program; multiple perspectives are presented if multiple relevant stakeholders are consulted and involved; the timing is appropriate because the program is ready for evaluation.

Questions for evaluation The aims of the evaluation are clear, well specified, and testable; the questions are feasible, significant, linked to the program, appropriate with respect to resources and audience, and derive logically from the conceptual foundations. Ingenuity and creativity are shown.

Findings and interpretation The conclusions are justified by the data analyses; the summary does not go beyond what the data will support; appropriate qualifiers are stated; the conclusions fit the entire analysis; equivocal findings are handled appropriately; the initial questions are answered; the interpretation ties in with the conceptual foundation; there is recognition of consistency with, or deviation from, the relevant literature; the presentation is understandable; the results have practical significance; the extent of program implementation is assessed.

Recommendations The recommendations follow from findings and are worth carrying out, affordable, timely, feasible, useful, and appropriate; the recommendations are shown to be relevant to the questions asked; the breadth or specificity of the recommendations is addressed. Any recommendations for future evaluations and/or for improvements in implementation are clearly presented.

Methods

Evaluation design

Design considerations include overall appropriateness; soundness; feasibility; funding and time constraints; generalizability; applicability for cultural diversity; assessment of the extent of program delivery; validity; feasibility for data collection; reliability of selected measurements; use of multiple measures of key concepts; and appropriateness of the sample. In addition, variables are clearly specified and fit with the questions and concepts; the design permits measurement of the extent of implementation of the program and answering of the evaluation questions.

Data collection Data are collected using appropriate units of measurement for analysis, controls for participant selection and assignment bias, and proper handling of missing data and attrition. Other considerations include use of an appropriate comparison group or control; adequate sample size, response rate, and information about the sample; a data collection plan; data collection that is faithful to the plan; attention to and cooperation with the relevant community; project confidentiality; and consistency in data collection. The quality of the data, (including the quality of any extant data sets used in the study) and the efficiency of sampling are addressed. The data collection is appropriate to evaluation questions.

Data analysis Among the factors that the data analysis addresses are the way that attrition should be handled; the matching of the analysis to the design; the use of appropriate statistical controls; the use of methodology and levels of measurement appropriate to the type of data; and estimation of effect size. The analysis shows sensitivity to cultural categories; appropriate generalizability of inferences; and choice of an analysis type that is simple and efficient.

Crosscutting Factors The following crosscutting factors are likely to be important at all stages of a report: clarity, presentation, operating at a state-of-the-art level, appropriateness, understandability, innovativeness, generalizability, efficiency of approach, logical relationships, and a discussion of the report's limitations. The report should also address ethical issues, possible perceptual bias, cultural diversity, and any gaps in study execution.

Footnotes

Chapter 1 Footnotes

1. Two PHS Agencies, FDA and IHS, use program funds instead of the 1 percent set-aside to conduct their evaluations. FDA programs are principally authorized by legislation other than the PHS Act, specifically the authority of the Federal Food, Drug, and Cosmetic Act, and funds are appropriated under the Agriculture, Rural Development, Food and Drug Administration and Related Agencies Appropriations Act. IHS programs are authorized under the Indian Health Care Improvement Act and the Indian Self-Determination Act, and funds are appropriated under the Department of the Interior and Related Agencies Appropriations Act.
2. In addition to using the set-aside authority for the PHS Agencies and OASH, Congress has used it over the past decade to support special program activities of the National Center for Health Statistics and the Agency for Health Care Policy and Research. Furthermore, the Office of the Secretary has used the authority to support public health-related surveys and high-priority evaluation activities.

---

Chapter 2 Footnotes

1. The National Institute of Allergy and Infectious Diseases (NIAID), the National Heart, Lung, and Blood Institute (NHLBI), and the National Institute of Environmental Health Sciences (NIEHS).
2. Oral poliovirus vaccine; hepatitis B vaccine; and measles, mumps, and rubella.
3. Diphtheria, tetanus, pertussis vaccine.
4. ATSDR, an agency of the Public Health Service, is closely aligned with but not part of CDC. For the purposes of this study, ATSDR was defined as a CDC activity.

---

Chapter 3 Footnotes

1. American Indians and Alaska Natives will hereinafter be referred to as American Indians.