Performance Improvement 1996 is the first in a series of annual reports on evaluation activities of the U.S. Department of Health and Human Services (HHS). As a report to Congress, it summarizes the findings of HHS evaluations completed in fiscal 1995. In that year, HHS agencies produced 148 evaluation reports and supported more than 200 evaluation projects in progress.
In general, the report is intended for three audiences: decisionmakers who need information on program results across the broad spectrum of health and human services; program managers who need information on how they can make improvements in program operations and outcomes; and the health and human services community, which can benefit by applying the knowledge and lessons learned from HHS evaluations.
In the Foreword, Secretary Donna Shalala and Principal Deputy Assistant Secretary for Planning and Evaluation David Garrison stress that evaluation is the primary source of knowledge for improving the performance of HHS programs, which is the theme of this report. Every year, program managers are engaged in efforts to determine whether programs and services reach intended populations or communities and achieve desired results. The information obtained from HHS evaluations is useful for planning new directions and adapting programs to meet new demands.
Chapter I provides an overview of evaluation at HHS. Described are the variety of evaluation activities; sources of funding; and evaluation management--including planning procedures, project management, quality assurance, dissemination, and effective use of results. The last section of the chapter discusses future directions for HHS evaluation. First, in coming years, HHS will examine the transformations now taking place in health and human services. Second, the transformations underscore the need for HHS to play a leadership role in developing and applying performance measures. Third, HHS agencies remain committed to seeking information to continuously improve the operation of current programs.
Chapter II highlights 11 HHS evaluations completed during fiscal 1995 and selected by the panel of Senior Editorial Advisors as evaluations considered as particularly useful to the public health and human services community.
Strengthening Families and Neighborhoods: A Community-Centered Approach
This Administration for Children and Families (ACF) report describes the design, implementation, and evaluation of the Iowa Patch project, which was undertaken to improve the quality, accessibility, and accountability of social services to children, youth, and families in a small "patch," or geographic area, of Cedar Rapids, Iowa.
Youth With Runaway, Throwaway, and Homeless Experiences: Prevalence, Drug Use, and Other At-Risk Behaviors
This report, also produced by ACF, describes the most detailed and generalizable study undertaken to date on runaway, throwaway, and homeless youth in the United States, and their associations with high-risk behaviors such as substance abuse, suicide attempts, unsafe sexual behavior, and criminal activities, both inside and outside the home.
Real People, Real Problems: An Evaluation of the Long-Term Care Ombudsman Programs of the Older Americans Act
This Administration on Aging report describes a 1-year project to evaluate long-term care ombudsman programs, which investigate complaints made by or on behalf of residents of nursing facilities and board and care homes.
Understanding and Choosing Clinical Performance Measures for Quality Improvement: Development of a Typology
Increasingly, private purchasers of health care, consumer groups, Federal and State agencies, and health care plans are searching for methods to compare clinical performance among providers and health care systems. This Agency for Health Care Policy and Research project develops a framework for evaluating the structure and content of measures currently used to assess clinical quality so that users can select measures that will help guide choices, ensure accountability, provide data for quality improvement, and track changes in the health care utilization and quality.
The Effect of Mandated Managed Care for Medicaid Populations on the Practice of Public Health: The Example of Childhood Lead Poisoning Prevention
This Centers for Disease Control and Prevention study examined the likely effects of managed care reforms on the delivery of childhood lead poisoning prevention programs. The results suggested that as managed care increases, appropriate childhood lead screening will decrease, use of public health laboratories for testing will decrease, and followup programs will be performed outside the traditional health department.
Medicare Participating Heart Bypass Center Demonstration: Evaluation Design
This negotiated bundled payment demonstration was implemented in May 1991 by the Health Care Financing Administration (HCFA) as one of many cost-containment demonstrations launched in response to increases in Federal spending under Medicare. The study found that both Medicare and hospitals could benefit from this type of payment arrangement through reduced costs and better coordination of care without sacrificing quality of care.
Monitoring the Impact of Medicare Physician Payment Reform on Utilization and Access
This report to Congress prepared by HCFA examined the impact of legislation that reformed Medicare payment methodology to physicians. As Congress intended, the legislation shifted utilization away from surgical procedures and toward medical evaluation and management. In addition, the study showed that the legislation did not create new barriers to care for vulnerable patient groups. The report documents the strong racial and income disparities in utilization.
Performance Indicators for Government Performance and Results Act: Initial Assessment of Health Resources and Services Administration Programs
There are increasing expectations across the Federal Government that agency programs, with a given set of resources and through a series of actions and decisions, will produce the desired outcomes and products for the intended audiences or beneficiaries. This project of the Health Resources and Services Administration helped the Agency begin planning for performance and outcomes measurement.
Evaluation of Diabetes Services Provided by IHS Model Diabetes Program: Final Report
This study evaluates the utility of a model diabetes demonstration program created by the Indian Health Service in 1979. The critical element of the model program was a team of health professionals who augmented primary care by providing a combination of education, outreach, service delivery, and referrals.
Fetal Alcohol Syndrome: Diagnosis, Epidemiology, Prevention, and Treatment
A committee of experts was convened by the National Academy of Sciences, Institute of Medicine, to evaluate published studies on fetal alcohol syndrome and related disorders. The evaluation synthesis was requested by the National Institute on Alcohol Abuse and Alcoholism at the National Institutes of Health.
Consequences of Whistleblowing for the Whistleblower in Misconduct of Science
The Office of Research Integrity (ORI) in the Office of Public Health and Science conducted a mail questionnaire survey of individuals who had made allegations of scientific misconduct to ORI to better understand the personal consequences of such actions.
Chapter III provides an overview of the evaluation activities of the HHS agencies, including information on the evaluation program, a summary of evaluations completed during fiscal 1995 and evaluations in progress, and a discussion of future directions for agency evaluations.
Administration for Children and Families (ACF)
ACF administers a broad range of entitlement and discretionary programs. The objectives of its evaluations are to provide information on the program design and operations; to test service delivery approaches; to conduct policy analyses; and to disseminate and apply evaluation results. In fiscal 1995, evaluations were completed on welfare reform demonstrations; welfare dependency reduction; foster care placement programs; runaway/homeless youth; family strengthening; child maltreatment; and low-income energy assistance. Major evaluations in progress include welfare-to-work programs; responsible fatherhood initiatives; teen pregnancy; intergenerational dependency; the Head Start Program; foster care placement; and women and infant nurturing services.
Administration on Aging (AoA)
AoA supports studies that provide information on the implementation of the Older Americans Act of 1992 (OAA). The major evaluation completed in fiscal 1995 examined the effectiveness of State long-term care ombudsman programs. Results documented the public purpose served by these programs and endorsed the continuation of their mandate. A second AoA study developed the framework and design for two studies authorized by OAA to examine the quality of care in board and care facilities and the quality of home care services for older and disabled individuals. AoA is currently evaluating the Elderly Nutrition Program.
Agency for Health Care Policy and Research (AHCPR)
The goals of the AHCPR evaluation program are to assess agency effectiveness in meeting major and long-term priorities; to obtain information to respond to critical agency and departmental concerns; and to conduct internal evaluations to improve the efficacy of key program areas. Evaluation projects completed in fiscal 1995 focus on clinical practice guidelines; clinical performance measures; and consumer assessments of health care services. In progress are evaluation activities to generate information to improve the health care system, such as surveys to collect data on consumers' satisfaction with access to and use of health services. AHCPR also is developing methods to disseminate this information for improved consumer choice.
Centers for Disease Control and Prevention (CDC)
CDC places high priority on evaluations seeking to answer policy, program, and strategic planning questions related to its mission. The major evaluations in fiscal 1995 include data policy/surveillance studies such as the injury risk factor surveillance system and the influenza vaccination coverage. Program evaluations include the Medical Examiner/Coroner Information Sharing Program, the Fatality and Assessment and Control Program, and the Lead Poisoning Prevention Program. Evaluations of educational interventions were completed in the areas of diabetes and adolescent school health. Projects in progress are evaluating data management and reporting systems; surveillance systems for drug-resistant Streptococcus pneumoniae; occupational health hazards; epidemiology training; and dengue hemorrhagic fever prevention and control in San Juan, Puerto Rico.
Food and Drug Administration (FDA)
Evaluation at FDA is largely a line management responsibility. Projects focus on performance management, customer participation, and rigorous rulemaking. In fiscal 1995, FDA identified performance measures jointly with industry customers as part of the drug and biologic review processes necessary for implementation of the Prescription Drug User Fee Act. In addition, FDA completed an assessment of the Mammography Quality Standards Act of 1992. This program is FDA's second major user fee, performance-oriented, Government Performance and Results Act-style initiative. During fiscal 1995, FDA program managers were evaluating their performance measures in light of GPRA standards.
Health Care Financing Administration (HCFA)
The Office of Research and Demonstrations performs and supports HCFA evaluations to develop new health care financing policies and provide information on the impact of HCFA's programs. In fiscal 1995, HCFA completed evaluations in four areas: monitoring the impact of Medicare physician payment reform on utilization and access; the Medicare participating Heart Bypass Center demonstration evaluation; the Essential Access Community Hospital/Rural Primary Care Hospital Program; and the Rural Health Care Transition Grant Program. The major evaluations in progress include examinations of Medicaid 1915(b) waivers; Community Support Living Arrangements; and evaluations of four demonstration projects: Medicare Cataract Surgery Alternate Payment, Medicare Case Management, Medicaid Uninsured, and Medicare Alzheimer's Disease.
Health Resources and Services Administration (HRSA)
The objectives of the HRSA evaluation program are to enhance strategic planning, budget decisions, and legislative planning, and to improve program management. In fiscal 1995, HRSA reviewed its performance measurement capacity for all line programs and bureaus. Completed evaluations included studies in the following areas: services for prevention and treatment of HIV/AIDS; the cost-effectiveness of the Community Health Centers Program; and grants to foster a permanent infrastructure for child and adolescent injury prevention. Ongoing evaluations include projects to enhance performance measurement capacity; the J-1 visa program; the National Health Service Corps; the future of primary care; the Healthy Start Program; and the role of telemedicine in improving the availability and quality of care to underserved populations.
Indian Health Service (IHS)
The IHS evaluation program serves the Agency's program and policy objectives, developed in consultation with the tribal communities. IHS's major evaluations for fiscal 1995 concentrated on family violence in Native American communities; the IHS Model Diabetes Program; the IHS Mid-Level Health Providers; and child abuse and neglect in American Indian and Alaska Native communities. Projects in progress include the development and implementation of the IHS Strategic Planning Model; an analysis of underreporting of American Indian and Alaska Native deceased persons on State death certificates; mental health service delivery models for urban Native Americans; the IHS Adolescent Regional Treatment Centers; and the IHS substance abuse treatment programs for American Indian/Alaska Native women.
National Institutes of Health (NIH)
Evaluation is an integral part of the role of NIH in supporting biomedical research, training, and public education. In fiscal 1995, NIH's evaluations focused on fetal alcohol syndrome; nursing research training; an assessment of the National Cancer Program; measuring social inequalities in health; evaluating outreach activities of the National Library of Medicine; and public and private funding for basic bioengineering research. The major evaluations in progress are the National Research Service Award Research Training Program; the Minority Access to Research Careers Research Training Program; an assessment of NIH implementation of Section 491 of the Public Health Service Act mandating a program of protection of research subjects; the utilization and regulation of genetic tests; a methodology study to assess the impact of National Heart, Lung, and Blood Institute research; the National Cancer Institute's Physician Data Query Program; an assessment of the facilities and resources for laboratory animals; and a survey of academic research equipment.
Office of the Assistant Secretary for Planning and Evaluation (ASPE)
ASPE functions as principal advisor to the Secretary on policy development, conducting evaluations and policy research studies. It is also responsible for HHS-wide coordination of legislative, planning, and evaluation activities. During fiscal 1995, ASPE's evaluations looked at adolescent sexuality and parenthood; privacy of private-sector health records; substance abuse problems of individuals served by HHS programs; and transition to work for welfare recipients. The major evaluations in progress include a national study of assisted living; an analysis of the effect of regulations on the quality of care provided in board and care homes; the costs of domestic violence to the health care system; family preservation programs; health care technology assessment; physician information use; future long-term care needs; managed care and people with disabilities; the JOBS Program; performance measurement for the State service delivery systems; managed care and the pharmaceutical marketplace; programs to promote responsible fathering; and the cost and quality of subacute care.
Office of Public Health and Science (OPHS)
OPHS provides advice, policy and program coordination, and leadership in the implementation, management, and development of HHS public health and science activities. Several major OPHS-supported evaluations were completed in fiscal 1995 on the following: blood supply safety; HHS standards for methadone programs; clinical trials for the experimental drug fialuridine and its parent drug fiacitabine; Healthy People 2000 baseline data for disease prevention activities; measuring State and local public health expenditures; HIV/AIDS prevention within minority communities; breast and reproductive cancer prevention efforts at community and migrant health centers; hepatitis B virus education and outreach efforts; competitive health plans and academic medical centers; and the supply, training, and distribution of primary care providers. Evaluations and policy studies in progress include dietary supplement labels; framework for Healthy People 2010; school health programs; public health infrastructure; cost-effectiveness methodology; and public health performance measures.
Substance Abuse and Mental Health Services Administration (SAMHSA)
SAMHSA is committed to evaluating its programs and grant projects to assess the effectiveness of prevention, treatment, and rehabilitation approaches and systems of care. In fiscal 1995, SAMHSA completed evaluations on mental health services for children and adolescents with serious emotional disabilities, including the effectiveness of case management services and the impact of multisystemic family preservation therapy on family functioning and delinquent adolescent behavior. Ongoing evaluations include evaluations of SAMHSA's demonstration programs: the Job Corps program of enriched substance abuse treatment for adolescents; the services integration experiment for chronically mentally ill homeless persons; and the National Treatment Improvement Evaluation Study. Evaluations are also under way to examine the impact of managed care on access, cost, and quality of substance abuse treatment.
A complete inventory of the 148 HHS evaluations completed in fiscal 1995 is provided in Appendix A, and HHS agency projects currently in progress are listed in Appendix B.