Performance Improvement 1997 is the second annual report of the U.S. Department of Health and Human Services (HHS) on its evaluation activities. As a report to Congress, it summarizes the findings of HHS evaluations completed in fiscal year (FY) 1996. In that year, HHS agencies produced 87 evaluation reports and supported more than 250 evaluation projects in progress.
In general, the report is intended for three audiences: decisionmakers, who need information on program results across the broad spectrum of health and human services; program managers, who need information on how they can improve program operations and outcomes; and the health and human services community, which can benefit by applying the knowledge and lessons learned from HHS evaluations.
In the Foreword, Secretary Donna Shalala and David Garrison, Principal Deputy Assistant Secretary for Planning and Evaluation, stress that evaluation is an important tool for producing the knowledge necessary to develop and improve the performance of HHS's activities to meet the needs of the 21st century. HHS program managers are continuously engaged in efforts to determine whether programs and services reach the intended populations or communities, perform efficiently, and achieve desired results.
Chapter I provides an overview of evaluation at HHS. It describes a variety of evaluation activities; sources of funding; and evaluation management policies, including planning and coordination, project management, quality assurance, dissemination, and effective use of results. The last section in this chapter discusses future directions for HHS evaluation. In the coming years, the Department will examine the transformations now taking place in health and human services. The transformations underscore the need of the Department to play a leadership role in developing performance measures for implementation of the Government Performance and Results Act of 1993 (GPRA). HHS agencies remain committed to seeking information to continuously improve the operation of their programs.
Chapter II highlights 11 evaluations completed during FY 1996 and identified by the panel of Senior Editorial Advisors as particularly useful to the public health and human services community. For the selection criteria, see appendix C.
Evaluation of Child Access Demonstration Projects: Report to Congress
This Administration for Children and Families (ACF) Office of Child Support Enforcement report evaluates different forms of interventions to bring noncustodial parents closer to their children after divorce and separation. The interventions included mediation, parenting training, counseling, enforcement of visitation, and monitoring of visitation. The report found that mediation where both parties attend resulted in parenting plans in 65 to 70 percent of the cases. These parenting plans stimulated more visitation by noncustodial parents and better compliance with child support. A majority of both parents were satisfied with mediation. Other forms of intervention for more longer term and problematic cases did not register impact.
Evaluation of Child Support Guidelines
This ACF report evaluated the implementation of State child support guidelines mandated by the Family Support Act of 1988. The investigation concluded that States need to increase the consistency of guideline application, including income verification procedures, extended custody and visitation arrangements, and consideration of multiple family situations, health insurance, and day care expenses in the calculation of child support awards. Adopting a standardized support order would improve documentation of departures from guideline award calculations.
Serving Elders at Risk: The Older Americans Act Nutrition Programs. National Evaluation of the Elderly Nutrition Program, 1993-1995
The Elderly Nutrition Program sponsored by the Administration on Aging provides grants to State units on aging to subsidize the provision of daily meals to people 60 years of age or older. This evaluation report describes participant characteristics, compares the characteristics of recipients of home-delivered meals with those receiving meals in congregate settings, assesses how well the program reaches disabled and poor elderly, estimates the impact on nutritional intake and social contacts of participants, and examines program costs and other sources of funding. Findings suggest that participants have higher daily intakes of nutrients and more social contacts per month than a comparable group of nonparticipants. The Elderly Nutrition Program succeeds in targeting older people who are poor, live alone, are nutritionally "at risk" because they are overweight or underweight, or are more functionally disabled than their age-group peers. Federal expenditures were found to be highly leveraged with State, local, and private funds.
Oregon Consumer Scorecard Project (OCS)
The final report on the Agency for Health Care Policy and Research (AHCPR)-supported Oregon Consumer Scorecard Project describes the Oregon Consumer Scorecard Consortium's development, pilot testing, and revision of a user-friendly guidebook, "A Consumer Guide to Selecting a Health Plan." The prototype scorecard was designed to help consumers choose a health plan consistent with their individual needs and their health care service delivery preferences. The model may also help purchasers and State policymakers develop their own comparative reports about health plan performance.
Georgia State Clinic Assessments of Immunization Coverage: Impact of Management and Clinic Immunization Practices
Between 1986 and 1994, the State of Georgia implemented a program of audits of the immunization practices of all State public health immunization clinics in an effort to improve child immunization rates. During this time, the immunization rate for preschoolers increased from 31 percent to 90 percent. The Centers for Disease Control and Prevention's (CDC's) evaluation of the program found that the performance of these audits was significantly associated with higher immunization rates for preschool children in the clinic catchment areas. This and other report findings formed the basis of CDC recommendations to all States.
Ventilator-Dependent Unit Demonstration: Outcome Evaluation and Assessment of Post-Acute Care
This Health Care Financing Administration evaluation assessed whether specialized units serving chronic ventilator-dependent patients deliver better clinical outcomes at a reasonable cost to Medicare than conventional hospital settings. Findings included improved patient outcomes for patients treated in ventilator-dependent units and lower hospital daily costs, yet higher overall costs to Medicare because of greater patient longevity. The analysis of costs and outcomes provides important information to policy makers and health care providers about a seriously ill population with intensive resource needs.
Evaluation of the Bureau of Health Professions Strategic Directions: Development of a Cross-Cutting Performance Monitoring System
This Health Resources and Services Administration (HRSA) evaluation reports on Phase II of a three-phase effort to develop a fully integrated computerized data system to facilitate planning and evaluation of the programs administered by the Bureau of Health Professions. A performance monitoring system is proposed to facilitate ongoing program management. Such a performance system is central to the ability of the Bureau of Health Professions to measure whether program support is meeting its national health workforce objectives, and to signal where program course correction is necessary.
Analysis of the Effect of Regulation on the Quality of Care in Board and Care Homes
This evaluation report, sponsored by the Office of the Assistant Secretary for Planning and Evaluation (ASPE), assesses the effects of State board and care home regulations on the quality of care provided by those institutions. The investigation also attempts to identify the characteristics of the board and care environments in the States selected for the study and to determine whether licensed and unlicensed homes differed in terms of those characteristics, as well as in the quality of care provided. The evaluation concludes that extensive State regulation and licensure can improve the quality of care provided by board and care homes and can increase the number of these facilities that hold licenses.
Health Care in Transition: Technology Assessment in the Private Sector
This report sponsored by ASPE briefly describes a number of private-sector health organizations that are engaged in technology assessment. Technology assessment is one of a family of evaluative activities circulating within these health care organizations. Among private-sector organizations conducting technology assessment, rigorous evaluation of clinical effectiveness based on a systematic review of scientific and clinical evidence has become the norm, and technology assessment has increasingly become "full service," encompassing drugs, medical devices, and clinical procedures.
Subacute Care: Policy Synthesis and Market Area Analysis
The report of this ASPE study of subacute care presents an overview and historical perspective of subacute care, a description of the current state of the art and how it varies across selected market areas, and an examination of evidence about its cost and effectiveness. Findings suggest that the emerging concept of subacute care holds promise but also poses many challenges for the public and private sectors.
Cost-Effectiveness in Health and Medicine
Cost-effectiveness analysis (CEA) is one tool available to decisionmakers to help determine the relative value of different approaches to improving health or life expectancy. For a variety of reasons, however, CEA has not been widely used in health policy settings. This Office of Public Health and Science report represents a review of the methodological and theoretical aspects of CEA, its effectiveness, and its expected outcomes and uses. Recommendations are made regarding framing and designing such studies, identifying and valuing outcomes, assessing effectiveness, estimating costs, discounting, reflecting analytical uncertainty, and reporting analyses.
Chapter III provides an overview of the evaluation activities of the HHS agencies, including information on the evaluation program, a summary of evaluations completed during FY 1996 and evaluations in progress, and a discussion of future directions for agency evaluations.
Administration for Children and Families
ACF administers a broad range of programs to support children and families and targeted populations, such as the developmentally disabled and Native Americans. The objectives of ACF evaluations are to provide information on the program design and operations, to test delivery approaches of various services, to conduct policy analyses, and to disseminate and apply evaluation results. In FY 1996, evaluations were completed on child support guidelines, child access demonstration projects, Head Start family self-sufficiency initiatives, home-based services for runaway youth, gang families in public housing projects, foster youth mentors, child protective and preventive services, family-based placement prevention programs, and a case management enhancement project. Major evaluations in progress include the Arkansas prenatal and postnatal paternity project, responsible fatherhood initiatives, home visitor services for teen parents, community development demonstrations, the Head Start program, and foster care placement. Two new National Child Welfare Research Centers have also been established.
Administration on Aging
The Administration on Aging (AoA) supports studies that provide information on the implementation of the Older Americans Act of 1992. This year, AoA completed a major evaluation of the Elderly Nutrition Program, which provides an average of one million meals per day to older Americans. Results showed that the program has been successful in improving the nutritional intakes of elderly people and in decreasing their social isolation. The report also provided guidance to help the program change to meet future needs of the elderly population and adapt to changes under way in the health care system. AoA is now helping many State and area agencies conduct evaluations of programs supported by the Older Americans Act.
Agency for Health Care Policy and Research
The goals of the AHCPR evaluation program are to assess agency effectiveness in meeting major and long-term priorities; to obtain information to respond to critical agency and departmental concerns; and to conduct internal evaluations to improve the efficacy of key program areas. Evaluation projects completed in FY 1996 included an evaluation of practice guidelines and science-based measures of clinical quality; an evaluation of CONQUEST 1.0, a data base of clinical performance measures useful for matching measures appropriate for different clinical conditions; the Oregon Consumer Scorecard Project; and a synthesis of findings from AHCPR's Medical Treatment Effectiveness Program (MEDTEP). In progress during FY 1996 were evaluation activities to assess the health care environment; evaluations of AHCPR publications for research dissemination and training for minority health services researchers; and several evaluation design studies, such as an assessment of data release strategies for the AHCPR's Medical Expenditure Panel Survey.
Centers for Disease Control and Prevention
CDC places high priority on evaluations seeking to answer policy, program, and strategic planning questions related to the Agency's mission. This year's major evaluations include data policy and surveillance studies, such as developing a comprehensive monitoring system to track the impact of health care reform. Program evaluations completed include CDC's National Nosocomial Infections Surveillance Program and the Dengue Hemorrhagic Fever Prevention and Control Program in San Juan, Puerto Rico. CDC also assisted the State of Georgia in a study of immunization auditing procedures and found positive results for increasing child immunization rates. CDC's evaluations in progress include evaluations of data reporting systems, such as one for public health laboratories and CDC's National Electronic Telecommunications System for Surveillance. Program evaluations are under way in the areas of occupational health hazards, epidemiology training, and prevention centers. Other specialized projects are focusing on tuberculosis outreach workers, minors' access to tobacco, and violence prevention.
Food and Drug Administration
Evaluation at the Food and Drug Administration (FDA) is largely a line-management responsibility. Projects focus on performance management, customer participation, and rigorous rulemaking. In FY 1996, FDA continued to identify performance measures jointly with industry customers as part of the drug and biologic review processes necessary for implementing the Prescription Drug User Fee Act. In addition, FDA completed two evaluation projects on food labels. The first reports on the impact of the new food label regulations on consumers, and the second assesses the status of processed, packaged food labels subsequent to publication of the regulations. During FY 1996 FDA program managers were engaged in evaluating their performance measures in light of GPRA standards. Other evaluation projects at FDA include assessing FDA's penalty-reduction and inquiries assistance program for small businesses, influenza virus vaccines, and the national evaluation of x-ray trends.
Health Care Financing Administration
The Office of Research and Demonstrations performs and supports Health Care Financing Administration (HCFA) evaluations to develop new health care financing policies and to provide information on the impact of HCFA's programs. In FY 1996, HCFA completed evaluations of ventilator-dependent hospital unit care, Medicaid drug rebate policies, the Arizona health care cost-containment system, telemedicine, home health prospective payments, physician payment reform, and pharmacy benefit managers. HCFA has over 34 projects in progress, with 24 scheduled to be completed in FY 1997. These projects are monitoring the Medicare and Medicaid programs through demonstrations. Some of the demonstrations involve the Medicaid managed care programs or waivers, Medicare case management demonstrations, ambulatory patient groups, Medicaid uninsured demonstrations, rural health clinics, community-supported living arrangements, Medicare Alzheimer's disease demonstration, long-term care for persons with developmental disabilities, and health plan report cards.
Health Resources and Services Administration
The objectives of the Health Resources and Services Administration (HRSA) evaluation program are to improve program management and policy development and to provide information for strategic planning, budget decisions, and legislation. During FY 1996, HRSA continued to enhance its performance measurement capacity for all line programs, with a notable study completed at the Bureau of Health Professions. Special evaluation projects were performed on quality assurance procedures for Ryan White Comprehensive AIDS Recovery Emergency (CARE) Act grantees; managed care and the Federally Qualified Health Centers; the impact of case management in community health centers; emergency medical services for children; international medical graduates; advanced practice nursing education; and primary care residency programs. Ongoing evaluations in FY 1996 include managed care and community and migrant health centers; health maintenance organization, or HMO, staffing strategies in underserved areas; casemix differences in health centers; assessment of multilingual services offered at community and migrant health centers; the Healthy Start Program to reduce infant mortality; Ryan White CARE Act programs; organ transplantation; telemedicine; and the supply of nurses.
Indian Health Service
The Indian Health Service (IHS) evaluation program serves the agency's program and policy objectives, developed in consultation with the tribal communities. IHS's major evaluations completed in FY 1996 concentrated on developing a methodology for adjusting IHS mortality data for inconsistent classifications; the IHS adolescent regional treatment centers; and the second phase of an evaluation of alcohol and substance abuse services for American Indian and Alaska Native women. Projects in progress are focusing on the impact of a promotional initiative on elderly wellness and use of comprehensive services; a study of prior trauma care of intoxicated patients as a predictor of subsequently fatal injury; an assessment of Resource Requirements Methodology as an IHS management tool; the development of an IHS health services research agenda; and the continuing evaluation of alcohol and substance abuse treatment services for women.
National Institutes of Health
Evaluation is an integral part of the role of the National Institutes of Health (NIH) in supporting biomedical research, training, and public education. In FY 1996, NIH's evaluations focused on the NIH shared instrumentation grant programs, clinical research at the National Institute of Diabetes and Digestive and Kidney Diseases, an evaluation of the Navajo alcohol rehabilitation demonstration, and the National Institute on Drug Abuse's study on the development of medications for the treatment of opiate and cocaine addictions. The major evaluations in progress during FY 1996 were an evaluation of the career status and satisfaction of NIH grant applicants with grant application processes and procedures; comprehensive school health programs in grades K-12; an evaluation of laboratory animal use; and the development of a data base for NIH-supported research in rare diseases.
Office of the Assistant Secretary for Planning and Evaluation
The Assistant Secretary for Planning and Evaluation (ASPE) functions as principal advisor to the secretary on policy development, conducting evaluations and policy research studies. It is also responsible for HHS-wide coordination of legislative, planning, and evaluation activities. During FY 1996, ASPE's evaluations looked at health care technology assessment, information to guide physician practice, assumptions underlying insurance participation modeling, board and care homes, subacute care, integrating acute and long-term care for children with disabilities, interdisciplinary professional training on disabilities, substance abuse treatment for parents and welfare recipients, community response to domestic violence, minority male violence prevention, and American Indian demography. The major evaluations in progress include monitoring changes in health and human services, assessing data needs, managed care and people with disabilities, assisted living, family preservation services, moving welfare recipients to work, promoting involvement of fathers in family life, and trend data on the well-being of children.
Office of Public Health and Science
The Office of Public Health and Science (OPHS) provides advice, policy and program coordination, and leadership in the implementation, management, and development of HHS public health and science activities. Several major OPHS-supported evaluations were completed in FY 1996 on the cost-effectiveness methodology; quantification of State public health expenditures; health needs of recently arriving refugees; and scientific misconduct and the consequences of being accused. Evaluations and policy studies in progress include dietary supplement labels; framework for Healthy People 2010, school health programs, public health infrastructure, and public health performance measures.
Substance Abuse and Mental Health Services Administration
The Substance Abuse and Mental Health Services Administration (SAMHSA) is committed to evaluating its programs and grant projects to assess the effectiveness of prevention, treatment, and rehabilitation approaches and system of care. In FY 1996, SAMHSA completed two evaluation projects. The first was an overview of addiction treatment effectiveness; the second was a study of the employment outcomes of indigent clients receiving alcohol and drug treatment in Washington State. Ongoing evaluations include evaluations of SAMHSA's demonstration programs, such as the Job Corps program of enriched substance abuse treatment for adolescents; the services integration experiment for chronically mentally ill homeless persons; and the National Treatment Improvement Evaluation Study. Evaluations are also under way to examine the impact of managed care on the access, cost, and quality of substance abuse treatment.
A complete inventory of the 81 HHS evaluations completed in FY 1996 is provided in appendix A, and HHS agency projects currently in progress are listed in appendix B. Appendix C describes the Senior Editorial Advisor's review criteria for assessing program evaluations. A glossary of acronyms used throughout this report is presented in appendix D.