Performance Improvement 1998 is the third annual report of the U.S. Department of Health and Human Services (HHS) on its evaluation activities. As a report to Congress, it summarizes the findings of HHS evaluations completed in fiscal year (FY) 1997. In that year, HHS agencies produced 155 evaluation reports and supported more than 335 evaluation projects in progress.
In general, the report is intended for three audiences: decisionmakers, who need information on program results across the broad spectrum of health and human services; program managers, who need information on how they can make improvements in program operations and outcomes; and the health and human services community, which can benefit by applying the knowledge and lessons learned from HHS evaluations.
In the Foreword, Secretary Donna Shalala and Dr. Margaret A. Hamburg, Assistant Secretary for Planning and Evaluation, stress that evaluation is an important tool for producing the knowledge necessary to develop and improve the performance of HHS programs and services to meet the needs of the 21st century. HHS program managers are continuously engaged in efforts to determine whether programs and services reach the intended populations or communities, perform efficiently, and achieve desired results.
Chapter I provides an overview of evaluation at HHS. The overview contains information about the types of evaluation activities; evaluation resources; and evaluation management, including planning and coordination, project management, quality assurance, and dissemination. The last section of the Chapter discusses four ways in which HHS makes effective use of evaluation results for strategic planning and program and policy development. These are performance measurement and data systems to monitor progress in achieving departmental goals and strategies as well as program outcomes; evaluations to assess the effectiveness of HHS strategies and programs used in achieving goals and objectives; environmental assessments to study changes in the larger society and their impact on departmental programs and strategies; and evaluation resources to improve program management and support evaluation efforts.
Chapter II highlights nine HHS evaluations completed during FY 1997 and selected by the Evaluation Review Panel as potentially useful to the larger health and human services research and practice community. For the selection criteria, see Appendix C. Summaries of each report are presented according to three general purposes of evaluation: performance measurement or assessment, program management and development, and policy analysis and development.
Performance Measurement or Assessment
LEAP: Final Report on Ohio's Welfare Initiative to Improve School Attendance Among Teenage Parents
The goal of Ohio's Learning, Earning, and Parenting (LEAP) Program is to use financial incentives to increase the proportion of pregnant teenagers and teen parents on welfare who graduate from high school or receive a General Equivalency Diploma (GED), find jobs, and ultimately achieve self-sufficiency. The immediate goals were to induce dropouts to return to high school or to enroll in General Educational Development programs, and to promote better attendance among those already enrolled in school. These short-term goals were accomplished--statistically significant increases in school enrollment and attendance were observed. Longer-term goals were to increase the rate of high school graduation, GED attainment, employment, and to reduce welfare payments. These long-term goals were only partly successful. The evaluation of LEAP is timely in light of new changes to national welfare laws which increases emphasis on school attendance and on more employment-oriented activities for teens and other welfare recipients.
National Treatment Improvement Evaluation Study: Final Report
The congressionally mandated 5-year National Treatment Improvement Evaluation Study (NTIES) examined the impact of three federally-sponsored drug and alcohol treatment demonstration programs. The study found that the demonstration programs significantly reduced drug and alcohol use, and that treatment had lasting benefits. Clients reported increases in employment and income, improvements in mental and physical health, decreases in criminal activities and homelessness, and modification of sexually risky behavior.
Program Management and Development
Self-Sufficiency Project Implementation Manual: Lessons Learned from Eight Years of Office of Community Services Demonstration Partnership Programs
This manual summarizes lessons learned during eight years of the Demonstration Partnership Program (DPP), which supports innovative approaches toward increasing the self-sufficiency of the poor, tests and evaluates the approaches, and encourages their replication through dissemination of project results and findings. These projects were divided into five areas: (1) case management, (2) micro-enterprise development, (3) minority males, (4) homelessness, and (5) youth at risk. The manual was designed as a step-by-step guide for designing and implementing Self-Sufficiency Projects by community action agencies, community-based organizations, and local community program planners.
The Effects of Informatics Tools and Decision Aids to Help Patient Decision-making about Medical Screening and Treatment
This study identified, evaluated, and synthesized research on our knowledge base on the effectiveness of tools intended to help inform patients about their medical choices, their treatment alternatives, and the risks and benefits of those alternatives. These included interactive computer disks, videotapes, audiotapes, brochures, and computer-generated fact sheets to help patients make decisions about medical screening and treatment. The research found that although some studies do point to the promise these tools hold for helping patients and consumers, few controlled and comparative studies have been done.
Syphilis in the South: A Case Study Assessment in Eight Southern Communities
This project provided a comparative case study of local-level syphilis prevention efforts in eight communities in Alabama, Mississippi, South Carolina, and Tennessee. The project focused on groups perceived to be at high risk of becoming infected with syphilis, the extent to which public health activities target such groups, and identification of factors that affect the reach of services to these groups. African Americans were found to be at greatest risk of syphilis infection. The report highlights the cultural, programmatic, and political barriers that restrict the prevention and control of syphilis and other sexually transmitted diseases for those at greatest risk. Cultural barriers include local norms that restrict public discussion of human sexuality, distrust of the public health system among African Americans, and a low priority of health relative to other issues of poverty in the community.
Stakeholders Revisit Healthy People 2000 to Maximize the Impact for 2010
This report presents an assessment by Healthy People 2000 stakeholders on how to restructure the goals and objectives of Healthy People 2000 for the next decade. Healthy People 2000 is a set of national objectives for improving the health of Americans through effective prevention strategies. These objectives form a model framework for results-oriented performance measurements that are relevant to both population-based and individual-based health care initiatives. The objectives also serve as benchmarks against which performance can be measured over time. The report explores new approaches for the development of Healthy People 2010, taking into consideration the many major changes in health care since 1990, and past experience with Healthy People 2000 and the health objectives for 1990.
Policy Analysis and Development
Impact of the Medicare Fee Schedule on Teaching Physicians
This study assessed the impact of a new Medicare Fee Schedule (MFS) on the revenues of teaching physicians. The new schedule was based on the Resource-Based Relative Value Scale and altered the relative prices paid to physicians. Relative payment levels were greatly increased for visits (less intensive relative value units, RVUs), and reduced for most types of diagnostic tests and surgical procedures (more intensive RVUs). Relative payment levels were raised in rural areas and lowered in large urban areas. It was hypothesized that teaching physicians would be adversely affected by the MFS because they perform more high-technology procedures and provide comparatively fewer primary care services. Teaching physicians also are located in large urban areas and experience customary charges that may be higher than the area-wide historical payments calculated for fee schedule transition. The study analyzed all Medicare Part B claims associated with discharges during 1991-1993 from a sample of teaching and non-teaching hospitals. Part B revenues and selected price variables were assessed for different categories of service. No evidence was found to suggest that teaching physicians had been adversely affected by the MFS.
Informal and Formal Kinship Care
This study used existing national data sources to describe the characteristics of children in kinship living arrangements, or living with relatives in the absence of a parent, and to define recent trends in the pattern of kinship caregiving. Comparisons were made between formal kinship arrangements in which foster care was provided under State auspices, and informal kinship arrangements in which foster care was provided without State intervention. Over 2 million, or 3 percent, of all children in the United States are in kinship care (i.e., a child living in a relative's household without a parent present). Kinship caregiving and the children living in single mother families both appear to be adaptations to family disruption and the decline of the traditional two-parent family.
Market Barriers to the Development of Pharmacotherapies for the Treatment of Cocaine Abuse and Addiction
This study analyzed market barriers to the development of drug therapies for substance abuse and addiction, and for cocaine use and addiction in particular. Using market analysis, quantitative modeling, case studies, and industry interviews, the study concluded that the development of a new cocaine pharmacotherapy was not economically viable for the pharmaceutical industry under current conditions. The industry faces three critical market barriers: a small and uncertain market for cocaine addiction and abuse drug therapy, a substance abuse treatment system that limits access to this market, and limited and uncertain payment possibilities.
Chapter III provides an overview of HHS agency evaluation activities, including information on the evaluation program, summaries of evaluations completed during FY 1997, and evaluations in progress.
Administration for Children and Families
The Administration for Children and Families (ACF) administers a broad range of programs to support children, families, and other targeted populations. ACF evaluation objectives include providing information on program design and operations, testing services delivery approaches, conducting policy analyses, and disseminating and applying evaluation results. In FY 1997, evaluations were completed on welfare-to-work strategies; responsible fatherhood programs; a paternity acknowledgment in child support program; the Head Start Program; protective, preventative, and reunification services for children and their families; special needs adoptions; child maltreatment; and the Child Abuse and Neglect Prevention Programs. Evaluations in progress during FY 1997 include State welfare reform programs, employment training programs for welfare recipients, child impact studies related to welfare reform, community services demonstrations, child support programs, various aspects of the Head Start Program, child care and welfare reform, homeless youth programs, and quality of foster care.
Administration on Aging
The Administration on Aging (AoA) supports studies that provide information on the implementation of the Older Americans Act of 1992. During FY 1997, AoA initiated a FACE="Times New Roman">n evaluation of the Supportive Services and Senior Centers provisions of the Act (Title III-B), which provides home- and community-based services for all elderly persons, particularly those in great economic or social need. Most home- and community-based services fall under three broad categories: access services, in-home services, and other community-based services. Many State and Area Agencies on Aging also are conducting evaluations of their Older Americans Act service programs.
Agency for Health Care Policy and Research
The goals of the evaluation program at the Agency for Health Care Policy and Research (AHCPR) are to assess the Agency's effectiveness in meeting major and long-term priorities, to obtain information to respond to critical agency and departmental concerns, and to conduct internal evaluations to improve the efficacy of key program areas. Evaluation projects completed in FY 1997 included assessments of private sector outcomes and effectiveness research, informatics as a tool for patient diagnosis and treatment decisions, quality of care measurement, data on managed care organizations for health services research, and customer satisfaction with AHCPR's research dissemination efforts. Ongoing evaluations during FY 1997 include projects to examine AHCPR's grant application and review processes and survey data collections procedures.
Agency for Toxic Substances and Disease Registry
The Agency for Toxic Substances and Disease Registry (ATSDR) was created as a Federal agency by the Comprehensive Environmental Response, Compensation, and Liability Act (CERCLA), more commonly known as Superfund legislation. The evaluation program is coordinated with the Agency's strategic planning process and implementation of the Government Performance and Results Act (GPRA). During FY 1997, ATSDR developed an agencywide inventory of evaluation activities, conducted an assessment of its site-specific evaluation needs, and developed a guidance document for State cooperative agreement evaluation activities. ATSDR also initiated an evaluation of its Hazardous Substances Emergency Events Surveillance (HSEES) program to measure the sensitivity and reliability of the system.
Centers for Disease Control and Prevention
The Centers for Disease Control and Prevention (CDC) places high priority on evaluations seeking to answer policy, program, and strategic planning questions related to the Agency's mission. In FY 1997, CDC's completed projects include major program evaluations, such as the National Laboratory Training Network, the Business Responds to AIDS Program, and suicide interventions for Native American communities. Several data policy/surveillance studies were completed, such as one on the medical certification process for death certificates. CDC also finished a major project on performance measurement to support GPRA implementation. CDC's evaluations in progress include program assessments in such areas as diabetes control, field epidemiology training, teen pregnancy prevention, and sexually-transmitted disease partner notification. Evaluation of data policy and surveillance systems are important at CDC. There are four projects currently underway to develop performance measurement systems, several of which are in the area of human immunodeficiency virus (HIV) prevention programs. CDC also invests it evaluation resources in more specialized assessments, such as evaluating the activities of tuberculosis outreach workers and making recommendations on bicycle helmet use.
Food and Drug Administration
The evaluation program at the Food and Drug Administration (FDA) continues to be aligned with the agency's policy, program and strategic planning initiatives as well as with the implementation of GPRA. FACE="Times New Roman">FDA managers also are identifying further opportunities to involve their customers in the design and testing of alternative ways of doing business. In FY 1997, FDA continued to identify performance measures jointly with industry customers as part of the drug and biologic review processes necessary for implementation of the Prescription Drug User Fee Act. In addition, FDA completed evaluation projects on mammography quality standards, diagnostic medical devices, and food safety issues. FDA's evaluation projects in progress include continuing studies on the Prescription Drug User Fee Act, mammography quality standards, and the assistance program for small businesses. New studies were initiated on food safety standards and the President's tobacco initiative.
Health Care Financing Administration
The Office of Strategic Planning, the research arm of the Health Care Financing Administration (HCFA), performs, coordinates, and supports research and demonstration projects (through intramural studies, contracts, grants, and waivers) to develop and implement new health care financing policies and to provide information on the impact of HCFA's programs. The scope of HCFA's research and demonstration activities embraces all areas of health care: costs, access, quality, service delivery models, and financing approaches. In FY 1997, HCFA completed 20 evaluations on the Medicare Fee Schedule, Medicaid quality of care, effectiveness of the Operation Restore Trust demonstrations, inpatient hospital services, alternative payment of Medicare cataract surgery, ambulatory patient groups, Medicare case management demonstrations, nursing home quality indicators, and health plan report cards. HCFA has over 22 projects in progress, with 13 of them scheduled for completion in FY 1998. Some of the demonstrations involve the Medicaid managed care programs or waivers, Medicaid uninsured demonstrations (particularly the expansion of insurance to children), rural health clinics, Medicare Alzheimer's disease demonstration, use of telemedicine, and long-term care for persons with developmental disabilities.
Health Resources and Services Administration
The purposes of the Health Resources and Services Administration (HRSA) evaluation program are to enhance strategic and performance planning, to strengthen budget and legislative development, and to improve program management. Evaluation priorities include developing and supporting performance measurements, assessing program implementation, and conducting crosscutting policy analysis and research. During FY 1997, HRSA completed evaluations related to its mission to eliminate barriers to care, which included assessments of rural applications of telemedicine and living-related kidney transplant operations. Related to HRSA's mission to assure quality of care, HRSA completed a study on human immunodeficiency virus/acquired immunodeficiency syndrome (HIV/AIDS) treatment services and tuberculosis treatment programs in major cities. Another major study assessed the availability of primary care services under Medicaid managed care. Ongoing evaluations in FY 1997 include effectiveness of community health centers, development and refinement of tools for monitoring cultural competence in managed care; the Healthy Start Program to reduce infant mortality; Ryan White CARE Act programs; and development of a comprehensive performance monitoring system for the Bureau of Health Professions.
Indian Health Service
The evaluation program of the Indian Health Service (IHS) is managed by the Office of Public Health's Staff Office of Planning, Evaluation, and Research. This Office provides national leadership and consultation for IHS and Area Offices on strategic and tactical planning, program evaluation and assessment, public health and medical services, research agendas, and special public health initiatives for the agency. One IHS evaluation completed in FY 1997 focused on the impact of the Alaska Tribal Health Compact on programs and services in Anchorage, Alaska, and another project looked at the prior trauma care of intoxicated patients as a predictor of subsequently fatal injury. Major projects in progress include an examination of the IHS Resource Requirement Methodology, a process for determining needs to deliver adequate care; an assessment of aftercare services provided by IHS regional substance abuse treatment centers; and a study of the IHS capacity for epidemiologic surveillance.
National Institutes of Health
The National Institutes of Health (NIH) Evaluation Program provides information to assist the NIH Director and the various Institute, Center, and Division Directors in determining whether NIH goals and objectives are being achieved and to help guide policy development and program direction. In FY 1997, NIH evaluated the intramural research program at the National Institute of Mental Health; the Bridges to the Future Program, a minority student initiative; and the NIH Consensus Conference on physician prescribing patterns for treatment of peptic ulcer disease. Other major evaluations looked at the status of biomedical research facilities; the National Research Service Award Research Training Program; article citation and patent references at the National Heart, Lung and Blood Institute; the rare disease clinical research database; laboratory animal use, facilities, and resources; safe and effective genetic testing; the Physician Data Query database at the National Cancer Institute; research instrumentation needs; toxicology and environmental health information resources; the international cooperative biodiversity groups; and the Federal Interagency Forum on Aging-Related Statistics. Projects in progress include the Surgeon General's Report on Oral Health, an evaluation of the NIH Human Resources Management System, completion of the Study of National Needs for Biomedical Research Personnel, and the biennial Survey of Scientific and Engineering Research Facilities at Colleges and Universities.
Office of the Assistant Secretary for Planning and Evaluation
The Assistant Secretary for Planning and Evaluation (ASPE) functions as principal advisor to the Secretary on policy development and conducts evaluations and policy research studies. It is also responsible for HHS-wide coordination of legislative, planning, and evaluation activities. During FY 1997, ASPE's major evaluation projects looked at pharmacotherapies for treating cocaine abuse and addiction; public health laboratories; performance measures for public health, substance abuse, and mental health; insurer response to health care market changes; privacy and health research; State-assisted living policies; HIV/AIDS home- and community-based services; consumer-directed personal assistance services; welfare-to-work program approaches; welfare reform and teenage parents; informal and formal kinship care; child welfare; and domestic violence. Major evaluations in progress include understanding private sector changes; improving data and analytic capability; disability, aging, and long-term care trends; promoting active lifestyles and maximum independence; managed care and people with disabilities; assisted living; moving welfare recipients to work; measuring implementation and outcomes of welfare reform, child support, child and youth development, and teen pregnancy programs.
Office of Public Health and Science
The Office of Public Health and Science (OPHS) conducts evaluations to support the Surgeon General and the Assistant Secretary for Health in his dual role as the Nation's top doctor and senior advisor to the Secretary on public health and science matters. OPHS evaluations completed in FY 1997 focused on developing a framework of national health goals and disease prevention objectives in anticipation of the development of the Healthy People 2010 plan; public health expenditures of nine State and local public health, mental health, substance abuse and environmental agencies; and the Office of Minority Health-sponsored program for Historically Black Colleges and Universities. Evaluations and policy studies in progress include dietary supplement labels, school health programs, public health infrastructure, a new model of coordinating and integrating HIV-prevention and primary care services, the Minority Health Resource Center, the Bilingual/Bicultural Service Demonstration Grant Program, and linguistically and culturally appropriate community health promotion programs in local health departments.
Substance Abuse and Mental Health Services Administration
The Substance Abuse and Mental Health Services Administration (SAMHSA) is committed to evaluating its programs and grant projects to assess the effectiveness of prevention, treatment, and rehabilitation approaches and system of care. In FY 1997, SAMHSA completed the National Treatment Improvement Evaluation Study, a comprehensive assessment of the results of 157 three-year substance abuse treatment demonstrations, and an assessment of the Job Corps program, which provides enriched substance abuse treatment for adolescents. Also completed were studies on substance abuse prevention services for pregnant and postpartum women and their infants; the National Training System for planning and delivering substance abuse prevention services; the Faculty Development Program to enhance research, teaching, and clinical practices on issues of alcohol, tobacco, and other drugs; and a program for mental health care provider education in HIV/AIDS programs. Evaluations in progress include a services integration experiment for the chronically mentally ill homeless persons; the impact of managed care on access, cost, and quality of substance abuse treatment; an outcome evaluation of the Community Partnership Program; evaluation of substance abuse treatment improvement protocols; the long-term effects of drug abuse treatment; and an evaluation of opioid treatment program accreditation.
A complete inventory of HHS evaluations completed in FY 1997 is provided in Appendix A, and HHS Agency projects currently in progress are listed in Appendix B.
The criteria used by the Evaluation Review Panel to select the reports highlighted in Chapter II are located in Appendix C.