Performance Improvement 1997

Foreword

The Department is pleased to present Performance Improvement 1997: Evaluation Activities of the U.S. Department of Health and Human Services--the second annual report documenting the evaluation efforts of all the HHS agencies and offices. The report has three audiences: decision makers, who need information on program results; program managers, who want to know how other service programs are operating and how performance can be improved; and the community of researchers, advocates, and practitioners, who will use the program information and evaluation tools.

The mission of the Department of Health and Human Services (HHS) is to enhance the well-being and health security of Americans by providing for effective health and human services and by fostering strong, sustained advances in the sciences underlying medicine, public health, and social services. To accomplish this mission, the Department manages an array of programs in basic and applied science, public health, child and adolescent development; programs that foster economic self-sufficiency and support working families; and programs that finance health, mental health, and social services.

Our programs are in a continuous state of reinvention and refinement, informed by the knowledge generated through our evaluation program. The Health Care Financing Administration (HCFA), for example, has multiple evaluations under way to understand the effects of Medicaid waivers. Policy makers and program managers at the Federal and State levels will be able to use information from these evaluations as they develop health care policy into the 21st century. HCFA also completed a study this year of hospital units for ventilator-dependent patients, described in the second chapter, that illustrates the usefulness of our evaluation knowledge for the larger health and human services community. The study's cost analysis found that these special units successfully demonstrated an integration of care for seriously ill patients that could hold promise for further application in managed care.

Increasingly, our programs are managed in partnership with State and local governments. Evaluation studies are an important tool for helping these partners improve program effectiveness. For example, the Centers for Disease Control and Prevention (CDC) assisted the State of Georgia by examining the effectiveness of a new audit method to improve childhood immunization rates. The study showed that the audits contributed significantly to the State's increase in child immunization rates (from 31 percent to 90 percent between 1994 and 1996). Georgia's audit method has been adopted with substantial success in Colorado, Illinois, and South Carolina. Another example is an evaluation by the Administration for Children and Families that helps States implement the child support guidelines established by the Family Support Act of 1988. This study led to greater consistency of guideline application in such areas as health insurance and day care expenses used in determining child support awards.

Performance Improvement 1997 contains information on projects completed in fiscal year (FY) 1996, HHS agency and office evaluations in progress, and future directions for evaluation. The report is organized into three chapters. Chapter I describes the organization of HHS evaluations--activities, resources, planning and management, and future directions. Chapter II highlights the results from 11 FY 1996 evaluations, selected for their potential application by the health and human services community. Chapter III presents the evaluation activities of the 11 HHS agencies and the Office of the Secretary, including information on their evaluation programs, evaluations completed in FY 1996, evaluations in progress, and future directions. A complete inventory of the 87 HHS evaluation projects completed in FY 1996 is provided in appendix A, and the HHS agency projects currently in progress are listed in appendix B.

We hope that you will find this report useful and informative.

Donna E. Shalala
Secretary
U.S. Department of Health and Human Services

David F. Garrison
Principal Deputy Assistant Secretary for Planning and Evaluation

Senior Editorial Advisors

The following individuals provided editorial guidance in developing Performance Improvement 1997:

Heather Becker, Ph.D.
   Professor, School of Nursing
   University of Texas
   Austin, Texas

Frank Caro, Ph.D.
   University of Massachusetts at Boston
   Gerontology Institute and Center
   Boston, Massachusetts

Michael Hendricks, Ph.D.
   MH Associates
   Bethesda, Maryland

Holly Korda, Ph.D.
   Evaluation Consultant
   Boston, Massachusetts

John Kralewski, Ph.D.
   Director, Institute for Health Services Research
   School of Public Health
   University of Minnesota
   Minneapolis, Minnesota

Mark Krushat, Ph.D.
   Office of Inspector General
   Department of Health and Human Services
   Washington, D.C.

Lynne Lackey, Ph.D.
   Professor, Department of Health Sciences
   Towson State University
   Towson, Maryland

Kenneth McLeroy, Ph.D.
   Health Promotion Sciences Department
   University of Oklahoma
   Oklahoma City, Oklahoma

Donna Mertens, Ph.D.
   Professor
   Gallaudet University
   Washington, D.C.

Ira Raskin, Ph.D.
   Health Policy Consultant
   EEI Communications
   Alexandria, Virginia

Mary Ann Scheirer, Ph.D.
   Evaluation Consultant
   Washington, D.C.

Acknowledgments

Performance Improvement 1997: Evaluation Activities of the U.S. Department of Health and Human Services describes the continuous efforts of the various agencies of the Department of Health and Human Services (HHS) to examine service and research programs for the efficiency of their operations and their effectiveness in achieving objectives. The planning, development, and coordination of those evaluations is largely the responsibility of the following HHS planning and evaluation offices:

Administration for Children and Families
   Office of Policy and Planning
   Howard Rolston, Director

Administration on Aging
   Office of Program Operations and Development
   Edwin Walker, Director

Agency for Health Care Policy and Research
   Office of Planning and Evaluation
   Phyllis Zucker, Director

Centers for Disease Control and Prevention
   Office of Program Planning and Evaluation
   Jeff Harris, Director

Food and Drug Administration
   Office of Planning and Evaluation
   Paul L. Coppinger, Associate Commissioner

Health Care Financing Administration
   Office of Research and Demonstrations
   Barbara S. Cooper, Acting Director

Health Resources and Services Administration
   Office of Planning, Evaluation and Legislation
   Ronald H. Carlson, Director

Indian Health Service
   Office of Planning, Evaluation and Legislation
   Leo Nolan, Director

National Institutes of Health
   Office of Science Policy and Technology Transfer
   Lana Skirboll, Associate Director

Office of the Assistant Secretary for Planning and Evaluation
   Office of Program Systems
   Susanne A. Stoiber, Deputy Assistant

Secretary for Program Systems
   Office of Public Health and Science
   Carol Roddy, Senior Program Advisor

Substance Abuse and Mental Health Services Administration
   Office of Policy and Program Coordination
   Frank Sullivan, Director

Preparation of this report and the noted appendices was coordinated by staff in the Office of the Assistant Secretary for Planning and Evaluation, under the supervision of Susanne A. Stoiber, Deputy Assistant Secretary for Program Systems. Responsible staff included Mike Herrell, Paul Johnson, Colleen Monaghan, Tricia Quiller, Carolyn Solomon, and Joan Lee Turek. Development and production of the report were supported, under contract, by the services of Deborah Burke, Lydia Paddock, and the staff of EEI Communications.

The following persons from HHS agencies contributed to writing, collecting information on evaluation activities, and reviewing the report:

Administration for Children and Families
   James V. Dolson
   Helen Howerton

Administration on Aging
   David Bunoski
   Saadia Greenberg

Agency for Health Care Policy and Research
   Irma Arispe
   Wendy Perry

Centers for Disease Control and Prevention
   Wilma Johnson
   Nancy Cheal

Food and Drug Administration
   Kathleen McEvoy
   Mary Gamunev

Health Care Financing Administration
   William Saunders
   Sydney Galloway

Health Resources and Services Administration
   Anabel Crane
   Susan Hammersten
   Mary Jean Skinner

Indian Health Service
   Frank Marion
   Sarah Crazythunder

National Institutes of Health
   John Uzzell
   Joan Bailey
   David Chananie

Office of Public Health and Science
   Carol Roddy
   Linda Bailey

Substance Abuse and Mental Health Services Administration
   Anna Marsh
   Maureen L. Moran

Executive Summary

Performance Improvement 1997 is the second annual report of the U.S. Department of Health and Human Services (HHS) on its evaluation activities. As a report to Congress, it summarizes the findings of HHS evaluations completed in fiscal year (FY) 1996. In that year, HHS agencies produced 87 evaluation reports and supported more than 250 evaluation projects in progress.

In general, the report is intended for three audiences: decisionmakers, who need information on program results across the broad spectrum of health and human services; program managers, who need information on how they can improve program operations and outcomes; and the health and human services community, which can benefit by applying the knowledge and lessons learned from HHS evaluations.

In the Foreword, Secretary Donna Shalala and David Garrison, Principal Deputy Assistant Secretary for Planning and Evaluation, stress that evaluation is an important tool for producing the knowledge necessary to develop and improve the performance of HHS's activities to meet the needs of the 21st century. HHS program managers are continuously engaged in efforts to determine whether programs and services reach the intended populations or communities, perform efficiently, and achieve desired results.

Chapter I

Chapter I provides an overview of evaluation at HHS. It describes a variety of evaluation activities; sources of funding; and evaluation management policies, including planning and coordination, project management, quality assurance, dissemination, and effective use of results. The last section in this chapter discusses future directions for HHS evaluation. In the coming years, the Department will examine the transformations now taking place in health and human services. The transformations underscore the need of the Department to play a leadership role in developing performance measures for implementation of the Government Performance and Results Act of 1993 (GPRA). HHS agencies remain committed to seeking information to continuously improve the operation of their programs.

Chapter II

Chapter II highlights 11 evaluations completed during FY 1996 and identified by the panel of Senior Editorial Advisors as particularly useful to the public health and human services community. For the selection criteria, see appendix C.

Evaluation of Child Access Demonstration Projects: Report to Congress

This Administration for Children and Families (ACF) Office of Child Support Enforcement report evaluates different forms of interventions to bring noncustodial parents closer to their children after divorce and separation. The interventions included mediation, parenting training, counseling, enforcement of visitation, and monitoring of visitation. The report found that mediation where both parties attend resulted in parenting plans in 65 to 70 percent of the cases. These parenting plans stimulated more visitation by noncustodial parents and better compliance with child support. A majority of both parents were satisfied with mediation. Other forms of intervention for more longer term and problematic cases did not register impact.

Evaluation of Child Support Guidelines

This ACF report evaluated the implementation of State child support guidelines mandated by the Family Support Act of 1988. The investigation concluded that States need to increase the consistency of guideline application, including income verification procedures, extended custody and visitation arrangements, and consideration of multiple family situations, health insurance, and day care expenses in the calculation of child support awards. Adopting a standardized support order would improve documentation of departures from guideline award calculations.

Serving Elders at Risk: The Older Americans Act Nutrition Programs. National Evaluation of the Elderly Nutrition Program, 1993-1995

The Elderly Nutrition Program sponsored by the Administration on Aging provides grants to State units on aging to subsidize the provision of daily meals to people 60 years of age or older. This evaluation report describes participant characteristics, compares the characteristics of recipients of home-delivered meals with those receiving meals in congregate settings, assesses how well the program reaches disabled and poor elderly, estimates the impact on nutritional intake and social contacts of participants, and examines program costs and other sources of funding. Findings suggest that participants have higher daily intakes of nutrients and more social contacts per month than a comparable group of nonparticipants. The Elderly Nutrition Program succeeds in targeting older people who are poor, live alone, are nutritionally "at risk" because they are overweight or underweight, or are more functionally disabled than their age-group peers. Federal expenditures were found to be highly leveraged with State, local, and private funds.

Oregon Consumer Scorecard Project (OCS)

The final report on the Agency for Health Care Policy and Research (AHCPR)-supported Oregon Consumer Scorecard Project describes the Oregon Consumer Scorecard Consortium's development, pilot testing, and revision of a user-friendly guidebook, "A Consumer Guide to Selecting a Health Plan." The prototype scorecard was designed to help consumers choose a health plan consistent with their individual needs and their health care service delivery preferences. The model may also help purchasers and State policymakers develop their own comparative reports about health plan performance.

Georgia State Clinic Assessments of Immunization Coverage: Impact of Management and Clinic Immunization Practices

Between 1986 and 1994, the State of Georgia implemented a program of audits of the immunization practices of all State public health immunization clinics in an effort to improve child immunization rates. During this time, the immunization rate for preschoolers increased from 31 percent to 90 percent. The Centers for Disease Control and Prevention's (CDC's) evaluation of the program found that the performance of these audits was significantly associated with higher immunization rates for preschool children in the clinic catchment areas. This and other report findings formed the basis of CDC recommendations to all States.

Ventilator-Dependent Unit Demonstration: Outcome Evaluation and Assessment of Post-Acute Care

This Health Care Financing Administration evaluation assessed whether specialized units serving chronic ventilator-dependent patients deliver better clinical outcomes at a reasonable cost to Medicare than conventional hospital settings. Findings included improved patient outcomes for patients treated in ventilator-dependent units and lower hospital daily costs, yet higher overall costs to Medicare because of greater patient longevity. The analysis of costs and outcomes provides important information to policy makers and health care providers about a seriously ill population with intensive resource needs.

Evaluation of the Bureau of Health Professions Strategic Directions: Development of a Cross-Cutting Performance Monitoring System

This Health Resources and Services Administration (HRSA) evaluation reports on Phase II of a three-phase effort to develop a fully integrated computerized data system to facilitate planning and evaluation of the programs administered by the Bureau of Health Professions. A performance monitoring system is proposed to facilitate ongoing program management. Such a performance system is central to the ability of the Bureau of Health Professions to measure whether program support is meeting its national health workforce objectives, and to signal where program course correction is necessary.

Analysis of the Effect of Regulation on the Quality of Care in Board and Care Homes

This evaluation report, sponsored by the Office of the Assistant Secretary for Planning and Evaluation (ASPE), assesses the effects of State board and care home regulations on the quality of care provided by those institutions. The investigation also attempts to identify the characteristics of the board and care environments in the States selected for the study and to determine whether licensed and unlicensed homes differed in terms of those characteristics, as well as in the quality of care provided. The evaluation concludes that extensive State regulation and licensure can improve the quality of care provided by board and care homes and can increase the number of these facilities that hold licenses.

Health Care in Transition: Technology Assessment in the Private Sector

This report sponsored by ASPE briefly describes a number of private-sector health organizations that are engaged in technology assessment. Technology assessment is one of a family of evaluative activities circulating within these health care organizations. Among private-sector organizations conducting technology assessment, rigorous evaluation of clinical effectiveness based on a systematic review of scientific and clinical evidence has become the norm, and technology assessment has increasingly become "full service," encompassing drugs, medical devices, and clinical procedures.

Subacute Care: Policy Synthesis and Market Area Analysis

The report of this ASPE study of subacute care presents an overview and historical perspective of subacute care, a description of the current state of the art and how it varies across selected market areas, and an examination of evidence about its cost and effectiveness. Findings suggest that the emerging concept of subacute care holds promise but also poses many challenges for the public and private sectors.

Cost-Effectiveness in Health and Medicine

Cost-effectiveness analysis (CEA) is one tool available to decisionmakers to help determine the relative value of different approaches to improving health or life expectancy. For a variety of reasons, however, CEA has not been widely used in health policy settings. This Office of Public Health and Science report represents a review of the methodological and theoretical aspects of CEA, its effectiveness, and its expected outcomes and uses. Recommendations are made regarding framing and designing such studies, identifying and valuing outcomes, assessing effectiveness, estimating costs, discounting, reflecting analytical uncertainty, and reporting analyses.

Chapter III

Chapter III provides an overview of the evaluation activities of the HHS agencies, including information on the evaluation program, a summary of evaluations completed during FY 1996 and evaluations in progress, and a discussion of future directions for agency evaluations.

Administration for Children and Families

ACF administers a broad range of programs to support children and families and targeted populations, such as the developmentally disabled and Native Americans. The objectives of ACF evaluations are to provide information on the program design and operations, to test delivery approaches of various services, to conduct policy analyses, and to disseminate and apply evaluation results. In FY 1996, evaluations were completed on child support guidelines, child access demonstration projects, Head Start family self-sufficiency initiatives, home-based services for runaway youth, gang families in public housing projects, foster youth mentors, child protective and preventive services, family-based placement prevention programs, and a case management enhancement project. Major evaluations in progress include the Arkansas prenatal and postnatal paternity project, responsible fatherhood initiatives, home visitor services for teen parents, community development demonstrations, the Head Start program, and foster care placement. Two new National Child Welfare Research Centers have also been established.

Administration on Aging

The Administration on Aging (AoA) supports studies that provide information on the implementation of the Older Americans Act of 1992. This year, AoA completed a major evaluation of the Elderly Nutrition Program, which provides an average of one million meals per day to older Americans. Results showed that the program has been successful in improving the nutritional intakes of elderly people and in decreasing their social isolation. The report also provided guidance to help the program change to meet future needs of the elderly population and adapt to changes under way in the health care system. AoA is now helping many State and area agencies conduct evaluations of programs supported by the Older Americans Act.

Agency for Health Care Policy and Research

The goals of the AHCPR evaluation program are to assess agency effectiveness in meeting major and long-term priorities; to obtain information to respond to critical agency and departmental concerns; and to conduct internal evaluations to improve the efficacy of key program areas. Evaluation projects completed in FY 1996 included an evaluation of practice guidelines and science-based measures of clinical quality; an evaluation of CONQUEST 1.0, a data base of clinical performance measures useful for matching measures appropriate for different clinical conditions; the Oregon Consumer Scorecard Project; and a synthesis of findings from AHCPR's Medical Treatment Effectiveness Program (MEDTEP). In progress during FY 1996 were evaluation activities to assess the health care environment; evaluations of AHCPR publications for research dissemination and training for minority health services researchers; and several evaluation design studies, such as an assessment of data release strategies for the AHCPR's Medical Expenditure Panel Survey.

Centers for Disease Control and Prevention

CDC places high priority on evaluations seeking to answer policy, program, and strategic planning questions related to the Agency's mission. This year's major evaluations include data policy and surveillance studies, such as developing a comprehensive monitoring system to track the impact of health care reform. Program evaluations completed include CDC's National Nosocomial Infections Surveillance Program and the Dengue Hemorrhagic Fever Prevention and Control Program in San Juan, Puerto Rico. CDC also assisted the State of Georgia in a study of immunization auditing procedures and found positive results for increasing child immunization rates. CDC's evaluations in progress include evaluations of data reporting systems, such as one for public health laboratories and CDC's National Electronic Telecommunications System for Surveillance. Program evaluations are under way in the areas of occupational health hazards, epidemiology training, and prevention centers. Other specialized projects are focusing on tuberculosis outreach workers, minors' access to tobacco, and violence prevention.

Food and Drug Administration

Evaluation at the Food and Drug Administration (FDA) is largely a line-management responsibility. Projects focus on performance management, customer participation, and rigorous rulemaking. In FY 1996, FDA continued to identify performance measures jointly with industry customers as part of the drug and biologic review processes necessary for implementing the Prescription Drug User Fee Act. In addition, FDA completed two evaluation projects on food labels. The first reports on the impact of the new food label regulations on consumers, and the second assesses the status of processed, packaged food labels subsequent to publication of the regulations. During FY 1996 FDA program managers were engaged in evaluating their performance measures in light of GPRA standards. Other evaluation projects at FDA include assessing FDA's penalty-reduction and inquiries assistance program for small businesses, influenza virus vaccines, and the national evaluation of x-ray trends.

Health Care Financing Administration

The Office of Research and Demonstrations performs and supports Health Care Financing Administration (HCFA) evaluations to develop new health care financing policies and to provide information on the impact of HCFA's programs. In FY 1996, HCFA completed evaluations of ventilator-dependent hospital unit care, Medicaid drug rebate policies, the Arizona health care cost-containment system, telemedicine, home health prospective payments, physician payment reform, and pharmacy benefit managers. HCFA has over 34 projects in progress, with 24 scheduled to be completed in FY 1997. These projects are monitoring the Medicare and Medicaid programs through demonstrations. Some of the demonstrations involve the Medicaid managed care programs or waivers, Medicare case management demonstrations, ambulatory patient groups, Medicaid uninsured demonstrations, rural health clinics, community-supported living arrangements, Medicare Alzheimer's disease demonstration, long-term care for persons with developmental disabilities, and health plan report cards.

Health Resources and Services Administration

The objectives of the Health Resources and Services Administration (HRSA) evaluation program are to improve program management and policy development and to provide information for strategic planning, budget decisions, and legislation. During FY 1996, HRSA continued to enhance its performance measurement capacity for all line programs, with a notable study completed at the Bureau of Health Professions. Special evaluation projects were performed on quality assurance procedures for Ryan White Comprehensive AIDS Recovery Emergency (CARE) Act grantees; managed care and the Federally Qualified Health Centers; the impact of case management in community health centers; emergency medical services for children; international medical graduates; advanced practice nursing education; and primary care residency programs. Ongoing evaluations in FY 1996 include managed care and community and migrant health centers; health maintenance organization, or HMO, staffing strategies in underserved areas; casemix differences in health centers; assessment of multilingual services offered at community and migrant health centers; the Healthy Start Program to reduce infant mortality; Ryan White CARE Act programs; organ transplantation; telemedicine; and the supply of nurses.

Indian Health Service

The Indian Health Service (IHS) evaluation program serves the agency's program and policy objectives, developed in consultation with the tribal communities. IHS's major evaluations completed in FY 1996 concentrated on developing a methodology for adjusting IHS mortality data for inconsistent classifications; the IHS adolescent regional treatment centers; and the second phase of an evaluation of alcohol and substance abuse services for American Indian and Alaska Native women. Projects in progress are focusing on the impact of a promotional initiative on elderly wellness and use of comprehensive services; a study of prior trauma care of intoxicated patients as a predictor of subsequently fatal injury; an assessment of Resource Requirements Methodology as an IHS management tool; the development of an IHS health services research agenda; and the continuing evaluation of alcohol and substance abuse treatment services for women.

National Institutes of Health

Evaluation is an integral part of the role of the National Institutes of Health (NIH) in supporting biomedical research, training, and public education. In FY 1996, NIH's evaluations focused on the NIH shared instrumentation grant programs, clinical research at the National Institute of Diabetes and Digestive and Kidney Diseases, an evaluation of the Navajo alcohol rehabilitation demonstration, and the National Institute on Drug Abuse's study on the development of medications for the treatment of opiate and cocaine addictions. The major evaluations in progress during FY 1996 were an evaluation of the career status and satisfaction of NIH grant applicants with grant application processes and procedures; comprehensive school health programs in grades K-12; an evaluation of laboratory animal use; and the development of a data base for NIH-supported research in rare diseases.

Office of the Assistant Secretary for Planning and Evaluation

The Assistant Secretary for Planning and Evaluation (ASPE) functions as principal advisor to the secretary on policy development, conducting evaluations and policy research studies. It is also responsible for HHS-wide coordination of legislative, planning, and evaluation activities. During FY 1996, ASPE's evaluations looked at health care technology assessment, information to guide physician practice, assumptions underlying insurance participation modeling, board and care homes, subacute care, integrating acute and long-term care for children with disabilities, interdisciplinary professional training on disabilities, substance abuse treatment for parents and welfare recipients, community response to domestic violence, minority male violence prevention, and American Indian demography. The major evaluations in progress include monitoring changes in health and human services, assessing data needs, managed care and people with disabilities, assisted living, family preservation services, moving welfare recipients to work, promoting involvement of fathers in family life, and trend data on the well-being of children.

Office of Public Health and Science

The Office of Public Health and Science (OPHS) provides advice, policy and program coordination, and leadership in the implementation, management, and development of HHS public health and science activities. Several major OPHS-supported evaluations were completed in FY 1996 on the cost-effectiveness methodology; quantification of State public health expenditures; health needs of recently arriving refugees; and scientific misconduct and the consequences of being accused. Evaluations and policy studies in progress include dietary supplement labels; framework for Healthy People 2010, school health programs, public health infrastructure, and public health performance measures.

Substance Abuse and Mental Health Services Administration

The Substance Abuse and Mental Health Services Administration (SAMHSA) is committed to evaluating its programs and grant projects to assess the effectiveness of prevention, treatment, and rehabilitation approaches and system of care. In FY 1996, SAMHSA completed two evaluation projects. The first was an overview of addiction treatment effectiveness; the second was a study of the employment outcomes of indigent clients receiving alcohol and drug treatment in Washington State. Ongoing evaluations include evaluations of SAMHSA's demonstration programs, such as the Job Corps program of enriched substance abuse treatment for adolescents; the services integration experiment for chronically mentally ill homeless persons; and the National Treatment Improvement Evaluation Study. Evaluations are also under way to examine the impact of managed care on the access, cost, and quality of substance abuse treatment.

Appendices

A complete inventory of the 81 HHS evaluations completed in FY 1996 is provided in appendix A, and HHS agency projects currently in progress are listed in appendix B. Appendix C describes the Senior Editorial Advisor's review criteria for assessing program evaluations. A glossary of acronyms used throughout this report is presented in appendix D.

Chapter I. Evaluation in the Department of Health and Human Services

The mission of the U.S. Department of Health and Human Services (HHS) is to enhance the well-being and health of Americans by providing for effective health and human services and by fostering strong, sustained advances in the sciences underlying medicine, public health, and social services. The following HHS agencies and offices accomplish this mission through their program activities:

• Administration for Children and Families (ACF)
• Administration on Aging (AoA)
• Agency for Health Care Policy and Research (AHCPR)
• Agency for Toxic Substances and Disease Registry (ATSDR)
• Centers for Disease Control and Prevention (CDC)
• Food and Drug Administration (FDA)
• Health Care Financing Administration (HCFA)
• Health Resources and Services Administration (HRSA)
• Indian Health Service (IHS)
• National Institutes of Health (NIH)
• Office of the Secretary
• Substance Abuse and Mental Health Services Administration (SAMHSA)

The Assistant Secretary for Planning and Evaluation (ASPE), located in the Office of the Secretary, coordinates evaluation activities throughout HHS.

Evaluation plays an integral role in carrying out the HHS mission: assessing various aspects of program performance of the HHS agencies enables staff to identify ways to improve that performance. The HHS evaluation function has three goals:

1. To provide information on HHS programs that helps government officials and members of Congress make decisions related to programs, policies, budgets, and strategic planning;
2. To help HHS managers improve program operations and performance; and
3. To disseminate evaluation results and methodological tools useful to the larger health and human services community of State and local health and human service officials, researchers, advocates, and practitioners for improving the performance of their programs.

This last goal is very important to HHS. Its various agencies have an important obligation to foster the development of new knowledge about the effectiveness of health and human services programs, interventions, and evaluation tools for use by the larger health and human services community. Although the findings and recommendations of HHS evaluations are usually first used by the Administration and the Congress, they can also be applied by others in the research and practice communities to improve the performance of programs at the State and community levels. The purpose of this report is to disseminate information about recent HHS evaluations and to make sure that the potential for wider application is realized.

This chapter describes the organization and operation of evaluation at HHS. It first provides an overview of the kind of evaluation activities supported by HHS agencies and then describes the resources and funding mechanisms used to support them. It details HHS evaluation management, including planning procedures, project management, quality assurance, dissemination of results, and effective uses of evaluation. The chapter concludes with a discussion of future directions for evaluation at HHS.

HHS Evaluation Activities

HHS defines evaluation as the assessment of program performance (efficiency, effectiveness, and responsiveness) through the analysis of data or information collected systematically and ethically, and the effective use of resulting information in program or policy decision making and program management. This definition encompasses a range of evaluation activities that include the full spectrum of evaluation methodologies developed over the last quarter century. The classification of HHS evaluation activities presented in figure I-1 summarizes that diversity.

Figure I-1. Range of HHS Evaluation Activities

Evaluation projects

1. Outcome evaluations: assessing the immediate or intermediate effects of a program with respect to the stated goals or objectives
2. Impact evaluations: assessing the broader results, intended or unintended, of a program on populations or institutions involved
3. Implementation or process evaluations: assessing the nature of program inputs and outputs and their relationship to stated goals and obectives
4. Policy assessments: examining health policies with respect to their development, implementation, or impact on public health or program activities
5. Cost-benefit or cost-effectiveness analyses: developing methodology and its application to assess the relationship of program results to program costs (direct and indirect), often in comparison with alternative programs
6. Survey data analyses: evaluating the results of HHS programs or policies by analyzing data obtained from surveys
7. Performance measurement and data systems: identifying and testing the validity and reliability of process, output, and outcome indicators to measure the performance of programs and develop data systems supporting implementation of the Government Performance and Results Act
8. Simulations and models: using computer simulations and modeling techniques to analyze the impact of policy changes on service delivery systems and beneficiaries
9. Management studies: examining the effectiveness or efficiency of the administration or operation of HHS programs and offices
10. Evaluation syntheses: integrating the results from multiple independent evaluation studies within a defined program or policy area in a fashion that improves the accessibility and application of those results

Methodology projects

1. Evaluation feasibility studies: assessing the clarity and importance of program goals and objectives, the consensus of program stakeholders on the potential utility of evaluation information, and the availability of relevant performance data before committing to a full-scale program evaluation
2. Evaluation design projects: procuring assistance in the development of an evaluation design, measurement tool, or analytic model in preparation for fully implementing an evaluation
3. Instrument development projects: developing evaluation instruments (design, measurement, or analytic) for a specific HHS program or for general use by the health and human services community

Evaluation support activities

1. Evaluation technical assistance: providing assistance to HHS program managers or office directors on any aspect of evaluation planning, project design-implementation-analysis, or use of results
2. Evaluation dissemination: identifying target audiences and mechanisms to inform program constituencies and evaluation stakeholders about evaluation results
3. Evaluation training/conferences: maintaining the professional skills and expertise of evaluation staff through training opportunities, as well as promoting the dissemination of HHS evaluations through symposia

Evaluation Resources

Evaluation activities of the various HHS agencies are largely supported through two funding mechanisms: direct use of programs funds and use of special legislative set-aside authorities for evaluation. The first is a common mechanism giving program managers discretionary authority to use appropriated program funds to support contracts that will design, implement, and analyze evaluation data. In some cases, a program's legislative authority calls for a special mandated evaluation, and program funds are used directly to support the evaluation.

The second mechanism for evaluation funding is legislative set-aside authorities permitting the Secretary of HHS to use a proportion of overall program funds for evaluation purposes. The largest of such set-aside authorities is one established for evaluations conducted by several agencies of the U.S. Public Health Service (AHCPR, CDC, HRSA, NIH, and SAMHSA), ASPE, and the Office of Public Health and Science (OPHS) in the Office of the Secretary. The mechanism, called the 1-percent evaluation set-aside legislative authority, is provided for in Section 241 of the Public Health Service (PHS) Act. This authority was established in 1970 when Congress amended the Act to permit the HHS Secretary to use up to 1 percent of appropriated funds to evaluate authorized programs. Section 241 limits the base from which 1percent of appropriated funds can be reserved for evaluations of programs authorized by the PHS Act. Excluded are funds appropriated for FDA, IHS,¹ and certain other programs that are managed by PHS agencies but not authorized by the Act (e.g., HRSA's Maternal and Child Health Block Grant and CDC's National Institute for Occupational Safety and Health).

In FY 1996, HHS invested more than $33 million in set-aside evaluation funds to carry out evaluation activities related to relevant public health programs. In FY 1997, HHS estimates that it will use approximately $35 million in PHS evaluation set-aside funds to continue current evaluation activities and to initiate new evaluation projects. Table I-1 provides a breakdown of the usage for FY 1996 by PHS agencies, ASPE, and OPHS, as well as the estimates for FY 1997.

Table I-1. Agency Use of Evaluation Set-Aside Funds,
in Thousands of Dollars

	FY 1996	FY 1997
Agency evaluation use:
AHCPR	$0	$250
CDC	2,000	2,000
HRSA	6,680	6,868
NIH	4,300	4,500
SAMHSA	995	1,423
ASPE	15,500	15,500
OPHS	3,852	4,552
Total use	$33,327	$35,093

Evaluation Management

The management of HHS evaluations, carried out on a regular basis by the HHS agencies and offices and coordinated by ASPE, involves five basic functions:

1. Evaluation planning and coordination
2. Project management
3. Quality assurance
4. Dissemination of evaluation reports
5. Ensuring effective use of evaluation results

A description of each function in general terms follows. Additional information on the individual HHS agencies, ASPE, and OPHS evaluation functions can be found in chapter III.

Evaluation Planning and Coordination

The HHS agencies, ASPE, the Office of the Inspector General (OIG), and OPHS develop evaluation plans annually, in concert with HHS's program planning, legislative development, and budgeting cycles. Plan development is coordinated by ASPE. Each agency or office plan generally states the evaluation priorities or projects under consideration for implementation. Typically, HHS evaluation priorities include congressionally mandated program evaluations, evaluations of Secretarial program or policy initiatives, and assessments of new programs, programs that are candidates for reauthorization, or programs for which key budget decisions are anticipated.

HHS evaluation planning activities are required more specifically in two ways. First, those agencies and offices that use the PHS 1-percent evaluation set-aside authority--AHCPR, CDC, HRSA, NIH, ASPE, OPHS, SAMHSA--submit a formal plan to ASPE, which coordinates and develops the individual plans into the HHS report to Congress on the use of the PHS 1-percent authority. This report must be submitted to Congress before HHS can implement the plan.

Second, Congress has asked HHS to coordinate all of its research, demonstration, and evaluation programs to ensure that the results of these projects address HHS's program goals and objectives. ASPE and the Assistant Secretary for Management and Budget now work with HHS agencies to provide Congress with a special annual research, demonstration, and evaluation budget plan that coincides with the preparation of the President's fiscal year budget. The plan outlines each HHS agency's research, demonstration, and evaluation priorities as related to overall HHS program goals and objectives.

Project Management

The design and management of evaluation projects at HHS is primarily decentralized--the various HHS agencies, OIG, and ASPE are all responsible for executing annual evaluation plans, developing evaluation contracts, and disseminating and applying evaluation results. Even within agencies, while there is some oversight responsibility and execution capability in the Office of the Director or Administrator, the various subunits (centers, institutes, bureaus) conduct much of the day-to-day evaluation activity.

OIG performs independent evaluations through its Office of Evaluations and Inspections (OEI). OEI's mission is to improve HHS programs by conducting inspections that provide timely, useful, and reliable information and advice to decisionmakers. This information (findings of deficiencies or vulnerabilities and recommendations for corrective action) is usually disseminated through inspection reports issued by the Inspector General. A summary of individual inspection reports and other OIG reports can be viewed on the Internet at http://www.sbaonline.sba.gov/ignet. OEI also provides technical assistance to HHS agencies in conducting their evaluations.

Quality Assurance

Most evaluation projects are developed at the program level. The initial quality review is generally conducted by a committee of agency- or office-level policy and planning staff members. Before a project is approved, however, it is also reviewed for technical quality by a second committee of staff who are skilled in evaluation methodology. Technical review committees follow a set of criteria for quality evaluation practice established by each agency. Some HHS agencies also have external evaluation review committees composed of evaluation researchers and policy experts from universities and research centers. More details on the quality assurance procedures for the various HHS agencies, ASPE, and OPHS are presented in chapter III.

Dissemination of Evaluation Reports

Maintaining report libraries and distributing information on evaluation results is an important component of HHS evaluation management. Project information and reports are continuously submitted to the HHS Policy Information Center (PIC)--the departmental evaluation data base and library maintained by ASPE. As an information data base and library resource, the PIC contains nearly 7,000 completed and in-progress evaluation and policy research studies conducted by the Department, as well as key studies completed outside HHS by the U.S. General Accounting Office and private foundations.

Typically, the results of HHS evaluations are disseminated through targeted distribution of final reports, articles in refereed journals, and presentations at professional meetings and conferences. Although the individual HHS agencies have primary responsibility for disseminating results, there is a departmentwide effort under way to expand dissemination to the larger research and practice communities through centralized computer communications and publications. Abstracts of all studies maintained in the PIC evaluation data base are on the Internet through the HHS World Wide Web server, at the following URL: http://aspe.hhs.gov/PIC/gate2pic.htm. Information is available on completed and in-process projects; it includes the name and telephone number of the HHS official responsible for the project.

In addition, HHS is widely distributing copies of its first departmentwide annual report on evaluation (Performance Improvement 1996: Evaluation Activities of the U.S. Department of Health and Human Services). The report's theme of performance improvement represents the numerous changes and initiatives throughout HHS to increase the effectiveness and efficiency of health and human services programs. This report to Congress summarizes the findings of HHS evaluations completed during FY 1995. In that year, HHS agencies produced 148 evaluation reports and supported more than 200 evaluation projects in progress. The report provides summaries or abstracts of the completed project reports and contacts for further information.

In addition to providing the report to members of Congress, copies were sent to State and local public health and social services officials, various associations for schools of public health, social welfare, and social work, and other national health and social policy research and practice associations. A similar dissemination strategy is being used for this report, the second annual HHS report on evaluations. Both reports, Performance Improvement 1996 and Performance Improvement 1997, are also available on the previously described HHS home page in three computer formats for downloading information.

Ensuring Effective Use of Evaluation Results

HHS is committed to ensuring that evaluations yield a high return on the investment of available program funds. In the last decade, HHS evaluations were used primarily by program managers for internal purposes of improving program operations and efficiency. The major outcome or impact evaluations conducted by HHS were generally congressionally mandated and involved a substantial commitment of evaluation resources. In the 1990's, however, the demand for more program outcome and impact evaluations has increased, largely in response to fulfilling HHS's responsibility for implementing reform legislation. To understand the effects of these transformations on HHS programs at the system and individual levels will require a substantial investment of evaluation resources in outcome or impact studies. In addition, HHS is giving priority to evaluation efforts related to implementation of the Government Performance and Results Act (GPRA), which concerns all HHS programs. The expectations for GPRA are that by FY 1999 HHS will have in place the capacity to measure the performance of its programs with data that are complete, accurate, and consistent enough to support decision making at various management levels. The measures should be tied to program performance goals and should have the capability of measuring program results in achieving those goals.

Future Directions in HHS Evaluation

Last year's annual report, Performance Improvement 1996, stated that HHS agencies, ASPE, and OPHS would focus future evaluation efforts in three directions: (1) the impact of transformations in health and human services; (2) the development of performance measures; and (3) overall program performance improvement. In FY 1997, those directions continue to be the principal evaluation priorities for HHS, and its evaluation resources will be used to generate new knowledge to improve the performance of its programs.

Impact of Transformations

The Secretary remains committed to an evaluation strategy to examine the transformations now taking place in health and human services and the impact of those changes on the well-being of Americans--especially the vulnerable populations that are a high priority for HHS programs, such as disadvantaged or low-income children and families, the elderly, racial and ethnic minorities, and individuals with disabilities.

The financing and delivery of health care in the United States is being transformed by the growth and evolution of managed care and by other innovations in the financing, management, and delivery of health services. HHS agencies need to better understand how these transformations directly affect their programs. For example, HCFA is now developing a comprehensive monitoring and evaluation plan to examine the Medicare and Medicaid programs on such dimensions as access to care, quality, efficiency, cost, and beneficiary satisfaction. They will implement new demonstrations that test alternative payment and health care delivery models and evaluate the results.

Public health programs supported by HHS are an essential part of the health care safety net for the uninsured and a vital element in protecting the health of all Americans. For HHS, an increasing challenge is to understand how managed care and other financing and service delivery innovations affect public health systems, public health programs, and the clients those programs serve. For example, HRSA and SAMHSA have already made significant investments to study the effects of new financing and service delivery mechanisms, especially managed care, on their programs, services, and clients. In upcoming years, ASPE will complement their efforts by undertaking additional evaluations focused on the public health system and programs. Particular emphasis will be given to evaluation issues that address the public health infrastructure; how changes affect vulnerable populations served by the public health system; and how program transformations taking place outside the health arena, such as welfare reform, interact with and affect the public health care system. In addition, HHS will continue to support evaluations to determine the effect of managed care on individuals with disabilities who have both health and long-term care needs.

In parallel, human services programs are undergoing transformations in their purpose, organization, and financing. Enactment of the Personal Responsibility and Work Opportunity Reconciliation Act of 1996 eliminated individual entitlement to cash assistance and replaced it with a fixed block grant to States. The law also placed a 5-year time limit on benefits, imposed strict work requirements on recipients, reduced benefits and services available to legal immigrants, and greatly expanded States' authority over welfare programs. The Nation has a great deal at stake in the success of this program, and timely and careful evaluation will be important to help ensure that success.

Studies are needed to evaluate the combined effects of changes taking place in health care delivery systems and in income supports for low-income children and families who often have special needs. For example, one issue is how to help substance abusers or victims of domestic violence whose entitlements and access to services will change under the terms of new welfare reform legislation. Under the new law, welfare recipients generally will be required to work to receive benefits and will face overall time limits on welfare benefits. In the past, many clients identified as substance abusers or victims of domestic violence were not required to participate in work activities. While some of these clients will be exempt under the new law, others must enter the workforce. Case managers need better tools to assess clients' barriers to sustaining employment and to assist these clients in addressing those barriers.

Children's issues will also be a priority. HHS will identify and evaluate effective tools--including methods of linking children and families under the new welfare system with health and family support services--and on understanding the extent and manifestations of health problems among the welfare population.

The interaction of health and welfare changes also raises questions about the impact of welfare changes on the health status and well-being of selected populations of disabled children, low-income mothers with disabilities, and elderly people with health and long-term care needs. For many in these groups, access to health care and supportive services was previously dependent on entitlements that may no longer exist. Studies are needed to examine how these populations obtain necessary health and long-term care services once their entitlements are terminated, as well as the extent to which their health status and quality of life changes. For example, the AoA is interested in examining how State aging agencies use funds under the Older Americans Act, along with other sources of funding, to develop comprehensive and coordinated systems of community-based services for the elderly. These issues become particularly important in light of the demographic changes facing the Nation as the population ages.

Development of Performance Measurements

Implementation of GPRA requires HHS to focus a substantial portion of its evaluation activities on the development of performance measures for its programs and on information systems necessary to produce the data expected to support GPRA's performance measurement requirements. By the end of FY 1997, HHS will have in place a departmentwide strategic plan. The plan will contain the HHS mission statement, the long-term strategic goals, and descriptions of how HHS intends to achieve those goals with its program resources. Next, HHS will develop its annual performance plan for FY1999, which will link the HHS strategic goals to specific program activities. Beginning with FY1999, the third major GPRA element, the Program Performance Report, will present performance data related to the strategic goals based on the previously identified performance indicators.

The performance plan for FY 1999 will identify the performance measures that HHS agencies and offices will use to assess program performance related to achieving the goals. GPRA has several expectations for these measures:

1. They should to be tied to program goals and demonstrate the degree to which the desired results are being achieved.
2. The number of measures for each goal should be limited to avoid confusion in assessing accomplishments and making adjustments.
3. Measures should be responsive to multiple priorities, such as quality, cost, customer satisfaction, and stakeholder concerns.
4. Performance measures should be linked directly to the offices responsible for making programs work in order to reinforce accountability.

Since GPRA was passed in 1993, HHS agencies have utilized evaluation projects to develop performance measures related to its health and human services programs. For example, AHCPR has initiated projects to develop and test the validity and reliability of agencywide performance measures of its health services research grants process and dissemination mechanisms. CDC now has projects to develop indicators to assess performance of its HIV-prevention community-planning programs and its State-based diabetes control cooperative agreements program. HRSA has invested its evaluation resources in training and technical assistance to program managers to develop measures for GPRA implementation. Featured in chapter II is an HRSA report about the efforts of the Bureau of Health Professions to develop a set of outcome-based performance measures and a plan for a monitoring system to support program management.

In addition to performance measures for HHS programs, the Federal grant relationships to States in public health are evolving into health outcomes-based performance management. For the past two years, ASPE and OPHS have collaborated on several cross-cutting efforts to involve States, communities, and service recipients in identifying such a data system. A major project with the National Research Council is identifying measures that can be used jointly by HHS agencies and States to measure their shared performance in grant management. The project will also recommend specific steps that can be taken to improve State and local community data capabilities. A related project is developing approaches to obtain comprehensive baseline and trend data on public health infrastructure. A third study is building on a Robert Wood Johnson Foundation project to assess the quality of data that States collect and determine whether there is a foundation on which to build a multi-State health data system that can be shared among States to improve health policy decision making.

HHS will continue to invest its evaluation resources in performance indicators to ensure a successful implementation of GPRA. The evaluation strategies of HHS agencies, mentioned in chapter III, include the priorities of projects that examine program performance objectives and develop useful measures of program outputs and outcomes. GPRA offers HHS agencies an opportunity to develop performance measurement systems that will eventually link program evaluation activities to budgeting. HHS's evaluation set-aside authority, such as the 1-percent authority for some PHS agencies, is an important resource to help program managers identify performance objectives and test the validity and reliability of progress indicators.

Performance Improvement

Program managers must initiate evaluations useful for improving the performance of HHS programs and ensuring that program operations are efficient and effective. Evaluations are an essential resource for HHS's Continuous Improvement Program, aimed at supporting development and operation of information systems and special studies to enable program managers to measure customer satisfaction with HHS services.

Several in-progress or planned projects illustrate HHS's evaluation priority of continuous improvement of services. AHCPR is initiating a study to examine the Medical Expenditure Panel Survey regarding the efficiency of its new design and its effectiveness in answering key health policy questions. They will look at the survey process, including data support contracts and the capacity of the survey data for developing health services analytical models.

FDA is examining its program to assist small businesses in complying with FDA rules, which includes determining FDA's success in coordinating different sources of agency advice and the appropriateness and timeliness of the advice. The evaluation will consider the training and materials available to FDA staff to help them answer questions from small businesses, identify barriers to timely and useful responses, and develop procedures for tracking inquiries.

One of HRSA's priorities is to improve access to the continuum of care for persons with HIV/AIDS through the Ryan White Comprehensive AIDS Recovery Emergency (CARE) Act programs. One project concerns the management of persons infected with both HIV and tuberculosis (TB) in metropolitan areas to identify models of successful collaboration between HIV/AIDS and TB control programs. The purpose is to determine the extent to which Ryan White CARE Act grantees are offering guidance to providers on TB screening and prophylactic and therapeutic regimes.

NIH is currently conducting a survey of individuals who applied to NIH for a research grant in FY 1992. The study will obtain information on the career progress and research productivity of a sample of grant applicants. The purpose is to assess the respondent's satisfaction with the grants application and award process and to get their opinions on specific modifications NIH is considering for improving policies and procedures.

SAMHSA's priority of evaluating its Knowledge, Development, and Application (KDA) grants is also a good example of HHS's commitment to continuous improvement of programs. Evaluating the current KDA grants will generate new data on the effectiveness of alternative models of managed care for treatment and prevention of substance abuse and mental disorders, including specific programs for homelessness among adults with serious mental illnesses and brief interventions for marijuana dependence.

Notes

1. FDA programs are principally authorized by legislation other than the PHS Act, specifically the authority of the Agriculture, Rural Development, Food and Drug Administration and Related Agencies Appropriations Act. IHS programs are authorized under the Indian Health Care Improvement Act and the Indian Self-Determination Act; they are appropriated under the Department of the Interior and Related Agencies Appropriations.

Chapter II. Highlights of Selected Evaluations Completed During Fiscal Year 1996

In this chapter, the U.S. Department of Health and Human Services (HHS) highlights evaluations of general interest to the public health and human services community and illustrates the diversity of HHS evaluations completed in fiscal year (FY) 1996. Included are summaries of 11 evaluation projects selected by the Senior Editorial Advisors on the basis of criteria identified in appendix C and applied to 34 reports nominated by HHS agencies. These criteria are as follows:

• Is the report important? Does it address a significant issue or problem for which evaluation would help confirm or change program direction? Does it measure program impact? Are the findings likely to be useful and generalizable?
• Is the report methodologically sound? Are its concepts, designs, data collection, and analyses conducted and reported in a competent manner?
• Is the report faithful to the data? Do the conclusions and recommendations logically follow from the data and analyses, and are they relevant to the questions asked?

Each summary includes a brief abstract; a description of the study, including its purpose, background, methods, findings, and use of results; the names of any publications that resulted; and the name and phone number of the person to contact for additional information.

Administration for Children and Families

Evaluation of Child Access Demonstration Projects: Report to Congress

Highlights

This study evaluated the results of eight Child Access Demonstration Projects from two waves of demonstrations testing the effectiveness of mediation, counseling, education, and visitation monitoring programs designed to facilitate noncustodial parents' access to their children following divorce and separation. The incidence of access problems ranged from 31 percent to 4 percent of cases. The nature of access disputes varied widely; such disputes were reported by both the custodial and noncustodial parents. Making both parents attend mediation sessions is seen as critical and difficult. Where both parties attended, mediation visitation increased, relitigation was low, and there was increased child support compliance for the experimental vis-a-vis the control groups. Other interventions (e.g., parenting classes, counseling) which were done for severely conflicted parties or at a distance from the divorce/separation were not seen as making an additional impact in these areas. Also, the timing of access disputes was unaffected by the demonstrations.

Purpose

As set forth in the Family Support Act of 1988, this evaluation explored the effect of two waves of Child Access Demonstration projects on the amount of time required to resolve access disputes; reductions in litigation related to access disputes; improvements in compliance with court-ordered child support amounts; and promotion of the emotional adjustment of children. It also assessed the extent and nature of child access disputes as well as parental satisfaction with the demonstrations.

Background

Recent research in child psychology shows generally that close, frequent, and positive contact with the father following divorce and separation is beneficial for the child.

Child access is also important for child support enforcement. Recent Census data and research studies have indicated that where noncustodial parents have visitation rights or joint custody they tend to be more compliant with child support orders, although it is difficult to show cause and effect since the parents wanting to see the child may also be the better payers. Desire for increased child contact may follow child support payment rather than vice versa. Moreover, denial of visitation is seen as the major reason for nonpayment of child support for noncustodial parents who have money to pay child support.

There has been considerable pressure for the system to give support to the needs of noncustodial as well as custodial parents. Over 43 States authorize joint custody. There are currently over 200 court-based divorce mediation programs and over 280 fathers' rights groups organized throughout the country to facilitate child access by noncustodial parents.

Congress responded to the continuing public debate about the problem of noninvolvement by noncustodial parents and resulting litigation by directing HHS to conduct State demonstration projects relating to a variety of means of facilitating continuing involvement by the noncustodial parent.

In 1996 a new Federal grant program for child access and visitation programs was established nationwide.

Methods

Projects involved control and experimental group testing of different interventions (e.g., mediation, parenting education, community services for the noncustodial parent, group counseling, telephone monitoring of visitation) in a variety of public and private, court and administrative settings in Idaho, Florida, Indiana, Massachusetts, Arizona, and Iowa. Evaluation was provided for randomly selected experimental and control groups on such issues as relitigation, relitigation timing, visitation, and child support compliance. There were over 2,400 cases in the baseline and 1,500 cases in follow-up interviews.

Findings

The incidence of child access problems for divorced and separated parents ranged from a low of 4 percent to almost 30 percent of the cases by site. This is consistent with findings of other studies. The nature of child access problems are varied and may be longstanding for both parents. Some frequent problems include insufficient amount of visitation time, being cut out of a child's life, scheduling visitation, fights during pick-up and drop-off, concerns for the child's safety, concerns about the other parent's parenting style and negativity, and continuing problems with the parents' relationship.

There is a big problem of nonattendance by one or the other spouse at mediation or related interventions; and some effort should be made to compel attendance. However, where both parents attend mediation there is a high (65 percent to 70 percent) rate of arriving at parenting plans.

Mediation did resolve conflict in many, but not all, cases, and problems decreased after interventions. Visitation days increased for most sites and child support compliance increased to over 20 percent. A majority of both parents indicate satisfaction with interventions. However, the speed of dispute resolution and incidence of relitigation were not affected. In addition, nonmediation experiments did not seem to have an impact. It was difficult to assess improvements in emotional child development where tested.

Use of Results

The results should assist State, local, and private agencies in establishing programs to improve the continued involvement by the noncustodial parent. Mediation interventions at the time of the divorce--when both parents can be made to attend--are most effective. When disputes are high charged or considerable time has elapsed, it is more difficult to make an impact. Scarce resources can be focused on cases with problems before they become intractable. The results of these demonstrations are more relevant for divorce and formal separation cases than for unwed situations.

Agency sponsor:

Office of Child Support Enforcement

Federal contact:

David Arnaudo
202/401-5364
PIC ID: 5972.2

Performer organization:

Policy Studies Inc., Denver, CO

Evaluation of Child Support Guidelines

Highlights

This study evaluated the implementation of State child support guidelines mandated by the Family Support Act of 1988. It concluded that States need to increase the consistency of guideline application, including income verification procedures, extended custody and visitation arrangements, and consideration of multiple family situations, health insurance, and day care expenses in the calculation of child support awards. Adoption of a standardized support order would improve documentation of departures from guideline award calculations. State reviews of guidelines could be more thorough if carried out with sufficient resources, case surveys, and analysis of the cost of raising children. The study did not recommend the adoption of a national child support guideline.

Purpose

This evaluation explored three issues: how State child support guidelines are applied; the extent, amount, direction, and causes of deviations from State guideline award formulas; and how States account for special family circumstances in determining child support awards. Research focused on the process of applying guidelines; the extent, cause, and documentation of deviation from guideline formulas; the consideration of special family circumstances (such as child day care, health expenses, and multiple families) in adjusting awards; and actions taken as a result of the mandated guideline reviews prepared by the States.

Background

In 1984, in order to ensure appropriate and equitable child support awards by courts and other decisionmakers, Congress required every State seeking Federal funding for public welfare programs to establish child support guidelines. These initial guidelines were only advisory. However, the Family Support Act of 1988 requires that the State guidelines be used in determining the proper amount of support. Deviation from the presumptive guideline amount of award requires written justification.

Several Federal requirements help States develop child support guidelines. These include need for uniformity within each State; use of numerical formulas that consider all of a payer's income and provide for health care needs; use of guidelines to determine rebuttable, presumptive award amounts in judicial or administrative proceedings; use of guidelines for subsequent award modification; application of guidelines to all cases; and consideration of the child's best interests in State criteria for deviating from a guideline. The Family Support Act also requires that States reevaluate their guidelines at least once every 4 years to ensure the appropriateness of child support awards. States are to consider the cost of raising children and to sample case data on deviations from the guidelines. States must review child support orders for cases handled by a State child support agency at least once every 3 years.

Methods

This evaluation employed various sources of data--child support case records, stakeholder interviews, analysis of State guideline review studies, analysis of the Current Population Survey (CPS), and an expert panel. A sample of case records was drawn from 2 counties in each of 11 States. Although not representative of all States, over 4,000 cases were selected to reflect diverse guideline model types and child support formulas. The study conducted 215 unstructured telephone interviews with State and local officials, parents, and other stakeholders to gain a real-life perspective of the overall operation of State guidelines. Case studies of child support guidelines in 20 States were analyzed to assess the review process and the extent of required documentation for deviation from the guideline award amount. Over 2,000 cases in the March-April Matched CPS file with established child support awards in 1992 were analyzed to determine if presumptive State guidelines affected child support award amounts. Finally, an expert panel was assembled to provide advice on the evaluation and to make recommendations based on study findings.

Findings

Analysis of sample case record data indicated that only 17 percent of cases involved deviations from the guideline award amount, although the percentage varied considerably across the counties of 11 States. The four most common reasons for deviation were (1) agreement between the parties; (2) second households or multiple families; (3) extensive or extraordinary visitation or custody expenses; and (4) low income of the payer. The majority involved downward deviations--the average award decreased 34 percent. The ordered amount of support differed from the calculated guideline amount of support in a significant percentage of cases. Differences were found both in cases that followed the guideline calculations and in deviation cases. However, reasons for these differences were not documented in the case records. The consistency of the guideline application varied considerably. While some factors (such as income and health care expenses) were considered in nearly all orders, the consideration of other factors (such as multiple families and child care expenses) was quite variable across counties and within the same State.

Unstructured interviews with decisionmakers and other stakeholders in 21 study counties found a consensus that child support issues involving the impact of multiple families and second marriages, income imputation, and support for postsecondary education required further investigation.

Examination of State guideline reviews indicated that only 20 States collected and analyzed case data. Fewer than half of the States consider economic data on the cost of raising children. The extent to which these States took action as result of the data is unclear. When a particular factor, such as multifamily situations, is mandated by a numerical formula in the guidelines, there generally is a more uniform and frequent consideration of that factor in child support actions by the award decisionmaker. Verification of parental income, which is important in calculating accurate child support, is not frequent across counties; parties are often not prepared with the proper forms at hearings. When health insurance was ordered for one or both of the parents, the cost was rarely included in the child support calculation in States that have a mandatory numerical adjustment for such costs. Child day care expenses were included in the child support calculations for the cases that mentioned these expenses. There was no clear consensus in State reviews about the provision of postsecondary education support. Tax exemptions are commonly allocated in a flexible and equitable manner. The States handle extended custody and visitation arrangements without consistency or equity. These arrangements were commonly used reasons for deviation in child support awards.

Analysis of CPS data indicated that support awards may have increased by a small amount after the shift from voluntary to presumptive guidelines in 1988. However, the demographic composition of custodial parents changed over the period and the income of the noncustodial parent was unavailable, preventing any definitive assessment of the impact of mandatory guidelines.

The panel of experts reached the following conclusions:

1. No steps should be taken at this time to adopt a national child support guideline because States are still in an experimental phase.
2. Presumptive State guidelines should continue, with emphasis on the conduct of required State guideline reviews every 4 years (in accordance with Federal regulations).
3. The Federal Government should provide technical assistance and point out areas in the guideline review process for States to consider.

Use of Results

The findings and recommendations of this evaluation should benefit those States considering the revision or improved consistency of guidelines for awarding child support payments. Other States should be able to use this study to accelerate the implementation of guidelines, periodic case record reviews, and other related actions required by Federal regulations. Examples of specific State actions that can be taken on the basis of this study include (1) more consistent application of guidelines across cases; (2) improved case documentation of departures from the guideline award amount and reasons for the deviations; (3) improved independent verification of parental income; (4) adoption of a standardized support order to ensure that factors common to the majority of child support actions are addressed consistently; (5) use of a mandated numerical formula to adjust for various factors; and (6) coordination and consistency of policy within and between agencies and courts with regard to procedures and documentation.

Agency sponsor:

Office of Child Support Enforcement

Federal contact:

David Arnaudo
202/401-5364
PIC ID: 5983

Performer organization:

CSR, Incorporated, Washington, DC

Administration on Aging

Serving Elders at Risk: The Older Americans Act Nutrition Programs. National Evaluation of the Elderly Nutrition Program, 1993-1995

Highlights

The Elderly Nutrition Program provides grants to State units on aging to subsidize the provision of daily meals to people 60 years of age or older. This evaluation study describes participant characteristics, compares the characteristics of recipients of home-delivered meals to those receiving meals in congregate settings, assesses how well the program reaches disabled and poor elderly, estimates the program's impact on the nutritional intake and social contacts of participants, and examines program costs and other sources of funding. The study showed that participants have higher daily intakes of nutrients and more social contacts per month than a comparable group of nonparticipants. The Elderly Nutrition Program is successful in targeting older people who are poor, live alone, are nutritionally "at risk" because they are overweight or underweight, or are more functionally disabled than their age-group peers. Federal expenditures are highly leveraged with State, local, and private funds.

Purpose

This study was conducted to inform policymakers about the effectiveness of the Elderly Nutrition Program (ENP), the largest Federal program aimed at meeting the nutrition service needs of the elderly. Four major issues were evaluated: (1) the ENP's effects on participants' nutrition and socialization, (2) who is using the ENP and how effectively the program serves targeted groups most in need of its services, (3) how efficiently and effectively the ENP is administered and delivers services, and (4) the range of ENP funding sources and the allocation of ENP funds among its components.

Background

America will face critical challenges in the coming decades as it attempts to provide long-term care services to the nation's elderly population. As the large group of individuals born after World War II ages, a much higher proportion of Americans will be elderly and will require more health services and long-term care. In this context, it is important to examine whether existing long-term care service programs are producing their intended outcomes and whether their services are directed to those who need them the most.

One very important component of the overall package of home- and community-based services available for elderly people is nutrition services. A critical step toward meeting this need was taken in 1972 with the creation of the Elderly Nutrition Program under Titles III and VI of the Older Americans Act. Through Title III, State Units on Aging and Area Agencies on Aging implement a system of coordinated, community-based services targeted to older individuals. The ENP is administered by the Administration on Aging (AoA) of the U.S. Department of Health and Human Services. The AoA awards funds to State Units on Aging, which in turn distribute the funds to Area Agencies on Aging on the basis of State-determined formulas that reflect the proportion of older people in their areas and other factors. The area agencies provide services directly or award grants to and contract with nutrition projects that provide nutritional and supportive services. In addition, area agencies receive financial support from State and local governments, in-kind contributions, private donations, and voluntary contributions from participants.

Title III of the Older Americans Act authorizes the provision of nutrition and supportive services, such as meals, nutrition education, transportation, personal and homemaker services, and information and referral. Congregate meals and supportive services are provided at nutrition projects' meal sites (e.g., senior centers, religious facilities, schools, public or low-income housing sites, or residential care facilities). Alternatively, home-delivered meals are provided to homebound clients, either by the congregate meal sites and affiliated central kitchens or by nonaffiliated food service organizations. ENP meals are targeted to persons aged 60 or older. While there is no means test for program qualification, services are targeted at older persons with the greatest economic or social need. Similar nutrition and supportive services for American Indians, Alaskan Natives and Native Hawaiians are authorized separately under Title VI. Indian Tribal Organizations and agencies serving Native Hawaiians receive grant awards directly from the AoA.

Methods

Mathematica Policy Research, Inc., undertook this study for the AoA. A comprehensive research design was used, including in-person surveys of participants and eligible nonparticipants; in-person surveys of staff of congregate meal sites; in-person observation of sanitation and health practices at congregate meal sites, central kitchens, caterers, and home-delivered meal distribution sites; telephone surveys of staff at State Units on Aging, Area Agencies on Aging, Indian Tribal Organizations, and nutrition projects; review of program records on amounts of funding transfers; review of scientific literature related to nutritional requirements for the elderly population; extraction of relevant census data; and extraction of results from the previous national ENP program evaluation published in 1981.

All State Units on Aging were sampled. The sample of Area Agencies on Aging and nutrition projects was selected as a geographically clustered sample to maximize efficiency in conducting surveys. For Title VI programs, the agency sampling was simplified by the fact that each ENP grant is awarded directly to the Indian Tribal Organization, which in turn consists of a single nutrition project, often operating from a single site. In all, data were collected at 197 randomly selected meal sites. Approximately 2,500 interviews were conducted with elderly ENP participants (both home-delivered meal recipients and congregate site meal recipients) and nonparticipants. The nonparticipant sample used for comparison with the Title III program participant sample was selected from lists of Medicare recipients in the same ZIP Code areas as the participant sample, screening for income and disability status to ensure that the samples were matched as closely as possible.

Findings

The evidence collected in this evaluation suggests that the ENP has substantial positive effects on participants. Title III participants have higher daily intakes of key nutrients and more social contacts per day than a comparable group of nonparticipants. The average ENP meal meets the program's requirements to provide at least one-third of the relevant Recommended Daily Allowances. Both congregate and home-delivered participants have about 14 more social contacts per month than the comparison group; this finding and interviews of participants suggest that the program increases socialization opportunities for participants. Over 60 percent of all Title III and Title VI participants are satisfied with program services. Approximately 40 percent of Title III ENP service providers have waiting lists for home-delivered meals, suggesting a significant unmet need for these meals.

The evaluation shows that the program has achieved considerable success in efforts to target ENP services to populations in the greatest need. Compared with the overall population in the United States age 60 and older, ENP participants tend to be older, poorer, more likely to be members of racial or ethnic minorities, and more likely to live alone. About one-third of Title III congregate participants and one-half of Title III home-delivered participants have incomes at or below the poverty threshold. More than one-half of Title VI meal participants have incomes at or below this level. The comparable figure for the overall population age 60 and older is 15 percent. Nearly four times as many Title III participants and nine times as many Title VI participants are low-income minorities, compared with the overall population age 60 and older.

Agencies at the various administrative levels of the program have forged close links with other parts of America's emerging home- and community-based long-term care system, primarily through cross-referrals and coordination of service delivery at all levels within the aging network. The ENP provides a continuum of services, including nutrition screening, assessment, education, and counseling. Title III funds support 37 percent of congregate costs and 23 percent of home-delivered costs and are highly leveraged by other funding sources, including U.S. Department of Agriculture commodities or cash in lieu of commodities, State, local, and private funds. Participant contributions support another 20 percent of the program costs. The leveraging rate for Title VI is considerably lower, with over 60 percent of the costs supported by Title VI grants.

Use of Results

This report will provide policymakers with the information they need to determine the future scope of activities of the ENP, as well as future funding levels. The information in this report will also assist service providers in their efforts to improve the operation of their ENP projects and to obtain additional State, local, and private funding to supplement Federal grants.

Agency sponsor:

Office of Program Operations and Development

Federal contact:

Jean Lloyd
202/619-2005
PIC ID: 6175

Performer organization:

Mathematica Policy Research, Inc., Plainsboro, NJ

Agency for Health Care Policy and Research

Oregon Consumer Scorecard (OCS) Project

Highlights

The final report on the Oregon Consumer Scorecard (OCS) Project describes the development, pilot testing, and revision of a user-friendly guidebook, "A Consumer Guide to Selecting a Health Plan," by the Oregon Consumer Scorecard Consortium. The prototype scorecard is intended to help consumers choose a health plan consistent with their individual needs and health care service delivery preferences. The model also may help purchasers and State policymakers develop their own comparative reports about health plan performance.

Purpose

A contract from the Agency for Health Care Policy and Research was awarded in January 1995 to the Oregon Health Policy Institute through the University of Washington for this Federal-State collaboration in policy making and knowledge-building. There were five project objectives:

1. Identify information consumers want and need to make better choices.
2. Identify different information needs, based on geographic location and health conditions.
3. Identify relevant health plan performance measures.
4. Pilot test various formats for displaying quality, access, and satisfaction measures to consumers.
5. Make recommendations for designing a broader statewide demonstration to evaluate a preferred scorecard format for statewide implementation.

The model scorecard was intended to aid consumer decision making in two primary ways. First, it was to provide comparative health plan descriptions based on the expressed information preferences of Oregon consumers. Second, it was designed to educate consumers about health plan function, management, and performance. The project was guided by the presumption that people will make more prudent choices among health plans if they have access to objective, reliable measures of technical quality, health plan performance, and consumer satisfaction.

Background

Oregon has one of the most extensive managed care markets in the United States. After Oregon received approval for a Federal Medicaid waiver, the State legislature enacted a series of reforms known as the Oregon Health Plan (OHP). Since its implementation, OHP has extended coverage to approximately 130,000 individuals, including many previously without health insurance coverage who must now navigate an unfamiliar health care delivery system.

These reforms led to the establishment of the Oregon Office of the Health Plan Administrator (OHPA), with responsibilities that included "to assist OHP consumers in selecting a health care provider or a health care plan" (SB 5530, Oregon Statutes of 1993). The OHPA facilitated the creation of the Oregon Consumer Scorecard Consortium, a public-private partnership committed to developing a high-quality, user-friendly health plan guide.

Methods

A series of activities was undertaken to develop and pilot test stimulus materials that would subsequently inform the design of a prototype consumer scorecard to be evaluated on a statewide basis. The first step was the formation of the Oregon Consumer Scorecard Consortium, comprising more than 50 stakeholders from the public and private sectors that may benefit from or be affected by the scorecard.

Second, a work plan was established for reviewing and synthesizing policy-related information, conducting focus groups, and reporting on all other Consortium-sponsored data collection activities. The information review and synthesis included analyzing the Oregon Office of Medical Assistance Programs' annual consumer satisfaction survey, designed during the project year in consultation with the OCS Project staff and the Consortium's Technical Committee; pilot testing measures from Medicaid HEDIS (Health Plan Employer Data and Information Set), a widely used health plan performance measurement tool; and gathering objective, comparative, and descriptive navigational information to help consumers understand how a health plan really works.

Through a subcontract to Oregon Health Decisions, Inc., two rounds of four focus groups were conducted with OHP-Medicaid consumers. Targeted at consumers living in rural areas and persons with chronic health care conditions and disabilities, the first-round focus groups were designed to gain understanding of consumers' expressed information needs and preferences for selecting a health plan. In the second round, consumers who had a choice between two or more competing health plans in their counties were given a model guidebook and a set of alternative formats and media presentations to use and critique.

Third, the first set of prototype scorecard materials was produced. Finally, using real health plan data but fictitious health plan names, the guidebook was presented to Oregon consumers in a series of focus groups for their review and assessment.

Findings

Consumers expressed a desire for more accountability from their health plans. They wanted objective information on satisfaction and health plan performanceþinformation that is understandable, reliable, meaningful, and geographically sensitive. There is a need for unambiguous descriptive data, free of marketing bias, that show real differences in how health plans manage care.

Materials developed during this project were well received. Consumers were interested in multimedia presentations of choice information, using interactive computer kiosks, videos, and telephone advice. The availability of a trained, live individual was seen as an important backup to the scorecard.

The report noted that consumers and health professionals think differently about quality. Plan-level information was less important to consumers than provider-level information. Consumers preferred information that was relevant to their individual experience, including how they personally use health care services. Condition-specific indicators in population-based performance measures (such as HEDIS) were found to be less well understood, and therefore not as useful to consumers in selecting a health plan.

Despite the advantages of this voluntary collaboration, producing a reliable and uniform consumer scorecard in a timely fashion proved difficult. The report recommended the establishment of uniform standards and data specifications that all health plans and purchasers agree to abide by, with an independent "audit" to ensure compliance.

The report also noted that the political nature and the potential economic consequences of the comparative health plan information made it critically important to have an objective, disinterested third party responsible for producing a scorecard. In addition, because scorecard development is politically and methodologically complex, the report advised that future efforts should proceed in stages, with more modest goals and an examination of the considerable costs of producing a scorecard.

Use of Results

This project provides a unique model for integrating major types of health plan performance information into a consumer-oriented guidebook and adds to the emerging body of knowledge on consumer information needs and preference modeling. This contribution to the science base for statewide quality assessment and reporting can be used by other States in their efforts to evaluate quality of care.

Major policy and methodological issues to pursue in the future are addressed in the report, including the need to address differences in how professionals and consumers think about quality; the need to balance the focus on health plans as the unit of comparison for scorecard purposes with the interest of most consumers in the attributes and performance of individual clinicians and the facilities in which they practice; the need to reconcile population-based measures and personalized, consumer-relevant information; the need to tailor information to consumers with special concerns; and the need to critically evaluate the costs and benefits of various forms of presentation. The report recommends as a top public policy goal the development of a real and meaningful quality feedback loop that extends from the State to health plans to consumers and back to the State.

Agency sponsor:

Office of Planning and Evaluation

Federal contact:

Sandra Robinson
301/594-1349
PIC ID: 5960

Performer organization:

Oregon Health Policy Institute, Portland, OR

Centers for Disease Control and Prevention

Georgia State Clinic Assessments of Immunization Coverage: Impact of Management and Clinic Immunization Practices

Highlights

Between 1986 and 1994, the State of Georgia implemented a program of audits of the immunization practices of all State public health immunization clinics (227 in 1994) in an effort to improve child immunization rates. During this time period, the immunization rate for preschoolers increased from 31 percent to 90 percent. This evaluation found that the performance of these audits was significantly associated with higher immunization rates for preschool children in the clinic catchment areas.

Other management and medical practices were found to be significantly associated with increased immunization rates. Greater accessibility, informed vaccine administration practices (such as following only true contraindications), adequate clinic resources, and staff participation in leadership were also associated with higher immunization rates. The report lists 11 recommendations associated with the findings; in particular, the audits of the clinics should continue, and efforts to improve immunization rates should be comprehensive in scope and continuous in application. Findings from the evaluation of this highly successful program formed the basis of Centers for Disease Control and Prevention (CDC) recommendations to all States.

Purpose

The purpose of this study was to evaluate how audits conducted by the State of Georgia affected immunization rates among the preschool population, to assess management and organization factors associated with immunization coverage, to identify aspects of immunization clinic practice and organizational or managerial processes associated with changes in immunization coverage levels, and to make recommendations regarding the implementation of similar programs in other projects in the United States.

Background

The Public Health Service (PHS) objectives for the year 2000 include 90 percent immunization coverage levels for children up to the age of 24 months. The diseases against which immunization is sought are diphtheria, tetanus, pertussis, poliomyelitis, measles, mumps, rubella, Hemophilus influenza type B, and hepatitis B. The CDC's Comprehensive Childhood Immunization Initiative was developed to address this objective through an intensified strategy that includes service delivery, assessment, information and education, operational research, and surveillance.

To improve its child immunization rates, the State of Georgia conducted a program of annual audits between 1986 and 1994. The program allowed and encouraged district and clinic immunization staff at the State's public health clinics to implement interventions to increase clinic-specific coverage levels. Various approaches and combinations of interventions to change clinic practices were undertaken and evaluated.

Methods

Data for the study were drawn from clinic audit data, 1987-94, and from a self-administered mail survey of key immunization personnel in all public health clinics in the State of Georgia (227 in 1994). Individuals surveyed included district health directors or program managers, district immunization coordinators, and clinic immunization coordinators or clinic lead nurses. The survey response rate, due to intensive follow-up procedures, was 100 percent. Univariate and multivariate methods were used for analyses relating information on clinic practices from the surveys to information on audit rates from medical files.

Findings

The study found substantial improvement in coverage rates following the audit program, with median immunization rates at the clinics improving from 31 percent to 90 percent. Staff participation in the audit process resulted in their increased awareness of the performance levels of their offices, and of how their clinic compared with state averages; this, in turn, appeared to prompt improved immunization rates at the clinics.

Accessibility factorsþsuch as short waiting times, fixed fees, and convenient hours of serviceþwere associated with higher immunization rates at the clinics. Clinics with higher immunization rates also used improved vaccine administration practices, suggesting an association between appropriate clinical knowledge by key staff and higher coverage rates. Early outreach, aggressive reminder and recall, and close coordination with Women, Infants, and Children (WIC) and other programs were also associated with higher immunization rates.

Clinics with adequate resources had higher coverage rates. Computer technology, appropriately supported and implemented, appeared to benefit clinic rates. Lower coverage rates were more likely to be found at sites in the process of computerization than in those that were fully automated, although clinics without computers had even lower immunization rates. These findings suggest the importance of support and training to the process of computerization, as well as the importance of computerization in the clinic setting.

The study also found that staff participation and management leadership were associated with higher immunization rates. The investigators conclude that sustained efforts to improve immunization rates can make a difference.

Use of Results

Eleven recommendations based on study findings are included in the report, supporting continuing efforts to improve immunization rates through strategies that are long term, comprehensive in scope, and continuous in application. The report recommends continuing the audits, which were a critical element helping to improve these rates, and furnishing audit results to clinic providers and managers. Recommendations also target ready accessibility to clinics, early outreach, and aggressive reminder and recall. Staff training and development were encouraged to provide accurate information about contraindications and to minimize missed opportunities for immunization. Clear management practices that support and recognize the importance of immunization, that foster staff participation in decisions about improving immunization rates, and that provide adequate staff and financial resources were also recommended. The report recognizes the important role that programs such as WIC can play in improving immunization rates, and it calls for ongoing study of the role of computerization and of improvement efforts in other settings as they relate to coverage rates.

These recommendations have had far-reaching application, providing a national model for immunization practice. Georgia staff report that the evaluation findings have stimulated new staff thinking about immunization practice, as well as changes in policies to carry out the study's recommendations. Georgia's successful strategyþusing routine assessment and feedback to achieve high sustainable coverageþwas adopted and produced successful results in public health clinics in Colorado, Illinois, Iowa, Kansas, and South Carolina. It also worked in private and managed care provider settings in Arizona, Massachusetts, New York, and Washington.

Several reports of early study findings have appeared in the Morbidity and Mortality Weekly Report and CDC information sheets. Workshops and presentations on the project have been conducted throughout the term of the project, resulting in replication of this approach in other States. Furthermore, CDC reports that the recommendations of this study now provide the basis for their national immunization program: Since 1995, all States and localities receiving Federal funds for vaccine programs have been required to conduct annual assessments of vaccine rates in public health clinics.

Agency sponsor:

National Immunization Program

Federal contact:

Andrew Baughman
404/639-8239
PIC ID: 5354

Performer organization:

Battelle Centers for Public Health Research and Evaluation, Arlington, VA

Health Care Financing Administration

Ventilator-Dependent Unit Demonstration: Outcome Evaluation and Assessment of Post-Acute Care

Highlights

Chronic ventilator patients experience high mortality, and their care is exceedingly expensive. The purpose of this evaluation was to determine whether specialized units serving chronic ventilator-dependent patients deliver better clinical outcomes at a reasonable cost to Medicare. The evaluation compared chronic ventilator patients treated in ventilator-dependent units (VDU's) at four demonstration sites with similar patients treated in conventional hospital settings, usually in intensive care units. The evaluation found that patients treated in VDU's have improved clinical outcomes and lower hospital daily costs, but higher overall costs to Medicare because of greater longevity. For example, while the patients' median survival from the time of admission was about 5 months longer, and their functional status superior at discharge, the overall cost to Medicare was increased by 35 percent, due to longer life expectancy. The findings across the four demonstration sites varied greatly, however. Because of the variability in clinical outcomes and the difficulty of screening patients for placement in VDU'sþboth of which can lead to greater costs without commensurate clinical benefitsþthe evaluation concluded that nationwide implementation of VDU's is not recommended. However, the evaluation did find a small number of special VDU centers of excellence to be warranted. The analysis of costs and outcomes provides important information to policymakers and health providers about a seriously ill population with intensive resource needs.

Purpose

The purpose of the evaluation was to assess whether chronic ventilator patients covered by Medicare were better served at a reasonable cost by specialized hospital units called ventilator-dependent units. These units, studied at four Medicare demonstration sites, have a rehabilitation focus and are staffed by a highly trained multidisciplinary team of health professionals. The daily cost of care in a VDU is about $500 to $700 less than that of an intensive care unit, where most hospitals care for chronic ventilator patients. The evaluation compared costs and clinical outcomes of 211 chronic ventilator patients treated in VDU's with 401 comparison cases treated in conventional hospital settings. The evaluation also projected the overall costs of providing care in VDU's to Medicare patients nationwide.

Background

Mechanical ventilation is a life-sustaining technology for patients suffering from acute respiratory failure. The ventilator is an apparatus that delivers air to the lungs of patients unable to breathe on their own, usually because of failure of other organ systems or severe chronic illnesses. Because of the severity of their underlying illnesses, 49 percent of Medicare ventilator patients died in FY 1988. The 133,500 Medicare patients discharged in FY 1994 after an episode of mechanical ventilation had an average length of stay of 23.1 days, with total Medicare expenditures amounting to $3.5 billion. Thus, ventilator patients' high mortality, coupled with the high cost of treatment, warranted further scrutiny by the Health Care Financing Administration (HCFA), the agency that administers Medicare.

The focus of the evaluation was on chronic ventilator patients, that is, the small subset of ventilator patients who need ventilation for extended periods. Chronic ventilator patients are especially important to study because they are even more seriously ill and their care is correspondingly more costly. In this evaluation, a chronic ventilator patient was defined as one receiving at least 20 days of ventilation.

This evaluation was conducted under an exemption to Medicare's prospective payment system to allow demonstration hospitals to be paid for VDU care on a cost-reimbursement basis, with some incentives to control costs. Under Medicare's prospective payment system, Medicare normally pays a flat fee for chronic ventilator patients. Because of earlier reimbursement problems, patients are now grouped under one of three distinct diagnosis-related groups specifically tied to their receipt of chronic ventilation, rather than to their underlying illnesses. Yet problems still exist with the revised groupings, because hospitals and patients continue to have strong incentives to discharge patients to another setting. Understanding costs and effective treatments therefore permits further refinement to the prospective payment system for chronic ventilator patients.

Methods

The evaluation of costs and outcomes relied upon a constellation of data bases, including Medicare's enrollment and claims data and special clinical data sets. The four VDU demonstration sites were the Mayo Clinic; Temple University Hospital; RMS Health Providers in Hinsdale, Illinois; and Sinai Hospital of Detroit. New clinical instruments were developed for these sites to obtain a comprehensive portrait of the course of ventilator episodes. Claims data were used to track the VDU and the comparison group through 18 months following hospital admission.

The comparison group was drawn from a HCFA pilot data base known as the Uniform Clinical Data Set System, which is no longer in place. The data on chronic ventilator patients were derived from hospital records in five States. A major difference between VDU and comparison cases was that the former were screened for entry into VDU's, while the latter were not. Screening according to criteria set by HCFA excluded medically unstable patients and those with poor prospects for rehabilitation. Multivariate models were employed to control for differences between the VDU and comparison cases.

Findings

Three sets of findings were generated by the study: admissions, clinical and cost outcomes, and national implementation findings. The admissions findings revealed that, in many instances, patients were improperly admitted to VDU's. In other words, it was difficult for VDU sites to meet all of HCFA's complex criteria for entry, which relate to medical stability and reasonable potential for rehabilitation. Across VDU sites, there was great variability in the application of admission criteria.

With respect to outcomes, the evaluation analyzed 15 clinical outcomes and 17 expenditure outcomes. On many of the clinical outcomes, VDU cases fared better than comparison cases. Among the most important findings was that median survival from the point of hospital admission was 258 days for VDU cases and 106 days for comparison cases, a statistically significant increase of about five months. The mortality rate within the hospital was 34 percent for VDU cases and 48 percent for comparison cases. When patients were discharged, VDU cases were in better condition: 34 percent of VDU cases were discharged to their home, compared with 27 percent of comparison cases. Their functional status at discharge was superior, using a special index based on activities like bed mobility, locomotion, toileting, and eating. About 18 percent of VDU cases at discharge scored in the highest functional group, compared with 11 percent of comparisons. At two of the demonstration sites, however, some clinical outcomes were not significantly improved relative to the comparison group.

While daily Medicare expenditures for VDU cases were lower, overall expenditures were about 35 percent higher because patients lived longer. Total hospital expenditures averaged $123,000 per VDU case and $91,000 per comparison case. The mean daily spending within the hospital was $1,468 for VDU cases and $1,740 for comparison cases, because VDU's were generally less costly than alternatives.

In the national implementation analysis, costs were projected for covering VDU's for all eligible Medicare patients. The evaluation found that 67 percent to 80 percent of comparison cases would have been admitted to a VDU if one had been available, yielding an estimate of between 24,000 and 41,000 total admissions nationwide. Total Medicare expenditures for their care were estimated between $0.6 and $2.2 billion, depending on assumptions and duration of coverage. In light of the difficulty of controlling admissions and the variability in clinical outcomes, the evaluation concluded that nationwide implementation of VDU's would be very costly and of questionable benefit.

Use of Results

The evaluation recommends continuation of VDU's at selected centers of excellence to refine rehabilitative treatment for chronic ventilator patients. It also recommends further exploration of financing mechanisms that could encourage integration of care, rather than moving patients from one setting to another. Managed care, particularly through case management, is seen as holding promise for such integration. The evaluation also pointed to research questions that warrant further attention. Finally, by providing detailed information about costs and clinical outcomes for a population at high risk of death, the evaluation raises for policymakers and health care providers the difficult question of whether and how resources should be allocated to a seriously ill population in a manner that optimizes duration and quality of life, without being economically wasteful and prolonging suffering.

Agency sponsor:

Office of Research and Demonstrations

Federal contact:

Michael Henesch
410/786-6685
PIC ID: 4590

Performer organization:

The Lewin Group, Fairfax, VA

Health Resources and Services Administration

Evaluation of the Bureau of Health Professions Strategic Directions: Development of a Cross-Cutting Performance Monitoring System

Highlights

This evaluation reports on Phase II of a three-phase effort to develop a fully integrated computerized data system to facilitate planning and evaluation of the programs administered by the Bureau of Health Professions. Phase I assessed indicators of the Bureau's progress in meeting the goals and objectives defined in its strategic plan. Phase II analyzed the feasibility and appropriateness of the goals, outcomes, and indicators developed by the Bureau and identified strategies the agency can use in refining them with periodic input from grantees in the field. A performance monitoring system was proposed to facilitate ongoing program management. The system described in this study builds on the Bureau of Health Professions Grants Management Application System. The study recommended that the proposed performance monitoring system be piloted with several programs before it is fully implemented.

Purpose

The overall purpose of this project was to help the Bureau of Health Professions develop a set of outcome-based performance measures and design a performance monitoring system, both to measure whether supported programs are meeting national health workforce objectives and to identify necessary program course corrections. The following issues were addressed in the study:

1. Can the outcomes and indicators identified to monitor program progress in meeting the agency's goals be reliably collected and analyzed?
2. Do grantees believe that the identified goals, outcomes, and key indicators reflect the most valuable contributions of their projects?
3. What data sources and collection tools are currently available?
4. What kind of system should be developed to support the requirements of the Government Performance Results Act (GPRA) of 1993 and of other performance monitoring and reporting measures?

Background

The Bureau of Health Professions provides both policy leadership and support for enhancement of the health professions workforce and development of its educational infrastructure. This evaluation follows earlier work to assess and refine a set of goals with respect to workforce quality, supply, diversity, and distribution, as well as the outcomes and indicators of performance identified to measure and monitor progress toward those goals. This strategic planning process was undertaken by the Bureau in response to internal and external pressures for more effective targeting of scarce Federal resources toward those programs and activities that support and have a demonstrable effect on national workforce priorities. The development of a more explicit outcome-oriented system that identifies measures of performance related to Bureau-funded efforts will help address such concerns and will be very useful in future planning and program management.

The cross-cutting goals developed by the Bureau of Health Professions address (1) the development of a health care workforce that has the mix of competencies and skills needed to deliver cost-effective, quality care; (2) the need for educational programs that will yield professionals who can meet the needs of vulnerable populations; (3) the need for cultural diversity in the health professions; and (4) the need to stimulate and monitor the education system's ability to respond to the changing demands of the health care marketplace. Expected outcomes and indicators of success were developed and refined for each goal as part of phaseI of this evaluation effort. This evaluation was undertaken to address the next set of questions that needed to be examined by the Bureau in order to develop a performance measurement and management system that could link individual grantee level information in all of the Bureau's programs to one or more of the cross-cutting goals, associated outcomes, indicators, and performance-related functions.

Methods

The investigators employed several approaches to achieve the evaluation objectives. Among these were ongoing discussions among evaluators, Bureau of Health Professions program administrators, and other agency officials and committees whose input would inform the design of the performance evaluation system. The evaluators also consulted outside experts representing a range of health workforce training perspectives on feasibility issues related to data collection from grantees for program monitoring and evaluation purposes. An initial survey of data sources was then undertaken to explore existing systems that may be applicable to the ongoing data collection needs of the Bureau's performance monitoring system.

Findings

This report focuses on the design of a Comprehensive Performance Monitoring System (CPMS). The study calls for a system that will provide information to answer a basic performance question: Can the Bureau of Health Professionsþwith available funding and guiding legislation and through planned and funded grant activitiesþmeet national health workforce objectives for targeted populations? In capturing the information necessary to answer this question, the system would track essential inputs, processes, outputs, and outcomes for Bureau programs. The system would provide that information in regular reports to internal decisionmakers and to the Bureau's external customers, including Congress and grantees. The CPMS should support the following functions: monitoring and measuring performance goals; analyzing and assessing indicator data; identifying successes and problems; reviewing key program processes; and identifying opportunities.

A detailed set of recommended steps was developed for the Bureau's preparation for implementing the CPMS. These include completing final specifications of cross-cutting indicator definitions; specifying performance benchmarks and a process for assessment; identifying external indicators that must be tracked (e.g., market interest, availability); identifying sources of data and pilot testing new data collection; specifying hardware and software requirements of the system; and developing a pilot version of the system for beta testing.

The processes essential for maintaining the functions of the CPMS will require data, hardware, software, and human resources. The analytic and reporting functions of the system would be largely automatic once designed and implemented. Obtaining key information resources will require collecting data that are readily available and developing data collection instruments or surveys to collect data that are not currently reported by all grantees on a systematic basis. Other data collection resources managed outside of the Bureau may also be integrated into the system. The CPMS could exist in a variety of forms, ranging from paper files and reports to computer data base systems. The study recommended that an add-on module linked to the Grants Management Application System may yield the most efficient system. The system could be built as a relational data base, with linkage variables for cross-referencing data. The types of resources that would be needed for a computer-based performance monitoring system could include computer hardware; system software, including data base system software; and trained software and hardware systems support staff to develop, maintain, and update the capabilities of the system as needed. The data base could be on a stand-alone computer or could reside on a network.

The study concluded that in the final phase of this effort, the Bureau of Health Professions must address the questions necessary to perform key functions of the CPMS for each of the cross-cutting goals, outcomes, and indicators. Furthermore, the Bureau must decide what data elements should be collected from grantees to support the CPMS. Many data elements may already be collected and captured in the Grants Management Application System; thus, the CPMS should be integrated with this system and the staff with these responsibilities should coordinate their efforts.

Use of Results

This Bureau of Health Professions study on performance monitoring to support program management and Bureau-wide planning also fits into the context of broader efforts within the Health Resources and Services Administration and HHS to conduct strategic planning for resource investment and compliance with the GPRA. Policymakers and program staff in other Federal agencies within and outside HHS can benefit from the efforts described in this evaluation.

Agency sponsor:

Bureau of Health Professions

Federal contact:

Jennifer Burke
301/443-1590
PIC ID: 5497

Performer organization:

The Lewin Group, Fairfax, VA

Office of the Assistant Secretary for Planning and Evaluation

Analysis of the Effect of Regulation on the Quality of Care in Board and Care Homes

Highlights

This study assessed the effects of State board and care home regulations on the quality of care provided by those institutions. The investigators also attempted to identify the characteristics of the board and care environments in the States selected for the study and to determine whether licensed and unlicensed homes differed in terms of those characteristics or in the quality of care provided. The evaluation concluded that extensive State regulation and licensure can improve the quality of care provided by board and care homes.

Purpose

This investigation examined the quality of care in board and care homes in 10 Statesþ5 with extensive regulatory systems and 5 with more limited regulatory systems. For the purposes of this study, board and care refers to nonmedical, community-based residential settings that house two or more unrelated adults and provide such services as meals, medication supervision or reminders, organized activities, transportation, or help with activities of daily living.

The principal purpose of the study was to analyze and compare State board and care regulations and their effects on the quality of care received by board and care residents. Investigators attempted to do the following:

• Identify the characteristics of the board and care environments in the selected States, including the characteristics of the homes, operators, staff, and residents.
• Determine the effect of State regulation on the quality of care and the experience of board and care residents.
• Explore differences between licensed and unlicensed homes, particularly with respect to quality of care.

To accomplish these objectives, the study analyzed data on State regulatory systems, constructed a sample frame of licensed and unlicensed homes, and implemented a complex multistage sample design. In addition, the investigation compared data with the findings of earlier studies of board and care and with current data on the characteristics of other residential long-term care settings to clarify the role played by board and care homes.

Background

Changes in population demographics and a number of policy initiatives have increased demand for residential facilities that offer support services for the aged and disabled. Chief among these are a rapidly growing elderly population with significant levels of physical disability and mental impairment, an almost universal rejection of nursing home care by younger persons with disabilities and their advocates, and a strong preference among the elderly for in-home and community-based services as opposed to traditional nursing home care. While family efforts continue to be the primary source of long-term care for elderly and disabled loved ones, different types of residential settings with support services have emerged to supplement the efforts put forth by families.

There are approximately 34,000 licensed board and care homes in the United States, with more than 613,000 beds. These homes fall into one of three basic types of licensed facilities: those serving mentally retarded or developmentally disabled persons, those serving mentally ill persons, and those serving a mixed population of physically frail elderly, cognitively impaired elderly, and persons with mental health problems. Not all board and care homes are licensed, however, and by some estimates, unlicensed homes are as numerous as licensed facilities. Thus, the total number of persons living and receiving care in all types of board and care homes may be as high as one million.

While the Federal Government has traditionally played only a limited role in monitoring or regulating the quality of services provided by board and care homes (leaving primary oversight to the States), the Office of the Assistant Secretary for Planning and Evaluation (ASPE) in HHS has maintained a long-standing interest in the potential of board and care homes to meet the needs of elderly and disabled persons for residential services. This becomes particularly important in light of the strengthened Federal oversight of nursing home quality required by the Omnibus Budget Reconciliation Act of 1987.

The present analysis stems from Federal concerns regarding the lack of systematic information on board and care residents, changing levels of disability among the population served, adequacy of care, protection from health and safety risks, and the significant numbers of unlicensed and unregulated homes in the United States.

Methods

To accomplish the goals of the investigation, the study design incorporated several activities, including a major collection of new data. However, all activities focused on the main study goals of facilitating cross-sectional comparisons among facilities and residents based on the licensure status and regulatory environment under which the homes operated.

After reviewing current State regulatory approaches, the investigators selected 10 States for inclusion in the study; these States represented the extremes on a continuum of regulatory systems, ranging from very extensive regulation (California, Florida, New Jersey, Oklahoma, and Oregon) to very limited regulation (Arkansas, Georgia, Illinois, Kentucky, and Texas). A stratified, multistage, cluster design was used to select probability-based samples of homes, staff, and residents in each State. A sampling frame of eligible unlicensed homes was created using the Social Security Administration's State Data Exchange (SDX) tapes and network sampling of State and local agencies.

Primary data collection was then undertaken in 385 licensed and 129 unlicensed board and care homes, including interviews with 490 operators, 1,138 staff, and 3,257 residents. Site visits were conducted at each facility and extensive information was collected on characteristics of the home, including patient and payer mix; characteristics of the staff, including knowledge of aging and caregiving; resident demographics, health status, and satisfaction; and quality of care, including the physical environment, adequacy of staff, unmet health needs, and patient satisfaction. Data were analyzed using descriptive statistics and logistic and linear regression models.

Findings

This inquiry found substantial and widespread positive effects of both regulation and licensure on the quality of care in board and care homes. States with extensive regulatory systems had a significantly smaller proportion of unlicensed facilities than States with limited regulation (7 percent versus 25 percent). Extensive regulation also had a positive effect on several quality-of-care and quality-of-life indicators, such as lower use of psychotropic drugs and medications contraindicated for the elderly, more operator training, and greater availability of social aids (e.g., reading materials and community rooms) and supportive devices (e.g., grab bars in showers and call buttons in bathrooms).

Licensure also had a positive effect on many of the quality-of-care measures explored. For example, licensed homes were more likely to have operators with training and to make more social aids and supportive devices available to residents. Licensure also enhanced the availability of key services and the prevalence of safety features. A significant finding is that licensure was effective in raising homes above a minimum level of acceptable performance, or "preventing" the worst performance. Thus, licensed homes were less likely to have the lowest scores on such measures as availability of social aids, physical amenities, safety features, and an environment with little diversity and a very institutional atmosphere.

The study also found that neither extensive regulation nor licensure had a positive effect on some aspects of quality, including a requirement for preservice training of staff and staff knowledge of care, monitoring, and medication management. Regulation also had no significant effect on the cleanliness of homes and the availability of amenities or the likelihood that a home would have any licensed nurses (registered nurses or licensed practical nurses) on staff. Furthermore, there was little variation among homes on such issues as unmet health care needs, residents' rights, and indicators of resident satisfaction.

Finally, the report noted that the niche for board and care homes appears to be different in States with extensive regulatory systems compared with States with limited systems. States with extensive regulatory systems have higher-than-average board and care bed supply and lower nursing home bed supply. Homes in States with extensive regulation have residents with higher levels of disability than homes in States with limited regulatory systems. Thus, it appears that States with extensive regulatory systems were using board and care beds as substitutes for nursing home beds, particularly in comparison with States with limited regulatory systems.

Use of Results

The study findings have several important implications for key participants in the board and care sector. Findings point to a board and care population that is considerably more frail and disabled than it was 10 years ago. Furthermore, the mix of physically frail elderly, cognitively impaired elderly, and residents with mental illness and developmental disabilities (some of whom are nonelderly) presents a complex caregiving challenge. These factors should prompt a reexamination of the health and safety issues that confront board and care providers and the States' systems for regulating the industry. Of primary importance are the range of services, staffing patterns, and staff training and knowledge needed to meet the needs of today's residents.

This study confirms that there is a well-defined role for board and care homes in the provision of long-term care. The findings specifically suggest the following:

1. States can improve the quality of care in board and care homes through appropriate regulation.
2. States can improve other aspects of quality by requiring licensure of board and care homes.
3. The Federal Government can support State and provider efforts to improve the quality of care by developing and disseminating information.
4. The board and care industry should work closely with State Governments to improve the quality of care.

Agency sponsor:

Office of Disability, Aging, and Long-Term Care Policy

Federal contact:

Floyd Brown
202/690-6613
PIC ID: 4720

Performer organization:

Research Triangle Institute, Research Triangle Park, NC

Health Care in Transition: Technology Assessment in the Private Sector

Highlights

Medical technology embraces innovations in medicine, such as new drugs, biologics, medical devices, and procedures, as well as existing therapeutic and diagnostic capabilities. Medical technology has been identified as a major factor driving the increase in national expenditures for health care services. Thus, the evaluation of the clinical and cost-effectiveness of medical technology is of substantial interest to many parties. Technology assessment is the term most often applied to such evaluations. A strong national system of technology assessment has not emerged in the wake of a series of failed efforts by the Federal Government. With rapid changes in health care, particularly the rise of managed care, the demand for technology assessment has increased in the private sector. It is not clear if technology assessment activities in the private sector are replacing the assessment functions once expected of Government.

This report briefly describes a number of private-sector health organizations that are engaged in technology assessment. Technology assessment is one of a family of evaluation activities occurring within these health care organizations. Private-sector technology assessment is highly evidence based. Rigorous evaluation of clinical effectiveness based on a systematic review of scientific and clinical evidence has become the norm among private organizations conducting technology assessment. Technology assessment in this sector has increasingly become "full service," encompassing drugs, medical devices, and clinical procedures.

Purpose

This report considers the use of technology assessment by managed care organizations, which constitute the most significant change in the financing, organization, and delivery of health care. Managed care organizations have strong incentives to hold down the costs of care, and one way to do so is to avoid providing unnecessary, inappropriate, or inadequately tested medical technologies and procedures. This study aims to answer the following questions: What is the demand for technology assessment? Who are the performers of technology assessment? What characterizes the conduct of technology assessment? How are the results of technology assessment used? What unmet needs for technology assessment might be met by cooperation between the public and private sectors of health care?

Background

Much attention has been given in the past two decades to a strong governmental role in technology assessment, a term used to describe the evaluation of the clinical and cost-effectiveness of medical technology. Yet the Government bodies established to conduct such evaluations have not survived the budget cuts and political environment of the 1990's. At the same time, large corporate purchasers of health care have become more active in seeking to rein in the growth of health care expenditures. Managed care organizations have responded to market opportunities by enrolling an increasing proportion of patients, reducing excess hospital capacity, and shifting care away from physician specialists to primary caregivers. In this context, the evaluation of clinical practice for cost-effectiveness has assumed greater market value, and the demand for technology assessment has increased. There are two main sources of increased demand for technology assessment: the press for cost containment and performance evaluation and the need to be responsive to a changing health care marketplace.

With the failure of health care reform efforts in the early 1990's, and with the emergence of managed care, new questions have emerged about the extent of technology assessment activity in the private sector and whether these activities fulfill the functions that were once expected of the Federal Government. This study briefly describes a number of private-sector health organizations engaged in technology assessment.

Methods

The scope of this study was narrowed to focus on managed care. Data collection was accomplished through site visits, interviews, attendance at meetings, document review, and follow-up interviews. Since the Blue Cross/Blue Shield Association and The Emergency Care Research Institute had long been involved in technology assessment, they were the primary contacts for data collection. In total, 46 interviews were conducted with 56 individuals in 29 different organizations, including performers of technology assessment, third-party insurers, managed care organizations, professional organizations, and other settings. This report is not a comprehensive enumeration of all the organizations engaged in technology assessment, but is a selective picture of the major players; thus, the extent to which these practices are diffused is unknown.

Findings

In the past decade, a strong, decentralized technology assessment capability has developed in the private health care sector of the United States, consisting of a small number of national organizations and a growing number of larger health plans. Private organizations are conducting technology assessment in response to private-sector demands. These assessments are often designed to serve special clientele and require a significant number of internal staff resources.

Some national insurers and managed care organizations conduct centralized technology assessment efforts that serve member plans. Several membership organizations have constituency-oriented technology assessment programs, and some individual managed care organizations have significant technology assessment capabilities. In short, the private-sector technology assessment system is a mix of distributed centralized (national subscription, plan, and constituency efforts) and decentralized (regional and local health plans) efforts. Managed care has strong incentives to support and conduct technology assessment.

Several issues in technology assessment appear to have been settled in the past decade, including the organizational decisions supported by such assessments, the evaluative criteria for assessing a technology, the means of setting priorities for conducting assessments, and the stage of technology that is the object of the assessment. However, issues of how technology assessment applies to clinical trials and the development and maintenance of data bases are yet to be resolved.

In general, the evaluative context within which technology assessment is conducted has changed, focusing more on effectiveness, outcomes, appropriateness, and clinical practice guidelines. There has also been a shift toward evidence-based assessment, that is, a deemphasis on intuition and more emphasis on systematic clinical experience as sufficient grounds for clinical decision making. Although consensus processes are used to interpret and apply the results of technology assessments, the evidence-based movement discounts very heavily the use of consensus to assess the scientific bases of clinical practice.

Driven by cost-containment and quality-assurance objectives, health plans are increasingly examining the clinical effectiveness of interventions in relation to their financial implications. The use of technology assessment in the managed care sector is primarily to support coverage decision making. However, a number of innovative developments go beyond this narrow range of issues. These include benchmarking for member organizations of evaluative activities believed to be needed for survival in the current marketplace and providing a forum for reviewing the evidence related to competing technologies. They also include educational efforts in some plans, directed at changing physician behavior toward evidence-based medicine. Finally, the working limits of technology assessment have been broadened to include the evaluation of drugs after Food and Drug Administration approval for marketing.

Use of Results

An examination of private-sector technology assessment leads back to questions about the appropriate role of the Federal Government in technology assessment and the appropriate division of labor between the public and private sectors. One strong implication is that the Federal Government, as purchaser of health care services, should be no less competent and effective than the private sector in supporting and conducting technology assessment. However, this does not necessarily imply a centralized Federal Government leadership role, which does not appear feasible at present. Although the Federal Government has not shown a leadership role in technology assessment, it can play a significant role as sponsor of technology assessment-related research.

Agency sponsor:

Office of the Assistant Secretary for Planning and Evaluation

Federal contact:

Cheryl Austein-Casnoff
202/690-6102
PIC ID: 6180

Performer organization:

RAND Corporation, Santa Monica, CA

Subacute Care: Policy Synthesis and Market Area Analysis

Highlights

This investigation examined policy-relevant issues pertaining to the development of more effective and efficient health care delivery systems. The evaluation report provides an overview of and historical perspective on subacute care, which generally refers to skilled care for patients with complex needs outside of the acute care, or hospital, setting. In addition to describing the current state of the art in subacute care and how it varies across selected market areas, the evaluation examines the evidence about its cost and effectiveness. Implications of the study's findings are discussed, and key policy issues related to the future of subacute care are presented. The study concludes that the emerging concept of subacute care holds promise in terms of better care and lower cost, but that it also poses many challenges for the public and private sectors.

Purpose

This study was initiated by ASPE to examine policy-relevant issues pertaining to the development of more effective and efficient health care delivery systems and involving substantial public and private expenditures. These issues encompass the definition of subacute care; variations in subacute care patients, providers, and settings; incentives and barriers to the growth of subacute care; and the current and potential size of the market. The primary purpose of the study was to provide useful information to both the public and the Department, including a synthesis of subacute care research, practices, and trends and an assessment of public policies related to subacute care. The way in which the term "subacute care" has evolved and the extent to which the ideal vision is currently implemented in practice are key subjects of the study.

Background

Public spending for post-acute care services increased in the past decade from $3.4 billion to over $12.1 billion, an average annual growth rate of nearly 25 percent. In contrast, Medicare spending for acute care services grew by only 6 percent per year during the same period. If the current system is unchanged, Medicare expenditures for post-acute care and acute care are expected to rise even more rapidly as the number and proportion of the older adult population increases. Escalating costs and shifting demographics have prompted some providers to advance the concept of "subacute care" as a cost-effective alternative to inpatient acute hospital care. Proponents believe that subacute care has the potential to generate savings and improve patient outcomes.

At the time of this study, there were many definitions of subacute care. Historically, "subacute" described care provided for hospitalized patients who did not meet established criteria for a medically necessary acute stay. Currently, the term refers almost exclusively to patients treated somewhere other than in acute-care beds. The concept now focuses on shorter stays that result in improved rehabilitation for lower cost, using specific procedures designed to achieve better outcomes for patients whose service needs fall somewhere between those traditionally provided by acute care hospitals and nursing facilities.

Little reliable information is available about the likely impact of subacute care on costs, quality, and access. This lack of data may be related both to differing definitions of "subacute care" and to the recent and rapid expansion of the industry.

Methods

Initiated in August 1994, this study had several methodologies: a comprehensive literature search and review of published and unpublished documents; the establishment of an advisory group comprising public- and private-sector experts; interviews conducted with additional national experts from a variety of fields and settings; site visits to 19 institutional providers in four market areas (Los Angeles, Miami, Boston, and Columbus, Ohio); interviews with seven home health care providers; telephone interviews with eight home care firms; and a series of stakeholder interviews.

Findings

The study team concluded that insight into the different ways in which "subacute care" is defined is critical to understanding the subacute care phenomenon. The term commonly refers to skilled care for patients with complex needs that some nursing facilities, home care providers, and others have been providing for years under a variety of names. There is, in addition, a growing movement to create a new type of subacute care. The core elements of this idealized, prototypical form of subacute care include a program organized around particular disease categories, specific interventions, or homogeneous patient characteristics; a focus on achieving measurable outcomes in a more efficient and lower cost manner; special resources, such as certain physical plant characteristics and more and better trained staff; and a set of techniquesþfor example, the use of interdisciplinary teams, case managers, care maps, or critical pathway protocols; evaluation of outcomes; and an emphasis on continuous quality improvement.

Estimates of the number of days of subacute care provided each year in this country range from 1.2 million to 8.1 million. It was not possible to develop a reliable estimate of the current amount of subacute care provided. Distinguishing characteristics of subacute care are not captured in any large national data base. In addition, the study team encountered difficulties in identifying and comparing subacute care providers, in finding subacute care providers in reportedly better developed markets, and in obtaining data on subacute patients.

Factors shaping the development of subacute care nationally include the growth of managed care; efforts of managed care providers to find more cost-effective services; the implementation of new Medicare payment policies applicable to acute and post-acute care providers; changes in patient preferences, such as a desire for a homelike environment for patients with minimal nursing needs; and the growth of publicly owned, for-profit, post-acute care companies. Industry growth is also being driven by facility-based subacute care providersþskilled nursing facilities, freestanding and hospital-based nursing facilities, rehabilitation hospitals, distinct-part rehabilitation units, and long-term hospitals. The establishment of minimum standards and broadly defined quality guidelines by the Joint Commission on Accreditation of Health Care Organizations and the Rehabilitation Accreditation Commission has also influenced the subacute care industry.

The study team found that home infusion therapy and full-service home health agencies were providing a product with many of the elements of the prototypical subacute care facility. In response to the growing potential for substituting home health care for facility-based subacute care, many interviewees expressed concern about the quality of care, staffing qualifications, the role of managed care, access issues for lower-income patients, the perception of a "no-care zone" for patients with lower acuity, and the Medicare reimbursement policy for home infusion therapy.

Little evidence was discovered to support the premise that shifting patients earlier from hospitals to subacute care will save money. It was noted that a national study of potential subacute care savings for Medicare was based on several questionable assumptions. There are only a few empirical studies on quality in subacute care facilities; these have generally found poorer outcomes for patients in skilled nursing facilities than for those in rehabilitation hospitals. The four available studies comparing functional status outcomes for rehabilitation patients in hospitals and skilled nursing facilities involved a limited number of facilities; none was nationally representative.

In conclusion, the study team determined that the emerging concept of subacute care is appealing because of the new attention it has focused on some types of patients and the programs envisioned in the ideal. While site visits identified some subacute care providers that are successfully applying elements of the concept, many other services labeled "subacute care" fell short of the ideal. Despite aggressive marketing, the study team found that much that is called "subacute care" is little more than a new name for care provided to higher acuity, medically complex patients or to those requiring more intensive therapies. The idealized approach to subacute care promises similar or better care for lower cost but poses major challenges to both the public and private sectors.

Use of Results

Significant proposed legislative changes to Medicare and Medicaid have heightened the need for accurate and reliable information about subacute care, including a rigorous examination of the potential savings. This kind of information can assist Federal and State Government policymakers in determining their role in the evolution of subacute care and in understanding the implications of their actions on public and private expenditures, quality of care, patient access, and provider equity. This study underscores the importance of finding replicable technologies for producing better value, not just cheaper care, and of developing appropriate payment policies that achieve both savings and value.

Agency sponsor:

Office of Disability, Aging, and Long-Term Care Policy

Federal contact:

Jennie Harvell
202/690-6613
PIC ID: 5951

Performer organization:

The Lewin Group, Fairfax, VA

Office of Public Health and Science

Cost-Effectiveness in Health and Medicine

Highlights

Cost-effectiveness analysis (CEA) is one tool available to decisionmakers to help determine the relative value of different approaches to improving health or life expectancy. For a variety of reasons, however, CEA has not been widely used in health policy settings. This study represents a review of the methodological and theoretical aspects of CEA, its effectiveness, and its expected outcomes and uses. The resulting report is a consensus among many of the top researchers in the field and provides guidance to the conduct of CEA. The report sets forth several overlapping recommendations: some intended to improve the overall quality of CEA and some intended to set a research agenda for further improvement of CEA. Recommendations are made in the areas of framing and designing the study, identifying and valuing outcomes, assessing effectiveness, estimating costs, discounting, reflecting analytical uncertainty, and reporting analyses. It also discusses practical constraints to CEA's policy uses, carefully taking into account the needs of its varied audiences.

Purpose

The purpose of this study was to assess the current state of the art of the use of CEA in health care and to recommend steps to improve its quality and comparability. The Panel on Cost-Effectiveness in Health and Medicine, working closely with methodologists and liaisons from the Public Health Service and the Health Care Financing Administration attempted to identify a number of methodological problems in existing CEA's that limit the analyses' policy relevance and usefulness.

Background

Society is confronting many difficult choices regarding the provision of health care services and the funding of public health programs to improve the Nation's well-being. The aging of the population and the costs of expanding medical technologies have placed growing pressure on health care resources. The private sector, which has long paid for health insurance for a significant proportion of the population, is confronted by competitive challenges and shifting nationwide economic trends. In the public sector, there are competing demands for tax dollars to fund health programs that range from the clinical care supported by Medicare and Medicaid to the population-based approaches employed by public health programs. The Nation is faced with an apparent series of tradeoffs among priorities, as we seek to make wise investment decisions to improve health.

There are many ways to resolve these choices, none of them straightforward and none without compromise. To make the most informed decisions, information is needed about the impact of programs and interventions, their costs, and what is given up when one program is implemented at the cost of another. This is true for Government, managed care organizations, providers, employers, and consumers. As the pressures to control U.S. health care spending have accelerated, the term "cost-effective" has found an increasingly broad and interested audience. But this term often has different meanings to groups as disparate as the U.S. Congress, business leaders, managed care organizations, the pharmaceutical industry, and the public. A more systematic and reliable approach to determining relative values through CEA is required.

Methods

This report summarizes the work of the Panel on Cost-Effectiveness in Health and Medicine, a non-Federal expert panel appointed in 1993 by PHS. The panel consisted of 13 scientists and scholars with expertise in CEA and related methods. The panel was charged with assessing the current state of the science of CEA and with providing recommendations for the conduct of studies to improve its policy relevance and utility. The panel met from 1993 through 1995 in collaboration with methodologists from Public Health Service agencies and the Health Care Financing Administration.

The panel organized itself into nine working groups focusing on specific topics: (1) the role of CEA in decision making; (2) CEA's theoretical foundations; (3) the appropriate framing of a study; (4) measuring effectiveness; (5) valuing outcomes; (6) measuring costs; (7) discounting considerations; (8) handling uncertainty within a study; and (9) guidelines for reporting analyses. Panel members and staff drafted papers on topics addressing the major controversies in each area. The papers contained recommendations that were debated by panel members in successive meetings until consensus was reached and formal recommendations could be issued or, in a few cases, until it became clear that consensus could not be reached. In areas where consensus was elusive, the report describes the full range of arguments presented.

Findings

CEA furnishes a framework for evaluating the economic and health impacts of different types of investments and can inform a wide range of policy decisions. The results of a CEA are typically summarized as a cost-effective ratio, for example, $138,000 per quality-adjusted life-year, in which the denominator reflects the gain in health from a specific intervention (e.g., years of healthy life gained, premature births averted, or sight-years gained) and the numerator reflects the cost in dollars of obtaining that gain.

A well-constructed CEA provides information that helps decisionmakers identify which strategies will best serve their programmatic and financial objectives. Depending on the type of analysis conducted, a CEA can compare similar or very different approaches to a given problem, such as strategies for decreasing cardiovascular disease. For example, State and Federal decisionmakers with responsibility for Medicaid programs, as well as for public health systems, may wish to know where investments of public dollars will have the largest impact in decreasing premature death from heart disease. In the private sector, a managed care organization might wish to know how extending its coverage to include cardiovascular disease prevention programs would affect the organization's expenditures.

Some decisionmakers may also wish to make comparisons among health conditions and programs; for example, they may wish to know the efficiencies of investments in interventions to improve neonatal outcomes relative to investments targeted at decreasing cardiovascular disease in adults. Ideally, CEA should allow its users to assess the relative value of dissimilar health-producing programs or treatments.

Unfortunately many CEA's are inadequate to these tasks. Reviews of CEA's chronicle a troublesome failure of CEA practitioners to adhere to basic analytic principles; there also is wide variation in sources and the quality of information on costs and effectiveness. Concerns about the discretionary nature of the application of analytic methods and the economic incentives of commercial sponsors of some analyses have resulted in a decision by at least one medical journal to restrict publication of any CEA where authors may have a financial conflict of interest.

Methodological inconsistency has often stymied the application of CEA in important public policy decisions. For example, in the Oregon Medicaid experiment, in which the State legislature attempted to expand coverage to uninsured Oregonians by basing inclusion of services on the cost-effectiveness of different treatment-condition pairs, deficiencies in CEA technique and knowledge were partly responsible for the fact that the initial list of included services lacked credibility. In a more narrow application to policy needs, a review by the National Cancer Institute of an extensive literature on the cost-effectiveness of screening mammography found that the study results ranged from a finding that mammography would be cost saving to a finding that it would cost nearly $84,000 per year of life saved. Confusing results can lead to confusing policy.

The imprecision attached to the term "cost-effective" stems in part from the variety of masters the concept serves: purchasers of health care, who use the term to convey a careful assessment of the value of different health care services; producers of health care technologies and programs, who use the idea to support marketing claims; and advocates for particular constituencies or illnesses, who use the term to garner resource investments. But the imprecision also comes from the manner in which methods of performing CEA have evolved over the past three decades. CEA is an analytic tool whose fundamental purpose is to provide information to decisionmakers about the relative value of different approaches to improving health, life expectancy, or both. Architects of the field and analysts who apply CEA methods come from a number of academic disciplines, including economics, medicine, operations research, medical sociology, psychology, public health, and ethics. Each discipline brings a particular set of concepts and a unique language that have been melded in the building of the CEA technique.

Use of Results

The panel has produced a book, Cost-Effectiveness in Health and Medicine, summarizing CEA methodology with the objective of improving its usefulness to policymakers. The report will be distributed to Federal decisionmakers and analysts to inform them about the interpretation of existing CEA and ways to improve studies conducted by and for the Government. The interdepartmental group is planning broader dissemination and discussion of the panel's work through an international conference. The conference will explore the implications of the panel's recommendations and will identify next steps in moving toward the production and use of high-quality comparable CEA.

Agency sponsor:

Office of Disease Prevention and Health Promotion

Federal contact:

Kristine McCoy
202/205-8180
PIC ID: 5619

Performer organization:

Office of Disease Prevention and Health Promotion, Washington, DC

Chapter III. HHS Agency Evaluation Activities

The 11 agencies of the Department of Health and Human Services (HHS) and the Office of the Assistant Secretary for Planning and Evaluation (ASPE) and the Office of Public Health and Science (OPHS) in the Office of the Secretary each maintains its own evaluation program--including the functions of evaluation planning and policy review, quality assurance through technical review, project coordination and management, dissemination of reports, and utilization of results. This chapter provides an agency-by-agency summary of the evaluation activities. Each summary begins with an overview of the relevant evaluation program, describing its philosophy, policies, and procedures. Next is an overview of the major evaluations completed in fiscal year (FY) 1996 and the evaluations that are currently in progress. Finally, each agency offers a discussion of new directions for its evaluations, including priorities for future program and policy evaluation projects. Abstracts and contact persons for all evaluations completed in FY 1996, organized by agency, are presented in appendix A. Listings by agency of all evaluations in progress are located in appendix B.

Administration for Children and Families

MISSION: To promote the economic and social well-being of families, children, individuals, and communities.

Evaluation Program

The Administration for Children and Families (ACF) administers a broad range of entitlement and discretionary programs, including programs designed to move families from welfare to work (Temporary Assistance for Needy Families); child support; children and family services (Head Start, Child Welfare, Family Preservation and Support, youth programs); four block grants; and special programs for targeted populations, such as the developmentally disabled, immigrants, and Native Americans.

The objectives of ACF's evaluations are to furnish information on designing and operating effective programs; to test new service delivery approaches, building on the success of completed demonstrations; to apply evaluation data to policy development, legislative planning, budget decisions, program management, and strategic planning and performance measures development; and to disseminate findings of completed studies and promote application of results by State and local governments.

ACF actively engages with other Federal agencies, State and local policy and program officials, national organizations, foundations, professional groups and practitioners, and consumers to maintain currency on the emerging issues affecting its programs and to identify questions for evaluation studies. Systems changes and how they affect vulnerable populations, particularly the well-being of children, are of primary concern. The movement toward devolving responsibility for health and human services to State and local organizations--in particular, the Personal Responsibility and Work Opportunity Reconciliation Act of 1996--offer both tremendous opportunities and unprecedented challenges in redefining and implementing services to families.

Evaluation study designs are carefully negotiated with the States and other interest groups. Studies are frequently funded as joint ventures with ASPE and other Federal agencies and foundations. Such collaborations enable efforts that are better informed, more representative of varying perspectives, and larger in scale. Proposals are reviewed by multidisciplinary experts. Work groups of various kinds are used throughout the projects to monitor progress and to advise on refinements in designing and presenting the findings.

Summary of Fiscal Year 1996 Evaluations

ACF's major evaluations are linked closely to its two strategic goals:

• Measurably improve the economic independence and productivity of families by reforming the welfare system and by stimulating the changes in attitude and behavior necessary to achieve results.
• Increase the number of children, youth, and families who have improved health, development, and well-being and who live in stable communities.

Described below are several major evaluations completed by ACF in FY 1996 that relate to these two strategic goals.

Economic Independence and Productivity of Families

Two studies addressed child support enforcement issues. First, Evaluation of Child Support Guidelines, highlighted in chapter II, evaluated the impact of shifting from voluntary to presumptive guidelines using the Current Population Survey Child Support and Alimony Supplement of 1992. Little impact on child support orders was found when States treated guidelines as presumptive rather than voluntary. Review of State studies of deviations from guidelines revealed that while deviation ranges from 3 percent to 81 percent, most States deviate in 25 percent of cases or less.

Second is an evaluation also highlighted in chapter II, Evaluation of Child Access Demonstration Projects: Report to Congress. This ACF Office of Child Support Enforcement report evaluates different forms of interventions to bring noncustodial parents closer to their children after divorce and separation. The interventions included mediation, parenting training, counseling, enforcement of visitation, and monitoring of visitation. The report found that mediation where both parties attended resulted in parenting plans in 65 to 70 percent of the cases. These parenting plans stimulated more visitation by noncustodial parents and better compliance with child support. A majority of both parents were satisfied with mediation. Other forms of intervention for more longer term and problematic cases did not register impact.

Healthy Development of Children and Families

Two evaluations completed during FY 1996 focused on ACF's Head Start program. First, Survey of Head Start Family Self-Sufficiency Initiatives explored ways programs could work with families to improve employability and literacy and deal with substance abuse. The study found that Head Start programs use formal surveys and assessments to identify a family's need for literacy and employability services but assess the need for substance abuse services more informally. Also, most program directors perceive the greatest need for employability services and the least need for substance abuse services.

Second, Survey of Head Start Family Self-Sufficiency Initiatives: Case Studies in Six Communities found that although most program directors were concerned about substance abuse and its effect on families, assessing and meeting families' service needs was difficult. According to the case studies, program directors believe their Head Start families need literacy services, although the underlying causes of illiteracy vary across sites. The six programs recognized and addressed employability in varying ways. For example, some programs offered information and referral, but only when parents requested assistance.

Several evaluation projects were targeted to programs for family members at risk, National Evaluation of Home-Based Services Programs for Runaway Youth reports on five demonstration programs--Baltimore, Maryland; Nashville, Tennessee; Kauai, Hawaii; San Diego, California; and Tucson, Arizona--targeting at-risk youth in dysfunctional families. In each, short-term interventions were applied to link the families to existing community resources. The projects reported difficulties in dealing with multiproblem families, lack of community resources, lack of support from other community agencies, and staff turnover. Effective service practices included bilingual and culturally sensitive staff, round-the-clock staff availability, and using a cotherapist approach.

Second, Gang Families in a Public Housing Project studied families having more than one gang member in a low-income Mexican-American community in Los Angeles. The project studied relationships among macrostructural and economic forces and household organization, family childrearing practices, sibling and relational influences, socialization of street children, culture and traditions, and levels of acculturation, especially in the colonization or marginalization process. The study identified family patterns and processes leading to gang membership and provided a better understanding of the dynamics of families with more than one gang member, how childrearing practices and street culture are transmitted to children, and how gang habits and values are transmitted among family members.

Length of Service and Cost-Effectiveness in Four Family-Based Placement Prevention Programs used an experimental design to study the impact of length of service on outcomes in family preservation programs in Portland and Pendleton, Oregon, and Baltimore, Maryland. The study found that, overall, participants (a total of 460 families) experienced low out-of-home placement rates, low rates of maltreatment, and significant improvement in family, parent, and child functioning. Six-month periods of in-home family treatment provided to families with older children and significant histories of prior services were the most effective.

Last, Final Evaluation Report for the Case Management Enhancement Project at the East Orange District Office of the New Jersey Division of Youth and Family Services (DYFS) documents a 4-year test of the effectiveness of a personal computer-based system to record child protective services case flow information. The overall impact has been to facilitate communication between caseworkers and supervisors, office staff, and outside agencies. Supervisors can now examine case records directly from their PC's, identify problems, assess risks, consult with caseworkers, and redirect case management in a timely manner; the network fax is used to transmit information directly from the computer. The Deputy Attorney General's office in Essex County is able to transmit affidavits, court documents, and complaints by computer sharply reducing the time needed to edit and finalize documents. East Orange office staff can also notify the prosecutor about child abuse and neglect cases on a timely basis. The project positively affected staff knowledge and skill levels and strengthened their motivation to tap the potential of computer technology. The DYFS client-tracking system has already proven its replication potential, both within the State of New Jersey and beyond. The experiment was expanded to all 39 district offices in New Jersey; Connecticut and Wisconsin have inquired about the East Orange system.

Evaluations in Progress

Described below are some of ACF's evaluations in progress that relate to the strategic goals.

Economic Independence and Productivity of Families

One study is looking at child support enforcement issues. Arkansas Prenatal and Postnatal Paternity Project tests the success of in-hospital and postnatal establishment of paternity. Early findings indicate the following: (1) Paternity is established in about half the cases; (2)38 percent of established paternities are for children in the IV-D caseload (Title IV, Part D of the Social Security Act authorizes Federal matching funds to be used for support obligations by locating nonresident parents, establishing paternity, establishing child support awards, and collecting money.); (3) 30 percent of child support cases receive some financial support; (4) 27 percent of child support paternity cases had closed; (5) 95 percent of paternities were established within 30 days of birth; and (6) 44 percent of parents who acknowledged paternity were cohabiting at the time of birth. Mothers at prenatal clinics were generally Medicaid-eligible and poor. Seventy-five percent of them said they wanted paternity established when their child was born and predicted that 71 percent of fathers would acknowledge their paternity; however, only 32 percent of the fathers actually signed paternity acknowledgments. Twenty-five percent of mothers at prenatal clinics did not want to establish paternity for reasons that included the following: (1) not wanting the father involved; (2) the father already gives money to the mother; (3) the mother does not know the location of the father; (4) the mother fears the father's reaction; (5) the mother does not know the identity of the father; (6) the mother sees no reason to establish paternity; or (7) the mother does not want to lose benefits. Programs to establish interstate paternities at the borders and programs to provide job and social services for unemployed fathers both succeeded. Models for deemphasizing child support and the use of child support staff were unsuccessful.

Two ACF studies under way address parenthood in welfare families. First, Responsible Fatherhood: Theoretical and Empirical Foundations for Policy and Program Development, a joint project with ASPE, focuses on a wide range of fathers, including disadvantaged, never-married, noncustodial fathers; separated or divorced noncustodial fathers; and fathers living with their children. By developing a theoretical underpinning to guide empirical research, program development, and program evaluation, this project intends to help inform policymakers about what is necessary to enable fathers to support and nurture a child.

A second study, Home Visitor Services Demonstration: Home Visiting for Teen Parents Required to Participate in JOBS, is testing the effectiveness of combining the Job Opportunities and Basic Skills Training (JOBS) Program with weekly home visits by paraprofessionals. The first-time teen parent welfare recipients are required to participate in education, training, and employment-related activities through the JOBS program, including home visiting. This controlled experiment is designed to evaluate whether home visiting helps teen parents to increase their participation in JOBS activities; improve parenting; experience fewer repeat pregnancies and births; and increase the use of preventive health care (including immunizations) both for themselves and their children. The evaluation is funded by ACF and the Henry J. Kaiser Family Foundation.

Eleven ACF evaluations projects relate to the Empowerment Zone/Enterprise Community Initiative. The Office of Community Services supports evaluations of 10 projects funded under the Job Opportunities for Low-Income Individuals (JOLI) Program. ACF also provides training and technical assistance to JOLI grantees. The services will help the grantees develop project designs and finalize evaluation plans.

Bridgeport Artisan Center, a project of Action for Bridgeport Community Development and its collaborative network of public and private agencies, is creating jobs and enterprise opportunities for low-income artisans in the inner city and surrounding neighborhoods of Bridgeport, Connecticut, an Enterprise Community.

Through its Harlem Railyards Transportation (HRT) Program, the South Bronx Overall Economic Development Corporation provides well-paying employment and business development opportunities for at least 102 recipients of Aid to Families with Dependent Children (AFDC) and other low-income individuals in the New York City Empowerment Zone. The HRT program, a not-for-profit trucking venture, operates a cost-effective transportation system that transfers loaded rail cars to trucks for delivery to New York City area businesses.

Venture, a highway construction program of the Rural Advancement Fund, trains 100 AFDC recipients for highway construction careers in three North Carolina counties. Venture capitalizes on Federal and State highway construction mandates aimed at employing AFDC recipients and other low-income people, particularly women and minorities.

JOLI Project of the Women's Self-Employment Project (WSEP) grows microenterprises, some incubated under a previous JOLI grant, to provide employment for AFDC recipients in Chicago, Illinois. The project identifies up to 30 employers to receive technical assistance aimed at expanding their businesses and creating 100 new jobs. WSEP increases access to financial services--such as loan capital, savings, and investment vehicles--by providing loans through its Revolving Loan Fund.

JOLI Project is enabling Bethel New Life to expand its 10-year-old Chicago home care service and to create 40 new upwardly mobile jobs. Bethel New Life plans to increase business by 25 percent by broadening its service population and diversifying its payer mix to include private-pay individuals and managed care provider contracts. It also plans to establish a career ladder to advance people from welfare to minimum-wage jobs as homemakers or home health aides and to living-wage jobs as certified nurse assistants, allied health workers, licensed practical nurses, and registered nurses.

Avenues (Avenidas), a project of the Mi Casa Resource Center for Women in Denver, Colorado, trains low-income persons, primarily women, for jobs and apprenticeships in highway construction and maintenance. Recruiters target the unemployed, public housing residents, and homeless persons. Keys to the project's success are a steering committee of collaborators to guide program implementation and facilitate job placements; motivational marketing and outreach to develop interest in nontraditional employment; comprehensive assessment, case management, and linkage to supportive services; a rigorous 6-week training program; and peer support and mentors.

Green Institute's JOLI Project is creating new job opportunities for 100 low-income residents in the Phillips neighborhood of Minneapolis, Minnesota, through the incubation of green businesses. The project focuses on five areas: (1) creating jobs that build on the success of the Green Institute's ReUse Center; (2) an incubator program for businesses that develop new products from industrial waste products; (3) businesses offering products and services to promote energy efficiency and conservation; (4) a program for youth entrepreneurs to develop environmental businesses and services through the Science Magnet Program at South High School; and (5) businesses focused on environmental technology and alternative energy that also provide better paying unskilled and semiskilled jobs.

Through Project RISE, Yakima Valley Opportunities Industrialization Center is expanding its housing development company into the Yakima Valley Rural Enterprise Community by creating 16 permanent, year-round jobs and safe, affordable housing for low- and moderate-income families. The project targets at-risk youth, public assistance recipients, displaced workers, and individuals enrolled in the JOBS program or in a program funded by the Job Training Partnership Administration. In 3 years, 56 homes will be constructed, weatherized, or rehabilitated on 8 acres developed by the project.

JOLI Project of Black Dollars Days Task Force in Seattle, Washington, is creating 71 jobs through three businesses: (1) NW ServiceMaster, a cleaning franchise expansion of the Handyman Connection, which provides JOLI participants opportunity for self-employment; (2) a new home health services business; and (3) a new driving and delivery service cooperative. The project is based on the Task Force's Multifaceted Business Development program, bringing together public-private partnerships, streamlined resources, and effective skills training to help people make the transition from welfare to self-sufficiency.

HOMECARE Co-op is a JOLI project of the San Jose Development Corporation. This self-employment house-cleaning services, business training, and business cooperative for AFDC recipients in Santa Clara County, California, is creating 40 new business enterprises which are, in turn, creating 60 new full-time jobs.

National Study of Protective, Preventive Reunification Services Delivered to Children and Their Families will examine the number and percentages of children and families in the child welfare system receiving protective, preventive reunification, out-of-home care, or aftercare services. The study will also obtain national data on the number, types, and dynamics of the services provided. Researchers will abstract case records on a nationally representative sample of 3,000 children and their families served by public child welfare agencies.

Foster Youth Mentors project is examining factors characteristic of successful relationships between foster youth and older citizen mentors by comparing 250 successful matches with 250 unsuccessful ones. Data is being collected on the characteristics of mentors, foster youth, and the mentoring program itself. Findings to be disseminated to independent living programs throughout the United States are expected to facilitate the use of mentors in older youths' transition from foster care.

Last, Evaluation of the Impact of Homelessness on ACYF Programs will identify service demands on ACYF programs serving homeless families, children, and youth; key strategies for increasing the effectiveness of ACYF programs; and methods to help reduce the risk of homelessness. Data will be collected from 40 communities in which a small-scale longitudinal study was done of homeless people and from case studies in five local programs.

Healthy Development of Children and Families

ACF has several studies under way that address Head Start issues. First, Descriptive Study of the Head Start Health Component describes health-screening, examination, referral, treatment, and follow-up procedures across four domains of the Head Start health component: medical, dental, nutrition, and mental health. Case records of 1,200 4-year-old children enrolled in a national sample of 40 randomly selected Head Start programs were reviewed, and parents were interviewed concerning their child's health status and health services utilization patterns.

Next, through a consortium of local evaluators, the Evaluation of the Head Start Family Service Center Demonstrations project, conducted by Abt Associates, Inc., is evaluating 41 Family Service Center Demonstrations. The project focuses on how Head Start can collaborate with community programs to help meet the needs of Head Start families that must deal with problems like illiteracy, substance abuse, and unemployment.

The Study of the Characteristics of Families Served by Head Start Migrant Programs is profiling Head Start migrant families in the main migratory streams; identifying unique services issues; documenting the availability and coordination of services for Head Start families during migration; and providing a national estimate of the number of eligible migrant children. Findings will inform policy decisions on both Head Start migrant programs and the new Early Head Start program for infants and toddlers.

The Early Head Start Research and Evaluation project is evaluating the effectiveness of the Early Head Start (EHS) program in 15 diverse communities. The study examines child, family, staff, and community outcomes in a sample of 3,400 children and their families, randomly selected into project and comparison groups when the mothers are pregnant or before the children reach the age of 12 months. Children, families, and child care environments will be assessed when children are 14, 24, and 34 months of age. Service use interviews will be conducted semiannually and programs visited annually. The study will produce the following reports: (1) Descriptive Study of EHS Programs; (2) Study of Program Variations; (3) Pathways to Early Head Start Quality; (4) Interim Study of Outcomes; (5) Longitudinal Study of EHS Outcomes; and (6) Selected Policy Papers.

Descriptive Study of Families Served by Head Start is examining policy-relevant issues with a nationally representative sample of families served by Head Start in 40 programs across the country. Employing survey and case study methods, the project is charting families' demographics, strengths, needs, expectations, and experiences in Head Start and programmatic efforts to join in partnership with families.

The National Child Welfare Research Center in the School of Social Welfare at the University of California, Berkeley, will serve as a knowledge-building and information-disseminating resource for improved child welfare services. The Center will give special attention to (1) child abuse and child welfare; (2) family preservation and maintenance; (3) foster care and adoption; (4) drug- and AIDS-affected children; and (5) organizing, financing, and evaluating child welfare services.

One ACF project focuses on family protective services. Evaluation of Nine Comprehensive Community-Based Child Abuse and Neglect Prevention Programs, a contract with CSR, Incorporated, is (1) designing and implementing a process and impact evaluation of nine comprehensive community-based child abuse and neglect prevention projects funded by the National Center for Child Abuse and Neglect; (2) providing technical assistance to the projects in meeting evaluation requirements; and (3) helping the programs design and implement their own internal program evaluations. Because each project has up to 10 service components, many differing across projects, the contractor has developed a series of experimental designs for each service component.

Two ACF studies are looking at family services from an international perspective. First, Transfer of International Innovations--Development of a Clinical Monitoring System to Support Foster Care in Michigan features a computerized system based on a model combining structured and systematic monitoring of each individual child with the aggregation of this information across the whole agency. Developed and now mandated in Israel, and modified for the U.S. foster care system, the system provides an integrated response to the needs of all partners in the agency, including administrators, managers, and policymakers.

A second project, Social and Educational Development of Tribal-Based Communities of the Sonoran and Neger Deserts, is a knowledge-transfer project assessing the application of a preteen Bedouin Arab dropout prevention program in a tribal community in the Negev Desert of Southern Israel to the Pascua Yaqui Indian Tribe in the Sonoran Desert of Southern Arizona. The assessment will examine the effectiveness of a cooperative community-university model for empowering economically disadvantaged minority communities to respond to the unique needs of their at-risk children.

Two ACF projects address cross-cutting issues of substance abuse treatment and gang membership. The first project, Women and Infant Nurturing Services (WINGS), is designed to counteract the upward spiral of female incarceration and substance abuse. This demonstration project at the Rose M. Singer Correction Facility on Riker's Island, New York City, targets pregnant, substance-abusing inmates and uses incarceration as a point of treatment intervention. The evaluation tests the effectiveness of a comprehensive service program, including substance abuse treatment, prenatal health and nutritional care, HIV education, parenting classes, mental health services, and assistance with entitlement preparation.

The second project, Factors Related to Gang Membership Resistance, is gathering data on gangs from two contrasting Los Angeles communities, one with higher-than-average Hispanic and African-American gang activity, the other with lower-than-average gang activity. The project is designed to increase understanding of how youth in urban areas with high levels of street gang activity avoid gang involvement. Expected products include replicable interview protocol, data tapes for other researchers, a final report including implications for prevention programming, and plans for extended validation and replication.

New Directions for Evaluation

ACF's evaluation activities will focus on questions of family economic independence and child well-being that arise from States exercising their new authority to determine public assistance policy and program design under the Personal Responsibility and Work Opportunity Reconciliation Act of 1996. Key questions will concern not only issues of cash assistance and employment policy but also a range of other issues, including interactions among public assistance systems and child and family welfare programs. Assessing the impact of welfare reform on low-income families calls for continuing efforts to develop measures of child well-being; to explore administrative data bases as informational sources for evaluation; to identify methods for measuring outcomes and performance; to stimulate the field to benchmark, measure performance, and track results; and to provide information rapidly to local and State decisionmakers and to national associations.

Initiating and completing evaluations of programs begun as State welfare reform demonstrations is necessary to provide timely information about public assistance strategies and to add to the knowledge base for Federal, State, and local policymakers. Principal descriptive questions include the following: How are Temporary Assistance for Needy Families (TANF) programs implemented at the State and local levels? What is the extent of devolution of decision making to local government? How are child care supply and quality affected? Impact analyses of key policies and interventions (e.g., income disregard strategies and time limits) must follow descriptive studies to examine the effectiveness of these changes and to help States modify policies and approaches.

The effects of interaction and coordination between and among TANF, child development, child welfare, child support, and communitywide interventions constitute another key element of ACF's evaluation agenda. The economic security and overall well-being of disadvantaged children and families may be improved not only by how States design their TANF programs but, more likely, by how States integrate public assistance systems with programs and resources to address child development, family stability, child welfare, and parenting development. Efforts to create a complete picture of public assistance and family and child well-being outcomes will include mechanisms to collect accurate data from States. ACF will work with the Census Bureau and other organizations to ensure that national surveys address issues of concern and will work with States to improve quality and linkages of administrative data.

This evaluation agenda--describing State systems changes, assessing impacts on affected populations, and monitoring State and local program interaction--can only be realized through partnerships with State and local governments, professional organizations, service providers, and others in both the public and private sectors.

Administration on Aging

MISSION: To foster the development of services to help older persons maintain their independence.

Evaluation Program

The Administration on Aging (AoA) is the Federal focal point and advocate agency for older persons and their concerns. The AoA administers key Federal programs mandated under various titles of the Older Americans Act. These programs help vulnerable older persons to remain in their own homes by providing supportive services. Other programs offer opportunities for older Americans to enhance their health and to be active contributors to their families, communities, and the Nation through employment and volunteer programs. AoA works closely with its nationwide network of regional offices and State and Area Agencies on Aging to plan, coordinate, and develop community-level systems of services that meet the unique needs of individual older persons and their caregivers. The AoA collaborates with Federal agencies, national organizations, and representatives of business to ensure that, whenever possible, their programs and resources are targeted to the elderly and coordinated with those of the network on aging.

As the responsibilities of this nationwide network of State and Area Agencies on Aging continues to grow, it is essential that they have the necessary information to meet these responsibilities.

The overall evaluation priorities of AoA are to support studies that provide information on the following:

• the success of existing program implementation in meeting the goals of the Older Americans Act
• the design and operation of effective programs
• issues relevant to policy development, legislative planning, and program management.

Summary of Fiscal Year 1996 Evaluations

One evaluation was completed in FY 1996, the evaluation of the Elderly Nutrition Program (ENP) under TitleIII and TitleVI of the Older Americans Act. This evaluation is highlighted in chapter II and is summarized below. The results of the evaluation show that the ENP has succeeded in accomplishing its mission of improving the nutritional intakes of elderly people, as well as in decreasing their social isolation. The evaluation also shows that the program is evolving to meet the changing needs of older people brought on by shifting demographics and changes in the health care system. There are indications of unmet needs for the program's services, as well as signs that there may be new roles for the program in the future.

The ENP provides an average of one million meals per day to older Americans. These meals are targeted toward highly vulnerable elderly populations, including the very old, people living alone, people below or near the poverty line, minority populations, and individuals with significant health conditions or physical or mental impairments. On average, the meals provided easily meet the recommended daily allowance requirements and significantly increase the dietary intakes of ENP participants. The ENP also reduces the social isolation of older Americans in both the congregate and home-delivered programs and links participants with other needed services. Agencies at all levels have forged close links with other parts of America's emerging home and community-based long-term care system. Federal dollars are highly leveraged. Despite participant's low income levels, their contributions account for 20percent of both congregate and home-delivered meal costs. Local donations and volunteer time, often from program participants, account for 14 percent of costs.

Evaluations in Progress

Evaluation is an important part of the AoA program, and a number of evaluation activities are currently under way, although not on a national level. Many State and Area Agencies on Aging are conducting evaluations of their Older Americans Act service programs. These studies generally focus on such issues as the needs of the target population, the quantity and quality of services delivered, and the impact of the services on the older person. Results are used to better design and target Older Americans Act services. Regional offices of the AoA also conduct assessments of the needs of the States for training and technical assistance. Finally, demonstration projects funded by the AoA have evaluation components to enable them to assess their progress in meeting the objectives of their demonstrations.

New Directions for Evaluation

Given the evolving roles of AoA and State and Area Agencies on Aging and the continued (and projected) growth of the Nation's elderly population, AoA's evaluation efforts will continue to focus on effective program planning and service delivery, as well as on continued monitoring of the program's effectiveness in addressing the goals of the Older Americans Act.

The changes occurring in the aging service-delivery network present an opportunity to learn critical lessons in program planning and system development. These changes include more systematic focus on home- and community-based long-term care, growing sophistication in addressing a variety of needs of the older population, the growth of managed care, and the implementation of sophisticated program information systems. Future evaluations will need to consider the impact of these and similar developments on the delivery of Older Americans Act services to our Nation's older persons.

Agency for Health Care Policy and Research

MISSION: To generate and disseminate information that improves the health care system.

Evaluation Program

The Evaluation Program within the Agency for Health Care Policy and Research (AHCPR) is designed to respond to three types of evaluation needs:

• the need for comprehensive information to assess the agency's effectiveness in meeting its major and long-term priorities and goals
• the need for information from "fast-track" or quick turnaround projects to respond to critical agency and departmental concerns
• the need for information from internal evaluations to improve the efficiency with which the agency performs its work

To address these needs, evaluation components are built into virtually all AHCPR activities. Among the evaluation mechanisms used by the agency are targeted evaluation studies undertaken through contracts or grants; peer review of grant applications and technical review of contract proposals for scientific rigor and integrity; efforts to obtain feedback from "customers" on the usefulness of AHCPR research efforts; feedback from AHCPR's User Liaison Program (which provides information and technical assistance to State policymakers, health departments, and officials); and other targeted efforts, such as focus groups and surveys to provide baseline information, inform the design of future agency work, and assess progress toward goals.

AHCPR received delegated authority to review evaluation projects in 1992. In carrying out that authority, it established a two-tier system for assessing proposed evaluation projects eligible for 1-percent set-aside funds. The first phase of the process, an executive-level review assessing the policy relevance and relative priority of proposed projects, is conducted by the administrator and senior staff. The second, a technical merit review, assesses policy-relevant proposals for technical feasibility, soundness of design, costs, potential importance of the findings, and relation to ongoing evaluation activities. This second review is conducted by the Task Force on AHCPR Evaluation Projects, a group consisting of one individual with evaluation expertise from each Office and Center in the agency.

Summary of Fiscal Year 1996 Evaluations

The four high-priority evaluation projects completed in FY 1996 directly reflect the agency's commitment to enhancing the scientific underpinnings of the Nation's efforts to measure and improve the quality of health care services. Each provides information to enhance the quality of the health care system: information that can improve clinical quality, inform consumer choice, and focus future research on what works best in health care.

Evaluating Practice Guidelines and Developing Science-Based Measures of Clinical Quality

The first of the four completed projects assessed the feasibility of deriving clinical quality evaluation and improvement tools from evidence-based practice guidelines. The project responds to the country's increasing demand for valid, reliable, science-based indicators of clinical quality that can be used in a variety of health care settings. A contract with the Center for Clinical Quality Evaluation (formerly the American Medical Review Research Center) in Washington, DC, translates three practice guidelines (for urinary incontinence, acute postoperative pain, and benign prostatic hyperplasia) into performance measures that can be used to determine the extent to which clinical care is consistent with scientific recommendations and to target specific opportunities for quality improvement.

With the assistance of Medicare Peer Review Organizations (PRO's) from four States, the project developed guideline-based measures and tested them for use among the Medicare population. The PRO's also served as educational facilitators in a quasi-experimental design component of the project focusing on the introduction of the benign prostatic hyperplasia (BPH) guideline in four "continuum-of-care" settings (clinics and provider groups providing both ambulatory and inpatient care).

The project found that it is feasible to develop guideline-based performance measures that could be used to evaluate and improve clinical performance. The BPH guideline was favorably received by both physicians and patients, and improvements in clinical performance made for specific areas were the focus of educational interventions.

Enhancing the Science of Clinical Quality Measurement

The second project, the Computerized Needs-Oriented Quality Measurement Evaluation System (CONQUEST 1.0), enhances the science of clinical quality measurement and evaluation by creating a standardized and rigorous approach to understanding performance measures, as well as for identifying, evaluating, and selecting measures most appropriate to the users' needs. The project collected 53 sets of clinical performance measures (tools that assess the delivery of clinical services) and summarized their content and construction so that the measures could be compared directly on factors such as their data needs, the aspect of clinical care each measure evaluates, their scoring and analysis considerations, reliability and validity testing, and the extent of their use to date. The information collected is summarized in a "measure" data base.

In addition to information on clinical performance measures, the project summarizes information on 50 common clinical conditions into a "condition" data base. Information in the condition data base includes age groups affected by the condition; forms and prevalence of the condition; cost and utilization associated with the condition; comorbidities, risk factors, and potentially preventable adverse events associated with the condition; information on clinical services used to diagnose and treat the condition; and information on provider settings and health care professionals associated with the primary condition. A key feature of the condition data base is its inclusion of recommendations from clinical practice guidelines and findings from medical effectiveness research. Information from the condition data base can be used to identify performance measures for a given condition, facilitate the use of individual performance measures for broader quality measurement and improvement uses, and interpret findings from performance data relative to information from practice guidelines and medical effectiveness research.

Initial response to CONQUEST 1.0 has been extremely favorable; the agency has distributed over 3,000 copies of the first version of the software and user manual and is currently developing a "run time" version that will function in Windows 95. In addition, the agency is building on this initial effort by sponsoring a multiyear contract to develop a Quality Measurement Network, a clearinghouse and technical resource for those interested in identifying and using measures to evaluate and improve clinical performance.

Helping Consumers Understand Health Care Quality Information and Evaluate Health Care Plans

The third project, described in chapter II, develops an approach for States to use in evaluating the effect of health care delivery changes on access to, quality of, and satisfaction with health care. Through the AHCPR Rural Health Research Center at the University of Washington, and its subcontractor the Oregon Health Sciences University, the Oregon Consumer Scorecard (OCS) Project assists the State of Oregon in developing an approach to evaluate access to and quality of care provided under the Oregon Health Plan. The Oregon Health Plan is a State-supported health insurance program designed to provide universal access to care for low-income residents. The project assisted in the development and pilot testing of measures of clinical quality and consumer satisfaction and assessed methods for communicating quality information to consumers in a way that would help them evaluate and select health care plans.

The project found that the information needs of consumers and health professionals differ markedly. While consumers judge health plans on such issues as how they gain access to specialty services for various acute or chronic conditions, health care professionals use population-based performance measures to judge quality of care. These population-based measures focus more extensively on clinical processes and outcomes; while these measures currently may have little meaning to most consumers, they have great potential in shaping the quality of health care and health plan accountability. The project found that consumers are interested in a variety of formats for reviewing health plan data and that the availability of a "personal guide" or a trained individual to assist consumers in understanding information would be highly beneficial. With respect to health plan information needs, the project found that health plans face a "data burden" that is costly and could be relieved by the establishment of uniform standards that are used by all health plans and purchasers.

In addition to providing technical and substantive assistance to the State of Oregon, the project develops the science base for statewide quality assessment and reporting, an effort which can be used to help other States evaluate quality of care.

Synthesizing Information From AHCPR-Sponsored Research

The last of the four projects is a multiyear effort to track and synthesize findings from AHCPR-sponsored research in the Medical Treatment Effectiveness Program (MEDTEP). AHCPR has been working with Walcoff and Associates, Inc., of Fairfax, Virginia, to collect and analyze information on MEDTEP research, including clinical practice guidelines. The project has collected information from published professional literature; presentations to and proceedings of conferences and meetings; trade journals in the health care delivery field; reports of congressional testimony and deliberation; reports of State and local legislation or executive testimony and deliberation; magazine and newspaper articles; radio and television programs; and reports of other Federal agencies. During the period of the project, findings from these tracking efforts have been published in the AHCPR-sponsored newsletters Research Activities and MEDTEP Update. Most recently the project has produced comprehensive summaries of MEDTEP projects, a bibliography of Publications of the Patient Outcomes Research Teams projects, and a data base of information on these AHCPR-sponsored projects.

Evaluations in Progress

Ongoing evaluations at AHCPR focus on three broad areas: environmental scans to identify research needs and potential partnership opportunities, evaluations of ongoing agency work, and evaluation planning and design studies to guide future efforts. These activities are part of AHCPR's emphasis on developing and implementing measures of agency performance and on ensuring that agency-sponsored research efforts are most useful to a wide range of audiences.

Assessing the Health Care Environment

Three projects examine ongoing health care policy and research activities in the public and private sectors. These "environmental scans" help AHCPR identify emerging health policy and research issues, facilitate agency strategic planning, and avoid duplication and waste of resources. They also identify areas in which Federal resources, tools, data, and methodologies can help answer questions about the effects of health care policy changes on the cost, quality, and access to health care services, as well as the effectiveness of those services.

One project builds on the work of the Department of Health and Human Services Data Council's Committee on Health Data Standards and the Information Infrastructure Task Force's Health Information Working Group to collect and summarize information on the information standards activities of various Federal agencies. This project will update a previous report, Current Activities of Selected Health Care Informatics Organizations. Another project will identify and describe the range of outcomes research activities conducted by various private-sector organizations and focus on approaches for identifying research topics, outcomes research methodologies, and translation of findings into clinical practice. A third project in this group will identify approaches to achieving productive public-private research collaborations. This effort will use publicly available data to identify examples of public-private collaborations and to examine such facets of the project as methods for identifying research topics, treatment of investment and financial records, and reporting research results.

Evaluating Ongoing Agency Work

Three projects in the agency's current portfolio evaluate aspects of AHCPR's ongoing research program activities. The first is a readership survey assessing customer satisfaction with selected AHCPR publications. The study focuses on the AHCPR publication with largest distribution, Research Activities, and will assess both the format and content of the publication. Subsequent surveys will target other agency products, with the objective of developing more effective means of communicating agency research findings. Another project synthesizes the results of AHCPR research aimed at implementing and evaluating of clinical practice guidelines, physician behavior change strategies, and other quality improvement tools. The third project in this category evaluates AHCPR's training activities for minority health services researchers. It will also examine other (non-AHCPR) minority health services research efforts to identify desirable elements and characteristics of success.

Evaluation Designs and Planning Studies

Four projects are ongoing in the area of evaluation design and planning. These projects assess the current state of knowledge on a given health care topic area, identify research gaps, and develop and test approaches and indicators that could be useful for assessing the quality of agency-sponsored programs. The first project, an assessment of data release strategies, examines how data from AHCPR's Medical Expenditure Panel Survey and other national data bases have been used to support health policy and health services research. The project will delineate the strengths and weaknesses of the various data sets and identify ways to increase their utility. It will also provide needs assessment information essential for maximizing the usefulness of AHCPR's large-scale data efforts.

Another project examines the feasibility of compiling an encounter-level managed-care research data base by identifying currently available public- and private-sector data and by identifying data analysis and data release problems. The third project in this category focuses on better understanding research needs in the area of appropriateness of care. This study will explore the published literature on the causes of diagnostic inefficiency in primary care, interventions designed to address these inefficiencies, and the effect of interventions on the quality of care. The project will focus on errors in selected high-cost, high-prevalence areas of health care for which there may be considerable uncertainty about diagnosis, but accepted treatments. The final study in this category examines published and unpublished studies on the effects of consumer health informatics on patient decision making.

New Directions for Evaluation

In FY 1998 and beyond, AHCPR will focus its evaluation activities in three general areas: the development and use of agencywide performance measures, evaluation projects targeted at effective communication and implementation of health care quality and effectiveness findings, and evaluation of the effect of health care policy changes on the cost and quality of services, access to services, and the effectiveness of those services.

The first area emphasizes agencywide performance measures that can evaluate the quality of the agency's work and inform policy making, budget planning, and program management. Agencywide performance measures will help AHCPR address requirements of the Government Performance and Results Act, providing information that will improve the agency's performance in key program areas. Projects initiated in FY 1997 will produce and test measures that can be applied in FY 1998 and beyond. In addition to these efforts, the agency will be evaluating the effect of changes in programmatic operations. One example of this type of effort is a proposed evaluation of the effect of recent improvements to the AHCPR grants process.

The second area centers on building AHCPR's portfolio of research to translate research findings into forms of information that actively assist consumers, practitioners, payers, and others in making effective health care decisions. Of particular interest to the agency are needs assessment and formative evaluation efforts related to the new Evidence-Based Practice Centers program. This new initiative, which will replace AHCPR's guideline development program, creates research centers that will conduct systematic reviews and synthesize scientific evidence related to the effectiveness of various approaches, procedures, and technologies. These evidence syntheses can be used by practitioners and health care organizations to develop clinical practice guidelines and other quality improvement tools. Evaluation activities will first focus on assessing user needs in terms of topics, formats, and technical assistance. Subsequent evaluation activities will assess the usefulness of the new products for clinical quality improvement.

The third area will build on AHCPR's work to examine and evaluate changes in the health care system and how those changes affect access to care, use of services, quality of care, and patient outcomes. As the agency continues improving the Medical Expenditure Panel Survey, new evaluation questions and opportunities arise on such issues as the efficiency of the new design and its effectiveness in answering key health policy questions. The agency will be evaluating aspects of the survey process, including data support contracts and approaches to evaluating modeling capacities.

Centers for Disease Control and Prevention

MISSION: To promote health and quality of life by preventing and controlling disease, injury, and disability.

Agency for Toxic Substances and Disease Registry

MISSION: To prevent exposure and adverse human health effects and diminished quality of life associated with exposure to hazardous substances from waste sites, unplanned releases, and other sources of pollution in the environment.

CDC Evaluation Program

The Centers for Disease Control and Prevention (CDC) places high priority on evaluations seeking to answer policy, program, and strategic planning questions. Performance improvement studies are of particular interest and importance to the agency. Evaluation studies are developed and selected based on CDC's eight strategies to achieve its mission:

• Monitor health.
• Detect and investigate health problems.
• Conduct research to enhance prevention.
• Develop and advocate sound public health policies.
• Implement prevention methods.
• Promote healthy behaviors.
• Foster safe and healthful environments.
• Provide leadership and training.

CDC's evaluation philosophy is consistent with its overall focus on the practice and application of sound public health strategies. This orientation is exemplified by the agency's "bottom up" nature of project development. Projects are conceptualized, developed, and monitored by the public health professionals at CDC who are most closely aligned with the everyday practice of public health--program staff.

On an annual basis, the Director of CDC provides guidance to the various Center, Institute, and Office Directors on 1-percent set-aside evaluation activities. This memorandum generally includes information about the types of studies to be carried out with 1-percent evaluation funds. Each proposal undergoes multiple levels of review. Initial review is conducted by the Office of Program Planning and Evaluation. Subsequent reviews are completed by staff within the Office of the Assistant Secretary for Planning and Evaluation. Study authors are provided with comments, questions, and recommendations made by reviewers. In addition to providing their responses, authors are given the opportunity to revise their proposals at this time.

A panel of CDC evaluators, scientists, and program managers meets to review and rank proposals. Review criteria include (1) relevance to prevention effectiveness; (2) relative importance of the public health problem being addressed; (3) probability that the proposed project will accomplish its objectives; and (4) the extent to which other CDC programs will derive benefit from the project. Results from this panel review are converted into a comprehensive ranking, which is provided to the Director of CDC. Final funding decisions are made at this time.

Finally, staff within the Office of Program Planning and Evaluation work closely with program staff to ensure development of a clear statement of work for selected projects. Prior to initiation of procurements, a final ad hoc review of the project statement of work is completed.

ATSDR Evaluation Program

The Agency for Toxic Substances and Disease Registry (ATSDR) receives its funds from Environmental Protection Agency Superfund appropriations, rather than from public health appropriations; therefore, ATSDR does not receive a 1-percent evaluation set-aside. Nevertheless, ATSDR is responding to the changes mandated in its program planning and evaluation efforts by the National Performance Review and the Government Performance and Results Act. To meet those requirements, ATSDR staff members modify the agency's planning process, incorporating implementation strategies and outcome performance measures.

Prominent issues addressed in the new planning system emphasize ATSDR's priority of improving the health of people affected by hazardous substances polluting the environment. Using exposure assessments and demographic data to identify people at risk and, more directly, assessing and addressing the concerns of ATSDR customers represent some of the improvements. The new planning system provides the basis for measuring ATSDR performance and making systematic improvements as part of its internal evaluation activities.

Summary of Fiscal Year 1996 CDC Evaluations

A total of 17 evaluation projects were completed in FY 1996, 6 of which are reported in appendix A. Ten of these projects were funded through the 1-percent evaluation set-aside, and 7 were funded with program dollars. These studies are of four types: data policy and surveillance studies, program evaluations, evaluations of educational interventions, and assessments of specialized aspects of current or proposed programs.

Data Policy and Surveillance Studies

Three studies sought to assess the implications of policy decisions on various existing or proposed data and surveillance systems. One of these projects was part of an effort to develop a comprehensive monitoring system to track the impact of health reform and provide feedback to policymakers. Although such a system was expected to monitor the impact of the 1994 Health Security Act, the report suggests that a system could be designed to inform health policy, regardless of the ultimate shape and timing of health reform. A second study was undertaken to assist in developing an appropriate accountability system for the Vaccine for Children (VFC) program that is both rigorous and practical for use in clinical settings. As part of this study, data collection procedures for proposed systems were tested to determine their practicality and usefulness for collecting vaccine usage data. The study also provided information about the acceptability of any accountability system to health care providers, since a system that is perceived as overly burdensome could discourage participation in the VFC program. Another study evaluated whether and to what extent use of CDC WONDER/PC by State and local public health practitioners positively affects the practice of public health, along a variety of dimensions (e.g., efficiency, scope, quality, and timeliness of work).

Program Evaluations

Four program evaluation studies were completed in FY 1996. These studies included evaluations of CDC's Prevention Effectiveness Activity, the National Center for Injury Prevention and Control's injury grant program, and the National Nosocomial Infections Surveillance Program. In addition, a case study of the Henry J. Kaiser Family Foundation's Community Health Promotion Program was completed.

Evaluations of Educational Interventions

Three of the four evaluations of educational interventions completed in FY 1996 focused on provider educational issues. One of these projects addressed health care providers' use and perceptions about vaccine package inserts, in an effort to improve the utility of inserts in communicating the nature and extent of dangers imposed by vaccines. A second immunization educational intervention developed and field tested a quality assurance tool designed to facilitate provider self-assessment, emphasize the importance of following prevention guidelines, and collect data about provider practices. A third study of this type assessed the effectiveness of the domestic violence module of an innovative physician training program that is being implemented at the University of California-Los Angeles Medical School. The final education-related evaluation study examined a variety of questions related to the distribution, awareness, use, and perceptions of the Chronic Disease Prevention (CDP) File. The CDP File consists of the Health Promotion and Education Database, the Comprehensive School Health Database, the Cancer Prevention and Control Database, and the Chronic Disease Prevention Directory.

Assessments of Specialized Aspects of Current or Proposed Programs

Five studies relating to discrete aspects of current or proposed programs were completed in FY 1996. Two of these projects were developmental and resulted in training materials. One activity concluded with the provision of an evaluation training course and materials for CDC Planning and Evaluation Officers. The other project identified key factors in the successful development and implementation of CDC-funded youth violence prevention projects and compiled the findings into a document to help individuals, groups, or communities implement youth violence prevention and intervention projects. Another study examined the impact of mandated managed care on health care. Based on eight case study organizations, this evaluation found (1) little prevention activity; (2) no relationship, or a poor one, with the local health department in half of the organizations; and (3) a dispersion of activities among community organizations that challenges the traditional roles of State and local health departments.

The two final studies of this type were related to immunization. One examined issues surrounding consent for adolescent immunization. As adolescent vaccination expands, particularly hepatitis B vaccination, issues related to need for parental consent for receipt of indicated vaccines become increasingly important. Similarly, operational issues surrounding the provision of immunization services is critically important. The final study evaluated how audits conducted in the State of Georgia from 1986 to 1994 affected immunization coverage levels in the preschool population and determined medical and management policies and practices that influence immunization rates.

CDC Evaluations in Progress

CDC's evaluation studies, in progress during FY 1996, consisted of four types: evaluations of data policy and surveillance studies; program evaluations; development of performance measures; and assessments of specialized aspects of proposed or existing programs.

Evaluations of Data Policy and Surveillance Systems

Ten studies are under way that relate to data management or reporting systems. These studies include evaluations of the use of data transmitted through CDC's National Electronic Telecommunications System for Surveillance (NETSS) and the Public Health Laboratory Information System (PHLIS), as well as an assessment of the National Immunization Survey (NIS). Evaluations of specific issues critical to data quality are also under way. These studies include evaluations of racial and ethnic identification data, redesign of the National Health Interview Survey (NHIS), and an examination of the effectiveness of CDC surveillance for drug-resistant pneumococcal infections.

In the case of the study of NETSS and PHLIS, determinations about how surveillance data transmitted through each of these systems are used at local, State, and Federal levels will be made. Similarly, the centerpiece of the NIS study is to design an evaluation system that will focus on three main questions: How well does the NIS meet the program needs of the National Immunization Program for estimates of immunization coverage rates for population groups of special interest? Is there a way to simplify the analytic methods currently used on the survey without jeopardizing the statistical integrity of the survey? Is there a way to make the survey data collection less costly without jeopardizing the statistical integrity of the survey?

Both the evaluation of racial and ethnic identification data and the redesign of the NHIS represent efforts to understand and refine survey operational issues. Finally, the study evaluating CDC's current surveillance system for drug-resistant Streptococcus pneumoniae will evaluate this system's hospital-based sentinel surveillance system for sensitivity and representativeness. Active population-based surveillance for invasive pneumococcal infections in two geographically distinct areas will be conducted. Surveillance areas for this evaluation are either adjacent to a sentinel surveillance hospital or in a community with one or more hospitals sufficiently large to serve as a comparison for the rest of the community.

Program Evaluations

Eight program evaluations are currently in progress. Included in this number are evaluations of the National Institute for Occupational Safety and Health (NIOSH) Health Hazard Evaluation Program, the Field Epidemiology Training Program, the National Laboratory and Training Network, and the Prevention Centers Program.

The study of the Health Hazard Evaluation (HHE) Program is a process evaluation. As such, the overall purpose of the study is to develop a tool and procedure by which the NIOSH can conduct an ongoing evaluation of the effectiveness of the HHE Program, which responds to 400 to 500 requests for on-site health hazard evaluations each year from employers, employees, employee representatives, or other Federal, State, or local agencies.

By contrast, two of the other studies mentioned are outcome evaluations. The evaluation of the Field Epidemiology Training Program (FETP) will ascertain whether the program has achieved its objectives, which are to train public health professionals in applied epidemiologic skills, to promote the sustainability of autonomous FETP's, and to develop a global network of national programs.

The evaluation of the National Laboratory Training Network (NLTN) will assess the degree to which the NLTN achieves its goals and mission. Study questions fall into five general categories: offerings related to needs of laboratories and their staff; quality of the training provided; impact of the training provided; quality of outreach and marketing; and barriers to training.

Finally, the evaluation of the Prevention Centers Program will assess to what extent CDC-supported research is providing the public health community with workable strategies to address major public health problems. Prevention research issues related to innovation, relevancy, dissemination, application, and quality of research are the focus of this project.

Development of Performance Measures

Four ongoing projects involve the development of performance measurement systems. Two of these studies relate directly to the Government Performance and Results Act of 1993 (GPRA). A third study will develop indicators by which the fiscal and programmatic impact of HIV-prevention community planning may be measured. The final study of this type entails the development of an evaluation protocol for State-based diabetes control cooperative agreement programs.

Assessments of Specialized Aspects of Proposed or Existing Programs

Fourteen specialized studies relating to discrete aspects of current or proposed programs are in progress. These studies include an evaluation of tuberculosis (TB) Outreach Worker Activities, as well as a community context study seeking to understand minors' access to tobacco products. Two studies related to violence prevention are also included in this category. One of these studies relates to suicide in Native American communities, and the other considers various implementation aspects of youth violence-prevention programs.

Each of the studies mentioned has at its core a desire to understand particular human behaviors and motivations. A clear understanding of these issues must be present before wholehearted program and fiscal commitments are made. For example, the Advisory Committee for the Elimination of TB recommended increased funding for TB outreach and workers conducting the outreach. However, prior to committing resources to this strategy, it is incumbent upon CDC to have a clear understanding of how outreach work is currently conducted and what outreach workers do in their everyday work. It is through the evaluation of TB Outreach Workers that the National Center for HIV, Sexually Transmitted Disease, and TB Prevention will be provided with baseline data about how outreach workers function.

Similarly, a community context study is also being conducted that will refine CDC's understanding of the relationships between public policies prohibiting minors' access to tobacco, implementation and enforcement of such policies, tobacco vendor perceptions, actions that may influence the sale of tobacco products to minors and the use of tobacco products by minors.

Finally, two evaluations addressing violence-prevention issues are also under way. The first of these studies is being conducted jointly by CDC and the Indian Health Service. This study will evaluate surveillance systems, process indicators, and outcomes of four multifaceted suicide-prevention programs. The focus of the second violence-prevention study involves evaluating the effectiveness of a selected program for health care providers and battered womens' advocates. Specifically, the ability of this program to successfully diagnose, manage, refer, and otherwise assist female victims of intimate partner violence will be assessed.

New Directions for CDC Evaluation

CDC's evaluation program will continue to support studies aimed at improving public health programs. This focus on improvement is consistent with the congressional intent of GPRA. Efforts to develop and implement performance indicators are currently under way at CDC. Similarly, as programs have been developing and implementing performance indicators, projects that assess the effectiveness and efficacy of such indicators have been initiated.

CDC's evaluation priorities in upcoming years will be based on five cross-cutting strategies that address the programmatic needs of its 11 Centers, Institutes, and Offices.

Strengthen Core Public Health Functions

A number of key activities at CDC center around the agency's commitment to strengthening the Nation's core public health functions. Tangible results of this commitment are reflected by CDC's work with State and local health departments and other partners throughout the country to obtain the information necessary for monitoring and evaluating health, conducting epidemiologic and laboratory studies, developing new technologies, and providing training and technical assistance to ensure that States and communities can protect the health of their citizens. Ongoing and proposed evaluation studies seek to answer how well CDC is fulfilling this commitment.

Enrich Our Capacity to Respond to Urgent Threats to Health

Epidemiologic investigations and laboratory work done by CDC and State and local health departments enable CDC to address urgent threats to public health in a timely and effective manner. Similarly, in a world increasingly threatened by emerging infections, CDC's leadership role in this area is critical. A number of studies are under way that address emerging infections, including Group B Streptococcus, perinatal HIV, and drug-resistant pneumococcal infections.

Develop Nationwide Prevention Strategies

One of CDC's key roles is to translate knowledge about effective methods of preventing disease and injuries into nationwide strategies that reach people in communities throughout the country. Several studies under way seek to assess the extent to which guidelines developed by CDC affect disease prevention activities throughout the Nation.

Promote Women's Health

Recognizing that women's health issues have not always received the attention they have warranted, CDC has identified promotion of women's health as one of its top priorities. Although a number of projects are funded by CDC's Office on Women's Health, there are several ongoing evaluation studies that address important women's health issues. One such study is an evaluation of guidelines for preventing perinatal HIV infection by measuring how recommendations are translated into routine clinical practice. Successful implementation of recommendations depend on the ability to maximize the opportunities for HIV-infected women to learn their HIV infection status, to be offered and receive preventive therapy, and to gain access to health and social services for themselves and their infants.

Invest in Our Nation's Youth

CDC's commitment to investing in the youth of our Nation is exemplified by school health education programs. However, CDC's investment is not limited to school-based intervention programs. For example, one study is designed to assess the effectiveness of various teen pregnancy prevention interventions. This study will result in a systematic methodology to identify and evaluate community intervention components that show promise based on behavioral science theory, available evaluation information, and the consensus of leaders in the field.

Food and Drug Administration

MISSION: To protect and promote public health through food, drug, medical device, and cosmetic regulation.

Evaluation Program

Systemic changes in the government management environment are strongly influencing the setting, conduct, and utilization of evaluation activities in the Food and Drug Administration (FDA). Two forces related to the goals of the Government Performance and Results Acts (GPRA) are driving and reshaping the evaluation function within the FDA.

Performance Management

The reorientation of all government managers toward performance management has shifted the responsibility for program evaluation from specialized staff offices and contractor studies to the line managers. Picking appropriate program goals, establishing a valid measure of these goals, and collecting management information to record the measured progress toward these goals are now an integral part of the manager's responsibility. Relearning the role of management with regard to these shifted responsibilities is a key priority.

Customer Participation

Most of the FDA's management measures of its own performance are also measures of the performance of the regulated industries. Thus, while the beneficiary of FDA's performance is ultimately the general public, the FDA supplies industry with an essential component of commercial success. The FDA's approval of a new drug, for example, not only satisfies a legal requirement but also assures the public of the safety and efficacy of the drug. Collaboration between the FDA and its regulated customers regarding the design and coordination of the joint responsibilities to ensure effective high-quality products has been a revolutionary concept, but it is becoming the new norm under the customer-conscious GPRA directives.

An extension of both performance-based goals and customer participation also affects our response to small businesses.

Responding to the Needs of Small Business

A high proportion of industries regulated by the FDA are small businesses. In order to improve responsiveness to industry needs without compromising the protection of the public health, the FDA is emphasizing two strategies. One is to enhance industry input during the rulemaking process and to increase the rigor of analysis of possible regulatory strategies. The second is to improve compliance, where possible, through better education and remedial technical assistance. Both strategies extend the existing priority of customer participation and performance-based goals to focus on enhancing the participation and performance of small businesses.

In sum, the FDA's evaluation efforts are driven by the mandates of GPRA and its corollaries, are carried out by line managers rather than specialized evaluation staffs, and are focused in the areas of performance management and customer participation, with additional attention to the needs of small businesses. Because the FDA receives its funds from agriculture appropriations rather than Public Health Service Act appropriations, the FDA does not manage a 1-percent evaluation set-aside.

Summary of Fiscal Year 1996 Evaluations

FY 1996 marked the third year under the new evaluation paradigm. The following example illustrates the manner in which managers have integrated evaluation into their line responsibilities and conducted evaluation in cooperation with the affected customer.

Implementation of the Prescription Drug User Fee Act

FY 1996 marked the fourth year of the 5-year implementation of the Prescription Drug User Fee Act (PDUFA) of 1992, which authorized the collection of fees from the pharmaceutical industry in order to facilitate the FDA's timely review of human drug applications. The act mandates that the FDA prepare annual reports for each fiscal year in which fees are collected. The Fourth Annual Performance Report (covering FY 1996) was prepared in response to that mandate and summarizes findings on how well the FDA has accomplished its performance goals for FY 1996. These performance-related goals were jointly established between the FDA and the pharmaceutical industry and fall into three main categories: eliminating overdue backlogs, building excellence into the review process, and achieving high performance. Eighteen of the goals have already been accomplished and reported in previous years' performance reports. Four additional goals were in effect for FY1996, while seven more will be in effect for FY 1997.

The success of the fourth year of PDUFA is reported in the Fourth Annual Performance Report. Despite an increasing workload, the agency has exceeded every PDUFA performance goal, resulting in a record number of new product approvals and shorter approval times. Under PDUFA, during FY 1996, the FDA approved 131 new drug applications and product license applications, an increase of 56 percent over FY 1995's total (84) and a 96-percent increase over FY 1994's total (67). PDUFA has enhanced the working relationship between the FDA and its sponsors, resulting in higher quality applications that can be accepted immediately and reviewed more quickly. Ultimately, new products are reaching the market faster.

Consumer Use of Food Labels

The Nutrition and Labeling and Education Act (NLEA), passed by Congress in November 1990, led to the most far-reaching changes in food labeling since the Food, Drug, and Cosmetic Act of 1934. The new regulations, governing the content and format of food labels on nearly all FDA-regulated products, went into effect in 1994. The NLEA had multiple objectives that will have multiple short- and long-term impacts on consumers and the marketplace. To monitor some of these impacts, the FDA is tracking consumer awareness, attitudes, and reported behaviors through nationally representative sample consumer surveys, is examining changes in the marketplace by looking at sales trends for selected product categories, and is assessing compliance with NLEA regulations. Some preliminary findings are presented in The Impact of the NLEA on Consumers: Recent Findings From FDA's Food Label and Nutrition Tracking System, which was completed during FY 1996. This report offers some insights into the impact of the new food label regulations that occurred in the period shortly after the regulations were implemented:

• The data support the popular perception that Americans have embraced the new food label as an effective tool to enable them to make healthier food choices.
• Consumer confidence in nutrient content and health claim information also showed improvements.
• Information on food labels appears to be having a growing impact on consumer decision making.

For a small number of product categories in which fat-modified products are available, there has been an impressive, simultaneous increase in new product introductions and percentage of market share for these fat-modified products. While each product category has a unique history, these data mirror recent news reports that suggest both consumers and food manufacturers have responded to the impact of the nutrition facts label on food products. These preliminary market trend analyses are consistent with the views that increased availability of nutrition information, as mandated by the NLEA regulations, has contributed to a growing number of new fat-modified products in the marketplace and that these products are garnering an increasing share of their respective markets.

Food Labeling and Package Survey

The report, Status of Nutrition Labeling of Processed Foods--1995, completed in FY 1996, reviews the status of processed, packaged food labels subsequent to publication of regulations promulgated in response to the NLEA of 1990. The findings were based on data from the Food Label and Package Survey (FLAPS), started in 1977. The purpose of FLAPS is to produce a data base containing label and package information obtained from a sample of food products from the processed packaged food industry. The primary information gathered consists of the prevalence of nutrition and sodium labeling. Label and package information are obtained from the food labels of approximately 1,300 brands of food. The sampling frame for FLAPS is based on sales data obtained from the A.C. Nielson Company, from which sales-weighted estimates are subsequently derived. The major findings from this review include the following:

• Nutrition-labeled products account for an estimated 96.1 percent of the annual sales of processed, packaged food across 58 major product groups.
• There was a 27.2-percent increase in the proportion of nutrition-labeled products between 1993 and 1995.
• For processed meat products, 100 percent are now nutrition labeled, an increase of 47.9 percent over the rate in 1993.
• In three product groups, less than 90 percent of the products sold have nutrition labels--tea, salt (including seasoning and spices), and coffee.

The current study focuses only on the percentage of packaged food products sold annually that bear quantitative nutrition labeling, as well as data on health claims, nutrient content claims, and ingredients.

Evaluations in Progress

The evaluation agenda of managers during FY 1997 and in the future will be influenced by two objectives, performance management and customer participation, which reflect the forces shaping their evaluation role. Managers of all FDA programs are engaged in the process of evaluating their performance measures in light of the GPRA standards. Broad-based training of managers to enable them to implement the performance design of their responsibilities is under way. This multiyear process toward a new standard of performance measure will end its first phase with the formulation of the FDA's FY 1997 budget. FDA managers are also identifying further opportunities to involve their customers in the design and testing of alternative ways of doing business. The design and implementation of the PDUFA is the notable example of the successful pursuit of this objective. FY 1997 will mark the fifth annual performance evaluation report on this successful program.

The following projects are currently under way.

Implementation of the Prescription Drug User Fee Act

FY 1997 marks the fifth year of the PDUFA program. Seven goals have been established, delineating the most ambitious standards of this program. The effects of additional resources were evident in the FY 1996 program where, despite an increasing workload, the agency exceeded every performance goal, resulting in a record number of new product approvals and shorter approval times. There is also evidence that the quality of applications has improved. Based on last year's performance, it is anticipated that the FY 1997 phase of this program will not only meet the established goals, but substantially exceed those goals.

Assessment of the Mammography Quality Standards Act of 1992 (MQSA)

This program is the FDA's second major user fee, performance-oriented, GPRA-style initiative. Under this statute, the FDA is charged with ensuring that accessibility to quality mammography services is maintained for all facilities performing mammographies. The agency is required to certify and annually inspect more than 10,000 mammography facilities. Prior to passage of the MQSA, the FDA's oversight responsibility was limited to approving new mammographic processors, x-rays, and tubing. However, MQSA requires that facilities within its jurisdiction meet standards developed by the agency in order to ensure that a reasonable level of quality is maintained to achieve certification by the agency.

The FDA is concerned that recent trends by third-party payers to capitate payments for mammography services have severely limited the ability of facilities to absorb any cost increases caused by the quality standards. Studies have indicated that excess capacity exists within the industry. Therefore, more efficient provision of mammography services may result in low-volume facilities leaving the industry. The agency needs to assess these economic implications in order to ensure that patients will continue to have access to affordable, high-quality mammography services. This study addresses these issues in two ways. The first involves measuring the effect of standards on patient access to mammography, particularly in rural areas. Findings for this analysis were reported in the document Effects of Facility Closures on Access to Mammography. The second component is a cost-benefit analysis that will be used to predict the impact of proposed standards on the costs of providing mammography. These issues are addressed using data collected from facilities that provide mammography services. A draft report of the results of the cost-benefit analysis was presented in the spring of 1996, and comments are currently being reviewed. A final version will be available in the fall of 1997. Agency managers are using these results to develop a performance-oriented regulatory policy that will be coordinated with existing customers.

Food Labeling and Package Survey (FLAPS)

This is an ongoing study conducted by the FDA's Center for Food Safety and Applied Nutrition; it will continue through December 1999. In order to monitor labeling practices of domestic food producers, the FDA implemented FLAPS in 1977. The purpose of FLAPS was to produce a data base containing label and package information obtained from a sample of food products from the processed, packaged food industry. The primary information gathered consists of the prevalence of nutrition and sodium labeling. Label and package information is obtained from the food labels of approximately 1,300 brands of food. The sampling frame for FLAPS is based on sales data obtained from the A.C. Nielson Company, from which sales-weighted estimates are subsequently derived. The FDA uses these data to quantify the prevalence of nutrition and sodium labeling in the food supply, the extent of quantitative labeling for cholesterol and fatty acid content, the extent of use of contents product ingredient lists, and the extent and types of nutrition claims appearing on food products. FLAPS data enable FDA personnel to keep abreast of market responses to food labeling rules via changes in product package labels.

FDA Inquiries Assistance Program for Small Business

Under the Small Business Regulatory Enforcement Fairness Act, the FDA must establish a program for answering inquiries and providing assistance to small businesses and other small entities, regarding how to comply with FDA rules. The FDA currently provides assistance and advice to small businesses in a multitude of ways, including Center and regional small business representatives, inspectors, public affairs staff, and others. This project will assess the level of success achieved in coordinating these many sources of advice and usefulness and appropriateness of the assistance. Elements to be evaluated include the frequency, subjects, sources, and recipients of inquiries; the type and timeliness of response; and the number and characteristics of complaints or requests for second opinions. The evaluation will also consider the training and materials available to FDA staff to help them answer inquiries, the barriers to providing timely and useful responses, and issues related to tracking inquiries and responses. The report on this project evaluation is due to be completed by March 1998.

FDA Penalty-Reduction Program for Small Business

The Small Business Regulatory Enforcement Fairness Act requires the FDA to create a program or policy that specifies the circumstances under which the agency may reduce or waive the penalties levied against small business entities for noncompliance. The program must establish the defined circumstances under which alternative penalties can be assessed. This project involves assessing preexisting and revised policies related to monetary penalties and evaluating the scope of the small business penalty-reduction program, the number of enforcement actions against small businesses that qualified (or failed to qualify) for the program, and the total amount of penalty reductions or waivers. The report on this program will be completed by March 1998.

Influenza Virus Vaccines in High-Risk Patient Categories

This study was initiated in FY 1995 and will be ongoing through FY 1998. No reports or publications will be produced from the study. The findings from this project will be used to safeguard public health by investigating the reactogenicity and immunogenicity of influenza virus vaccines. The information derived from this study will also be used to facilitate the scientific review at the FDA's Center for Biologics Evaluation and Research of current influenza virus vaccines used in the United States. Serum samples collected from these studies provide data on antibody responses to the influenza viruses circulating in human populations, indicating whether current vaccines are likely to fail to protect against emerging influenza virus vaccines. This information is critical for making accurate recommendations for the selection of new strains to be used for influenza virus vaccines that are manufactured domestically, guaranteeing the annual availability of serological data to the Public Health Service, data which are unavailable from any other source.

Nationwide Evaluation of X-Ray Trends (NEXT)

Another ongoing domestic evaluation, this survey is sponsored through a cooperative agreement jointly conducted between the FDA and State regulation control agencies. No reports are produced from this study, and this program will continue indefinitely. The study estimates the radiation dosage from diagnostic x-ray examinations, using data collected from previous NEXT surveys of different types of x-ray examinations. Data are available on image quality, photographic processing quality, and types of practice. Public health and professional organizations use this information in setting policy. Some of the products generated in the past have included the National Surveys of Mammographic Facilities conducted during the 1980's and 1990's, a study dealing with poor mammographic quality associated with photographic processing, assessment of skin entrance kerma in the United States, and automatic film-processing analysis of 9 years of data.

New Directions for Evaluation

Changes in government management have created a systemic change in the FDA's evaluation function. The new paradigm--driven by the line manager's performance responsibilities, by the imperatives for involving customers, and by directives for analytically rigorous rulemaking--has replaced the traditional practice of evaluation guided and managed by centralized, specialized evaluation staffs.

Health Care Financing Administration

MISSION: To promote the timely delivery of appropriate, quality health care to the Nation's aged, disabled, and poor through administration of the Medicare and Medicaid programs.

Evaluation Program

The research arm of the Health Care Financing Administration (HCFA), the Office of Research and Demonstrations (ORD), performs and supports research and demonstration projects (through intramural studies, contracts, grants, and waivers) to develop and implement new health care financing policies and to provide information on the impact of HCFA's programs. The scope of ORD's activities embraces all areas of health care: costs, access, quality, service delivery models, and financing approaches. ORD's research responsibilities include evaluations of the ongoing Medicare and Medicaid programs and of demonstration projects testing new health care financing and delivery approaches. These projects address four major themes:

• Monitoring and Evaluating Health System Performance: Access, Quality, Program Efficiency, and Costs--HCFA's research program produces information and descriptive statistics on the infrastructure of the health system, on populations of health care users, and service and expenditure patterns; examines differences in costs, quality, and access to care; and assesses the effects of HCFA programs on beneficiary health status.
• Improving Health Care Financing and Delivery Mechanisms: Current Programs and New Models--HCFA performs research and demonstrations to develop and test new payment and delivery models intended to increase the efficiency and effectiveness of Medicare. ORD's evaluations of these demonstration projects provide policymakers with information about their impact.
• Meeting the Needs of Vulnerable Populations--HCFA's research and evaluation program includes projects that develop new approaches to improve access to cost-efficient and appropriate health services for vulnerable populations.
• Information to Improve Consumer Choice and Health Status--Improving beneficiaries' knowledge and ability to make informed health care choices, both in the health plans they select and in the services they use, is part of HCFA's commitment to improving communication of information to beneficiaries. ORD examines variations in the use of services and treatments and the impact of new information tools and technologies in making health care decisions and improving beneficiaries' health status.

Summary of Fiscal Year 1996 Evaluations

During FY 1996, HCFA completed 10 evaluations:

Evaluation of the Ventilator-Dependent Unit Demonstration looked at the cost of the service for patients who are being weaned from ventilators, which often exceeds the present-day payment system under prospective payment. The project evaluated four demonstration sites that provided care for chronic ventilator-dependent patients. The evaluation comprised three major components: case studies of the demonstration sites, including a comparison of Medicare reimbursement for patient care under the Tax Equity and Fiscal Responsibility Act (TEFRA) compared with reimbursement for the same care under the prospective payment system rules; outcome measures such as comparing the utilization of services, patient health, hospital charges, and Medicare expenditures for individuals admitted to demonstration sites and for patients selected to serve as a control group; and estimation of the effects of implementing a national ventilator-dependent unit program, under TEFRA reimbursement, on utilization and Medicare expenditures.

Evaluation of the Arizona Health Care Cost-Containment System looked at the continuing operation of the Arizona Health Care Cost-Containment System (AHCCCS), with particular emphasis on the implementation and operation of the Arizona Long-Term Care System (ALTCS), a new component of AHCCCS, which began in December1988. AHCCCS is a unique, State-sponsored capitation demonstration that provides public assistance medical care to residents of Arizona who are eligible for Aid to Families with Dependent Children and Supplemental Security Income cash payments. The major research questions included the following:

• Does competitive bidding and selective contracting result in a lower per unit long-term care service cost?
• How effective is the preadmission screening instrument used by ALTCS in identifying individuals who are at risk of being institutionalized?
• Can home- and community-based services be substituted for long-term institutional care for individuals who pass preadmission screening, and are those services less expensive than institutional care?
• Does case management of long-term care services result in lower cost and better coordination of care?
• What are the effects of capitating long-term care services?
• Is the ALTCS more cost-effective than a comparable State's fee-for-service long-term care program?

The results of the quality-of-care analysis indicate that ALTCS nursing home residents are more likely to experience a decubitus ulcer, a fever, or a catheter insertion than nursing home residents covered by NewMexico Medicaid, suggesting a lower quality of care for ALTCS nursing home residents than for those in NewMexico. However, the lack of pre-ALTCS data precluded an analysis of the improvements in quality since ALTCS began. The cost of the ALTCS program during its first 3years was somewhat less than the cost of a traditional program in Arizona (6percent in FY 1990; 13 percent in FY 1991). The AHCCCS acute care program cost also continued to be less costly than of a traditional fee-for-service program.

Assessment of the Impact of Medicaid Drug Rebate Policy on Expenditures, Utilization, and Access used decomposition analysis to determine the change in total drug expenditures before and after implementation of the Medicaid drug rebate program. The role of covered population changes, intensity (utilization rate) changes, changes in efficiency (drug product prices), changes in dispensing fees, changes in rebates, and administrative costs were evaluated. The impact on recipient access was assessed by constructing a person-level file of prescription drug claims, both pre- and post-Omnibus Budget Reconciliation Actof1990 (the legislation that mandated the drug rebate program). During the first two fiscal years the drug rebate amounts accrued were 10.3 percent of the total Medicaid drug expenditures.

Sustainable Support System for Telemedicine Research and Evaluation created an ongoing mechanism by which the cost, effectiveness, and utility of telemedicine services could be systematically evaluated. This was done through formation of a Clinical Telemedicine Cooperative Group, which was modeled after a successful cooperative multicentered research organization. Its functions included (1) providing operational and statistical support for telemedicine research and evaluation; (2) maintaining a communication system to link geographically distant telemedicine projects to share information and perform telemedicine research; (3) creating easily adaptable electronic data collection and tabulation instruments for use in telemedicine research; and (4) building a comprehensive on-line telemedicine information clearinghouse for gathering, storing, and disseminating information about the utility, effectiveness, and suitability of telemedicine for a broad range of medical and social applications.

Evaluation of the Home Health Agency Prospective Payment Demonstration examined the first phase of a program designed to test the effectiveness of using prospective payment methods to reimburse Medicare-certified home health agencies. In this demonstration a per-visit payment method that sets a separate payment rate for each of six types of home health visits (skilled nursing, home health aide, physical therapy, occupational therapy, speech therapy, and medical social services) was tested. The study looked at the effects of this payment method on agency operations, service quality, and expenditures. It also analyzed the relationship between patient characteristics and the cost and utilization of home health services. The findings suggested that demonstration sites had not decreased their cost per visit, had increased their total revenues and net revenues, or had altered their behavior in ways that affect the quality of home health care.

Evaluating the Effects of Physician Payment Reform on Access: Time Series Analyses of Hospitalizations for Ambulatory-Care Sensitive Conditions looked at the effects of physician payment reform (PPR) on access to care in the Medicare population by studying patterns of hospitalization for ambulatory-care sensitive (ASC) conditions. This project analyzed the trend in rates of hospitalization for selected ASC conditions to see whether there is a discontinuity in the time series associated with the implementation of PPR. Analyses were compiled for the trend in hospitalizations for one ASC condition, congestive heart failure (CHF). No significant discontinuity was found in hospitalizations for CHF with the implementation of PPR.

Effects of Predetermined Payment Rates for Home Health Care is a study of the Home Health Per-Visit Prospective Payment Demonstration that pays home health care agencies a prospectively set rate for home health visits, thus providing an incentive to these agencies to control their costs of delivering Medicare home health visits. The study shows that this incentive was largely overwhelmed by the current home health care environment, which is characterized by diversity, change, and competitive pressures. Nonetheless, the opportunity to earn a profit and the increased possibility of losses may have slightly increased the level of attention agencies gave to cutting costs. Prospective rate setting had no discernable effect on the number of visits provided by agencies or on patients' other Medicare costs, quality of care, access to care, or use of services not covered by medicare.

Trends in Access to Health Care Services for Selected Segments of the Medicare Population were developed for the years prior to, during, and after implementation of PPR. The focus was on vulnerable subgroups of the Medicare population, such as persons with low income, persons without supplemental medical insurance, and persons with acute and chronic conditions. Geographic differences also were examined. These trend data were derived from the National Health Interview Survey (NHIS) conducted by the National Center for Health Statistics. The years 1984, 1986, 1989, 1990, and 1991 were used to develop pre-PPR baseline data. The years 1992 and 1993 were used to develop post-PPR data. Pre- and post-PPR data from the NHIS showed that health insurance and health status are both important determinants of the use of physician services.

Assessment of the Impact of Pharmacy Benefit Managers looked at companies that apply managed care principles to prescription drug programs. Their objective is to ensure optimal and cost-effective drug prescribing and use. The project characterized these firms, compared the costs and quality of care (pharmacy benefits) in Medicaid programs versus pharmacy benefit managed care for the privately insured and for Medicaid enrollees in managed care. The project found that these firms did provide both administrative functions and drug use control, that they were dispersed throughout the country and covered substantial numbers of beneficiaries, that this aspect of the health care industry is complex and rapidly changing, and that they offer various programs (e.g., Medicaid, extensive provider networks, favorable market reimbursement rates for pharmacies, and sophisticated claims processing and data management systems).

The Evaluation of Medicare SELECT Amendments looked at a pilot Medicare supplemental insurance product under which full Medigap benefits are paid only when services are provided by the plan's provider network. Case studies were conducted. The analytical portion of the project compared cost and use of Medicare and supplemental services, selection effects, beneficiary satisfaction, and physician practice patterns with other Medigap options.

Evaluations in Progress

ORD currently supports 32 evaluation projects in progress, 24 of which are scheduled to be completed in FY 1997. These projects provide information for continued monitoring of the Medicare and Medicaid programs and assess the impacts of HCFA's Medicare and Medicaid demonstration projects.

Medicare Projects

Results from the following evaluations are expected in FY 1997.

The Evaluation of Medicare SELECT looks at a pilot Medicare supplemental insurance product under which full Medigap benefits are paid only when services are provided by the plan's provider network. Case studies are being conducted. The analytical portion of the project will compare cost and use of Medicare and supplemental services, selection effects, beneficiary satisfaction, and physician practice patterns with other Medigap options.

The Evaluation of the Medicare Case Management Demonstrations studies the appropriateness of providing case management services to beneficiaries with catastrophic illnesses and high medical costs. It will test case management as a way of controlling costs in the fee-for-service sector.

The Medicare Participating Heart Bypass Center Demonstration Extended Evaluation continues to study the feasibility of a negotiated all-inclusive pricing arrangement for coronary artery bypass graft surgery while maintaining high quality care. The project will look for any net cost savings to the Medicare program, any volume increases at the demonstration sites, the aspects of the demonstration that were attractive to beneficiaries and referring physicians, and whether the quality of care at the sites was equivalent to that provided prior to the demonstration.

Monitoring and Evaluation of the Medicare Cataract Surgery Alternate Payment Demonstration assists HCFA in tracking this set of demonstrations, which test the feasibility of an all-inclusive negotiated (bundled) price for cataract surgery. The price covers the physician, facility, and intraocular lens costs. The analysis portion of the project tests whether there were any net savings to the Medicare program, changes in the use of services included and excluded from the bundle, beneficiary satisfaction, and quality of care.

In the Impact of the Medicare Fee Schedule on Access to Physician Services, HCFA is evaluating the effect of the physician fee schedule on the beneficiary's access to care. Six different strata of beneficiaries are selected based on the relative size of the payment change under the fee schedule compared to the earlier payment scheme. The project is examining use of services, outcomes of services, and the change in the beneficiary's financial liability.

The Evaluation and Technical Assistance of the Medicare Alzheimer's Disease Demonstration assists with HCFA's projects that provide comprehensive in-home and community-based services to beneficiaries who have dementia. Two different models of care are involved in the demonstration, differing by the intensity of the case management and the amount of service costs covered each month. The analytical portion of the contract is attempting to identify the factors associated with cost-effectiveness, the services that appear to affect the health status and functioning of the patients, the effects on the caregiver (burden and stress), and whether the provision of the additional home care services delay or prevent institutionalization.

In later years, the following Medicare projects are expected to be completed.

The Evaluation of HMO Outlier Demonstration that examining the Outlier Pool Demonstration that is under way in the Seattle area. Participating plans are paid 97 percent of the adjusted average per capita cost, with 2 percent of the payments going into a pool. Plans with a higher than average incidence of high-cost cases will receive more from the pool than they paid in, and those with a lower incidence will receive less.

The Evaluation of the Medicare Choice Demonstration assesses the feasibility and desirability of new types of managed care plans for Medicare. These plans can be integrated delivery systems and preferred provider organizations.

The Medicare HMO Evaluation updates the findings of an earlier study of Medicare risk health maintenance organizations. That study found that 5.7 percent more was paid for plan enrollees than would have been spent under fee-for-service. The current study looks at disenrollment, beneficiary satisfaction, quality of care, and selection and savings.

HCFA is experimenting with paying skilled nursing facilities on a prospective basis. Currently, such facilities are reimbursed on a retrospective cost basis. This demonstration uses a case-mix classification, called Resource Utilization Groups, to classify patients. The Evaluation of the Nursing Home Case-Mix and Quality Demonstration seeks to estimate specific behavioral responses to the prospective payment and to test hypotheses about aspects of such responses. The main goal of the project is to estimate the effects on the health and functioning of the nursing home residents, their length of stay, and use of health care services; on the behavior of the facilities; and on the level and composition of Medicare expenditures.

Medicaid Projects

The following Medicaid-related evaluations are scheduled for completion in FY 1997.

The State of Delaware is experimenting with its Medicaid program under the assumption that by enrolling children into a managed-care system, they will reap the benefits of a higher level of coordinated care while benefiting from lowered costs. The Evaluation of the Demonstration Entitled Delaware Health Care Partnership for Children is examining this real-time test of the hypothesis.

The Evaluation of Medicaid-Managed Care Programs With 1915(b) Waivers will provide information on the extent to which various features of the waiver projects contribute to the ability of the Medicaid program to deliver cost-effective care to eligible populations.

The Evaluation of the Utah Prepaid Mental Health Plan: Coordinated Care Systems as Alternatives to Traditional Fee for Service looks at Utah's project which has three mental health centers providing mental health services to all Medicaid beneficiaries in their catchment areas (these areas include over 50 percent of all Utah Medicaid beneficiaries). The State hopes this program will control the cost inflation and improve patient outcomes.

The Evaluation of the Iowa Implementation of Ambulatory Patient Groups (APG's) studies a Medicaid outpatient prospective payment system that groups patients for payment purposes rather than paying on a cost basis. It involves a case study on Iowa's implementation of the APG system and an analysis of the reimbursement methodology.

The Comparative Study of the Use of Early and Periodic Screening, Detection, and Treatment and Other Preventive and Curative Health Care Services by Children Enrolled in Medicaid is examining the effects of the 1989 changes to this portion of the program. It is looking at the process of providing health services and the appropriateness of expenditures for services in four States. It compares Medicaid children with other Medicaid-enrolled children in the same State who are not receiving these services, with emphasis on preventive services. Using national survey data, it also compares Medicaid-enrolled children with non-Medicaid-enrolled children, insured and uninsured, on the use of and expenditure for preventive and other health services.

The Evaluation of the Demonstration for Improving Access to Care for Pregnant Substance Abusers is assessing the effectiveness of projects that improve outreach and assessment; expand, integrate and coordinate program services; and improve client case management. The evaluator will look at access to prenatal care, substance abuse treatment services, and other relevant services. It will assess the effects of services on the health of the drug-addicted pregnant women, any prevention of reduction or short-term impairments to their infants, and the impact on birth outcomes.

The Evaluation of the Medicaid Uninsured Demonstrations covers projects in Maine, South Carolina, and Washington State. They test the effects of extending Medicaid coverage to low-income families. The evaluator will look at the ability of the programs to enroll significant numbers of eligible persons, the conditions under which these persons are willing to participate, the program's ability to induce adequate numbers of providers to participate, the effect on service utilization and health outcomes, their cost-effectiveness, and the extent to which these demonstrations' interventions could be applied nationally.

The Examination of the Medicaid Expansions for Children will use enrollment and expenditure trends to ascertain the impact of the 1989 program changes. It will look the penetration of the target population, and the impact of State policies (and the eligibility group) on enrollment, expenditures, and utilization of services.

The Department is required to report to Congress on the relative quality of care in the Medicaid program. HCFA's Medicaid Quality of Care Study examines the necessity, appropriateness, and effectiveness of selected medical treatments and surgical procedures for Medicaid patients. It is assessing the variation in the rate of performance of selected treatments and procedures on Medicaid beneficiaries for small areas within and among States. It is determining underutilized, medically necessary treatments and procedures for which failure to furnish them could have an adverse effect on their health status.

The Community-Supported Living Arrangements Program: Process Evaluation is designed to test the effectiveness of developing a continuum of care concept as an alternative to the Medicaid-funded residential services provided to individuals with mental retardation and related conditions. The program serves individuals who are living in the community--either independently, with their families, or in homes with three or fewer other individuals receiving the same services.

The Project Demonstrating and Evaluating Alternative Methods to Assure and Enhance the Quality of Long-Term Care Services for Persons With Developmental Disabilities Through Performance-Based Contracts With Service Providers tries to determine whether and how well the implementation of new approaches to quality assurance, with outcome-based definitions and measures of quality, will replace input and process measures of quality in this population group.

In addition, HCFA has other major Medicaid evaluations whose results are anticipated in several years.

HCFA is sponsoring a wide variety of waiver-based demonstrations that give States the opportunity to experiment with their Medicaid programs. As these demonstrations are begun, HCFA undertakes examinations of their impact. The Evaluation of the Oregon Medicaid Demonstration, the Evaluation of the State Medicaid Reform Demonstrations (in Hawaii, Rhode Island, and Tennessee), and the Evaluation of the State Medicaid Reform Demonstrations (in Ohio and Minnesota), are three such projects. More will be started as more State demonstrations are begun.

The Drug Utilization Review Evaluation is looking at the impacts of retrospective and prospective review, which include the payment of pharmacists for cognitive services. Data from demonstrations in Iowa and Washington State and information from programs in other States serve as the basis for this study. Maryland and Georgia will serve as coexperimental and comparison States.

Other Health Financing Policy and Demonstration Evaluation Projects

In 1977 Congress authorized a new type of provider--the rural health clinic. A rural health clinic must be located in a rural health professional shortage area, medically underserved area, or governor-designated shortage area, and must make use of mid-level practitioners. Rural health clinics are reimbursed on a cost-basis by the Medicare and Medicaid programs. Their numbers have been growing at a relatively rapid rate. The Evaluation of Rural Health Clinics examines the reasons for this growth, the impact on access to care for these rural populations, and the costs to the Federal Government and the States.

Evaluation of the Essential Access Community Hospital/Rural Primary Care Hospital Program examines the development, implementation, and early operating experiences of this program. Known as EACH/RPCH, the program is supposed to assist States in maintaining access to health care services in rural areas. This is done through the development of rural health plans, establishment of rural health networks, and creation of a limited service alternative for communities that can no longer support a full-service hospital.

The Evaluation of the Community Nursing Organization Demonstration looks at a set of projects mandated in 1987. The legislation directs projects in four or more sites to test a capitated, nurse-managed system of care. The two fundamental elements of these demonstrations are capitation payment and nurse case management. They are designed to promote timely and appropriate use of community health services and to reduce the use of costly acute-care services.

The Evaluation of the Program of All-Inclusive Care for the Elderly Demonstration, known as PACE, examines the replicates of a unique model of managed care service delivery for very frail community-dwelling elderly, most of whom are dually eligible for Medicare and Medicaid coverage and all of whom are assessed as being eligible for nursing home placement. The core services include adult day health care and multidisciplinary team care management through which all health and long-term care services are arranged. The evaluator is looking at the demonstration sites before and after assumption of financial risk to see if the replicates are cost effective relative to the current Medicare and Medicaid services. They are also looking at the decision to enroll to understand how PACE enrollees differ from the eligible beneficiaries who choose not to enroll.

The Inspector General of the Department is conducting a demonstration of improved methods for investigating and prosecuting fraud and abuse. The Evaluation of the Effectiveness of the Operation Restore Trust Demonstrations will determine whether the more concentrated effort rendered through the demonstration's partnership model has a relatively greater impact on industry fraudulent behavior.

The Evaluation of the Impact of Health Plan Report Cards on Consumer Knowledge, Attitudes, and Choice in a Managed Competition Setting seeks to determine whether the dissemination of information about health plans to consumers (who choose health plans within a managed-care competition framework) will influence their knowledge of plan characteristics, attitudes toward the plans, or choice of plan.

New Directions for Evaluation

As the U.S. health care system continuously changes, there is a clear need for developing, designing, and testing new ways to monitor and evaluate its performance. It is important that monitoring and evaluation efforts for the Medicare and Medicaid programs include a number of critical dimensions (access to care, quality, efficiency, costs, and beneficiary satisfaction) to provide an understanding, on an ongoing basis, of how well the programs are performing. ORD is working to develop a comprehensive monitoring and evaluation plan for systematically examining the Medicare and Medicaid programs.

HCFA also will continue to develop a wider array of evaluation and measurement tools. The agency's evaluation activities will continue to examine specific policy issues within the HCFA programs. For example, as Medicare and Medicaid continue to pursue managed care options, ongoing work will examine the cost-effectiveness of, quality of, and beneficiary satisfaction with managed care. HCFA plans to carry out projects to monitor and compare the health status or health risk behaviors of beneficiaries in various delivery systems and how these change over time.

Finally, as HCFA develops and implements new high priority demonstrations that will test new payment and health care delivery models for the future, the agency will evaluate these programs and provide information to policymakers about the impacts of these alternatives.

Health Resources and Services Administration

MISSION: Improve the health of the Nation by assuring quality health care to underserved, vulnerable, and special-need populations and by promoting appropriate health professions workforce capacity and practice, particularly in primary care and public health.

Evaluation Program

The purposes of the Health Resources and Services Administration (HRSA) evaluation program are to enhance strategic planning, budget decisions, legislative planning, and program management. Consequently, major emphases during FY 1997 and beyond will be (1) performance measurement, (2) assessment of program implementation, and (3) cross-cutting policy analysis and research.

Performance measurement includes technical assistance and training activities to strengthen the agency's capacity to assess program performance, as well as studies to assess program outcomes. These activities, mandated by the Government Performance and Results Act (GPRA), will provide data for ongoing program monitoring and evaluation and for developing annual performance plans and budgets. A study completed in September 1995 established a performance measurement baseline for all operating programs, with the ultimate objective of ensuring that the HRSA specifies valid and useful indicators and measures for all programs by September 1998. Over the past two years, the HRSA has made significant progress in performance management, such as strengthening linkages among strategic planning, program activity, and the budget process.

Assessment of program implementation includes a broad range of efforts to assist the agency during a time of new directions from the Administration and Congress. HRSA programs are entering into new arrangements for delivering services, providing health professions education, and encouraging the development of systems reform within seven Program Priority areas. These priorities reflect the HRSA's focus on the communities where underserved populations live, and include academic and community partnerships to foster clinical training in community-based settings; new arrangements to bring poor, uninsured, rural, and chronically ill persons into the mainstream of managed care; assistance to communities in strengthening their health care infrastructure; and new activities with States. All agency studies relate to one or more of these Priorities, which also include improved comprehensiveness and integration of HIV/AIDS programs; school-based care for children and adolescents; and activities to enhance the health of people living along the U.S.-Mexican border.

Cross-cutting policy analysis and research includes efforts to build capacity and conduct studies to clarify the environmental shifts within which HRSA programs operate and to improve the HRSA's ability to document performance and impact through better measurement tools and data sources. Developing this capacity at the agency level and initiating some studies are special priorities for FY 1997; work in later years will build on these beginnings. "Cross-cutting" implies policy issues that bear on the programs of two or more HRSA Bureaus, such as trends in numbers and characteristics of the uninsured and the health of the safety net.

The main purpose of the HRSA's evaluation program is to provide accurate and relevant information on a timely basis to the administrator and other senior line managers. In working toward this objective, HRSA subjects study proposals to a formal review process, which begins with consideration by a committee comprised of senior line officials and is chaired by the Director of the Office of Planning, Evaluation and Legislation (OPEL). This committee's task is to assess the relevance of the proposals to important policy, budgetary, or legislative issues; its potential to answer questions about program effectiveness or impact; its degree of attention to cross-cutting topics; and its relative priority for funding. Following administrator decisions on approval of the recommended studies, a committee of senior analysts from the HRSA, the Agency for Health Care Policy and Research, the National Center for Health Statistics, and the Office of the Assistant Secretary for Planning and Evaluation (ASPE) reviews scopes of work and other substantive portions of documents needed for development of contracts. The role of this technical review group is to suggest ways to strengthen study design or data sources.

HRSA is continuing to place major emphasis on disseminating findings and information about use of study results. This is done through inclusion of abstracts in the ASPE evaluation data base (on the Internet), submission of articles to peer-reviewed professional journals, presentations by HRSA staff at professional conferences, and placement of final reports with the National Technical Information Service. In addition, the HRSA prepares annual volumes of summaries of studies begun or completed the preceding fiscal year. These are distributed within the department and their availability is noted on the HRSA OPEL web page at http://www.hrsa.dhhs.gov/oa.html#opel.

Summary of Fiscal Year 1996 Evaluations

During FY 1996, HRSA completed evaluations in the following areas.

Performance Measurement

Projects to build on the baseline study on performance measurement noted above were a major emphasis during the past year; they included technical assistance tailored to the needs of each of the four Bureaus. Illustrative of this work was the Evaluation of the Bureau of Health Professions Strategic Directions: Phase II, which developed a set of outcome-based performance measures and a plan for a performance-monitoring system to support ongoing program management and compliance with GPRA requirements. This study is featured in chapter II.

Relevant to HRSA's continued efforts to identify appropriate health outcomes indicators, the agency conducted a Consensus Conference on Health Status Gaps of Low-Income and Minority Populations. The report recommends potential measures and data sources for eight conditions, sets of conditions, or preventive approaches: diabetes; hypertension, cardiovascular, and cerebrovascular conditions; breast cancer; cervical cancer; infectious diseases, including AIDS, sexually transmitted diseases, and tuberculosis; immunization; asthma; and pregnancy outcomes.

Another HRSA effort developed quality assurance procedures for the Uniform Reporting System for Titles I and II of the Ryan White Comprehensive AIDS Resources Emergency (CARE) Act, which provide grants to disproportionately affected metropolitan areas and States. The purpose of the funding is to improve the availability and coordination of services for persons with HIV/AIDS. Under the Uniform Reporting System, grantees submit provider-level administrative data and data on client demographics and services received. Uniform data are used to assess the extent to which grantees are achieving the goals of the Act and to help metropolitan planning councils, State agencies, and State consortia target and monitor the provision of services to specific population groups. Through this study, data quality targets were established and manuals were written to help grantees compile high-quality data. HRSA and grantee staff are using the results to improve the quality of data submissions, which will support national and local program monitoring and evaluation.

Managed Care

Managed care will continue to have a major influence on HRSA programs. This topic was looked at in two studies: An Evaluation of the Impact of the Social Security Act Section 1115(a) Waivers on Federally Qualified Health Centers was designed to investigate how federally qualified health centers (FQHC's) have fared in Hawaii and Oregon, both of which have waivers involving placement of Medicaid eligibles in managed care organizations. The study found that the waiver programs expanded access to primary care and dental services, and that FQHC's continued to provide types and levels of services not included in the capitation rates, as well as to serve patients not covered under the waiver program. Study results are being used by State and Federal policymakers concerned with Medicaid managed care programs and have been disseminated to managed care organizations that are part of Medicaid managed care programs.

Additionally, tools were developed for monitoring the cultural competence of primary care providers practicing in managed care systems. The work was carried out through physician and patient panels in two California HMO's serving predominantly Latino or Chinese populations, and through a literature review. "Cultural competence" was defined as the level of knowledge-based skills required to provide effective clinical care to patients from a particular ethnic or racial group. A related concept, "cultural sensitivity," was defined as the psychological propensity to adjust one's practice style to the needs of different ethnic or racial groups. Study products include a patient satisfaction questionnaire, which a managed care plan could use to assess the degree to which enrollees with limited English believe their care is culturally competent; a provider self-assessment questionnaire; and behavioral ethnic identifiers that can be used in enrollment processes as a cue for designing culturally appropriate services.

Primary Care

Primary Care: America's Health in a New Era was a major undertaking carried out by the Institute of Medicine to define and describe the place of primary care in the changing health care environment and to develop a set of normative goals for primary care. Findings focus on interdisciplinary primary care teams, roles of the different providers that make up the teams, and ways in which team members could interact with each other and with providers outside the team. The study report is being used to inform the health care community about primary care issues for the next decade.

Assessing the Impact of Public Spending on the Health of Vulnerable Populations: A Framework for Evaluating HRSA Programs provides a structure for describing the linkages among HRSA dollars and HRSA-funded services, HRSA-enabled resource- and infrastructure-building programs, and the populations affected in terms of health outcomes. The analytic approaches developed here could be used by the HRSA or other operating divisions to explain the difficulties in establishing causal factors, to highlight areas with the greatest need for surveillance and monitoring, and to develop data systems to provide the needed information.

A study entitled Impact of Case Management on Health Status in Community and Migrant Health Centers (CMHC's) examined case management practices and their impact on health status of participants in the Comprehensive Perinatal Care Program (CPCP). The study consisted of three components: a control-site study of paired CMHC's, intensively using and not intensively using case management; a model-site study of seven CMHC's serving high-risk perinatal populations; and a two-part mail and telephone survey of a sample of centers receiving CPCP funding. The study found that case management had a positive impact on the continuity and adequacy of care, on the content of care in some areas (psychological risk assessment and health promotion), and on users' perceptions of care. The HRSA is using the findings to integrate case management in all primary care projects serving perinatal patients. The study also developed a self-assessment tool that grantees can use to evaluate their case management activities.

Finally, Technical Support for Need Designation and State Primary Care Access Plan Initiatives assessed the process for designating areas of primary health care shortage. Results are being used in finalizing new designation criteria for shortage, underservice, and access barrier areas.

Care for Mothers and Children

The Emergency Medical Services for Children (EMSC) program was begun in 1984 to help States develop and institutionalize emergency medical services for critically ill and injured children. This past year, HRSA completed a study to assess the extent to which seven States had sustained services funded with an EMSC grant, to identify attributes associated with successful institutionalization of EMSC in State emergency medical services systems, to characterize barriers to institutionalization, and to recommend approaches for Federal actions to improve EMSC systems. Recommendations, which include three areas (strategic planning and funding; grant guidance, including an increased emphasis on evaluation; and program development and intergovernmental relations), will be used by program managers in setting future directions and goals.

Health Professions

One project in the health workforce area developed the Report to Congress on the Process by Which International Medical Graduates Are Licensed To Practice in the United States. This study assessed the time required to verify the credentials of physicians applying for licenses and differences in the application process for international versus U.S. medical graduates. The study found that there is a time differential between these groups of graduates and that a national credentials verification system is needed to assist State medical boards. The study also recommended that uniformity among States in licensure requirements be expanded to a maximum degree.

Another study assessed the feasibility of increasing the production of advanced practice nurses by investigating the potential for nursing educational institutions to expand their capacity. "Advanced practice nurses" include nurse practitioners, clinical nurse specialists, nurse anesthetists, and certified nurse midwives. The study found that the single most significant factor in determining an institution's ability to expand capacity is the availability of sites for clinical education, although the availability of clinical faculty is also important. HRSA is using the results to guide future directions for advanced practice nursing programs and to provide technical assistance to grantees. The report has also been shared with schools of nursing with advanced practice programs.

A third health professions study was an exploratory effort to describe arrangements between primary care residency programs and ambulatory training sites affiliated with them. The most important findings are that (1) residency programs do not significantly offset the costs of ambulatory training experienced by the sites nor are there many written agreements in place; (2) residency programs are not aggressively seeking managed care training sites; and (3) cost pressures on ambulatory training sites may make decisionmakers less willing to assume or continue to absorb the cost of training. HRSA will use these findings, in conjunction with findings from two other studies on site-specific costs of ambulatory training, to expand understanding of the costs of ambulatory education and help HCFA in its deliberation on new approaches to GME funding.

Evaluations in Progress

During FY 1996, HRSA had evaluations under way in the following areas.

Performance Measurement

A study entitled Health Care Status Outcome Measures for the Bureau of Primary Health Care: The Assessment of Ambulatory Care Sensitive Conditions Through State Medicaid Recipient Files is comparing the relative risk of inpatient hospitalizations of community health center users with that of nonusers for ambulatory care-sensitive conditions. This project builds on the Consensus Conference noted above. The third study identified below under Managed Care, Meeting Industry Standards for Performance Measurement Through Clinical Outcome Measures at Community and Migrant Health Centers, also applies here.

Managed Care

A number of studies will continue several of the major themes and initiatives. For example, Evaluation of the Response to Medicaid Managed Care Initiatives by Federally Qualified Health Centers will describe strategies that FQHC's are using to incorporate Medicaid managed care into their practices and assess the impact of these strategies on FQHC organization, delivery of services, and market position. Findings will guide policy for CMHC's. Building on a study completed in FY 1995, the Performance Assessment of Managed Care in Community and Migrant Health Centers will develop case studies in seven centers. Center performance will be compared with that of other providers in the same HMO network on selected industry performance indicators.

Work will also continue to examine the extent to which CMHC's are using clinical outcome measures to assess quality of health care, to meet center needs in a managed care environment, and to improve and monitor the quality of care locally and nationally. Findings will be used to identify performance issues needing further research and development and to facilitate collaboration among private and governmental organizations focused on health outcome measures and underserved populations.

Two studies are examining staffing issues in a managed care context: HMO Staffing Strategies in Underserved Areas is describing strategies to ensure access to services for enrollees in underserved areas, as well as the implications of these strategies for overall access in these areas. The study involves interviews and other data collection at nine HMO's with significant Medicaid enrollment. Selected Aspects of Education in Managed Care is surveying managed care organizations to identify their practices and determine the potential for these organizations to provide high-quality learning experiences for medical students and residents.

Primary Care

The Community Health Center User and Visit Survey, which involves personal interviews of users at a representative national sample of centers and a review of patient records at the same centers, continues from FY 1994. One of the purposes is to gather data on characteristics of users and types of services they received on a basis that will permit comparison with the results of the National Health Interview Survey. A related study begun during FY 1996 is expanding the analysis of the User and Visit Survey data to compare managed care users with users of centers that are not part of managed care arrangements.

Casemix Differences in Health Centers and Other Providers and Their Relationship to Cost will recommend a method for measuring casemix (severity of illness) in ambulatory populations; develop a methodology for creating "casemix profiles" of patient characteristics and expenditure data; and test the methodology with Medicaid data from several States. This study should also provide insights into the relative importance of casemix compared with other factors that may affect expenditures for primary care in the ambulatory setting.

HRSA is continuing to work with the American Hospital Association Hospital Research and Educational Trust (HRET) on evaluation of the HRET-sponsored Community Care Network, a series of demonstrations in 25 sites across the United states. HRSA support for this collaboration was begun through a project completed last year, National Demonstration of the Community Care Network Vision: Development of a Program Evaluation. HRSA will use the findings from the evaluation as a guide for selecting model types or community characteristics that deserve detailed exploration to form hypotheses about attributes that may lead to the successful launching of collaborative activities and to assess implications for cost-effective approaches to measurement and data collection in future evaluations of HRSA programs.

Another study concerns the ability of HRSA-supported community-based organizations to integrate services for individuals with multiple risk factors, such as HIV/AIDS, substance abuse, and homelessness. This project has explored the use of multiple funding sources to provide integrated care, barriers to integration of services supported by separate funding sources, successful integration strategies, and ways in which HRSA might reduce barriers created through multiple funding streams. The results will inform policy on primary care and AIDS programs and will be shared with other operating divisions.

Attention to bilingual and multilingual services is also continuing through studies directed to services at CMHC's, such as the project, An Assessment of Bi/Multicultural Services Offered at Community and Migrant Health Centers. Bilingual Assistance Program: Evaluation of Strategies for Reducing Cultural and Linguistic Barriers to Health Care for Hispanic and Asian Pacific Islander Populations concerns bilingual services in the context of small grants to State and local health departments.

Health Care for Mothers and Children

The National Evaluation of the Healthy Start Program continues as a 5-year longitudinal study of the development, implementation, and outcomes of the comprehensive perinatal delivery systems at the initial 15 Healthy Start sites. This project includes an assessment of changes in the health status of pregnant women and infants across the sites. The study is designed to answer four questions: Did the Healthy Start initiative succeed? If so, why? If not, why not? What would be required for a similar intervention to succeed in other settings?

HIV/AIDS

A major priority of the HRSA is to improve access to a comprehensive continuum of care for persons with HIV/AIDS, through the Ryan White CARE Act programs. One ongoing study concerns the management of people infected with both AIDS and TB in metropolitan areas with Title I grants to identify models of successful collaboration between HIV/AIDS and TB control programs. The project is also determining the extent to which specific Title I grantees are providing guidance to funded providers on TB screening and prophylactic and therapeutic regimes. Another study is reviewing and synthesizing HIV-related consumer-level evaluations conducted by CARE Act grantees across the country. The intent is to identify useful and replicable models of studies for use in providing technical assistance to grantees, and to identify topics on which further studies are needed.

Organ Transplantation

One of the HRSA's roles is to oversee the national system for distributing organs for transplantation. An ongoing study is examining the reasons why African Americans remain on waiting lists longer than white patients. Anecdotal evidence indicates that African Americans are more difficult to contact within allowable time periods than whites and that African Americans are more likely to decline an organ offer. Studies on this subject have been inconclusive, however. Findings will be used to assess whether changes are needed in HRSA policy concerning the transplantation network.

Telemedicine

Another significant policy area involving ongoing work is the rural applications of telemedicine. In FY 1994, the HRSA began a baseline project to determine the current status of telemedicine in rural health; to explore the effects of telemedicine on access to care, practitioner isolation, and development of health care networks; to identify organizational factors that aid or impede the successful implementation of telemedicine systems; and to develop and refine data collection instruments for use in subsequent evaluations.

Health Professions

The Sixth National Sample of Registered Nurses will provide new data on the characteristics of the registered nurse workforce and will contribute to the data base underlying models for projecting national nurse supply and requirements. Data from the survey will be used by the HRSA to guide nurse education programs and to prepare biennial reports to Congress on characteristics of the registered nurse population. Results will also be used by the nursing community, which looks to the Department to maintain and update this unique data base.

The cost and location of training of physicians and others to provide primary care in ambulatory settings is a continuing concern. One project is obtaining quantitative data on the costs involved in training medical students and residents in over 30 ambulatory sites across the country.

A collaborative study with the Educational Commission on Foreign Medical Graduates and the Appalachian Regional Commission is tracking exchange students granted a J-1 visa to participate in graduate medical education in the United States. The HRSA portion of the project will examine the postresidency experience of exchange visitors who have secured waivers to remain in the United States following completion of training for the purpose of providing services in an underserved area.

New Directions for Evaluation

As in FY 1996, the HRSA will emphasize performance measurement, managed care, and such primary care programs as CMHC's. Care for mothers and children (exemplified by Healthy Start initiative), HIV/AIDS (including new approaches to financing and delivering care tested through the Special Projects of Regional and National Significance), and strengthening of the health care infrastructure at State and local levels will also be major areas of attention. Ongoing and new efforts that involve collaboration with other operating divisions and with such external entities as foundations and the American Hospital Association will also be continued or developed. Finally, the HRSA will ensure the further integration of evaluation with strategic planning and budget decisions and presentation, as called for in GPRA.

Indian Health Service

MISSION: To provide a quality health services delivery system for American Indians and Alaska Natives with opportunity for maximum tribal involvement in developing and managing programs to meet health needs.

Evaluation Program

The goal of the Indian Health Service (IHS) is to raise the health status of its principal beneficiary, American Indians and Alaska Natives, to the highest level possible. The importance of evaluation in supporting this goal has increased significantly in recent years. The IHS evaluation process seeks to include American Indians and Alaska Natives as primary stakeholders in defining the purpose, design, and execution of evaluations. Stakeholders are the users of the end product of evaluations and typically are the population or groups most likely to be affected by the evaluation findings. The principles of responsive evaluation practice have been adopted by the IHS because they can best address these needs and concerns of American Indians and Alaska Natives.

The purpose of the Office of Planning, Evaluation and Legislation (OPEL) is to advise the Director of the IHS on policy formulation; to conduct and manage program planning, operations research, program evaluation, health services researches, legislative affairs, and program statistics; to develop the long-range program and financial plan for the IHS in collaboration with appropriate agency staff; to coordinate with HHS, Indian Tribes, and organizations on matters that involve planning, evaluation, research and legislation; and to develop and implement long-range goals, objectives, and priorities for all activities related to resource requirements and allocation methodologies and models. OPEL serves as the principal advisory office to the IHS on issues of national health policy and coordinates these four evaluation functions.

• Health Program Evaluations--Collect and analyze information useful for assisting IHS officials in determining the need for improving existing programs or creating new programs to address health needs.
• Policy Analysis--Conduct analyses when a change in the IHS health service delivery system must be considered, when issues emerge in an area where no policy currently exists, or when current policies are perceived as inappropriate or ineffective.
• Health Services Research--Undertake analyses of the organization, financing, administration, effects, and other aspects of the IHS.
• Special Studies and Initiatives--Conduct studies and prepare special reports required by Congress in response to pending legislation or policies, often using a roundtable whenever an issue or a health problem requires immediate action and it is unclear what type of action should be taken.

OPEL meets part of the IHS evaluation needs with two major types of short-term studies: policy or program assessments and evaluation study. The policy study contributes to IHS decision making about budget, legislation, and program modifications and includes background information to support IHS initiatives. Evaluation studies are carried out at the program level, or area offices, and focus on specific program needs.

Annually, OPEL identifies the high-priority health care and health management issues and concerns through the submission of headquarters and area office proposals for assessment or evaluation. IHS area and associate directors submit proposals for possible areas of evaluation study. These proposals are reviewed and rated by a panel of subject-matter experts and evaluation experts and also reviewed by IHS staff for more specific concurrence with IHS strategic goals, objectives, and priority areas. The proposals are then ranked by priority and forwarded to the OPEL Associate Director for review and approval. The Director of the IHS reviews the final proposals and decides the respective funding levels.

Summary of Fiscal Year 1996 Evaluations

During FY 1996, OPEL completed three evaluation projects of several major Indian health topics.

Methodology for Adjusting IHS Mortality Data for Inconsistent Classification of Race-Ethnicity of American Indian and Alaska Natives Between State Death Certificates and IHS Patient Registration Records

The study findings indicate a large variation in the number of deaths reported from 1986 to 1988 across IHS area offices and extensive variation in the number of deaths reported in the matched IHS-National Death Index data across States of occurrence. There is also a great variation in the percentage of consistently reported American Indian and Alaska Native race identification on State death records and IHS patient registration records by State of a resident. Recommendations include replicating the study using data on deaths occurring since 1988, using the adjustment factors developed in the study, and working with States to decrease inconsistent race reporting.

Evaluation of the Indian Health Service (IHS) Adolescent Regional Treatment Centers

The principal conclusion based on this study's findings is that regional treatment centers have developed effective adolescent alcohol and substance abuse programs. The continuity of care and aftercare, however, is the biggest problem. The regional treatment centers need additional mental health staff resources, client charting improvements, and innovative ways to increase family involvement. Recommendations include improving the continuum of care to adolescent substance abusers, self-evaluation, and regional treatment center effectiveness and efficiency.

Evaluating the Effectiveness of Alcohol and Substance Abuse Services for Native American and Alaska Native Women: Phase II Final Report

This evaluation provides both qualitative and quantitative information about a group of women that has been traditionally underrepresented in research. The life conditions of women about whom information was gathered are extreme, and for many women, adverse or abusive childhood experiences and conditions have carried through to adulthood. The vast majority of women were exposed to various types of abuses--such as physical, sexual, and emotional abuse--from childhood to adulthood. Women entered treatment through a variety of ways. Those who were mandated tended to enter treatment as an alternative to incarceration. Women hear about the availability of services through the court system, word-of-mouth, or through a community or an American Indian and Alaska Native social service agency. Women in the focus groups tended to select their current alcohol and other drug treatment program over alternatives because of its focus on American Indian and Alaska Native tradition and culture. The women and staff also espoused the benefits of the family-like environment that the treatment centers promoted. The availability of women-centered, family-focused approaches to alcohol and other drug treatment is severely limited in the United States. Several barriers to services for potential participants exist. The leading obstacle for parenting women is the lack of child care for their children while in treatment. It was strongly emphasized that a woman's recovery was dependent on three key factors: herself, her social networks, and her community. One of the strongest impressions that has been derived from this evaluation is that the women who have participated in alcohol and other drug treatment programs are, overall, a cadre of strong women.

Evaluations in Progress

During FY 1996, the IHS funded more than ten projects, including health program evaluations, policy analysis projects, health services research, and special studies and initiatives. Examples of evaluations in progress follow.

Elderly Wellness

This study will focus on collecting data to determine if the provision of health care and health promotion services specially designed for the elderly will increase their use of these services and ultimately improve their health status. Comprehensive care remains an unmet need for the elderly. Recent reviews show that less than 50 percent of the elderly population residing in the Fort Peck Service Unit (Billings, Montana, area office) has had a health care visit of any kind in the last two fiscal years.

Prior Trauma Care of Intoxicated Patients as a Predictor of Subsequently Fatal Injury

In FY 1996, the IHS funded the first part of this study to begin the preliminary data collection, crude data reporting, and initial death certificate-hospital record linkage. The purpose of this study is to identify intervention opportunities associated with nonfatal, alcohol-related injuries reported in IHS emergency departments and clinics that could, over time, decrease alcohol-related injury death in the Billings, Montana, Service Units. This study will provide baseline data for postintervention comparisons by expanding the existing data base about alcohol-related injuries and death. The study will identify different intervention and prevention strategies directed at decreasing alcohol-related injuries and deaths in the Billings, Montana, Service Units. Injury-control efforts will include a new policy regarding referrals by emergency room treatment staff to alcohol treatment staff. Prevention of alcohol-related injuries and deaths will also include activities focused on informing youth about the relationship between alcohol consumption and high-risk behavior.

Resource Requirements Methodology Update

In the early 1970's, the IHS formulated the Resource Requirements Methodology (RRM) as a management tool to provide a comprehensive, systematic, and uniform process for estimating the level of resource requirements necessary to provide adequate health care to IHS customers and to assist in the allocation of unearmarked resources. To reaffirm the purpose of the RRM, a study was conducted in 1995 to determine the validity and accuracy of the present methodology for use in today's health care environment. Preliminary findings support the need to update the current methodology to meet the future program demands of the IHS. The update of the RRM will include a complete revision and documentation of the RRM and will consist of the following phases: (1) Update Staffing Criteria and Modules, (2) Formulate Needs Assessment Cost Model, and (3) Needs Assessment Model Training.

Development of a Health Services Research Agenda for American Indian and Alaska Native Populations

The IHS and the Agency for Health Care Policy and Research cosponsored a health services research conference as a first step in a long-term agenda-setting process to identify the most important health services research issues facing American Indian and Alaska Native communities and their health care systems over the next 5 to 10 years. The health services research agenda is intended to promote collaboration among American Indian or Alaska Native organizations, tribal and urban health systems, medical communities, foundations, and government agencies to increase communications and produce research information on health program services for the American Indian or Alaska Native patient. The health services research agenda is also intended to provide a forum for discussing health care reform changes that are creating new directions in the Indian health care system.

Evaluation of Indian Health Service-Supported Substance Abuse Treatment Program for American Indian/Alaska Native Women

This evaluation consists of Phases III and IV of a multiphase effort by the IHS to improve the health of American Indian and Alaska Native women who abuse alcohol or other substances. Phase I of this effort focused on designing the study, and Phase II collected descriptive data on the women who use IHS-supported substance abuse treatment centers. The primary objectives of this project are to assess and measure the treatment outcomes achieved by the women receiving treatment in the IHS-sponsored projects, to attempt to relate treatment outcomes to treatment services provided, and to describe the organization and provision of substance abuse treatment and aftercare services available for adult American Indian and Alaska Native women, identifying common strengths, problems, and recommendations for improvement.

New Directions for Evaluation

The IHS is responding to dramatic changes taking place inside and outside the Government. The causes of the many changes include a decrease in funding level of discretionary Federal programs, greater involvement of tribal governments in the Indian health care system, technological innovations, the changing patterns of disease to more chronic conditions, and the transfer of many Federal programs and resources to individual States.

These changes will affect the IHS evaluation strategy in the coming years. Nevertheless, the IHS remains committed to comprehensively community-based, preventive, and culturally sensitive projects that empower tribes and communities to overcome health issues. The Director of the IHS has increased emphasis on several initiatives to focus attention on specific health areas and to serve as a management tool to prioritize the IHS's workload. These initiatives focus on women's health, youth, traditional medicine, elder care, and establishment of working relationships with Federal and State governmental agencies. The initiatives will undoubtedly affect new directions for evaluation.

Research and evaluation proposals to be considered in upcoming years include the following topics: evaluation of the effects of medical nutrition therapy on patient outcomes among Native Americans with newly diagnosed type II diabetes mellitus, evaluation of the elders clinic at the Zuni (New Mexico) Ramah Service Unit, evaluation of the impact of the Northern Cheyenne End-Stage Renal Disease Prevention Project, and evaluation of IHS aftercare services provided by IHS's regional treatment centers.

National Institutes of Health

MISSION: To sponsor and conduct medical research that leads to better health for all Americans.

Evaluation Program

The National Institutes of Health (NIH) develops scientific knowledge that leads to improved means to prevent illness, cure disease, and treat disability. This is done by conducting medical research in its intramural laboratories and supporting research in universities, medical and health professional schools, hospitals, and other health research organizations. NIH fosters the widespread dissemination of the results of medical research, facilitates the training of research investigators, and ensures the viability of the research infrastructure. The NIH Evaluation Program is an integral part of how NIH manages its support and conduct of medical research.

Philosophy and Priorities

NIH evaluation activities assess program performance (efficiency, effectiveness, responsiveness); analyze both quantitative and qualitative results based on those assessments; and use the resulting information in policy making, strategic planning, budgeting, and program development and management.

The quantity and diversity of diseases, disorders, and biological systems in the NIH portfolio make strategic planning and evaluation at NIH a complex task that is continuously evolving. Priorities are set and research programs are planned and evaluated at two levels: the Institutes and Centers (IC's); and centrally by the NIH Director, with whom the ultimate responsibility lies for the course of NIH-funded medical research.

This two-level approach ensures that planning and priority setting specific to the mission of each IC are fully developed and implemented with a clear vision and within the fiscal constraints set by the IC budget and that there is central leadership for developing cross-cutting initiatives and promoting collaboration among the IC's. The NIH Evaluation Program provides information to assist the NIH Director and IC Directors in determining whether NIH goals and objectives are being achieved and to help guide policy development and program direction.

Policies and Operations

A distinguishing feature of the NIH Evaluation Program is the extent to which it employs a variety of evaluation strategies and inputs that include the use of national advisory councils, boards of scientific counselors, consensus development conferences, and ad hoc committees that help to chart scientific directions and select the most promising research to support.

Program evaluations are funded through both a trans-NIH mechanism, the 1-percent set-aside authority, and by individual IC program funds. A two-tier system is used to review project requests to use set-aside funding. The first tier involves a review and recommendations by the NIH Technical Merit Review Committee (TMRC) on the technical aspects of project proposals and whether a project fits within HHS guidelines for use of the set-aside fund. The second tier involves the NIH Evaluation Policy Oversight Committee (EPOC), which considers TMRC recommendations and makes final funding recommendations to the Director of NIH or his or her designee. It also conducts policy-level concept reviews of proposals for NIH-wide evaluation studies; establishes the overall NIH set-aside budget; and oversees the process. EPOC recommendations are approved by the NIH Director or his or her designee. Concurrently, the IC's fund program evaluations from their budgets that are used by various committees, working groups, task forces, workshops, conferences, and symposia to assist them and the NIH in program management and development.

One of the ways NIH sets priorities and evaluates its programs is discussion with extramural grantees, intramural investigators and other NIH staff, members of Congress and the Administration, and members of the public, including professional societies and voluntary health organizations. These individuals and groups provide valuable input on pressing public health needs, important scientific opportunities, knowledge gaps, and the balance between patient-oriented and laboratory research.

Summary of Fiscal Year 1996 Evaluations

The evaluations completed in FY 1996 addressed a cross-section of the NIH research program, including bench research, clinical research, a demonstration project, and drug research. Examples of some of these evaluations are described below.

Evaluation of NIH Shared Instrumentation Grant Programs: Reports From Users documents the results of surveys of users of costly, state-of-the-art instruments purchased through the National Center for Research Resources shared instrumentation program. The program provides equipment that is to be shared among mainly NIH-funded investigators (termed major users), and other researchers who do not have an NIH grant. The study found that 16,050 scientists nationwide used these instruments in 1993; that of these, 69 percent were NIH-supported investigators; and that three-quarters of the major users believed the instruments were essential to their research. The study also found that 70 percent of 11-year-old equipment was still in use, and that 4 out of 5 major users of this older equipment believed the equipment was adequate for their research.

Moving from bench science, the report Clinical Research in the National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK) examined how much clinical research was being conducted in NIDDK's research portfolio, under two definitions of clinical research. The first definition required that the patient and the physician had a consultation; this was the narrow definition. The broader definition, adopted by the Advisory Committee to the NIH Director on Clinical Research, subsumed the narrow definition and also included areas of epidemiology, behavioral studies of patients, outcomes research, health services research, and technology development. The report found that in FY 1995, NIDDK spent 31 percent of its extramural budget on clinical research using the narrow definition and 37 percent of its budget using the broader definition; NIDDK spent 29 percent of the intramural budget on clinical research using the narrow definition and 33 percent using the broader definition.

Turning to evaluation of a demonstration project, the report Navajo Alcohol Rehabilitation Demonstration Program examined a federally funded program to provide culturally sensitive treatment for Navajo Native Americans who suffer from alcoholism and the effects of alcohol abuse. It is the only program to use Navajo-speaking counselors in the treatment. The evaluation found a higher level of participation in culturally sensitive therapies after completion of an inpatient treatment program, low use of aftercare services, a high program completion rate, and a low readmission rate. Data on long-term effects were not available.

Another way to treat chemical dependency is medication. The report Development of Medications for the Treatment of Opiate and Cocaine Addictions: Issues for the Government and Private Sector examines the National Institute of Drug Abuse's program to coordinate and encourage academic, private, and Federal regulatory involvement in developing and marketing new medications to treat drug abuse. The study found that pharmacotherapy for the treatment of drug addiction has received little attention; only two additional drugs beyond methadone have been approved, both of which were developed at least 20 years ago; and there is no approved medication for the treatment of cocaine addiction. The report noted major disincentives to pharmaceutical research and development for antiaddiction medications, among which is a lack of sustained Federal leadership. The report gives recommendations to help remove these barriers.

Evaluations in Progress

NIH supports a number of evaluations in progress. They range from small- to large-scale assessments, from evaluability studies to comprehensive evaluations. A few examples to illustrate the range include an examination of the research grant application and award processes; school health programs in Grades Kþ12; evaluation of laboratory animal use, facilities, and resources; and development of a data base for NIH-supported rare disease research.

The study Career Status and Satisfaction With NIH Research Grant Application and Award Processes: A Sample Survey of FY 1992 R01 and R29 Applicants will survey individuals who applied to NIH for an R01 or R29 grant in FY 1992. The purpose is to obtain information on the current status of these individuals. Career progress and research productivity will be assessed by comparing: (1) funded principal investigators versus others supported on NIH grants; (2) principal investigators funded on grants from non-NIH sources; and (3) those who are unsuccessful in getting any support for their research. Respondents will also be asked to assess their satisfaction with the grants application and award process and to provide their opinions on specific modifications NIH is considering for improving policies and procedures.

The study Comprehensive School Health Programs in Grades Kþ12, conducted by the Institute of Medicine, will assess grade-school health education, school health promotion and disease prevention, and school-based health care services from kindergarten through high school. The study will identify and examine model programs to formulate strategies for successful implementation of school health programs at national, State, and local levels.

NIH is responsible for the nationwide infrastructure for animals in research, which includes not only laboratory animals, but also the resources and facilities required to ensure their availability, appropriate use, and humane care. The purpose of Evaluation of Laboratory Animal Use, Facilities and Resources--Phase II is to provide NIH and other government agencies with information and objective data to inform the current national discussion--involving the press, animal rights activists, scientists, and others--about laboratory animal use.

Congress requires NIH to report annually on clinical research it is sponsoring for rare diseases. Development of a Database for NIH-Supported Rare Disease Research will establish a data base of NIH-sponsored rare disease clinical research. The Oak Ridge Institute for Science and Education will provide the instrument for NIH to determine the presence of existing data bases and the nature and extent of data on rare disease studies funded by the NIH. The contractor will review the completed instruments and will develop and provide recommendations on estimated costs and on the best ways to make the data base accessible to the biomedical research community, practicing physicians and other health professionals, and the public.

New Directions for Evaluation

NIH will continue to engage in a wide variety of evaluation activities. Priority setting for future NIH evaluation activities includes input from a number of sources including the HHS Strategic Plan, the NIH Government Performance and Results Act Performance Plan, and the NIH Strategic Plan--all in the developmental phase; the NIH Director's Areas of Emphasis; the Administration's High-Priority Areas; recommendations of the NIH IC Directors; and the recommendations of the EPOC. As a result, NIH will give priority funding to the following evaluation areas of study.

• Medical Research--This area focuses on issues such as the economic impact of medical research; clinical research including the impact of managed care; the health of special populations; and the development of measures for medical research accomplishments.
• Data Collection and Dissemination--This area focuses on issues such as expanded data collections to address emerging priorities; the health information dissemination infrastructure and strategy; and technology transfer.
• Research Training--This area focuses on issues involving national needs for medical research personnel; the quality of NIH training programs; and diversity among trainees.
• Research Infrastructure--This area focuses on issues involving the maintenance and construction of buildings and facilities; computers in biomedical research and medicine; and informatics in medical research.

NIH evaluation studies provide a rational basis for managerial decision making and responding to public concerns for accountability in government. Additionally, it is through such studies that the NIH is able to determine progress in meeting its scientific objectives, strengthening research and administrative activities, and contributing to the mission of NIH to sponsor and conduct research that leads to better health for all Americans.

Office of the Assistant Secretary for Planning and Evaluation

MISSION: To provide analytical support and advice to the Secretary on policy development and to assist the Secretary with the development and coordination of Departmentwide program planning and evaluation activities.

Evaluation Program

The Office of the Assistant Secretary for Planning and Evaluation (ASPE) functions as a principal advisor to the Secretary on policy development and, in this capacity, conducts a variety of evaluation and policy research studies on issues of national importance. ASPE is also responsible for Departmentwide coordination of legislative, planning, and evaluation activities. In its evaluation coordination role, ASPE has the following tasks:

• Provide annual guidance to all HHS agencies and staff offices regarding evaluation priorities, procedures, and review requirements.
• Review evaluation priorities proposed by the HHS agencies, providing advice about the focus or method of proposed projects, identifying opportunities for collaboration, and ensuring effective use of resources.
• Prepare planning and summary reports on evaluation activities as required by Congress.

Through the departmental evaluation planning process, ASPE has the capacity to identify cross-cutting program or policy issues of particular concern to the Secretary and specific program and policy areas not covered by the HHS agency evaluation plans. In these instances, ASPE initiates evaluations or collaborates with the agencies to conduct evaluations or policy assessments. For example, in recent years ASPE has initiated projects to develop cost estimates for health financing issues in general, and specifically for the Medicare and Medicaid programs; the effects of managed care expansion on public health infrastructure; welfare-to-work approaches; long-term care alternatives; and studies evaluating alternative services for children at risk of harm from drug abuse, crime, child abuse, and other pathologies.

Another continuing ASPE evaluation objective is to support and promote the development and improvement of data bases that HHS agencies and ASPE use to evaluate health care programs and health trends. For example, ASPE has been the major initiator on collaboration with the National Center for Health Statistics at the Centers for Disease Control and Prevention (CDC) of the first comprehensive survey of people with disabilities in the United States. The first component of these new data was completed in FY 1996, and national prevalence data on disability are now available. ASPE provides support to the HHS Data Council, which is charged with integrating key national surveys, such as linking health status indicators with indicators of well-being. The Department needs more comprehensive data sources to assess anticipated transformations in health and human services.

Finally, ASPE uses evaluation funds to promote effective use of evaluation-generated information in program management and policy making. The latter is accomplished through the dissemination of evaluation findings and other activities, such as providing technical assistance to agencies in the development of performance measures.

Summary of Fiscal Year 1996 Evaluations

During FY 1996, ASPE completed a wide range of studies and reports on a number of issues, providing information useful to the Secretary and the HHS divisions for purposes of program planning and budget and legislative development.

Health Policy

• Health Care Technology Assessment--This project, also described in chapter II, contains current information on the demand for technology assessment. The study examined the performers of technological assessment, the methods of conducting assessments, the uses of the results, and the unmet needs that might be met by further cooperation between the public and private sectors of health care. In the past 5 years, the evaluation of clinical practice for its cost-effectiveness has assumed greater market value. The use of technology assessment in the managed care sector is primarily to support coverage decision making. This study also examined the appropriate role of the Federal Government in technology assessment and the appropriate division of labor between the public and private sector.
• Information to Guide Physician Practice--The purpose of this study was to understand the relationship between the increasing amounts of information that are being produced for physicians and the types of information that physicians actually use. In particular, this study assessed physician use of computerized health and medical information sources available, such as those on the World Wide Web and the Internet, and how access to such information influences medical practice. Among the findings are the following: (1) The wealth of information available to physicians causes them to aggressively screen information sources; this screening process depends significantly on the characteristics of the physician, on the financial and legal practice environment, and on the source and type of information being provided; (2) technology advances are making it more feasible for rural physicians to communicate with colleagues, but financial constraints prevent them from taking full advantage of resources; (3) potential barriers to use of the On-line Medical Networked Information resources include the fact that information resources are not of enough value to justify the time needed to access them.
• Assumptions Underlying Insurance Participation Modeling--This report provides background information to analyze policies designed to expand health insurance coverage, especially models that seek to estimate the impact of incremental health care reforms on the size and composition of the uninsured population. The study found that the number of uninsured persons increased from 30.5 million in 1979 to 39.6 million in 1994 (about one-third due to increases in the population); disagreement over estimates of the numbers of uninsured persons can be traced to several factors, including different survey methods and disparities between point-in-time and longitudinal estimates. The report concludes that the major contributor to the problem of uninsured workers lies in the area of insurance market supply, but the lack of demand on the part of some workers also contributes to the problem.

Disability, Aging, and Long-Term Care Policy

• Board and Care Homes--Nearly one million elderly or disabled individuals are dependent on personal services and supervision provided by board and care homes. ASPE carried out an analysis of the effect of State regulation on the quality of care provided in board and care homes in 10 States. The study, also described in chapter II, found that board and care residents are significantly older and more frail than a decade ago, making it a more complex challenge for board and care operators and staff. The study findings indicate that States with extensive regulatory and licensing requirements are more able to cope with this challenge on a number of dimensions including greater availability of supportive services in licensed homes, more operator training, and lower use of psychotic drugs and inappropriate medications.
• Subacute Care: Market Analysis, Cost, and Quality--There has been a huge increase in health care services provided in postacute care settings, as providers struggle to reduce inpatient hospital care. This exploratory study, also presented in chapter II, examined the extent to which subacute care appears to be a low-cost, appropriate alternative to hospital care. It found that even "state-of-the-art" subacute care programs had difficulty documenting their success in reducing costs and improving outcomes compared with traditional nursing home care. The study concludes that systematic evaluation is needed before the cost-effectiveness of subacute care is demonstrated.
• Research and Demonstration Strategy to Test Models for Integrating Acute and Long-Term Care for Children With Disabilities--This study documented "state-of-the-art" practices in developing managed care for children with disabilities and how States are coordinating managed care strategies with their Title V programs for children with special needs. The study concludes that there is little operational experience to guide States in developing managed care approaches for children with special needs, particularly programs that integrate acute and long-term care services. It also found that Medicaid managed care providers could benefit from the experience of Title V providers in serving special needs children as they create new managed care models.
• Americans With Disabilities--This project provided funds to the Census Bureau to analyze the 1990 Census questions on disability and to create a national baseline on disability data. Data were collected for the population age 16 and over on several types of disabilities: limitations in work, ability to work, mobility limitations, and self-care limitations. The tabulations are unique disability data for States, counties, and large metropolitan areas, including persons with disabilities living in the community and in institutions. The results--which include tables, graphs and maps--are available on the HHS home page available through the Internet at http://www.hhs.gov.
• Interdisciplinary Education and Training of Professionals Caring for Persons With Disabilities--People with disabilities generally use more health care services than people without disabilities and often require a range of both health and human service professionals. If managed care arrangements are to serve such populations effectively, providers must learn to integrate the services of a number of different professional disciplines. This study identified a number of promising programs where professionals receive interdisciplinary instruction and practicum experience in providing care to people with disabilities. It also found that few training programs address the needs of managed care organizations and that managed health care providers often lack the specialized knowledge and skills to serve people with significant disability.

Human Services Policy

• Alcohol and Other Drugs Treatment for Parents and Welfare Recipients: Outcomes, Costs, and Benefits--While alcohol and drug treatment have been shown to be effective for reducing crime and health care costs, very little is known about this relationship for welfare clients. This study analyzed the outcomes, benefits, and costs of substance abuse treatment for two overlapping groups: parents of children under 18 years of age and recipients of public income support such as Aid to Families with Dependent Children (AFDC). The study found that many people in treatment are both parents and welfare recipients. The data documented that the treatment benefits for these individuals exceeded the treatment costs.
• Coordinated Community Responses to Domestic Violence in Six Communities: Beyond the Justice System--Awareness of domestic violence has extended past the justice and domestic violence service systems to health care providers, child welfare and substance abuse service agencies, churches, and business. This study examined the coordinated responses to domestic violence in six U.S. communities. The study found that a strong community response requires well-coordinated policies and consistent application among community criminal justice agencies. A need was found for health care providers to develop screening protocols to identify women who are battered and to develop a system for referring victims to the most appropriate agency.

Special Populations

In addition, ASPE completed two evaluation projects during FY 1996 that focused on the health needs of special populations. The first project was an evaluation design effort with the Office of Minority Health in the Office of Public Health and Science, titled Family and Community Violence Prevention Program: Technical Assistance in Program Evaluation and Capacity Building Assessment. The project produced a design for the Family and Community Violence Prevention Program, created by the Male Minority Consortium, a group of 19 historically black colleges and universities (HBCU). The report recommends several evaluation strategies that individual HBCU's can employ to assess the program, such as measuring attitudes and beliefs about violence, identifying communication channels and influences, and measuring community participation.

The second project was an effort to synthesize knowledge about the demography of American Indian and Alaska Native populations, with a focus on the major health problems they face and their use of health care services. The report, Changing Numbers, Changing Needs: American Indian Demography and Public Health, documents the difficulties in tracing demographic and public health trends for Native American populations. The report synthesizes data on population growth since the turn of the century, including information on fertility rates; reduction of infant mortality; persistence of high death rates for American Indian youths; declines in infectious diseases; and increases in rates of chronic disease, injuries, and alcohol and drug abuse. One analysis presented in the report showed that although IHS facilities are well suited to serve rural Native American population, problems continue with adequate access to care in rural areas and sufficient levels of budgeted resources.

Evaluations in Progress

Health Policy

• Monitoring Changes in Health and Human Service Programs--With the rapid changes in the health and human services systems, ASPE is studying the effects of State Medicaid waivers and changes in the State welfare laws on HHS programs such as Medicaid, foster care, and child support enforcement. ASPE evaluations will help define key measurements with which to monitor the effects of both program changes and changes in the environment, such as the impact of managed care on nonelderly individuals with disabilities, a project being done in collaboration with the Health Care Financing Administration (HCFA), and the effect of welfare-to-work programs on the well-being of children, a project being done in collaboration with the Administration for Children and Families (ACF) and foundations.
• Understanding Private-Sector Changes--ASPE's objective is to understand the private-sector environments in which HHS programs operate and to be able to describe and monitor the ongoing changes. A joint project with HCFA on managed care outcome measures and collaborative efforts with some of the Public Health Service (PHS) agencies on health promotion, prevention services, and benefits coverage in a changing private-sector marketplace were initiated during FY 1996. ASPE also continues to analyze data and information on health insurance coverage on an intramural basis.
• ASPE Data Initiative--This is a series of projects directed at filling gaps in the data HHS needs to analyze and monitor the health and well-being of the population, particularly the potential policy changes in health and human service programs. These gaps include State-level data; improved data on providers and the supply of health services; data that allow the measurement of child well-being; and better linkages between health and human services data.
• Performance Data--The Federal grants relationships to States in public health are evolving into outcomes-based performance management. A health outcomes-based monitoring approach requires data system development. ASPE is coordinating three projects with HHS agencies to accomplish this data development. One, with the National Academy of Sciences, is working to identify options for a performance-based approach that States and others want to achieve. The project will recommend specific steps that can be taken to improve State and local community data capabilities. A related project is developing approaches to obtain comprehensive baseline and trend data on public health infrastructure. A third study is building on a Robert Wood Johnson Foundation project to assess the quality of data that States collect and determine whether there is a foundation on which to build a multistate health data system that can be shared among States to improve health policy decision making.
• HHS Agency-Related Priority Health Services Evaluations--During FY 1996 ASPE has collaborated with HHS agencies on several program evaluations and technical assistance projects in program areas such as CDC's domestic violence initiatives, Indian Health Service alcohol treatment programs, the FDA's food additive review process, Medicare and Medicaid payment policies, and the Minority Male Consortium for Violence Prevention with the Office of Minority Health.

Disability, Aging, and Long-Term Care Policy

• Managed Care and People With Disabilities--Managed care can provide unique opportunities, but it also has potential pitfalls for people with disabilities. ASPE has projects under way to determine the impact of managed care on access, quality, and satisfaction for people of all ages who experience disabilities. Of particular interest are two policy questions. First is a concern about the impact of extending Medicaid managed care to the population eligible for Supplemental Security Income. Second, the impact of managed care on elderly persons with physical and cognitive impairments needs to be examined, and a comprehensive program for capitating acute and long-term care services for children with disabilities must be initiated and evaluated.
• Assisted Living--ASPE has commissioned a national study of assisted living, or the residential settings that combine adapted housing, assistive technologies, personal assistance, and other supportive services for persons with disabilities. Assisted living is considered an important component of services to the aging American population and an effective response to the rising costs of nursing home care.

Human Services Policy

• Evaluation of Family Preservation Services--ASPE and ACF are now examining family preservation programs aimed at preventing children from entering substitute care and reunification services to speed the return of children to their homes after they enter substitute care. Measures of program success will include prevention of placement of children into substitute care (for preplacement services), successful reunification (for reunification services), reduction of the recurrence of child abuse and neglect, improved child psychological well-being, improved child behavior, and improved family functioning.
• Moving Welfare Recipients to Work--ASPE is also working with ACF to conduct a comprehensive, multiyear study of the Government's principal program for moving people off welfare and into employment--the Job Opportunities and Basic Skills Training (JOBS) Program. The evaluation addresses the long-term effects of different welfare reform approaches, including whether any approach significantly improves the economic circumstances of the most disadvantaged recipients; the effects of welfare-to-work programs on the children of welfare recipients; and the cost-effectiveness of different approaches.
• Promoting Father Involvement--Three evaluations are looking at various aspects of paternal involvement, especially among fathers who are not living with their children. One project is examining the relationship between various child custody, visitation, and support payment patterns and the effect that payment and contact have on child well-being. The second is developing a design for the evaluation of community-based programs for vulnerable fathers. The third project will develop a theoretical framework to assist in understanding and implementing programs and activities to promote responsible fathering. The last two projects are being conducted in collaboration with ACF.
• Trends in the Well-Being of America's Children and Youth--In FY 1995, ASPE provided funds for the first annual report on the health and well-being of America's children. The report presented up-to-date trends on how our Nation's children are faring, pulling together information on all available national trends in the lives of children and youth. This volume fills a crucial gap and is now an annual series to keep the Nation apprised of the well-being of its children and youth. ASPE is developing the second annual report, to be released in 1997.

In addition, human services policy assessments are under way to examine child welfare and domestic violence policies, the health and mental health of immigrant children, and factors in the growth and decline of AFDC caseloads.

New Directions for Evaluation

In upcoming years, ASPE will pursue the following evaluation priorities.

Impact of Program Transformation

The public health system and public health programs supported by HHS are an essential part of the health care safety net for the uninsured and are a vital element in protecting the health of all Americans. An increasing challenge for HHS agencies is to understand how health care system changes, such as managed care, interact with the dramatic changes taking place in welfare policies to affect the public health system, public health programs, and the clients those programs serve. In future studies, ASPE will compliment the efforts of the CDC, the Health Resources and Services Administration, and the Substance Abuse and Mental Health Services Administration by evaluating how health and welfare reforms create new challenges and responsibilities for the public health system and programs.

Managed Care and Disability

ASPE will continue to support evaluations to determine the effect of managed care on individuals with disabilities who have both health and long-term care needs. As mentioned above, ASPE, in collaboration with PHS and other HHS agencies, initiated a series of studies to identify and understand how State and community health care reform efforts are affecting people with disabilities. New studies will continue to examine the impact of selected State and community managed care initiatives on people with disabilities, including how such arrangements affect access to needed services, health status, well-being, and satisfaction. Investigations will also focus on developing better outcome indicators of health care quality for elderly persons with physical and cognitive impairments, including individuals with Alzheimer's disease and other dementias. In addition, a collaborative evaluation effort will be initiated with PHS and HCFA to investigate the effectiveness of new initiatives to integrate health and long-term care services.

Impact of Welfare Reform

ASPE will be initiating new studies in upcoming years to evaluate the combined changes taking place in health care delivery systems and in income supports for low-income children and families who often have special needs. One issue that will be studied is how to help substance abusers and victims of domestic violence whose entitlements and access to services will change under the terms of new welfare reform legislation. Under the Personal Responsibility and Work Opportunity Reconciliation Act, welfare recipients generally will be required to work to receive benefits and will face overall time limits on welfare receipt. In the past, many clients identified as substance abusers or victims of domestic violence were not required to participate in work activities. While some of these clients will be exempted under the new law, others must enter the workforce. Case managers need better tools to assess clients' barriers to sustaining employment and to assist these clients in addressing those barriers.

Other ASPE efforts to study the combined effects of changes in health care and income support policies on the public health system and its clients will focus on developing better State and national data on outcomes and the impact of the changes on the health and well-being of children and youth. Studies that address teen pregnancy prevention will build on work already undertaken and will examine how communities are addressing this issue as required in the new welfare law.

Quality-Related Studies

A major goal of the ASPE evaluation program, in relationship to HHS's ongoing responsibilities in the area of consumer protection and health quality standards, is both to understand the measurement of quality of care issues in health plans and providers and to improve the public availability and the comparability of such information. Working in conjunction with HCFA's Bureau of Health Standards and Quality, the Agency for Health Care Policy and Research, and other related PHS agencies, ASPE will develop projects to better measure health outcomes, relate health outcomes data to critical process of care data, evaluate the feasibility of developing new outcomes-based measures as part of administrative monitoring systems, and provide comparative-based health information to health consumers.

Improving Evaluation Data and Analytic Capability

HHS has taken a number of steps to strengthen its capacity in the data policy area and to work more effectively in collaborative data efforts with the health industry, the research community, and State and local governments. In the future, ASPE, working with the HHS Data Council, expects to initiate projects to support (1) development of State data strategies to monitor the impact of health and welfare transformations and to support evaluation of public health programs; (2) development of health data standards to support evaluation and monitoring in health care and public health programs; and (3) development and evaluation of HHS-wide information-privacy policy. These projects are largely responsive to new data policy requirements that the Kennedy-Kassebaum law places on HHS.

Office of Public Health and Science

MISSION: To provide advice on public health and science to the Secretary of Health and Human Services, to provide executive direction to program offices within OPHS, and, at the direction of the Secretary, to coordinate cross-cutting Public Health and Science initiatives in the Department.

Evaluation Program

The Office of Public Health and Science (OPHS) within the Department of Health and Human Services (HHS) provides advice, policy and program coordination, and leadership in the implementation, management, and development of activities related to public health and science, as directed by the Secretary. OPHS provides advice to ensure that HHS conducts broad-based public health assessments designed to better define public health problems and solutions to those problems. It assists other components of HHS in anticipating future public health issues and problems and provides assistance to ensure that HHS designs and implements appropriate approaches, interventions, and evaluations that will maintain, sustain, and improve the health of the Nation. OPHS provides a focus for leadership on matters including recommendations for policy on population-based public health and science and, at the Secretary's direction, leads or coordinates initiatives that cut across agencies and operating divisions. In addition, it communicates and interacts, on behalf of the Secretary, with national and international professional and constituency organizations on matters of public health and science. It also provides guidance and coordination across the Department on international results and domestic refugee health issues, and serves as the focal point for senior staff support in the governance of health agencies of the United Nations system.

In keeping with its newly defined role within the Department, OPHS has specifically sought to develop an evaluation plan that avoids duplication of efforts that might more appropriately and effectively be undertaken by the operating divisions of HHS or by the Office of the Assistant Secretary for Planning and Evaluation (ASPE). Thus, the direction of the FY 1997 evaluation strategy for OPHS is toward public health and science issues that cut across multiple interests of the operating divisions and that require a coordinated approach to achieve the most effective results. In addition, OPHS commits itself to seeking to carry out every project proposed as part of this year's strategy in collaboration with relevant operating divisions.

As to its specific responsibilities, OPHS conducts evaluations requested of the Secretary by Congress that are most appropriately managed by staff with medical or public health sciences credentials, and that cannot be assigned to an operating division. Also, evaluations are conducted to support the Assistant Secretary for Health in his or her role as senior advisor for public health and science to the Secretary. This role includes the responsibility to "assist the Secretary in developing a policy agenda for the Department to address major population-based public health, prevention and science issues" and to provide "leadership and a focus for coordination of population-based health, clinical preventive services and science initiatives that cut across operating divisions." In addition, OPHS conducts evaluations specific to the needs of the program offices located within OPHS, such as Women's Health, Minority Health, Disease Prevention and Health Promotion, International and Refugee Health, and Emergency Preparedness.

Summary of Fiscal Year 1996 Evaluations

In FY 1996, OPHS completed four evaluations in an effort to better inform policy decisions throughout the Public Health Service (PHS). Studies completed this year focus on performance and accountability of the public health and science communities and include such topics as improved methodologies for cost-effectiveness analysis and quantification of State public health expenditures, an examination of refugee health care issues, and the impact of research misconduct investigations. As illustrated below, OPHS efforts cut across a variety of PHS program areas.

In an effort to study and learn from past experiences, the cost-effectiveness analysis (CEA) report provides an assessment of the current state-of-the-science of CEA methodology and makes recommendations about ways to improve the quality, comparability, and utility of CEA's. CEA is a powerful tool for evaluating which strategies best serve programmatic and financial objectives. Unresolved methodological issues are discussed in the context of factors that limit the policy relevance of existing CEA's. The CEA report is described in more detail in Chapter II.

A second methodological report, Measuring Expenditures for Essential Public Health Services, was completed on quantification of State public health expenditures. Working with State and local public health, mental health, substance abuse, and environmental agencies in nine States and with Federal PHS agencies, a methodology was developed and tested for estimating investments in 10 essential public health service areas. For all nine participating States, the total expenditure for essential public health services was $8.8 billion in 1994-95, of which $2.7 billion is population based. Estimates of public health expenditures are provided for agencies within the States and by service areas. In addition to estimating the relative investment in public health in the United States, the report lays the groundwork for a system to track these expenditures over time, across agencies, and across levels of Government. The resulting financial data can be related public health outcomes, infrastructure, and workforce measurements. An article about this study was published in Morbidity and Mortality Weekly Report on February 21, 1997.

A report on the health needs of newly arriving refugees evaluated the health assessment and follow-up care available to refugees, Cuban and Haitian entrants, and Amerasian immigrants from Vietnam during the first 8 months following their arrival in the United States. During this 8-month period, the Federal Government provides funding for health care coverage to refugees to prevent the spread of health conditions that could affect the public health and to identify health conditions that could impede effective refugee resettlement. The findings indicate areas of success, as well as areas needing improvement. For example, the flexibility of Federal funding has enabled States to fill gaps in their refugee health programs. In addition, statewide infrastructure for refugee health care has been developed in States that contract with local providers to deliver health-screening services. Concerns are raised about the accuracy and completeness of screening conducted outside of refugee-specific clinics, the reporting of refugee health screening results to the Centers for Disease Control and Prevention, and the availability of follow-up care to refugees.

Scientific misconduct is a topic of great interest to the public health and science communities. In its report on the consequences of being accused of research misconduct, the Office of Research Integrity (ORI) examined the impact of misconduct investigations on exonerated researchers. About 70 percent of all cases of alleged scientific misconduct that come to the attention of ORI result in exoneration. Yet little was known about the adverse consequences for the accused and the extent to which institutions comply with confidentiality requirements. The findings raise questions about the confidentiality of ongoing investigations. In a survey of 54 individuals who were accused of scientific misconduct and exonerated (prior to 1995), 60 percent reported one or more negative consequences of being accused of scientific misconduct. Of these, 90 percent indicated that the negative actions began during the inquiry or investigation and 65 percent reported that these actions continued after they had been exonerated. Yet only 39 percent viewed the impact of accusation on their careers as negative; 39 percent believed there was a continuing stigma attached to having been accused of misconduct. A similar 38 percent of the respondents were dissatisfied with the efforts of their institutions to restore their reputations.

Evaluations in Progress

In reviewing evaluation projects for FY 1996, OPHS considered the following priority areas: congressional mandates, cross-departmental initiatives, OPHS strategic plan-related initiatives, program improvements, and the Government Performance and Results Act (GPRA). OPHS selected 17 program and policy evaluations for funding this year, which are described below.

The Commission on Dietary Supplement Labels, appointed by the President, is evaluating factors relevant to Food and Drug Administration regulation and possible legislation of label claims and statements for dietary supplements, including health claims as requested by Congress. A second nutritional study, undertaken by the National Academy of Sciences, is evaluating human nutrient requirements. The study will provide reference intakes, guidance about these values for clinical and public health use, and a listing of research needs.

An evaluation of the Cooperative Agreements for Demonstration Projects for Capacity Building at Historically Black Colleges and Universities (HBCU's) is under way. The setting will examine the extent to which the overall program and individual projects have resulted in the establishment of offices of sponsored programs and the adoption of uniform processes in participating schools. The evaluation will also assess how well the program has achieved increased funding for health-related research and training at the institutions and how effectively it has enhanced current research, training, and services.

The OPHS evaluation priorities related to strategic planning initiatives will focus on several areas. First is the organization of the Healthy People 2000 objectives. To promote an integrated public health message, OPHS is consulting a broad range of members of the Healthy People 2000 Consortium and using focus groups to evaluate the basic framework of national health goals and objectives in preparing for the Healthy People 2010 Plan. The study will collect insights on the current functions of the objectives framework and explore opportunities for making them more relevant to the Healthy People 2000 Consortium members, State and local agencies, managed care industry representatives, and Fortune 500 major purchasers of health care plans.

School health programs, both comprehensive and categorical, continue to be developed and implemented throughout the Nation. OPHS has funded a study to identify evaluations of school health programs and make available an updated compilation of methodologically sound studies and their findings to assist school boards, administrators, health personnel, and health educators trying to maximize limited resources for effective health programs.

OPHS will continue to expand efforts to improve nationally available data on the public health infrastructure, focusing on local public health capacity and workforce issues. The results will provide local health officials and planners with a validated instrument for estimating the level of support for essential public health functions. This local study of expenditures represents the community analog to the State study funded in FY 1996.

In the area of health data and the environment and in continuance of support for public health infrastructure revitalization, OPHS funded development of two data-tracking systems, one related to sentinel public health indicators, health outcomes, levels of health risks, and the health protection infrastructure; and the second on State and local environmental health data needs and sources of State and local environmental information. OPHS also funded an assessment of health plan involvement in community-based initiatives and a survey of clinicians to assess the level of provision of and the importance placed on preventive clinical services by primary care providers,

OPHS supports several projects to assist in improving program operations and management. These include an evaluation of a new model of coordinating and integrating HIV-prevention messages and primary care in high risk populations, an evaluation of the Minority Health Resource Center, and the assessment of the Adolescent Family Life Program.

The implementation of the GPRA of 1992 is a priority for OPHS. The development of performance measures for public health programs will be crucial for program planning, budgeting, and legislative development. OPHS, in collaboration with the Office of the Assistant Secretary for Planning and Evaluation, has targeted efforts in three program areas to develop these measures. First, it is supporting completion of the consultation begun in FY1995 with State and local health officials concerning the development of performance measurement systems that can be used to improve accountability for expenditure of Federal funds. OPHS will assist States and local governments in determining the impact Federal programs are having on improving the health status of Americans. OPHS will also discern the extent to which States and selected counties and cities have current data available to measure their own State year 2000 objectives and selected national objectives.

Two communication studies are also under way. The first focuses on the design and evaluation of interactive communication applications for consumer health information. A second study examines the dissemination of public health information from PHS agencies.

Finally, the Office of Minority Health is examining the efficiency and effectiveness of its Bilingual/Bicultural Service Demonstration Grant Program, which is intended to build the capacity of community-based organizations to address access to health services for limited-English-speaking minority populations.

New Directions for Evaluation

The next few years will continue to be a time of enormous transformation for this Nation's health care system. The need to manage skyrocketing health care costs and rein in the Federal budget deficit--while trying to address the health needs of over 40 million uninsured Americans and a general population that is becoming increasingly aged as well as linguistically and culturally diverse--presents enormous challenges to HHS, OPHS, and our partners in the public health community.

The shifting emphasis on managed care presents new opportunities coupled with new dilemmas regarding how best to ensure that, in the quest to manage the cost of care, overall quality of care is not unduly compromised. It raises new questions of how to ensure a workforce that is properly trained and fully capable of performing the essential services of public health to increasingly diverse populations in this dynamic health care environment. Furthermore, transformations in the health care and public health arena necessitate the monitoring and assessment (and the data systems to do so) of the effects and impacts of these changes on the financing, organization, and availability of population- and clinic-based preventive services and medical treatment.

OPHS has given highest priority in its FY 1997 strategic plan and evaluation efforts to addressing these issues. The FY 1997 OPHS strategic plan emphasizes the need for a strong foundation for public health in the 21st century. The priorities focus attention on (1) the information systems and workforce that comprise the Nation's public health infrastructure and that are necessary to effectively provide the essential services of public health; and (2) the impact of managed care arrangements on the resources available for this infrastructure and on the health of all Americans, especially those most vulnerable.

Future OPHS evaluation planning efforts will reflect the OPHS strategic plan, as well as the broader HHS programmatic priorities related to improving the health of all Americans and assessing the effects of health-sector transformation. Planned investigations in support of the public health infrastructure include an assessment of response to health and medical consequences of chemical and biological agent incidents; an evaluation of summary measures of health status; an evaluation of the electronic gateway to consumer health information; analyses of international year 2000 plans to inform the United States effort and the WHO Renewing Health for All strategy; and an examination of outreach efforts and potential uses of Healthy People Objectives for the Year 2010.

Other planned evaluation activities will focus on addressing specific health needs of our most vulnerable populations. These include evaluations of strategies and approaches for addressing the health of racial and ethnic minorities in the United States as compared with the United Kingdom, cultural competence in health care to address issues of measurement and linkages to health and patient outcomes, an inventory of State data on women's health, an inventory and assessment of linguistically and culturally appropriate services in managed care outreach organizations and an assessment of minority consumer experiences with State outreach activities related to Medicaid managed care.

Last, in response to the increased need to measure the performance and effects of public health programs and activities, OPHS will support efforts to enable performance measurement activities among local, State, and Federal health agencies.

Substance Abuse and Mental Health Services Administration

MISSION: To improve the quality and availability of prevention, treatment, and rehabilitation services for substance abuse and mental illness.

Evaluation Program

The Substance Abuse and Mental Health Services Administration (SAMHSA) is committed to evaluating its overall programs and individual grant projects to assess the effectiveness of prevention, treatment, and rehabilitation approaches and systems of care; the accountability of Federal funds; and the achievement of SAMHSA's programmatic and policy objectives.

To the greatest extent appropriate and feasible, SAMHSA will encourage the use of comparable data elements and instruments across its evaluations in order to work toward a comprehensive evaluation system and to minimize respondent burden.

SAMHSA conducts grant programs under a variety of legislative authorities. These authorities can generally be grouped into two types: (1) services and (2) knowledge development and application (KDA). The evaluation required for a particular grant program is dependent on the type and purpose of the program. SAMHSA will evaluate each of its service programs and provide information to program managers about the accountability of Federal funds. In addition, the evaluations of KDA programs will generate new knowledge to lead the field in the development of policies that improve services. Program and evaluation staff must work together to identify clearly the questions or goals each grant program will address and to propose appropriate evaluation strategies.

The two types of grant programs (service delivery and KDA) represent the two facets of SAMHSA's mission. SAMHSA's leadership in the field depends on the successful interaction of these two facets of its mission. Through evaluation, SAMHSA must identify effective approaches to prevention, treatment, and rehabilitation. Through service delivery funds, SAMHSA must provide incentives to the field to implement effective approaches. Major emphases of SAMHSA's mission are to develop, identify, and disseminate effective strategies and systems for prevention, treatment, and rehabilitation.

SAMHSA is implementing an integrated model of evaluation and planning. Strategic planning identifies priorities, such as managed care, that drive the development of grant programs and evaluations. In compliance with the Government Performance and Results Act (GPRA), SAMHSA is attempting to improve performance by identifying performance goals associated with its strategic plan. The formulation of programmatic and evaluation priorities includes consultation with the SAMHSA and Center Advisory Councils and with other experts in the fields of evaluation and service delivery. Early and continuous coordination of program planning and evaluation design will result in the articulation of program objectives that can be evaluated. Evaluations will show how well the overall grant programs have achieved their objectives, and SAMHSA will translate these results into information that can be used for program and policy development. The strategic planning and policy development processes will then use these results to refine SAMHSA's priorities and objectives.

This evaluation policy will help SAMHSA achieve its goal of continually informing policy and program development with knowledge culled from past performance. In this way SAMHSA can best serve its customers by enhancing the quality of public substance abuse and mental health services.

In compliance with Public Health Service (PHS) guidelines for the technical review of evaluations, SAMHSA has established a standing committee of PHS staff who are evaluation specialists. Representatives of the Office of the Assistant Secretary for Planning and Evaluation serve as ex officio members of the committee. The SAMHSA evaluation officer is the committee chair. The committee does not generally review the evaluation proposals of individual grantees. It does review proposals for broader, more comprehensive evaluations, such as the cross-project evaluations of grant programs.

Evaluation project proposals are generally prepared by SAMHSA program staff in the various Centers. The standing committee reviews each proposal on the following criteria: clarity of evaluation objectives and research questions; appropriateness and feasibility of the specifications for evaluation design and methods; appropriateness of the plans for dissemination of results; and use of previous relevant evaluations and existing program data systems. Each proposal must clearly state the relationship of the evaluation to SAMHSA's overall policies, priorities, and evaluation program.

Summary of Fiscal Year 1996 Evaluations

In FY 1996, SAMHSA completed a report entitled Overview of Addiction Treatment Effectiveness. This document presents an overview of the research literature substantiating the effectiveness of addiction treatment. The report describes specific treatment approaches, settings, components, and services and provides an analysis of the effectiveness of each. It was prepared by conducting comprehensive literature searches, reviewing meta-analyses, examining literature reviews, and reviewing several hundred research articles.

Also during FY 1996, SAMHSA completed a report entitled Employment Outcomes of Indigent Clients Receiving Alcohol and Drug Treatment in Washington State. This report was prepared by researchers at the University of Washington and the Washington State Division of Alcohol and Substance Abuse. It reports results from a project that evaluated substance abuse treatment outcomes for indigent clients served in Washington State. The treatment group included 499 individuals in various treatment modalities. The comparison group included 168 clients who were assessed but did not initiate treatment. Analyses were conducted on employment outcomes, using data obtained from official State employment records. The employment data cover a 12-month period before treatment and an 18-month period after treatment.

The study found that the duration of treatment is an important predictor of employment outcomes. Clients staying in treatment longer experience better outcomes, as do clients who complete treatment. A second finding was that the benefits of treatment, in enhanced earnings, compare favorably with the costs of treatment.

The policy implication of the study is that less treatment may not necessarily be less costly in the long run. Limiting coverage for substance abuse treatment for public clients will reduce short-run treatment costs, but may compromise important benefits of treatment, such as enhanced employment and increased earnings.

Evaluations in Progress

SAMHSA currently has several major evaluations under way in the following general areas: program accountability, knowledge development and application, and managed care.

Program Accountability

SAMHSA conducts evaluations for program accountability in compliance with the GPRA. These evaluations are undertaken to inform program management and help managers refine program operations. This is the primary type of evaluation conducted on SAMHSA's service grant programs. For example, the children's mental health service program has an ongoing evaluation. The evaluation will yield continuous information on program implementation and on outcomes for children and families served. This information will be used for reports to Congress, feedback to grantees, program development, and performance improvement.

Knowledge Development and Application

The primary purpose of SAMHSA's KDA program is to generate new knowledge to lead the field in the development of policies that improve substance abuse and mental health services. For example, evaluation results on substance abuse prevention for high risk youth will allow policymakers to draw inferences about the effectiveness of certain interventions for this population. An evaluation of the program for Access to Community Care and Effective Services and Supports will provide information for the design of ongoing service programs at the Federal, State, and local levels.

The National Treatment Improvement Evaluation Study is a cross-site evaluation study examining the effectiveness of demonstration grants funded by the Center for Substance Abuse Treatment. The purpose of the study is to assess the extent to which treatment enhancements improve substance abuse treatment outcomes over time. Preliminary results from this study provide strong support for the efficacy of substance abuse treatment in reducing levels of substance use, reducing involvement in criminal activities, and increasing employment. The final report will be available in FY 1997.

Managed Care

The Center for Substance Abuse Treatment has initiated an important managed care evaluation to assess the impact of State managed care initiatives on substance abuse treatment in terms of access, cost, and quality. Currently, there is little information at the State, provider, or client levels on the impact of managed care on the provision and outcomes of substance abuse treatment services. Several States will be selected for rigorous, statistical evaluation, which will include measures of costs, access, quality, and treatment outcomes. Results will be used to improve State substance abuse managed care programs and for Federal policy making related to managed care and health care reform.

New Directions for Evaluation

SAMHSA is designing its evaluation activities to guide programmatic and policy decisions. The evaluation activities will complement the SAMHSA and HHS strategic plans and will respond to emerging trends such as managed care.

One SAMHSA evaluation priority is knowledge development and application. SAMHSA's KDA program is designed to answer specific, important, policy-relevant questions. These questions will be designed to provide critical information to improve the Nation's mental health and substance abuse treatment and prevention services. Evaluations of the current KDA program will generate knowledge on such topics as the relative effectiveness of alternative models of managed care for treatment and prevention of substance abuse and mental disorders, the relative effectiveness of alternate models for preventing homelessness among adults with serious mental illnesses, and the efficacy of a brief intervention for marijuana dependence.

Another of SAMHSA's evaluation priorities is to assess the impact of managed care on the availability of services to populations in need, and to develop standards for measuring quality and outcome. SAMHSA's managed care evaluation strategy is designed to leverage important knowledge that directly affects the agency mission. This knowledge will be acquired through focused applied health services research, evaluations, demonstrations, and epidemiological and service capacity studies.

Appendix A. Abstracts of HHS Evaluations Completed in Fiscal Year 1996

The abstracts included in this appendix describe the evaluation reports completed by the Department of Health and Human Services (HHS) during fiscal year (FY) 1996. They are listed in alphabetical order by agency. The sponsoring agency, report title, abstract, and Federal contact person(s) are listed for each report. For more information on any of the reports, please call the contact person listed at the end of each abstract. Copies may be obtained by contacting either the Policy Information Center or the National Technical Information Service, addresses for which are given below.

Policy Information Center

Final reports for most of the evaluations listed in this appendix have been submitted to the HHS Policy Information Center (PIC), a centralized source of information on short-term evaluative research, policy-oriented projects, and in-process, completed, and ongoing HHS evaluations. The PIC maintains a resource data base containing information on more than 6,000 completed and ongoing studies sponsored by HHS, other Federal agencies, and private-sector entities. The PIC provides executive summaries (if available) at no cost.

The PIC identification number appears after the Federal contact's phone number. For information about using PIC resources, please contact

Policy Information Center
Office of the Assistant Secretary of Planning and Evaluation
Department of Health and Human Services
Room 438F, Hubert H. Humphrey Building
200 Independence Avenue, S.W.
Washington, DC 20201
(202) 690-6445

National Technical Information Service

The National Technical Information Service (NTIS) is an information clearinghouse run by the Department of Commerce. It provides reports in both paper and microform formats for a fee. If a final report has been registered with this service, the NTIS accession number follows the PIC ID entry. For more information about ordering copies of reports from NTIS, please contact

National Technical Information Service
Department of Commerce
5285 Port Royal Road
Springfield, VA 22161
(703) 487-4650

Administration for Children and Families

Contents

Evaluation of Child Support Guidelines

Evaluation of Four Child Access Demonstration Programs Funded by the Federal Office of Child Support Enforcement

Evaluation of the Child Access Demonstration Projects: Report to Congress

Evaluation of Nine Comprehensive Community-Based Child Abuse and Neglect Prevention Programs

Final Evaluation Report for the Case Management Enhancement Project at the East Orange District Office of the New Jersey Division of Youth and Family Services

Final Report: National Evaluation of Home-Based Services Programs for Runaway Youth

Gang Families in a Public Housing Project

Length of Service and Cost-Effectiveness in Three Intensive Family Service Programs: Final Report

Social and Educational Development of Tribal-Based Communities of the Sonoran and Negev Deserts

Survey of Head Start Family Self-Sufficiency Initiatives

Survey of Head Start Family Self-Sufficiency Initiatives: Case Studies in Six Communities

Transfer of International Innovations: Development of a Clinical Monitoring System To Support Foster Care in Michigan

Women and Infant Nurturing Services (WINGS)
 

TITLE: Evaluation of Child Support Guidelines

ABSTRACT NUMBER: 001

ABSTRACT: This study evaluated presumptive child support guidelines in order to (1) assess the impact of shifting from voluntary to presumptive guidelines using the Current Population Survey--Child Support and Alimony Supplement of 1992; (2) assess changes and activities of state Guideline Commissions using materials submitted by the States; (3) carry out a prospective administrative survey of 11 State guidelines applications; and (4) conduct unstructured interviews with State, local, and judicial personnel in the states participating in the survey. The report finds little impact on child support orders when States treat guidelines as presumptive rather than voluntary. A review of 48 formal State guidelines shows that States' inquiries range from questions about the guideline formula to the responsibility of new spouses to the definition of income. Furthermore, a review of State studies of deviations from guidelines shows that most States deviate in 25 percent or less of cases, but that the range of deviation is from 81 percent to 3 percent. A study of cases in 11 states and 21 sites (4,000 cases) shows that the formal deviation rate is 17 percent, but that there was a range of 50 percent to 0 percent by county. Reasons for deviations include agreement between parties, second-parent households, extended visitation and custody, and low income. Seventy-four percent of deviations decrease the award. Finally, the report finds that treatment of special factors--such as second families, income imputation, verified income, health insurance, extraordinary health, child care, postsecondary education, taxes, custody, and visitation--was low and extremely variable.

AGENCY SPONSOR: Office of Child Support Enforcement

FEDERAL CONTACT: David Arnaudo

PHONE NUMBER: (202) 401-5364

PIC ID: 5983

PERFORMER ORGANIZATION: CSR, Incorporated, Washington, DC
 

TITLE: Evaluation of the Child Access Demonstration Projects: Report to Congress

ABSTRACT NUMBER: 002

ABSTRACT: Since the passage of the 1984 Child Support Enforcement Amendments and the 1988 Family Support Act, courts and child support agencies have vastly increased their efforts to establish a child's paternity and establish and enforce child support orders. However, many noncustodial parents' groups have complained that there has not been a parallel increase in enforcement of their rights to parental access. This report presents information from child access demonstration projects in Florida, Idaho, and Indiana that began in October 1990 (Wave I). These projects were designed to determine whether mediation services for child access problems would reduce parent conflict, reduce interference with visitation rights, and encourage full, voluntary compliance with child support obligations. The Wave I report finds that: (1) the nature of access disputes is complicated and long-standing for one or both parents--insufficient time, visitation denial, feeling cut out of the child's life, scheduling problems, fights during drop-off and pick-up, problems caused by new relationships, concerns about proper supervision and safety; (2) nonattendance at mediation was a big problem; (3) parenting/visitation agreements were arrived at in 65 to 70 percent of the cases when both parents attended mediation sessions; (4) there were fewer problems after mediation than before; (5) visitation rates increased for experimental cases compared to control cases in most sites; (6) the speed of dispute resolution was not affected; (7) the incidence of relitigation was low--5 percent--but not affected by the experiments; and (8) compliance with child support increased for all experimental groups over the control groups and ranged up to 24 percent increases. See also PIC ID Nos. 5972.1 and 5972.2. (Final report: 239 pages, plus appendices.)

AGENCY SPONSOR: Office of Child Support Enforcement

FEDERAL CONTACT: David Arnaudo

PHONE NUMBER: (202) 401-5364

PIC ID: 5972

PERFORMER ORGANIZATION: Center for Policy Research, Denver, CO
 

TITLE: Evaluation of Four Child Access Demonstration Programs Funded by the Federal Office of Child Support Enforcement

ABSTRACT NUMBER: 003

ABSTRACT: Although attention has long been paid to issues of child support enforcement, problems of noncustodial parental access to their children have been less studied. However, there is substantial research demonstrating that parental contact and payment of child support are related. This report summarizes the result of the second (Wave II) set of Child Access Demonstration Projects funded by the Office of Child Support Enforcement. Four States participated in the demonstration. In Arizona, divorced couples under enforceable visitation orders received monitoring calls according to different frequencies and schedules, while parents in the control group received no calls. In Idaho, parents seeking divorce or modification of visitation orders took part in family skills education. Detailed access plans were developed by participants, or they were referred to mediation. The Iowa project provided counseling services to noncustodial parents who called the operating organization. In Massachusetts, parents seeking divorce or modification of child access participated in education programs or assessment and counseling. The report finds that (1) both custodial and noncustodial parents report problems concerning child access; (2) parents in Wave II experimental and control groups reported decreased access problems; (3) in Massachusetts, fathers in the experimental group were less likely to report that their ex-spouse made negative comments about them to their children, and custodial parents in Arizona and Massachusetts reported fewer incidences of children being upset at drop-off and pick-up times; (4) no statistically significant differences in payment of child support were evident after the Wave II projects; and (5) most parents expressed satisfaction with the demonstration projects. See also PIC ID Nos. 5972 and 5972.2. (Final report: 154 pages, plus appendices.)

AGENCY SPONSOR: Office of Child Support Enforcement

FEDERAL CONTACT: David Arnaudo

PHONE NUMBER: (202) 401-5364

PIC ID: 5972.1

PERFORMER ORGANIZATION: Center for Policy Research, Denver, CO
 

TITLE: Evaluation of Nine Comprehensive Community-Based Child Abuse and Neglect Prevention Programs

ABSTRACT NUMBER: 004

ABSTRACT: This evaluation (1) designed and implemented a process and impact evaluation of nine comprehensive community-based child abuse and neglect prevention projects funded by the National Center for Child Abuse and Neglect; (2) provided technical assistance to the nine projects in meeting the requirements of the evaluation; and (3) aided the programs in their efforts to design and implement their own internal program evaluations. The evaluation was conducted in three phases over a 3-year period. The 9 projects examined 10 service components, many of which were different across the programs. Therefore, a series of individual experimental designs were developed for each service component. Process and impact data were collected across programs and through a series of studies conducted in each site. All grantees were aided in refining their evaluation and research plans. An electronic bulletin board was established for use by the grantees. Two site visits were made each month and evaluation technical assistance was provided by telephone, electronic bulletin board, and mail. Grantee evaluators were advised on statistical techniques, instruments, and data collection methods. An evaluation plan, technical assistance overview, and bibliography have been submitted to the Administration on Children, Youth, and Families for review.

AGENCY SPONSOR: Administration on Children, Youth, and Families

FEDERAL CONTACT: Emily Cooke

PHONE NUMBER: (202) 205-8709

PIC ID: 5851

PERFORMER ORGANIZATION: CSR, Incorporated, Washington, DC
 

TITLE: Final Evaluation Report for the Case Management Enhancement Project at the East Orange District Office of the New Jersey Division of Youth and Family Services

ABSTRACT NUMBER: 005

ABSTRACT: In mid-1992, New Jersey's Division of Youth and Family Services (DYFS) requested applications that might demonstrate improvements in child welfare service delivery and program cost effectiveness through the computerization of many caseworkers' record keeping tasks. The East Orange District Office is located in a municipality contiguous to Newark, the State's largest city and, in mid-1992, supervised about 1,500 children in 760 families. This report describes the extent to which computerization changed the allocation of DYFS caseworkers' time. In the half-year leading up to the project's start, the East Orange District Office was noteworthy for having added a disproportionately large number of children to its caseload and for having placed disproportionately high percentages of these children in foster care for relatively long periods of time. During the project's life span, these trends were reversed, due to a combination of a new District Office manager and the introduction of a number of operational changes including computerization of the office. The report concludes that, between 1993 and 1995, the establishment of a new management team in the office, which stressed heightened accountability, tighter controls, and closer monitoring of individual staff performance, was largely responsible for eliminating previous delays in the disposition of new referrals and in reducing caseloads. The report postulates that these new procedures and a new work ethos were more responsible for beneficial trends than computerization. Furthermore, the time workers spent on various record-keeping activities did not decline between 1993 and 1995 as a consequence of automation, but increased at first as workers learned how to use the new computers. Thus, while computerization has no clear near-term benefits, it can be instituted as one of several components in a wide-ranging office reorganization.

AGENCY SPONSOR: Administration on Children, Youth, and Families

FEDERAL CONTACT: James Dolson

PHONE NUMBER: (202) 401-0133

PIC ID: 6397

PERFORMER ORGANIZATION: A.L. Nellum and Associates, Washington, DC
 

TITLE: Final Report: National Evaluation of Home-Based Services Programs for Runaway Youth

ABSTRACT NUMBER: 006

ABSTRACT: This report presents the results of five demonstration programs that sought to develop and implement home-based service models for at-risk youth. The first set of demonstrations, funded beginning in February 1989, developed and tested model programs of treatment for dysfunctional families to enable the family to remain the primary caregiver while protecting the interests of at-risk youth. The demonstrations were based in Baltimore, Maryland, and Nashville, Tennessee. The second set of demonstrations, funded beginning in September 1991, were based in Kauai, Hawaii; San Diego, California; and Tucson, Arizona. These programs implemented models to address and prevent runaway behavior among adolescents. The report finds that all projects provided families with relatively short-term interventions with the intent of linking them to existing resources in the community for longer-term services and generally served youth between 12 and 17 years of age. The projects encountered several problems with the provision of these services, including multiproblem families, a lack of community resources, a lack of support from other community agencies, and staff turnover. Effective service practices included employing bilingual and culturally sensitive staff; round-the-clock staff availability; and using a co-therapist approach. Furthermore, the report finds that (1) only the Kauai evaluation examined an array of participant outcomes; it reported success in preventing out-of-home placements for youth, decreased incidence of runaway and truant behaviors, improved family functioning, and improvements in child behaviors and emotional well-being; (2) the Nashville evaluation, although it used a control group, only assessed the outcomes of out-of-home placements, it was successful in preventing out-of-home placements; and (3) projects need more guidance on evaluation to ensure that cross-site analysis is possible. (Final report: variously paginated, plus appendices.)

AGENCY SPONSOR: Administration on Children, Youth, and Families

FEDERAL CONTACT: James Griffin, Ph.D.

PHONE NUMBER: (202) 205-8138

PIC ID: 5847

PERFORMER ORGANIZATION: KRA Corporation, Columbia, MD
 

TITLE: Gang Families in a Public Housing Project

ABSTRACT NUMBER: 007

ABSTRACT: This project examined families containing multiple gang members in a low-income Mexican-American community in Los Angeles. The project considered the relationships among macrostructural and economic forces and household organization, family childrearing practices, sibling/relational influences, socialization of street children, culture and traditions, and levels of acculturation, especially in the colonization or marginalization process. The study identified family patterns and processes leading to gang membership. Additionally, the study provided a better understanding of family dynamics in families with more than one gang member, how childrearing practices and street culture are transmitted to children, and how gang habits and values are transmitted among family members. Project findings were presented in a report targeted to service providers, as well as in a summary targeted to the general public. Famlies and individuals participating in the study were informed of the findings and referred to counselors and substance abuse groups. See also PIC ID No. 50421.1.

AGENCY SPONSOR: Administration on Children, Youth, and Families

FEDERAL CONTACT: Maria Candamil

PHONE NUMBER: (202) 205-8054

PIC ID: 5042

PERFORMER ORGANIZATION: James Diego Vigil, Whittier, CA
 

TITLE: Length of Service and Cost-Effectiveness in Three Intensive Family Service Programs: Final Report

ABSTRACT NUMBER: 008

ABSTRACT: This project examines the impact of the length of service provision on outcomes in family preservation programs located in Portland, Oregon; Pendleton, Oregon; and Baltimore, Maryland. While there is widespread anecdotal and legislative support for time-limited provision of family services, there are few scientific data on how or why limiting service delivery is more effective in family preservation programs. Thus, this project develops an experimental design to test the effect of length of service on client outcomes and cost-effectiveness. Families were randomly assigned to 3- and 6-month service contracts and, in one site, to a control condition with no specified length of service. Location of services (home or office), target populations, and staffing pattern (teams or individual workers, combined or separate case management) varied among study sites. In the Baltimore site, 193families participated; in Pendleton, 79 families participated; In each site, two equal groups were assigned to 3-month and 6-month contracts. In Portland, of the 188-family total, 60families were assigned to 3-month contracts, 55 to 6-month contracts, and 73 to the "no time limit" group. The report finds that, overall, study participants experienced low out-of-home placement rates, low rates of maltreatment, and significant improvement in family, parent, and child functioning. The only significant instance of length of service affecting outcomes was seen in the Portland site, where families in the 6-month group had a much lower placement rate in the year following the study than families in the 3-month group. Low-income families and those with child behavior problems achieved better outcomes in the 6-month group, even though most families preferred indeterminate time frames for service receipt. The report concludes that 6-month periods of in-home family treatment provided to families with older children and significant histories of prior services were most effective.

AGENCY SPONSOR: Administration on Children, Youth, and Families

FEDERAL CONTACT: Cecelia Sudia

PHONE NUMBER: (202) 205-8764

PIC ID: 4386

PERFORMER ORGANIZATION: University of Iowa, School of Social Work, IowaCity,IA
 

TITLE: Social and Educational Development of Tribal-Based Communities of the Sonoran and Negev Deserts

ABSTRACT NUMBER: 009

ABSTRACT: This project transferred an innovative, cooperative community-university model for empowering economically disadvantaged minority communities to respond to the unique needs of their children at risk of leaving school early. The model focuses on dropout prevention for preteen Bedouin Arab students in a tribal community in the Negev Desert of Southern Israel. This demonstration model has been transferred to the Pascua Yaquis in a tribal community in the Sonoran Desert of Southern Arizona. The model includes components for (1) in-school linguistically and culturally appropriate instruction; (2) community-based after-school programs; and (3) family education and training. The social service model refined and disseminated by this project addresses the dire needs of at-risk youth through the use of a community-based approach. The University of Arizona and Ben-Gurion University of the Negev committed their resources for technical assistance to the two communities in order to firmly establish such a capability. In keeping with the tenets of a true community-based approach, the transfer of the model from Israel to the United States has been spearheaded by the two indigenous communities themselves, through the Rahat Council and the Pascua Yaqui Tribe, with the universities playing only a support role. The project is family centered, focusing on locally developed educational and support strategies emanating from the participants, strategies which stress their rights and responsibilities in assuming leadership roles in decision making. The final project report is expected to influence policy and program decisions affecting socially and economically disadvantaged youth at risk of early school leaving.

AGENCY SPONSOR: Office of Policy and Evaluation

FEDERAL CONTACT: James Dolson

PHONE NUMBER: (202) 401-0133

PIC ID: 5978

PERFORMER ORGANIZATION: University of Arizona, College of Education, Tucson,AZ
 

TITLE: Survey of Head Start Family Self-Sufficiency Initiatives

ABSTRACT NUMBER: 010

ABSTRACT: The Head Start Bureau within the Administration for Children, Youth, and Families (ACYF) launched the Survey of Head Start Family Self-Sufficiency Initiatives in order to explore ways in which programs could work with families to improve their employability, literacy, and substance abuse. The report presents findings on self-sufficiency activities provided directly by Head Start programs as well as those presented indirectly through collaboration with local service providers. The report finds that: (1) Head Start programs use formal surveys and assessments to identify a family's need for literacy and employability services, but assess the need for substance abuse services more informally; (2) most program directors perceive the greatest need for employability services and the least need for substance abuse services; (3) programs meet families' literacy needs by providing information and educational materials, as well as assessment and identification of problems; outside agencies provide individual tutoring and group instruction; (4) programs provide employability services consisting of volunteer opportunities, vocational training, referrals to postsecondary education, and school or job application assistance; outside agencies provide career assessments and job-readiness classes; (5) programs address substance abuse needs by conducting educational and prevention activities or by providing assessment services; outside agencies provide support groups, assessments, education, and prevention activities; and (6) programs devote resources of staff time and training, space for activities, transportation activities, and child care assistance to help families with literacy, employability, and substance abuse needs. The report provides specific recommendations for how staff can meet the needs of families in areas that promote self-sufficiency. See also PIC ID No. 4983.1 and 4973.2. (Final report variously paginated plus appendices.)

AGENCY SPONSOR: Administration for Children, Youth, and Families

FEDERAL CONTACT: John Corrigan

PHONE NUMBER: (202)205-8403

PIC ID: 4973

PERFORMER ORGANIZATION: CSR, Incorporated, Washington, DC
 

TITLE: Survey of Head Start Family Self-Sufficiency Initiatives: Case Studies in Six Communities

ABSTRACT NUMBER: 011

ABSTRACT: This report presents the findings of six case studies carried out as part of the Survey of Head Start Family Self-Sufficiency Initiatives. This survey explored ways in which programs could work with families to improve their employability, literacy, and substance abuse. The case studies have been stripped of all identifying information in order to preserve the anonymity of program directors, staff, and community agency representatives. The case studies show that program directors believe that their Head Start families are in need of literacy services, although the underlying causes of illiteracy vary across sites. For example, one site is located in a rural county where 32 percent of the population did not graduate from high school and two other programs serve significant non-English speaking populations. The six programs recognized and addressed employability in varying ways. One Migrant Head Start program director did not think that clients needed employability services because they were migrant workers who already had jobs. Other sites offered information and referral, but only when parents requested assistance. Although one program was more active in working on employability issues, but the program did little to address families concerns about temporary work or low-paying jobs. Most program directors were concerned about substance abuse and its effect upon families, but programs have difficulties in assessing and meeting families needs for services in this area. The report also discusses conclusions drawn from the case studies in the following areas: (1) staff perceptions of their roles; (2) identifying the most effective service; (3) barriers to families receiving services; and (4) collaborations with other organizations. See also PIC ID nos. 4973 and 4973.1. (Final report variously paginated.)

AGENCY SPONSOR: Administration for Children, Youth, and Families

FEDERAL CONTACT: John Corrigan

PHONE NUMBER: (202)205-8403

PIC ID: 4973.1

PERFORMER ORGANIZATION: CSR, Incorporated, Washington, DC
 

TITLE: Transfer of International Innovations: Development of a Clinical Monitoring System To Support Foster Care in Michigan

ABSTRACT NUMBER: 012

ABSTRACT: Children in foster care do not always receive adequate care. They may "drift" in the system and may remain cut off from their families for unnecessarily extended periods of time. The aim of the project is to enhance effective and accountable services through the transfer of an innovative Clinical Information System for Foster Care. This computerized system is based on a model combining structured and systematic monitoring of each individual child with the aggregation of this information across the whole agency. Developed on a national level in Israel, the system's use is now mandated in that country. It has been modified and expanded to fit the U.S. foster care system. The system has been tested and integrated into the metropolitan Detroit offices of a statewide foster care agency, Lutheran Child and Family Services. It has become part of that agency's routine practice procedures throughout the State. The system facilitates ongoing monitoring and prioritization of treatment for children in care, coordination and shared language among practitioners and services, and processing of collected data for policy formulation, evaluation, and research. The system provides an integrated response to the needs of all partners in the agency, including administrators, managers, and policymakers. However, the main focus of the Integrated Information System is the information needs of the clinical staff of the agency, caseworkers, and clinical supervisors. The Integrated Information System includes three seamlessly integrated system modules: Foster Care, Foster Families, and Adoption. The conceptual framework and methodology used in the current system for foster care can easily be adapted to suit the needs of other child and family programs and to improve integration among such service delivery systems as child protective services, family preservation, youth residential care, and adoption programs.

AGENCY SPONSOR: Office of Policy and Evaluation

FEDERAL CONTACT: James Dolson

PHONE NUMBER: (202) 401-0133

PIC ID: 5976

PERFORMER ORGANIZATION: Wayne State University, Merrill-Palmer Institute, Detroit, MI
 

TITLE: Women and Infant Nurturing Services (WINGS)

ABSTRACT NUMBER: 013

ABSTRACT: The Women and Infant Nurturing Services program was intended to address the spiral of increasing female substance abuse and subsequent incarceration. The program targets pregnant, substance-abusing inmates using incarceration as a point of treatment intervention. Staffed by a social worker, a nurse practitioner, and student interns, the project coordinates and enhances services presently in place at the Rose M. Singer Correctional Facility on Riker's Island in New York City and, by collaborating with community-based case managers, secures appropriate postrelease services. The in-jail programs consist of substance abuse treatment; prenatal health and nutritional care; HIV education; parenting classes; mental health services; and assistance with entitlement preparation. If a woman delivers her child while still incarcerated, she is able to remain with the child at Singer's Baby Nursery. If a woman has not delivered by the time of her release, staff will secure community medical linkages and provide transitional supportive services. Women are linked to needed postrelease social services, including appropriate day or residential community substance abuse treatment facilities designed to meet their special needs.

AGENCY SPONSOR: Office of Policy and Evaluation

FEDERAL CONTACT: James Dolson

PHONE NUMBER: (202) 401-0133

PIC ID: 5979

PERFORMER ORGANIZATION: New York Department of Corrections, MiddleVillage,NY

Administration on Aging

 

TITLE: Serving Elders at Risk: The Older Americans Act Nutrition Programs. National Evaluation of the Elderly Nutrition Program, 1993-1995

ABSTRACT NUMBER: 014

ABSTRACT: The Elderly Nutrition Program provides grants to State Agencies on Aging to support the provision of daily meals to the el- derly, specifically targeting elders who have the greatest economic and social need. The 1992 Amendments to the Older Americans Act directed the Administration on Aging and the Assistant Secretary for Planning and Evaluation to evaluate the nutrition programs funded under Titles III and VI of the act, the Elderly Nutrition Program (ENP). Title III provides funds to State units on aging to help them provide meals and nutrition services to elders in congregate or home-based settings; Title VI provides funds to tribal organizations to help them deliver the same services to older Native Americans. This series of reports evaluates (1) the program's effects on participants' nutritional status and socialization; (2) the use and effectiveness of the program; (3) program administration efficiency and effectiveness; and (4) program funding sources. The report finds that (1) people who receive ENP meals have higher daily intakes of key nutrients than similar nonparticipants; (2) ENP meals provide 40 percent to 50 percent of participants' daily intake of most nutrients; (3) participants have more social contacts per month than nonparticipants; (4) between 80 percent and 90 percent of participants have income levels below 200 percent of the poverty level, live alone, and are either overweight or underweight; (5) the average cost of an ENP meal, including the value of donated labor and supplies, under Title III is $5.17 for a congregate meal and $5.31 for a home-delivered meal; comparable costs under Title VI are $6.19 and $7.18, respectively; and (6) ENP is part of a growing network of home and community-based care services. The report concludes that ENP successfully fulfills the nutritional and social needs of the elderly. (Executive summary: 32 pages; Volume I: 313 pages; Volume II: 188 pages; and Volume III: 21 pages, plus appendices.)

AGENCY SPONSOR: Office of Program Operations and Development

FEDERAL CONTACT: Jean Lloyd

PHONE NUMBER: (202) 619-2005

PIC ID: 6175

PERFORMER ORGANIZATION: Mathematica Policy Research, Inc., Plainsboro, NJ

Agency for Health Care Policy Research

Contents

CONQUEST 1.0: Overview of Final Report and User's Guide

Develop, Apply, and Evaluate Medical Review Criteria and Educational Outreach Based Upon Practice Guidelines: Final Project Report

Oregon Consumer Scorecard (OCS) Project: Final Report

Publications of the Patient Outcomes Research Teams (PORTS), May 1996, and Medical Treatment Effectiveness Projects
 

TITLE: CONQUEST 1.0: Overview of Final Report and User's Guide

ABSTRACT NUMBER: 015

ABSTRACT: This project develops a prototype computer-based framework for identifying, evaluating, and selecting clinical performance measures. The Computerized Needs-Oriented Quality Measurement Evaluation System (CONQUEST 1.0) consists of two interlocking data bases, one containing information on clinical performance measures and a second on clinical conditions. Information on attributes of clinical performance measures--such as the clinical rationale for a measure, requirements for constructing the denominator and numerator, the purpose and content of the measure, analysis considerations, and reliability and validity testing--is contained in the "measures" data base. The measures data base allows users to directly compare performance measures on one or more attributes in order to select the measures most appropriate to their needs. The condition data base synthesizes information for guidelines supported by the Agency for Health Care Policy and Research (AHCPR), as well as those developed by other organizations, and medical effectiveness research so that users can access scientific information to guide their selection of the most meaningful measures for their specific client populations. CONQUEST 1.0 was developed under the leadership of R. Heather Palmer, M.B., B.C.H., of the Center for Quality of Care Research and Education at the Harvard School of Public Health. The data base is available through AHCPR's Internet web site: http://www.ahcpr.gov. The User's Guide is available through the Policy Information Center.

AGENCY SPONSOR: Center for Quality Management and Improvement

FEDERAL CONTACT: Marge Keyes

PHONE NUMBER: (301) 594-1352

PIC ID: 5961

PERFORMER ORGANIZATION: Mikalix and Company, Lexington, MA
 

TITLE: Develop, Apply, and Evaluate Medical Review Criteria and Educational Outreach Based Upon Practice Guidelines: Final Project Report

ABSTRACT NUMBER: 016

ABSTRACT: This project evaluates three Agency for Health Care Policy and Research (AHCPR) clinical practice guidelines by assessing their usefulness for quality measurement of improvement purposes. The project derived medical review criteria and clinical performance measures for the guidelines on postoperative pain management (POP), urinary incontinence (UI), and benign prostatic hyperplasia (BPH); it also developed data specifications, and data collection and analysis procedures for the guidelines. The project also implemented and evaluated the BPH guideline-based performance measures using alternative quality improvement strategies. An earlier report (PIC ID No. 5973) presents findings related to the guidelines on POP and UI; this report examines BPH and educational outreach. The impact of the guidelines was examined in four ambulatory care settings in four states. The report finds that review criteria could be derived from the BPH guideline and that other elements of a valid, reliable performance measurement system could be developed. Furthermore, the application of this system to a sample of ambulatory records demonstrated that this highly structured, explicit review system is reliable and provides detailed information about conformance with the guidelines. The report examines the impact of the educational intervention on physicians' practice and attitudes. It finds that: (1) significant performance rate changes followed the educational intervention for some BPH criteria; (2) few differences in behavioral changes between urologists and primary care practitioners were observed; and (3) those physicians who received only the criteria and clinical practice guideline, rather than the educational intervention, did not exhibit marked changes in behavior. See also PIC ID No. 5973. (Final report 77 pages plus appendices.)

AGENCY SPONSOR: Office of Policy and Evaluation

FEDERAL CONTACT: Phyllis Zucker

PHONE NUMBER: (301) 594-1454

PIC ID: 5973.1

PERFORMER ORGANIZATION: Center for Clinical Quality Evaluation, Washington, DC
 

TITLE: Oregon Consumer Scorecard Project: Final Report

ABSTRACT NUMBER: 017

ABSTRACT: This project supported the State of Oregon in the development of a prototype scorecard that could assist consumers in selecting a health plan that best suits their needs. It also determined the extent to which such a scorecard could serve as a comparative performance measurement tool providing feedback to health plans, purchasers, and State policymakers about how well health plans are performing and meeting the expectations and needs of their consumers. The scorecard is designed to assist consumers in two ways: by providing comparative health information and by serving as an educational tool on health plan operations for consumers. The project provided a range of information about the gaps in current quality measurement efforts, the need to tailor information to a variety of consumer needs, and issues related to the interpretation of data. The project also collected information on health plan management issues, consumer survey information, and qualitative information through consumer focus groups. The report finds that (1) consumers were highly receptive to the materials presented and were eager for objective choice materials; (2) a major challenge confronting the project was how to present comparative health plan data on consumer satisfaction and performance that were meaningful, understandable, and reliable; (3) there are differences between how professionals and consumers think about quality; (4) consumers expressed a strong preference for choice information that was personally relevant and geographically sensitive; (5) uniform data specifications will help to consolidate and reduce the data burden on health plans; (6) information about the severity of condition or risk adjustment is needed if consumers are to get reliable and accurate comparative information about health plans; (7) consumers prefer multimedia presentations of the scorecard; and (8) the costs of producing a scorecard are formidable. (Final report: 51 pages, plus appendices.)

AGENCY SPONSOR: Office of Policy and Evaluation

FEDERAL CONTACT: Sandra Robinson

PHONE NUMBER: (301) 594-1349

PIC ID: 5960

PERFORMER ORGANIZATION: Oregon Health Policy Institute, Portland, OR
 

TITLE: Publications of the Patient Outcomes Research Teams (PORTS), May 1996, and Medical Treatment Effectiveness Projects

ABSTRACT NUMBER: 018

ABSTRACT: This project tracked the activities and effects of research conducted under the Medical Effectiveness Treatment Program (MEDTEP). Using secondary data sources such as newspaper and journal articles, the project collected information on the products of Agency for Health Care Policy and Research-sponsored medical effectiveness research and the diffusion of these findings into other sources. A bibliography entitled Publications of the Patient Outcomes Research Teams (PORTS): PORT and PORT II Projects provides information about publications arising from each PORT, listed by the condition that each addressed. Also included is information about the grant or contract number, project title, name and institution of the principal investigator, project start and completion dates, and a brief description of the project. A second bibliography entitled Medical Treatment Effectiveness Program Research Projects: 1989 through 1995 presents a comprehensive list of MEDTEP research projects. Projects are categorized under Clinical Studies, with two subcategories: Guideline Implementation and Evaluation; and Methods and Measures. Within each category, projects are in alphabetical order by the name of the principal investigator. PORT and PORT II projects are marked. As in the first bibliography, information includes the grant or contract number, project title, name and institution of the principal investigator, project start and completion dates, and a very brief description of the project. For those projects that have been completed, executive summaries and final reports may be available through the National Technical Information Center. (PORTS bibliography: 49 pages; MEDTEP bibliography: 73 pages.)

AGENCY SPONSOR: Center for Outcomes and Effectiveness Research

FEDERAL CONTACT: Mary Cummings

PHONE NUMBER: (301) 594-1485

PIC ID: 4311

PERFORMER ORGANIZATION: Walcoff and Associates, Inc., Fairfax, VA

Centers for Disease Control and Prevention

Contents

Assessment of Accountability Procedures for the Vaccines for Children Program: Final Report

Evaluation of the Community-Based Mosquito-Control Programs for Dengue Hemorrhagic Fever (DHF) Prevention and Control Program at the San Juan Laboratories, Puerto Rico

Evaluation of the NNIS System: Final Study Report

Georgia State Assessments of Immunization Coverage: Impact of Management and Clinic Immunization Practices (Three Volumes)

Key Monitoring Indicators of the Nation's Health and Health Care and Their Support by NCHS Data Systems

Private Sector Health Care Organizations and Public Health: Potential Effects on the Practice of Local Public Health
 

TITLE: Assessment of Accountability Procedures for the Vaccines for Children Program: Final Report

ABSTRACT NUMBER: 019

ABSTRACT: This study provided information to assist in developing an appropriate accountability system for the Vaccines for Children (VFC) program that is both rigorous and practical for use in clinical settings. As part of this study, data collection procedures for proposed systems were tested to determine their practicality and usefulness for collecting vaccine usage data. The study also provided information about the acceptability of any accountability system to health care providers, since a system that is perceived as overly burdensome could discouarge participation in the VFC program.

AGENCY SPONSOR: National Immunization Program

FEDERAL CONTACT: Wesley Hodgson

PHONE NUMBER: (404) 639-8222

PIC ID: 5935

PERFORMER ORGANIZATION: Battelle Centers for Public Health Research and Evaluation, Alexandria, VA
 

TITLE: Evaluation of the Community-Based Mosquito Control Programs for Dengue Hemorrhagic Fever (DHF) Prevention and Control Program at the San Juan Laboratories, Puerto Rico

ABSTRACT NUMBER: 020

ABSTRACT: Dengue hemorrhagic fever (DHF) is a mosquito-transmitted viral disease that has been present in the Americas for the last 200 years. There have been two major epidemics of DHF in recent years (Cuba, 1981, and Venezuela, 1989-90), and sporadic cases have been documented in 12 other locations in the hemisphere, including Puerto Rico. DHF causes severe headache, bone and joint pain, and prostration. In some cases, it may produce hemorrhagic manifestations and death. This report assesses a comprehensive cross-sectional evaluation of the impact of the community-based dengue prevention pilot programs in Puerto Rico on (1) dengue-related knowledge; (2) human behavior; and (3) indices of infestation of the mosquito (Aedes aegypti) carrying the disease. Surveys of knowledge and practices were administered to children and their parents, and surveys of the yards and homes of parents were conducted to describe the number and type of larval habitats of the Aedes aegypti mosquito. Focus groups and in-depth interviews were also conducted. Surveys for mosquito larvae were conducted in 703 premises across five different study sites from April to June 1995. The report finds that (1) overall, 18.3 percent of the premises had at least one container positive for Aedes aegypti larvae; 24.9 percent had at least one container positive for some type of mosquito larvae; (2) larval indices were higher in more rural housing; and (3) the most common containers for Aedes aegypti larvae were water-storage containers, plant-related containers, tires, small disposable containers, and animal drinking dishes. The report also finds that exposure to dengue prevention programs in Head Start, elementary schools, and the Children's Museum are associated with both higher levels of correct knowledge and lower levels of incorrect knowledge about mosquito life cycles, modes of disease transmission, larval container habitats, and control measures. Children also communicated their learning to their parents.

AGENCY SPONSOR: National Center for Infectious Diseases

FEDERAL CONTACT: Jose Rigau

PHONE NUMBER: (787) 766-5181

PIC ID: 5355

PERFORMER ORGANIZATION: Johns Hopkins University, School of Hygiene and Public Health, Baltimore, MD
 

TITLE: Evaluation of the NNIS System: Final Study Report

ABSTRACT NUMBER: 021

ABSTRACT: The Centers for Disease Control and Prevention (CDC), National Center for Infectious Diseases conducts the National Nosocomial Infections Surveillance (NNIS) System to create a national data base of infections associated with hospitalization in general adult and pediatric acute care hospitals in the United States. Therefore, it is important to develop an accurate method for determining the current rate of nosocomial infection within the entire NNIS system or within a specific hospital. This study develops a methodology for assessing NNIS hospitals' accuracy in identifying and reporting nosocomial infections in NNIS hospitals; it also assesses the utility of criteria used to diagnose these infections. A pilot test examined nosocomial infections in two hospitals, while Phase II of the study uses retrospective medical record reviews in eight hospitals located in the eastern United States. Infection control practitioners (ICPs) reviewed 1,252 records from the ten hospitals; a total of 1,326 infections were identified in 691 medical records. Pneumonia was the most prevalent infection (36.6 percent), followed by other infection sites (23.2 percent), urinary tract infections (18.7 percent), blood stream infections (12.8 percent), and surgical site infections (8.7 percent). The effectiveness of infection control practitioner performance was reviewed in cases where discrepancies between observers occurred. Of the 3,052 infection sites recorded on the initial reviews, 61 percent did not require adjudication. Nineteen percent of the remainder were judged to be false negative; 14 percent were false positives; 3.9 percent were misclassified; 1.3 percent were subject to a date discrepancy; and 0.1 percent were both misclassified and subject to a date discrepancy. The report recommends that NNIS manual criteria be updated or modified in regard to (1) the determination of the site of nosocomial infections and (2) pathogen codes. (Executive summary: 9 pages.)

AGENCY SPONSOR: National Center for Infectious Diseases

FEDERAL CONTACT: T. Grace Emori, R.N.

PHONE NUMBER: (404) 639-6438

PIC ID: 5349

PERFORMER ORGANIZATION: Analytical Sciences, Inc., Durham, NC
 

TITLE: Georgia State Assessments of Immunization Coverage: Impact of Management and Clinic Immunization Practices (Three Volumes)

ABSTRACT NUMBER: 022

ABSTRACT: One of the goals of the Public Health Service by the year 2000 is to have 90percent of preschool children immunized by the time they are 24 months of age. The diseases against which immunization is sought are diphtheria, tetanus, pertussis, poliomyelitis, measles, mumps, rubella, hemophilius influenza type B, and hepatitis B. In an effort to improve immunization rates, the State of Georgia conducted annual audits of clinic records from 1986 to 1994 to assess the coverage rates in public immunization clinics. This study evaluates how these audits have affected immunization coverage levels in the preschool population and the medical and management policies and practices that influence immunization rates. All 227 immunization clinics in Georgia were surveyed and audited. Thereport finds that between 1987 and 1993, the State's immunization rate improved from 31 percent to 90 percent. Analysis shows that participation in audits is associated with higher immunization rates. The report recommends that (1) comprehensive efforts to improve immunization rates should continue; (2) a system of periodic clinic audits should be instituted; (3) access to clinics should be assured; (4) early outreach and reminder procedures should be used; and (5) adequate staff and financial resources should be committed. Volume II contains appendices to the study, while Volume III contains computer documentation. (Final report: Volume I: 195 pages; Volume II: variously paginated; and Volume III: 359 pages.)

AGENCY SPONSOR: National Immunization Program

FEDERAL CONTACT: Andrew Baughman

PHONE NUMBER: (404) 639-8239

PIC ID: 5354

PERFORMER ORGANIZATION: Battelle Centers for Public Health Research and Evaluation, Alexandria, VA
 

TITLE: Key Monitoring Indicators of the Nation's Health and Health Care and Their Support by NCHS Data Systems

ABSTRACT NUMBER: 023

ABSTRACT: This project evaluates the adequacy and appropriateness of information collected in the National Center for Health Statistics (NCHS) data systems to support key monitoring indicators for health reform. The results of the evaluation are intended as guidance in strengthening and revising NCHS data systems to meet the need for a widely accepted set of key monitoring indicators for the Nation. The report notes that over the course of the project, major shifts in Federal policy focus and political support for broad health care reform occurred. Instead of a Federal-level program, much of the responsibility for health care reform is being shifted to the State level. Ongoing changes in public priorities and in private payer and provider initiatives make the need for good information all the more critical for policymakers. The report concludes that criteria for evaluating the quality of key health indicators, including data relevance and reliability, must address fundamental questions about health and the health care system components that will withstand rapid system change. Basic questions about indicators and information needs are a critical element in identifying a robust set of system components to be monitored, regardless of the ultimate shape, scope, and level of public funding of health programs. Public debate and shifts in the marketplace make the immediate need for accurate information systems all the more pressing. Baselines for future benefits must be developed if the positive or negative effects of health system reform are to be measured. The report: also addresses specific indicator identification and NCHS data systems evaluation, primarily in the area of population health and personal health care services. (Final report: 75 pages, plus appendices.)

AGENCY SPONSOR: National Center for Health Statistics

FEDERAL CONTACT: Ronald Wilson

PHONE NUMBER: (301) 436-7032

PIC ID: 5500

PERFORMER ORGANIZATION: The Lewin Group, Fairfax, VA
 

TITLE: Private Sector Health Care Organizations and Public Health: Potential Effects on the Practice of Local Public Health

ABSTRACT NUMBER: 024

ABSTRACT: This study examines the emerging role of private health care providers, especially health maintenance organizations, in delivering public health services. It takes as its conceptual framework the Institute of Medicine model of 3 core public health functions, and the Public Health Service elaboration of these functions into 10 essential public health services. The study identifies and presents case studies of private health care organizations providing these services. It synthesizes findings from site visits to eight health systems and managed care organizations that were identified as being active in public health and population-based activities in their communities. The report finds that (1) service delivery and assurance are two key essential services most likely to be conducted in the private sector; (2) private-sector health care delivery organizations are active in all three core public health functions (assessment, policy development, and assurance); (3) private organizations take on public health functions for a number of reasons, including altruism and business-related reasons; and (4) almost 40 percent of the public health activities of private organizations are carried out by the organization alone. Finally, although the report praises private efforts rooted in altruism, it cautions that the separation of these public health activities from the business strategy of an organization may mean that they are eliminated in an era when private health care organizations are increasingly pressured by the bottom line. (Executive summary: 15 pages; Final report: 39 pages, plus appendices.)

AGENCY SPONSOR: Office of Program Planning and Evaluation

FEDERAL CONTACT: Nancy Cheal

PHONE NUMBER: (404) 639-7095

PIC ID: 6213

PERFORMER ORGANIZATION: Macro International, Inc., Atlanta, GA

Food and Drug Administration

Contents

Fourth Annual Performance Report: Prescription Drug User Fee Act of 1992--Fiscal Year 1996 Report to Congress

Impact of the NLEA on Consumers: Recent Findings From FDA's Food Label and Nutrition Tracking System

Status of Nutrition Labeling of Processed Foods--1995: Food Label and Package Survey (FLAPS)
 

TITLE: Fourth Annual Performance Report: Prescription Drug User Fee Act of 1992--Fiscal Year 1996 Report to Congress

ABSTRACT NUMBER: 025

ABSTRACT: FY 1996 marks the fourth year of the 5-year implementation of the Prescription Drug User Fee Act (PDUFA) of 1992, which authorizes the collection of fees from the pharmaceutical industry in order to facilitate the Food and Drug Administration's (FDA's) timely review of human drug applications. The Act mandates that the FDA prepare reports each fiscal year for which fees are collected. The fourth annual report, covering FY 1996, was prepared in response to that mandate, presenting findings on how well the FDA has accomplished its performance goals for FY 1996. These performance-related goals were jointly established by the FDA and the pharmaceutical industry. They fall into three main categories: (1) eliminating overdue backlogs; (2) building excellence into the review process; and (3) achieving high performance. Eighteen of the goals have already been accomplished and reported on in previous years' performance reports. Four goals were in effect in 1996, while seven will be in effect for 1997. The report finds that the fourth year of PDUFA was successful, in that FDA has exceeded every PDUFA performance goal, despite its increasing workload. A record number of new product approvals and shorter review times during FY1996 enabled new products to reach the market faster.

AGENCY SPONSOR: Office of Planning and Evaluation

FEDERAL CONTACT: Paul Coppinger

PHONE NUMBER: (301) 443-4230

PIC ID: 6079.1

PERFORMER ORGANIZATION: Office of Planning and Evaluation, FDA, Rockville,MD
 

TITLE: Impact of the NLEA on Consumers: Recent Findings From FDA's Food Label and Nutrition Tracking System

ABSTRACT NUMBER: 026

ABSTRACT: The Nutrition Labeling and Education Act (NLEA) of 1990 was intended to provide consumers with better nutrition information on food labels and to enable them to make more healthful food choices. Regulations to implement NLEA were issued by the Food and Drug Administration in January 1993. This report examines the impact of the NLEA on consumers, using several measures of behavior. The report finds that NLEA caused a number of significant changes in reported label use practices and beliefs of the general public, suggesting that it is easier for consumers to use the quantitative nutrition information on the food label. For example, consumers say that they most often use the Nutrition Facts panel of the label, which contains quantitative information about nutrient amount and is the part of the label that showed the greatest increase in reported use. Consumers use the Nutrition Facts panel to see if a product is high or low in a particular nutrient, to judge its overall nutrition content, and to compare the nutrition profile of different brands. Furthermore, consumers report greater confidence in the veracity of nutrition and health claims made on labels. The report also finds that NLEA has had a significant impact on the availability and market share of food promoted on the basis of health and nutrition. In several instances, fat-modified versions of foods have been introduced to the market. For example, fat-modified cookies had a market share of 15 percent in 1995, up from nearly 0 percent in 1990; at the same time, fat-modified cheeses more than doubled their share of cheese sales. Finally, NLEA has also helped facilitate the flow of useful nutrition education messages to the public. Public awareness of, and interest in, dietary risk factors has steadily increased, indicating the continuing importance of the food label as a source of accurate nutrition information. See also PIC ID No. 4988. (Final report: 6 pages, plus appendix.)

AGENCY SPONSOR: Center for Food Safety and Applied Nutrition

FEDERAL CONTACT: Brenda Derby

PHONE NUMBER: (202) 205-5363

PIC ID: 4988.1

PERFORMER ORGANIZATION: Center for Food Safety and Applied Nutrition, FDA, Washington, DC
 

TITLE: Status of Nutrition Labeling of Processed Foods--1995: Food Label and Package Survey (FLAPS)

ABSTRACT NUMBER: 027

ABSTRACT: The Food Label and Package Survey (FLAPS) provides a data base containing label and package information recorded from the packages of a scientifically derived sampling of food products representative of the U.S. processed packaged food industry. The 10th FLAPS survey reflects the status of processed, packaged food labels subsequent to the publication of regulations promulgated in response to the Nutrition Labeling and Education Act of 1990. It focuses on the percentage of packaged foods sold annually that bear quantitative nutrition labeling, as well as on data on health claims, nutrient content claims, and ingredients. The survey found that (1) nutrition-labeled products account for about 96.1 percent of the annual sales of processed, packaged foods across 58 major product groups; (2)there was a 27.2 percent increase in the proportion of nutrition-labeled products between 1993 and 1995; (3) 100 percent of processed meat products surveyed are nutrition-labeled, an increase of 47.9 percent over 1993; (4) 44 of 58 major product groups have nutrition labels for all of the brands in the sample; (5) three product groups have less than 90 percent of the products sold with nutrition labels (tea; coffee; salt, seasoning, and spices)--all these items are exempt from the requirement to display nutrition labels because they contain insignificant amounts of all nutrients; and (6) there were increases of more than 30 actual percentage points between 1993 and 1995 for 14 nutrition-labeled product groups. See also PIC ID No. 5711. (Final report: 19 pages.)

AGENCY SPONSOR: Center for Food Safety and Applied Nutrition

FEDERAL CONTACT: Tom O'Brien

PHONE NUMBER: (202) 205-5432

PIC ID: 5711.1

PERFORMER ORGANIZATION: Center for Food Safety and Applied Nutrition, FDA, Washington, DC

Health Care Financing Administration

Contents

Assessment of the Impact of Medicaid Drug Rebate Policy on Expenditures, Utilization, and Access

Assessment of the Impact of Pharmacy Benefit Managers

Effects of Predetermined Payment Rates for Home Health Care

Evaluating the Effects of Physician Payment Reform on Access: Time Series Analyses of Hospitalizations for Ambulatory-Care Sensitive Conditions

Evaluation of Arizona's Health Care Cost-Containment System Demonstration: Final Report

Evaluation of the Home Health Prospective Payment Demonstration

Evaluation of the Medicare SELECT Amendments: Case Study Report

Evaluation of the Medicare SELECT Amendments: Final Evaluation Report

Sustainable Support System for Telemedicine Research and Evaluation

Trends in Access to Health Care Services for Selected Segments of the Medicare Population

Ventilator-Dependent Unit Demonstration: Outcome Evaluation and Assessment of Post-Acute Care
 

TITLE: Assessment of the Impact of Medicaid Drug Rebate Policy on Expenditures, Utilization, and Access

ABSTRACT NUMBER: 028

ABSTRACT: The purpose of this study was to assess the impact of the Medicaid drug rebate program on expenditures, utilization, and access to medications for Medicaid recipients. The study used a decomposition analysis to determine the change in total drug expenditures before and after implementation of the Medicaid drug rebate program. The role of covered population changes, intensity (utilization rate) changes, changes in efficiency (drug product prices), changes in dispensing fees and rebates, and administrative costs were evaluated. The impact on recipient access was assessed by constructing a person-level file of prescription drug claims both pre- and post-Omnibus Budget Reconciliation Act 1990 (legislation that mandated the drug rebate program).

AGENCY SPONSOR: Office of Research and Demonstrations

FEDERAL CONTACT: Kathleen Gondek

PHONE NUMBER: (410) 786-7765

PIC ID: 6319

PERFORMER ORGANIZATION: University of Minnesota, Minneapolis, MN
 

TITLE: Assessment of the Impact of Pharmacy Benefit Managers

ABSTRACT NUMBER: 029

ABSTRACT: Pharmacy Benefit Management (PBM) companies apply managed care principles to prescription drug programs to ensure optimal and cost-effective drug prescribing and use. The industry has furnished clients with administrative efficiencies and drug program savings through retail pharmacy contracts that provide discounts on drug prices and dispensing fees, manufacturer rebates, increased generic substitution, and drug utilization review. This report characterizes PBM's; compares costs and the quality of care in providing pharmacy benefits in Medicaid programs and PBM's for privately insured or Medicaid enrollees in managed care; and analyzes the potential impact of PBM's on Medicaid and other government programs and on the larger pharmaceutical market. The report finds that (1) PBM's provide both administrative functions and drug use control activities; and (2)PBM's are dispersed throughout the nation. Seventy-one PBM's cover fewer than 5million clients and a dozen cover more than 10 million clients; 6 PBM's have between 5 and 10 million clients. Seven PBM's participated in a case study to determine the character and impact of PBM's. The report concludes that the PBM industry is complex and dynamic and that it has been successful in several areas of drug program management, including techniques directed toward clients, pharmacists, and prescribers. PBM's offer government programs like Medicaid advantages in their extensive provider networks, success in extracting favorable market reimbursement rates from pharmacies, and sophisticated claims processing and data management systems. Disadvantages include lower rebates and the uniqueness and size of the Medicaid population. PBM's have the potential to affect pharmacies and pharmaceutical manufacturers by influencing their revenue streams, which has potential long-range effects on industry consolidation and reduced access or quality.

AGENCY SPONSOR: Office of Research and Demonstrations

FEDERAL CONTACT: Kathleen Gondek

PHONE NUMBER: (410) 786-7765

PIC ID: 6395

PERFORMER ORGANIZATION: University of California, Institute for Health Policy, SanFrancisco, CA
 

TITLE: Effects of Predetermined Payment Rates for Home Health Care

ABSTRACT NUMBER: 030

ABSTRACT: The Home Health Per-Visit Prospective Payment Demonstration pays home health care agencies a prospectively set rate for home health visits, thus providing an incentive to these agencies to control their costs of delivering Medicare home health visits. This report examines 47 home health care agencies. It shows that this incentive was largely overwhelmed by the current home health care environment, which is characterized by diversity, change, and competitive pressures. Nonetheless, the opportunity to earn a profit and the increased possibility of losses may have slightly increased the level of attention agencies gave to cutting costs. The agencies randomly assigned to prospective rate setting were more likely than the control group to hold their cost increases below inflation, enabling three-fourths of the prospective payment group to earn profits. However, the average size of the real cost reduction was small (about 4 percent). These effects were limited to agencies that served predominantly Medicare patients and were not controlled by a hospital or hospice. Prospective rate setting had no discernable effect on the number of visits provided by agencies or on patients' other Medicare costs, quality of care, access to care, or use of services not covered by Medicare. (Final report: 57 pages.)

AGENCY SPONSOR: Office of Research and Demonstrations

FEDERAL CONTACT: Elizabeth Mauser, Ph.D.

PHONE NUMBER: (410) 786-6665

PIC ID: 6396

PERFORMER ORGANIZATION: Mathematica Policy Research, Inc., Plainsboro, NJ
 

TITLE: Evaluating the Effects of Physician Payment Reform on Access: Time Series Analyses of Hospitalizations for Ambulatory-Care Sensitive Conditions

ABSTRACT NUMBER: 031

ABSTRACT: This project evaluated the effects of physician payment reform (PPR) on access to care in the Medicare population by studying patterns of hospitalization for ambulatory-care sensitive conditions (ASC). If there is a decrease in access in needed ambulatory care services associated with PPR, one would expect to see an increase in hospitalizations for ASC's following the implementation of PPR. The project analyzed the trend in rates of hospitalization for selected ASC conditions to see whether there is a discontinuity in the time series associated with the implementation of PPR. Analyses are compiled for the trend in hospitalizations for one ASC, congestive heart failure, for the overall Medicare population and for selected vulnerable subgroups. The results indicated no significant discontinuity in hospitalizations for this condition with the implementation of PPR.

AGENCY SPONSOR: Office of Research and Demonstrations

FEDERAL CONTACT: Thomas W. Reilly, Ph.D.

PHONE NUMBER: (410) 786-2196

PIC ID: 6315

PERFORMER ORGANIZATION: Health Care Financing Administration, Baltimore, MD
 

TITLE: Evaluation of Arizona's Health Care Cost-Containment System Demonstration: Final Report

ABSTRACT NUMBER: 032

ABSTRACT: The first HCFA evaluation of the Arizona Health Care Cost-Containment System (AHCCCS), completed in 1987, focused on the acute care program. The second, while following some acute care issues, focused on the long-term care program, the Arizona Long-Term Care System (ALTCS). It indicates generally positive findings for the use of capitation in delivering care to long-term care beneficiaries and continued success for the acute care program. Use of services appears appropriate and costs are considerably lower. This report finds that implementation of ALTCS has been generally successful: The State conducts preadmission screening and controls entry to the program, while capitated contractors determine a beneficiary's placement and manage, arrange for, and pay for a full range of acute, behavioral, health, home- and community-based, and institutional services. Although administrative costs are substantially higher than comparable Medicaid programs, the report finds that these costs support managed care activities that reduce medical service costs and result in significantly smaller overall program costs. The report also examines the AHCCCS acute care program and finds that it is very successful: Cost savings and competition are increasing, services utilization is appropriate, and the information management system has stabilized. Arizona beneficiaries have fewer hospital days, fewer procedures, and more evaluation and management services than traditional fee-for-service Medicaid programs. Overall intensity of service use is similar, but Arizona's pattern of use shows a distribution of services deemphasizing the use of institutional services and specialty care. AHCCCS annual program costs are 7 percent lower than a traditional Medicaid program for the acute care program and 16 percent lower for ALTCS. See also PIC ID Nos.2509-2509.4 and the 2158 series. (Final report: 157 pages, plus appendices.)

AGENCY SPONSOR: Office of Research and Demonstrations

FEDERAL CONTACT: Joan Peterson

PHONE NUMBER: (410) 786-0621

PIC ID: 2509.5

PERFORMER ORGANIZATION: Laguna Research Associates, San Francisco, CA
 

TITLE: Evaluation of the Home Health Prospective Payment Demonstration

ABSTRACT NUMBER: 033

ABSTRACT: This project evaluates Phase I of a demonstration designed to test the effectiveness of using prospective payment methods to reimburse Medicare-certified home health agencies for services provided under the Medicare program. In Phase I, a per-visit payment method that set a separate payment rate for each of six types of home health visits (skilled nursing, home health aide, physical therapy, occupational therapy, speech therapy, and medical social services) was tested. The contractor evaluated the effects of this payment method on agency operations, service quality, and expenditures. They also analyzed the relationship between patient characteristics and the cost and use of home health services. By October 1994, all demonstration agencies had exited the demonstration. The contractor submitted a preliminary impact report based on the findings from the first year of the demonstration. The findings suggested that treatment agencies had not decreased their cost per visit, increased their total revenues and net revenues, or altered their behavior in ways that affect the quality of home health care. The article Do Preset Per-Visit Payment Rates Affect Home Health Agency Behavior?, which discusses results from Phase I of the demonstration, appeared in the Health Care Financing Review, 16(1):91-107, Fall 1994.

AGENCY SPONSOR: Office of Research and Demonstrations

FEDERAL CONTACT: Elizabeth Mauser, Ph.D.

PHONE NUMBER: (410) 786-6665

PIC ID: 6318

PERFORMER ORGANIZATION: Mathematica Policy Research, Inc., Plainsboro, NJ
 

TITLE: Evaluation of the Medicare SELECT Amendments: Case Study Report

ABSTRACT NUMBER: 034

ABSTRACT: The Medicare SELECT program is an experimental supplemental insurance policy (Medigap) that HCFA allowed to be sold in 15 States for 3 years. SELECT insurers are exempt from regulations affecting other Medigap insurers regarding the 10 standard Medigap plans, from network prohibitions against selling Medigap policies, and from antikickback regulations for Part A hospital deductions. These exemptions allow SELECT policies to restrict services to network providers (for supplemental benefits to be paid) and to negotiate discounts with hospitals. This report provides a set of descriptive case studies of 13SELECT States and a synthesis of finding across plans. The key findings from the case studies are that (1) few insurers are participating in SELECT (only 40 of the 400 HMO's and Medigap insurers in the 13 States offer SELECT; (2) Humana, Inc., and Olympic Health Management Systems are aggressively pursuing SELECT; (3) many Medigap insurers are reluctant to sell SELECT because they believe that its developmental costs are too high and that the potential return on investment is inadequate; (4) many SELECT networks include only hospitals or hospitals and pharmacies; and (5) it appears unlikely that program savings will accrue to Medicare, but beneficiaries who participate may have less expensive premiums than if they purchased identical plans without network restrictions from the same insurers. (Final report: variously paginated.)

AGENCY SPONSOR: Office of Research and Demonstrations

FEDERAL CONTACT: Sherry Terrell

PHONE NUMBER: (410) 786-6601

PIC ID: 5966

PERFORMER ORGANIZATION: Research Triangle Institute, Research Triangle Park, NC
 

TITLE: Evaluation of the Medicare SELECT Amendments: Final Evaluation Report

ABSTRACT NUMBER: 035

ABSTRACT: The Omnibus Budget Reconciliation Act of 1990 authorized an experimental type of supplemental insurance (Medigap) policy. This policy, termed Medicare SELECT, was a demonstration project limited to 15 states for 3 years, beginning in 1992. This report addresses (1) implementation of the demonstration project; (2) consumer access, satisfaction, and informed consent; (3) premium affordability; and (4) impact on Medicare costs and use. The report finds that Medicare SELECT was implemented differently in each State and often differed significantly from the model implicitly expected by the legislation (the network model). As of November 1995, there were almost half a million Medicare beneficiaries enrolled in Medicare SELECT programs in 14 of the original 15 demonstration states (Massachusetts had no enrollees). Access to services and satisfaction with policies was the same for Medicare SELECT beneficiaries as for standard Medigap policy holders. There were no significant health differences between the two groups. In terms of affordability, 65-year-old SELECT beneficiaries generally have lower premiums than their counterparts enrolled in standard Medigap plans. However, as beneficiaries age, this price advantage reverses itself, so that by age 75, SELECT beneficiaries pay more for their policies than their counterparts in standard Medigap plans. Finally, Medicare SELECT was expected to reduce aggregate health care costs in the Medicare program because providers would have an incentive to establish cost-effective provider networks. It was not expected that SELECT would have any effect on utilization or Medicare program costs. However, in the first 3 years of the demonstration, 5 States show increased Medicare program costs, 4 show decreased costs, and 2 show no change. On average, implementation of Medicare SELECT resulted in a 5.7 percent increase in program costs over the 11 States. (Final report: variously paginated.)

AGENCY SPONSOR: Office of Research and Demonstrations

FEDERAL CONTACT: Sherry Terrell

PHONE NUMBER: (410) 786-6601

PIC ID: 5966.1

PERFORMER ORGANIZATION: Research Triangle Institute, Research Triangle Park,NC
 

TITLE: Sustainable Support System for Telemedicine Research and Evaluation

ABSTRACT NUMBER: 036

ABSTRACT: The goal of this evaluation support project was to create an effective, ongoing mechanism by which the cost, effectiveness, and utility of telemedicine services could be systematically evaluated. This was done through the formation of a Clinical Telemedicine Cooperative Group (CTCG). The CTCG was based at the Telemedicine Research Center in Portland, Oregon, a nonprofit public service research corporation formed to foster high-quality research in telemedicine. The CTCG was modeled after a successful cooperative multicentered research organization. Functions of the CTCG included (1) providing operational and statistical support for telemedicine research and evaluation; (2) maintaining a communication system to link geographically distant telemedicine projects to share information and perform research; (3) creating easily adaptable, electronic data collection and tabulation instruments for use in telemedicine research; and (4) building a comprehensive on-line telemedicine information clearinghouse for gathering, storing, and disseminating information about the utility, effectiveness, and suitability of telemedicine for a broad range of medical and social applications. This project is complete. Data collection instruments were developed for the Mercy Foundation and the Iowa Methodist Health System in Des Moines, Iowa, as well as for East Carolina University in Greenville, North Carolina.

AGENCY SPONSOR: Office of Research and Demonstrations

FEDERAL CONTACT: William England

PHONE NUMBER: (410) 786-0542

PIC ID: 6317

PERFORMER ORGANIZATION: Telemedicine Research Center, Portland, OR
 

TITLE: Trends in Access to Health Care Services for Selected Segments of the Medicare Population

ABSTRACT NUMBER: 037

ABSTRACT: Trend data on access to health care services were developed for the years prior to, during, and after implementation of physician payment reform (PPR). The focus of the study was on health care service access for vulnerable subgroups of the Medicare population, such as persons with low income, persons without supplemental medical insurance, and persons with acute and chronic conditions. Geographic differences were also examined. These trend data were derived from the National Health Interview Survey conducted by the National Center for Health Statistics. The years 1984, 1986, 1990, and 1991 were used to develop post-PPR baseline data. The years 1992 and 1993 were used to develop post-PPR data. Descriptive data for 1984, 1986, 1989, 1990, 1991, and 1992 have been developed by sociodemographic characteristics. Relative standard errors have been computed by using a software package that takes complex sample designs into account. A multivariate model with 1984, 1986, 1989, 1990, 1991, and 1992 data has been developed to assess the impact of specific factors on use of physician services. Pre- and post-PPR data from the NHIS showed that health insurance and health status are important determinants of the use of physician services.

AGENCY SPONSOR: Office of Research and Demonstrations

FEDERAL CONTACT: Renee Mentnech

PHONE NUMBER: (410) 786-6692

PIC ID: 6316

PERFORMER ORGANIZATION: Health Care Financing Administration, Baltimore, MD
 

TITLE: Ventilator-Dependent Unit Demonstration: Outcome Evaluation and Assessment of Post-Acute Care

ABSTRACT NUMBER: 038

ABSTRACT: Mechanical ventilation is an important life-sustaining technology for some of the most seriously ill patients in any hospital. Acute respiratory failure frequently occurs in concert with failure of other organ systems or severe chronic diseases. This report presents the findings from the outcome evaluation and post-acute care analysis of the Ventilator-Dependent Unit Payment Demonstration, which evaluated costs and clinical outcomes in four ventilator-dependent units (VDU's). For the outcome evaluation, a comparison group of patients were identified from Part A discharge claims, using HCFA's pilot Uniform Clinical Data Set System. The report finds that (1) although VDU patients were substantially more eligible than the comparison group, there were many instances of apparent violations of the VDU admissions criteria among their patients; (2) in general, VDU clinical outcomes were substantially better than those for the comparison group, having lower hospital mortality and significantly higher rates of weaning from ventilators; (3) mean Medicare costs and total expenditures for comparison cases were very high, but expenditures for VDU cases were much higher, largely due to longer lengths of stay for VDU patients; and (4) national implementation with effective controls on admission would have increased Medicare expenditures in 1994 by about $0.4 billion, but implementation with ineffective controls would have increased expenditures by about $1.25 billion. An analysis of VDU patients' postacute care shows that (1) virtually all patients who were discharged alive from the VDU or acute hospital used postacute care, most of them in multiple settings; and (2) most ventilator-dependent patients have at least one acute rehospitalization and some have multiple rehospitalizations. Work for the project was performed under grant numbers: 29-P-99397/5, 29-P-99408/5, 29-P-99404/3, and 29-P-99405/1. (Final report: variously paginated.)

AGENCY SPONSOR: Office of Research and Demonstrations

FEDERAL CONTACT: Michael Henesch

PHONE NUMBER: (410) 786-6685

PIC ID: 4590

PERFORMER ORGANIZATION: The Lewin Group, Fairfax, VA

Health Resources and Services Administration

Contents

Assessing the Impact of Public Spending on the Health of Vulnerable Populations: A Framework for Evaluating HRSA Programs

Community Care Network Demonstration Program: Identification and Evaluation of Lessons for Community Collaboration

Developing a HRSA Strategy for Technical Assistance to the States

Emergency Medical Services for Children: An Evaluation of Sustainability in Seven States

Evaluation of Learning Needs Using a Computerized, Standard Educational Needs Assessment Protocol for AIDS Education and Training Centers (AETCs): Final Report

Evaluation of the Bureau of Health Professions Strategic Directions: Phase II--Cross-Cutting Performance Monitoring System: Final Report

Evaluation of the Impact of the Social Security Act Section 1115(a) Waivers on Federally Qualified Health Centers

Evaluation of the Impact of the Social Security Act Section 1115(a) Waivers on Federally Qualified Health Centers: Case Study--Hawaii

Evaluation of the Impact of the Social Security Act Section 1115(a) Waivers on Federally Qualified Health Centers: Case Study--Oregon

Expanding the Capacity of Advanced Practice Nursing Education

Graduate Medical Education in Ambulatory Settings

Hospital Development and OPO Performance

Impact of Case Management on Health Status in Community and Migrant Health Centers

Primary Care: America's Health in a New Era

Quality Assurance Procedures for the Uniform Reporting System for Titles I and II of the Ryan White CARE Act

Report to Congress on the Process by Which International Medical Graduates Are Licensed To Practice in the United States

Study of the Health Care for the Homeless Program

Technical Support for Need Designation and State Primary Care Access Plan Initiatives

Test of Local-Level Health Care Reform Strategies in Selected California Counties

Tools for Monitoring Cultural Competence in Health Care
 

TITLE: Assessing the Impact of Public Spending on the Health of Vulnerable Populations: A Framework for Evaluating HRSA Programs

ABSTRACT NUMBER: 039

ABSTRACT: This project develops a conceptual framework for describing the linkages between HRSA dollars and HRSA-funded services, HRSA infrastructure-building initiatives, and changes in the health status of the populations served. Through the ability to apply an analytical process to understand these linkages, HRSA, and other HHS agencies, could better assess the potential impacts of prospective funding changes on access to care for the Nation's vulnerable populations. The health care environment in which HRSA's programs and the Public Health infrastructure operate is changing in ways that could increase access barriers for vulnerable populations and diminish the ability of providers to deliver recommended primary care services. The implementation of Welfare Reform, Medicaid managed care waivers, and other cost management strategies at the State level create the potential for an increase in the numbers of uninsured Americans and a corresponding increase in the demands for publicly funded services. HRSA, therefore, has sought to better understand the potential effects of reduced resources on the high-risk populations served by HRSA's programs and on HRSA's ability to serve greater numbers of uninsured persons. Therefore, the project explored various analytical approaches to measuring the relationship between HRSA-funded health services and health outcomes. A conceptual framework was developed to describe how specific health outcomes of vulnerable populations might be traced back to HRSA programs. As a specific application, the analytical approaches were applied to a Maternal and Child Health program administered by HRSA. In its conclusions, the authors of the report made suggestions for improvements in data measures that would facilitate the quantification of the linkages between HRSA programs and health outcomes, and proposed interim measures for assessing cutbacks in Medicaid and reductions in funding for health centers and for maternal and child health programs. The authors stated that obtaining the necessary data consistently across the Nation should be a high priority as the HHS agencies monitor changes in the American health care system and the impacts of those changes on the Public Health infrastructure. (Final report: 34 pages, plus appendices.)

AGENCY SPONSOR: Office of Planning, Evaluation and Legislation

FEDERAL CONTACT: Melissa Clarke

PHONE NUMBER: (301) 443-5277

PIC ID: 6348

PERFORMER ORGANIZATION: Urban Institute, Washington, DC
 

TITLE: Community Care Network Demonstration Program: Identification and Evaluation of Lessons for Community Collaboration

ABSTRACT NUMBER: 040

ABSTRACT: This project developed a conceptual framework for an evaluation of the Community Care Network (CCN) demonstration project. The CCN project is concerned with the development of local collaborative networks that address a comprehensive range of community health needs through innovative public-private collaborations. Evaluation of the CCN demonstration project is a major opportunity to learn how to protect access for vulnerable populations in a restructured health care system. The Health Resources and Services Administration (HRSA) provided a small amount of funding to help design the overall evaluation because several of the CCN program goals are consistent with aspects of HRSA's mission. The conceptual framework for the evaluation of the program will be reviewed and used to develop the actual evaluation design for the CCN demonstration. See also PIC ID No. 6361.

AGENCY SPONSOR: Office of Planning, Evaluation and Legislation

FEDERAL CONTACT: Michael Millman

PHONE NUMBER: (301) 443-0368

PIC ID: 6361.1

PERFORMER ORGANIZATION: Hospital Research and Education Trust, Chicago, IL
 

TITLE: Developing a HRSA Strategy for Technical Assistance to the States

ABSTRACT NUMBER: 041

ABSTRACT: The purpose of this study was to gain insight into the status and roles of State health departments in health reform at the State level, the types of assistance that would strengthen their capabilities, and options for delivering that assistance as responsibility and authority shifts from the Federal to the State level. The study uses focus groups with State health personnel in five States to identify deficiencies in capacity that could be addressed by various types of technical assistance. The study also explores the different methods for delivering technical assistance, such as interactive telecommunications systems, small targeting meetings, Federal technical assistance teams, leadership development activities, and clearinghouses. The report identifies critical challenges in States' information and technical assistance needs, including (1) responding to the changing role of public health and the impact of managed care; (2) overcoming barriers created by categorical funding and grant requirements; (3) enhancing State capacity to perform assessment and accountability functions; (4) improving the public health information infrastructure; (5) strengthening State leadership in building community-based systems; and (6) increasing the role of public health in the State policy-making process. The report also makes recommendations for a Health Resources and Services Administration State technical assistance strategy, which should (1)be viewed as an investment strategy; (2) build upon current resources and efforts; (3)include a variety of approaches and methods; (4) be implemented agencywide; (5) use newer technologies and approaches; (6) assess staff capabilities to provide TA and expand capacity; (7) include effective information-sharing mechanisms for interstate assistance; and (8) include evaluations and feedback loops. (Final report: 14 pages.)

AGENCY SPONSOR: Office of Planning, Evaluation and Legislation

FEDERAL CONTACT: Michael Millman

PHONE NUMBER: (301) 443-0368

PIC ID: 6203

PERFORMER ORGANIZATION: The Lewin Group, Fairfax, VA
 

TITLE: Emergency Medical Services for Children: An Evaluation of Sustainability in Seven States

ABSTRACT NUMBER: 042

ABSTRACT: The Emergency Medical Services for Children (EMSC) program was begun in FY 1984 to provide Federal funding to assist States in eliminating gaps in the care of critically ill and injured children. This study was designed to (1) determine the ability of States to use short-term Federal funding to develop and institutionalize emergency medical services for children; (2) identify attributes associated with successful institutionalization; (3)characterize barriers to institutionalization; and (4) make recommendations for improving EMSC programs. The study employed a case study approach in assessing EMSC projects before, during, and after Federal funding was provided. Sources included existing data and key informant interviews conducted during site visits. The study found that the HRSA grants funded activities in four areas: (1) assessment of emergency medical services capabilities and development of equipment inventories and protocols; (2) development of training and educational programs for providers; (3) development of injury prevention programs; and (4)data systems development. Overall, the grant program was successful in achieving its goals in the first two areas; it was less successful in the areas of injury prevention and development of data linkage systems. Seven indicators of successful EMSC projects were identified: (1) continuity of project staff, (2) strong leadership in project management, (3)adesignated coordinator to oversee training and education, (4) development of networks and coalitions around pediatric emergency care, (5) ability to leverage additional resources, (6)prior understanding of or experience with emergency medical system issues, and (7)project personnel with a strong advocacy orientation. Recommendations address three areas: (1) strategic planning and funding; (2) grant guidance, including setting of target issues and an increased emphasis on evaluation; and (3) program development and intergovernmental relations. Together with an assessment of the EMSC program conducted by the Institute of Medicine, this report will be used in setting future program directions and goals. (Final report: 54 pages.)

AGENCY SPONSOR: Bureau of Primary Health Care

FEDERAL CONTACT: Barbara Wells

PHONE NUMBER: (301) 594-4284

PIC ID: 6351

PERFORMER ORGANIZATION: George Washington University, Washington, DC
 

TITLE: Evaluation of Learning Needs Using a Computerized, Standard Educational Needs Assessment Protocol for AIDS Education and Training Centers (AETCs): Final Report

ABSTRACT NUMBER: 043

ABSTRACT: The human immunodeficiency virus (HIV) and its final stage--acquired immunodeficiency syndrome (AIDS)--have posed many challenges for health care providers. They have exposed deficiencies in access to health care and have severely burdened existing resources within the public health system. To help meet the need for trained and motivated health care professionals competent in treating HIV/AIDS-infected patients, the Health Resources and Services Administration established AIDS Education and Training Centers (AETC's). A network of 15 AETC's provides training for health care professionals, especially physicians, physician assistants, advanced practice nurses, registered nurses, dentists, and dental hygienists. Many AETC's have expanded their target audience to encompass allied health care providers, such as mental health workers, case managers, and other psychosocial workers. This study developed a methodology for an ongoing system of evaluating the training needs of key providers trained through AETC's. Based on the results of this evaluation, a standard needs assessment protocol was developed, which includes computerized assessment learning instruments. Presently, a learning needs assessment must be completed by each AETC every three years for key disciplines. The protocols and instruments are being implemented nationally. Once implemented, the findings from this study will make available, for the first time, a standardized nationwide data set regarding the AIDS learning needs of the general population of key primary providers caring for persons with HIV/AIDS. Broader applications of this survey process are also being considered for other areas of learning needs of health professionals, such as violence and substance abuse. (Final report: 53 pages, plus appendices.)

AGENCY SPONSOR: Bureau of Health Professions

FEDERAL CONTACT: Ruth Kahn

PHONE NUMBER: (301) 443-6785

PIC ID: 5604

PERFORMER ORGANIZATION: Macro International, Inc., Calverton, MD
 

TITLE: Evaluation of the Bureau of Health Professions Strategic Directions: Development of a Cross-Cutting Performance Monitoring System: Final Report

ABSTRACT NUMBER: 044

ABSTRACT: The Bureau of Health Professions (BHPr) provides policy leadership and support for health professions workforce enhancement and educational infrastructure development. This project, part of a three-phase effort to develop an integrated data system to facilitate planning and evaluation of BHPr programs, developed a set of outcome-based performance measures and a performance monitoring system that measures progress of the Bureau in meeting its strategic goals. In particular, four cross-cutting goals are examined: (1)promoting a health care workforce with a mix of the competencies and skills needed to deliver cost-effective, quality care; (2) supporting educational programs' ability to meet the needs of vulnerable populations; (3) improving cultural diversity in the health professions; and (4) stimulating and monitoring relevant systems of health professions education in response to changing health care marketplace needs. This report examines several issues including (1) whether the outcomes and indicators chosen to monitor progress can be reliably collected and analyzed; (2) whether grantees believe that identified goals, outcomes, and indicators reflect what they consider to be the most valuable contributions of their projects; (3) what data sources and collection tools are available; and (4) the kind of system that should be developed in support of required monitoring and reporting. The report also describes the plan for a monitoring system that would provide the functions needed for ongoing program management through a cycle of measurement and monitoring, especially as required by the Government Performance and Results Act. Key functions in the plan include (1) program grantee-level measurement and monitoring; (2) analysis and assessment of program performance relative to expectations; (3) identification of successes and programs that merit further investigation; and (4) data to support the process of reviewing performance at the grantee and program level. (Final report: 25 pages, plus appendices.)

AGENCY SPONSOR: Bureau of Health Professions

FEDERAL CONTACT: Jennifer Burks

PHONE NUMBER: (301) 443-1530

PIC ID: 5497.1

PERFORMER ORGANIZATION: The Lewin Group, Fairfax, VA
 

TITLE: Evaluation of the Impact of the Social Security Act Section 1115(a) Waivers on Federally Qualified Health Centers

ABSTRACT NUMBER: 045

ABSTRACT: This study described and assessed the possible impact of Social Security Act Section 1115(a) Medicaid demonstration programs on two Federally Qualified Health Centers (FQHC's) in Hawaii and two in Oregon, based on the first year of implementation. Approaches included a review of existing data and site visits to the FQHC's. The report finds that (1) a change in the method of payment from cost-based reimbursement to a combination of prepaid capitation and fee-for-service rates creates a major challenge for FQHC's; (2) FQHC's experienced changes in their delivery system; (3) changes in benefit packages varied between the two States; (4) FQHC involvement in waiver development and implementation varied depending on the State (however, in both States, the FQHC's responded to the 1115 waiver by partnering with other organizations to form an HMO network); and (5)administration and management of the FQHC's has been affected by their waiver participation, although the effects have varied depending on the size, target populations, and the FQHC's prior managed care experience. The report also finds that FQHC's see the waiver programs as increasing access for beneficiaries. In addition, the report notes that there have been both positive and negative financial consequences for FQHC's as a result of their waiver participation. In both States, FQHC's are being reimbursed at rates substantially below cost-based Medicaid. All four centers indicated that they were continuing to provide services not included in the capitation rate and that they were serving patients who were not enrolled with the center. See also PIC ID No. 5738.1. (Executive summary: 6 pages.)

AGENCY SPONSOR: Bureau of Primary Health Care

FEDERAL CONTACT: Beth Ann Tutunjian

PHONE NUMBER: (301) 594-4060

PIC ID: 5738

PERFORMER ORGANIZATION: The Lewin Group, Fairfax, VA
 

TITLE: Evaluation of the Impact of the Social Security Act Section 1115(a) Waivers on Federally Qualified Health Centers: Case Study--Hawaii

ABSTRACT NUMBER: 046

ABSTRACT: This is one of two case studies conducted as part of an evaluation of the impact of the Social Security Act 1115(a) waivers on Federally Qualified Health Centers (FQHC's). This case study focuses on two FQHC's in Hawaii and the State's waiver program, Health QUEST, which was implemented beginning in August 1994. The discussion of Health QUEST's impact on FQHC's is limited to the first year of waiver implementation. The case study finds that certain design and implementation features of the QUEST program have implications for FQHC's, including (1) the change from cost-based reimbursement to a combination of prepaid capitation and fee-for-service rates creates a major challenge for FQHC's; (2) QUEST enrollment experience differed from expectations; (3) there were changes in the FQHC delivery system; and (4) there was substantial improvement in benefits for enrollees formerly covered by a state-funded program, SHIP. The report finds that, although FQHC's were not actively involved in development of the waiver, following approval they began to prepare for QUEST, in part by developing their own managed care organization, AlohaCare. As of January 1995, about 17.3 percent of all QUEST enrollees were enrolled in FQHC's or in FQHC look-alikes. The report also finds that enrollment in QUEST has differed from the expectations of the two FQHC's: in one, the estimate of enrollment was too high, while in the other, the estimate was quickly exceeded. Furthermore, staffing changes in both FQHC's occurred as a result of QUEST's implementation, as did increases in paperwork and needs for management information. The report concludes that QUEST has increased access to care through broadening eligibility, and that the program has had varied financial impacts on the FQHC's: one FQHC is receiving payments well below reasonable cost reimbursement, while the other FQHC is receiving payments at rates similar to those based on reasonable costs. See also PIC ID Nos. 5738.1 and 5738.2. (Executive summary: 14 pages; case study: 69 pages.)

AGENCY SPONSOR: Bureau of Primary Health Care

FEDERAL CONTACT: Beth Ann Tutunjian

PHONE NUMBER: (301) 594-4060

PIC ID: 5738.2

PERFORMER ORGANIZATION: The Lewin Group, Fairfax, VA
 

TITLE: Evaluation of the Impact of the Social Security Act Section 1115(a) Waivers on Federally Qualified Health Centers: Case Study--Oregon

ABSTRACT NUMBER: 047

ABSTRACT: This case study is one of two that The Lewin Group prepared as part of a study that evaluated the impact of the Social Security Act Section 1115(a) waivers on Federally Qualified Health Centers (FQHC's). This case study focuses on two FQHC's in Oregon, which implemented its waiver program, the Oregon Health Plan (OHP), beginning in February 1994. The information contained in the case study reflects the results of the first year of OHP operations and the opinions of individuals at the State Medicaid agency, the Oregon Primary Care Association, selected managed care organizations, and staff of the two FQHC's included in the study. The case study finds that several implementation features of OHP have implications for FQHC's, including (1) a change from cost-based reimbursement to a combination of capitation and fee-for-service rates; (2) expanded Medicaid eligibility; (3)restricted beneficiary freedom of choice; and (4) changes in the former Medicaid benefits package resulting in more limited coverage (but with some new benefits) for previously eligible enrollees. These changes led to a greater reliance on primary care and risk management. Some specific changes occurring in the two FQHC's included in the case study are (1) increased requirements for 24-hour coverage; (2) greater paperwork requirements due to the referrals process; (3) increased needs for management information in order to monitor changes in patient loads, new enrollees and disenrollments, and costs and utilization of services; (4) increased numbers of new patients, requiring more time for visits and fewer appointments available for previous patients; (5) increased needs for staff to check eligibility; and (6) limitations in the management information systems available, which directly affect analyses of the financial repercussions of the waiver. The case study concludes that the impacts of capitation may vary between FQHC's and that current OHP rates do not cover the costs of services at FQHC's as presently constituted. See also PIC ID Nos. 5738, 5738.1, and 5738.2. (Executive summary: 16 pages, case study: 80 pages.)

AGENCY SPONSOR: Bureau of Primary Health Care

FEDERAL CONTACT: Beth Ann Tutunjian

PHONE NUMBER: (301) 594-4060

PIC ID: 5738.1

PERFORMER ORGANIZATION: The Lewin Group, Fairfax, VA
 

TITLE: Expanding the Capacity of Advanced Practice Nursing Education

ABSTRACT NUMBER: 048

ABSTRACT: Advanced practice nurses are registered nurses with specialized education and training beyond basic nursing education for the provision of care in clinical settings. This study examines the feasibility of increasing the production of advanced practice nurses by investigating the expansion capacity of the educational system that prepares them. Four specialty areas are under consideration: nurse practitioners, nurse-midwives, clinical nurse specialists, and nurse anesthetists. The report finds that (1) most programs are currently at capacity in some resource aspect of their program (e.g., faculty, facilities, clinical sites, etc.); (2) the availability of sites for appropriate clinical education is the single most significant factor in determining a program's ability to expand its capacity; (3) the availability of clinical faculty, including clinical coordinators and preceptors, is an important factor affecting a program's ability to expand; (4) programs report that they have sufficient faculty to teach didactic courses; (5) there is no shortage of applicants, and the student demographic mix in programs tends to mirror the supporting community; (6) advance practice nursing programs use a wide variety of educational technologies to assist with both didactic and clinical instruction and to support distance education; and (7) the effects of distance education approaches on expansion of program capacity are inconclusive thus far but appear to have potential to expand capacity significantly. The report indicates that when programs seek to expand they have accommodated in a variety of ways by using resources more efficiently, changing program structure and design, and/or seeking increased resources. The report recommendations for Federal policy directions include increased support for individuals to enable full-time study, increasing support for programs with greatest potential for expansion, and developing selective grant criteria geared toward those programs with potential for increasing overall numbers of graduates. (Final report: 38 pages, plus appendices.)

AGENCY SPONSOR: Bureau of Health Professions

FEDERAL CONTACT: Evelyn Moses

PHONE NUMBER: (301) 443-6315

PIC ID: 5746

PERFORMER ORGANIZATION: The Lewin Group, Fairfax, VA
 

TITLE: Graduate Medical Education in Ambulatory Settings

ABSTRACT NUMBER: 049

ABSTRACT: This report examines the affiliations between primary care residency programs and the ambulatory training sites associated with them. It studies flows of personnel, money, and non-monetary benefits between these entities in order to develop an understanding of how these residency programs operate within the current funding environment for graduate medical education (GME). Information collected in a series of interviews with a total of eight primary care residency programs (four family practice programs, two general internal medicine programs, and two general pediatrics programs) was synthesized according to several key elements, including (1) organization; (2) operations; (3) residents; (4) supervising physicians; and (5) the health care marketplace. The interviews reveal that (1) there is not a significant amount of cost sharing between the programs and their affiliated ambulatory training sites; (2) the programs operate fairly well within the current GME funding environment, despite limitations to GME funding mechanisms; (3) programs are unaware of their resource costs for resident training and are unaware of how their affiliated teaching hospitals use Medicare GME funding in the medical education process; (4) programs have varied their responses to the changing health care marketplace--those involved in health maintenance organizations are active in their response to managed care, while others are relatively passive; and (5) despite the fact that physicians employed by ambulatory training sites may not be as productive when they have to supervise residents, these physicians appear willing to participate in resident training. The report also notes that the number of graduating medical students seeking primary care residencies is growing, that funding for all residencies is decreasing, and that outpatient placement opportunities are likely to decrease, as cost pressure on ambulatory training sites may make them unwilling to absorb future training costs.

AGENCY SPONSOR: Office of Planning, Evaluation and Legislation

FEDERAL CONTACT: Jessica Townsend

PHONE NUMBER: (301) 443-1900

PIC ID: 6350

PERFORMER ORGANIZATION: Center for Health Policy Studies, Columbia, MD
 

TITLE: Hospital Development and OPO Performance

ABSTRACT NUMBER: 050

ABSTRACT: Information about the extent and effects of hospital development activities of organ procurement organizations (OPO's) is quite limited. This report examines this issue and analyzes the efficiency of OPO's. A mail questionnaire was sent to all 66 U.S. OPO's; data regarding 64 of these are included in the analysis. These data show that (1) the median OPO in the sample served 70 hospitals with 16.5 full-time equivalents and had operating expenses of over $3 million; (2) hospital development is an important activity for most OPO's, backed by a strong commitment from OPO executive directors and a high level of resources; (3) OPO's whose directors are more committed to hospital development, which are not hospital sponsored, which serve a larger population or a larger land area, and which have a service area with higher death rates, display higher levels of hospital development activity; and (4) there is limited evidence that hospital development activities are associated with higher levels of performance, but a more valid assessment is dependent upon better performance measures and data. Part II of the report uses Data Envelopment Analysis to assess the relative efficiency of OPO's. It finds that (1) overall, 55 percent of OPO's are efficient, compared with their peers; (2) 27 percent of the 22 larger OPO's are inefficient compared with other large OPO's, while 55 percent of the 42 smaller OPO's are inefficient by comparison; and (3) efficient OPO's recover significantly more kidneys and extrarenal organs, have higher operating expenses, and have more referrals, donors, and transplants than others. (Final report: 55 pages, plus appendices.)

AGENCY SPONSOR: Bureau of Health Resources Development

FEDERAL CONTACT: Larry Wannemacher

PHONE NUMBER: (301) 443-6560

PIC ID: 6349

PERFORMER ORGANIZATION: Virginia Commonwealth University, Department of Health Administration, Richmond, VA
 

TITLE: Impact of Case Management on Health Status in Community and Migrant Health Centers

ABSTRACT NUMBER: 051

ABSTRACT: In 1988, the Health Resources and Services Administration, Bureau of Primary Health Care, initiated the Comprehensive Perinatal Care Program (CPCP). The CPCP provides monies to selected community and migrant health centers (CMHC's) to develop or enhance their delivery of case management services to perinatal populations. This study evaluated the effects of case management in the CPCP on access to services and on maternal and infant health behaviors and outcomes. Case management is a process for locating and integrating primary care with other specialized services on behalf of the recipients. The study also documented trends in perinatal health indicators and examined the role of case management in dealing with pregnancies in women who are homeless, have HIV-positive status, or are substance abusers. The study consisted of three phases: (1) a case-control site study of paired CMHC's intensively using and not intensively using case management; (2) a model site study of seven CMHC's serving high risk perinatal populations; and (3) a two-part mail survey of CMHC's. The study found that case management had a positive effect on continuity and adequacy of care, awareness of other professional services available, and the user's perception of quality of care. Case management did not seem to affect the utilization rates for the Special Supplemental Food Program for Women, Infants and Children (WIC), or Medicaid eligibility assistance services. The report recommends that CMHC's (1) take the time and effort to develop a comprehensive perinatal program; (2) develop easy tools for psychosocial risk assessment, nutrition assessment, and health education; (3) develop simple tracking and follow-up systems; (4) develop patient-specific care plans; (5) develop communication systems among caregivers; (6) provide several opportunities for disclosure; and (7) establish boundaries for case management (i.e., deciding when to refer and when to treat). (Final report: 63 pages.)

AGENCY SPONSOR: Bureau of Primary Health Care

FEDERAL CONTACT: Thomas Coughlin

PHONE NUMBER: (301) 594-4450

PIC ID: 4078

PERFORMER ORGANIZATION: John Snow, Inc., Boston, MA
 

TITLE: Primary Care: America's Health in a New Era

ABSTRACT NUMBER: 052

ABSTRACT: "Primary care is the provision of integrated, accessible health care services by clinicians who are accountable for addressing a large majority of personal health care needs, developing a sustained partnership with patients, and practicing in the context of family and community." This definition, developed by the Institute of Medicine, was used as a guide for an Institute committee that was appointed to carry out a 2-year study addressing the opportunities for and challenges of reorienting health care in the United States. The report discusses the structural changes in the health care system that will influence the evolution of primary care and underscores the need for coordinated services and relationships with public health, mental health, and long-term care. The report also discusses the importance of the primary care team and of primary care for vulnerable populations. Recommendations create a comprehensive set of actions that would ensure the proper place of primary care in the Nation's health care system as well as an adequate supply of appropriately trained primary care providers. On the latter topic, the committee recommendations include continuing the current level of effort to increase the supply of primary care clinicians, efforts to improve their competency, and interdisciplinary training and rotations in sites such as community health centers. Other recommendations include the development of tools for monitoring and improving the quality of care and dissemination of performance information. Recommendations concerning primary care research and data needs are also included. (Final report: 262 pages, plus appendices.)

AGENCY SPONSOR: Office of Planning, Evaluation and Legislation

FEDERAL CONTACT: Jessica Townsend

PHONE NUMBER: (301) 443-1900

PIC ID: 5612

PERFORMER ORGANIZATION: National Academy of Sciences, Institute of Medicine, Washington, DC
 

TITLE: Quality Assurance Procedures for the Uniform Reporting System for Titles I and II of the Ryan White CARE Act

ABSTRACT NUMBER: 053

ABSTRACT: The Ryan White Comprehensive AIDS Recovery Emergency (CARE) Act of 1990 assists States, metropolitan areas, and providers in the development of services for people living with AIDS and HIV infection. Titles I and II of this Act provide grants to disproportionately affected metropolitan areas and to States to improve the availability and coordination of services for HIV-infected people. The Uniform Reporting System (URS) supports Bureau of Health Resources Development monitoring and evaluation of the activities taking place under these Titles. This report presents a set of quality assurance procedures that could be applied to the URS, a provider-level system that was phased into operation beginning in 1992. The data collected in the URS are used to evaluate the extent to which the programs are achieving the goals of the CARE Act and to assist planning councils, State agencies, and consortia in targeting and monitoring the provision of services to specific population groups. Site studies, conducted for six grantee programs, included (1)a review of errors observed in the data submitted as part of the URS field test; (2) getting feedback from data managers on data quality issues; and (3) getting feedback from data managers on the plan developed for monitoring and improving data quality. The products of the project included quality assurance manuals for dissemination to grantees, providers, and staff of the Health Resources and Services Administration and a software module addition to the URS program to automate the preparation of error profiles. The quality assurance manuals include material covering the data quality plan, the effective use of quality profiles and targets, guidance for obtaining quality data, strategies for identifying sources of errors, and recommendations for improving data quality. The products of the project will be used to improve the quality of data submitted by CARE Act grantees. (Final report: 15 pages.)

AGENCY SPONSOR: Bureau of Health Resource Development

FEDERAL CONTACT: Ram Jain

PHONE NUMBER: (301) 443-6315

PIC ID: 5602

PERFORMER ORGANIZATION: Mathematica Policy Research, Inc., Plainsboro, NJ
 

TITLE: Report to Congress on the Process by Which International Medical Graduates Are Licensed to Practice in the United States

ABSTRACT NUMBER: 054

ABSTRACT: This project examined the policies and practices of State medical boards with regard to the licensing of International Medical Graduates (IMG's) and U.S. Medical Graduates (USMG's). It also assessed how well the National Credentials Verification System (NCVS), established by the American Medical Association (AMA) in 1991, expedited the process of licensure by endorsement. The AMA discontinued the NCVS in 1994 due to the costliness of its operation. The data collection strategy consisted of an overall examination of licensure policies and practices and a survey of the processing and approval times of nine State medical boards. The report finds that (1) the time required to verify the credentials of physicians applying for licenses remains a critical element in creating differences in the application process for IMG's and USMG's; (2) it is more difficult for IMG's to obtain and for State boards to verify the credentials documentation required for licensure; (3) as of 1994, 34 licensing jurisdictions require more years of graduate medical education for IMG's than for USMG's; (4) States differ with respect to both the primary and additional requirements for licensure imposed on IMG's; and (5) very few applications, whether submitted by IMG's or USMG's, resulted in denials. The report recommends that a national credentials verification system be established. It finds that such a system is urgently needed to assist State medical boards in verifying the credentials of both international and domestic medical graduates applying for initial licensure by endorsement. (Final report: 20 pages, plus appendices.)

AGENCY SPONSOR: Bureau of Health Professions

FEDERAL CONTACT: C. Howard Davis

PHONE NUMBER: (301) 443-6785

PIC ID: 5742

PERFORMER ORGANIZATION: Macro International, Inc., Calverton, MD
 

TITLE: Study of the Health Care for the Homeless Program

ABSTRACT NUMBER: 055

ABSTRACT: This report assesses critical programmatic and policy aspects of the Health Care for the Homeless (HCH) program. It examines models of service delivery, efforts to improve access to a variety of services, barriers to effective treatment, the effects on the health status of the population, program costs and financing, and compliance with the enabling legislation. Study methodology included a literature review, an analysis of HCH program data, a survey of all HCH programs, follow-up telephone interviews with 43 HCH programs, and site visits to nine sites. The report finds that (1) almost half of HCH's are federally funded community health centers; one-fourth are public health departments, and the remainder are hospitals, shelter coalitions, and other entities. Barriers to care include lack of transportation, language and cultural conditions, and the general social conditions of homelessness. Almost 20 percent of HCH programs identified a severe shortage of mental health and substance abuse services as their greatest problem. Only 6 percent of HCH financing comes from Medicaid. Recommendations include (1) the development of outcome measures to standardize practices; (2) development of better data and information systems; and (3) greater efforts to maximize revenue from Medicaid. Specific efforts have been made in the Bureau of Primary Health Care to address some of these recommendations. A best practices manual will outline successful strategies for serving homeless people and will cover many of the specific issues concerning outreach, case management, and continuity of care highlighted by this evaluation. A working group on health outcome measures for the homeless was convened in April 1996 and the proceedings were distributed to all HCH grantees. (Executive summary: 19 pages, plus appendices; Final report: 111 pages, plus appendices.)

AGENCY SPONSOR: Bureau of Primary Health Care

FEDERAL CONTACT: Lynda Honberg

PHONE NUMBER: (301) 594-4450

PIC ID: 5595

PERFORMER ORGANIZATION: Solon Consulting Group, Silver Spring, MD, with the UCLA Center for Health Policy Research
 

TITLE: Technical Support for Need Designation and State Primary Care Access Plan Initiatives

ABSTRACT NUMBER: 056

ABSTRACT: The purposes of this study were (1)to bring together appropriate experts to discuss methods for improving the process of designating areas of primary health care shortage or underservice, (2) to compile and compare Primary Care Access Plan data from the States; and (3) to analyze available indicators of the need for and access to primary health care services at State and local levels. The Bureau of Primary Health Care is responsible for several related programs that provide primary care services to areas and populations with access barriers to such services. This study was intended to help determine currently available indicators of shortage, access, and underservice at county and subcounty levels that could be used in a new index of medical underservice. By compiling and analyzing a data base of State Primary Care Access Plan submissions, the study also identified (1) the most efficient ways to store the data; (2) specific indicators for developing measures of the effects of State health care reform efforts; and (3) report designs to illustrate the effects. The information from the study will be used to finalize new designation criteria and support development of new regulations. (Volume I: variously paginated; Volume II: 31 pages.)

AGENCY SPONSOR: Bureau of Primary Health Care

FEDERAL CONTACT: Richard Lee

PHONE NUMBER: (301) 594-4280

PIC ID: 6204

PERFORMER ORGANIZATION: MDS Associates, Inc., Wheaton, MD
 

TITLE: Test of Local-Level Health Care Reform Strategies in Selected California Counties

ABSTRACT NUMBER: 057

ABSTRACT: A core idea behind this project is the realization that health care reform at the local level can do as much, if not more, than national reform to improve access to high-quality health care for all members of a community. This idea is given strength by the fact that a significant proportion of the health care expenditures in any community comes from public funds, whether these expenditures are spent on groups for which the Government accepts responsibility (military families, veterans, the elderly, or the very poor) or on purchasing health insurance for public employees. In some communities, public expenditures for health care may equal 75 percent of the health dollars spent. This project provides a replicable model that can assist a local community (in this case, Alameda County, California) to develop and implement a local health care reform initiative. Alameda County, the sixth most populous county in the State, covers an area of 737,500 square miles. The population is 53 percent white, 17 percent black, 14 percent Asian, and 14 percent Hispanic. About 203,000 Alameda County citizens are eligible for Medicaid; about 300,000 are indigent and lacking health insurance, and about 100,000 work at some level of government. Alameda County has a history of innovative public programs, strong and visionary leaders, and active collaboration between the public, private, and academic sectors. The report finds that the county has a highly developed provider network in place; thus, the most promising strategies included the development of a value-purchasing cooperative and a countywide health information network. This report develops an implementation plan for each of these strategies. The study concludes that such systems are feasible in the county and that they are replicable in other sites. The report also discusses how the Federal Government could encourage such local-level health reforms. (Final report: variously paginated, plus appendix.)

AGENCY SPONSOR: Office of Planning, Evaluation and Legislation

FEDERAL CONTACT: Karen Thiel Raykovich

PHONE NUMBER: (301) 443-2204

PIC ID: 6205

PERFORMER ORGANIZATION: Fresno Regional Foundation/Institute for Health Futures, Fresno, CA
 

TITLE: Tools for Monitoring Cultural Competence in Health Care

ABSTRACT NUMBER: 058

ABSTRACT: This project examines the dimensions of culturally competent health care practices, defined as the level of knowledge-based skills required to provide effective clinical care to patients from a particular ethnic or racial group. "Cultural competence" is thus distinguished from "cultural sensitivity," defined as the psychological propensity to adjust one's practice styles to the needs of different racial or ethnic groups. The project had as its goal the development and testing of tools that can be used to assess the cultural competence of primary care providers practicing within managed care systems. Two expert panels were convened to assist in the development of these tools. The report provides (1) findings of the physician panels and patient panels, and an analysis of their contribution to our understanding of the culturally competent model of health care delivery in the context of the two largest minority-run health maintenance organizations in California; (2) a patient satisfaction survey (tested with a sample of enrollees of the two HMO's) that can be used to evaluate the degree to which a managed care plan is providing culturally competent health care to its limited-English-speaking minority enrollees; (3) a provider self-assessment survey for use with physicians and other health care providers working in a managed care system; and (4)behavioral ethnic identifiers that can be used in enrollment forms as a cue for the provision of culturally appropriate health care services. (Final report: 61 pages, plus appendices.)

AGENCY SPONSOR: Office of Planning, Evaluation and Legislation

FEDERAL CONTACT: Michael Millman

PHONE NUMBER: (301) 443-0368

PIC ID: 6352

PERFORMER ORGANIZATION: Latino Coalition for a Healthy California, SanFrancisco, CA

Indian Health Service

Contents

Evaluating the Effectiveness of Alcohol and substance Abuse Services for Native American and Alaska Native Women: Phase II Final Report

Evaluation of the Indian Health Service (IHS) Adolescent Regional Treatment Centers (RTCs)

Methodology for Adjusting IHS Mortality Data for Inconsistent Classification of Race-Ethnicity of American Indians and Alaska Natives Between State Death Certificates and IHS Patient Registration Records
 

TITLE: Evaluating the Effectiveness of Alcohol and Substance Abuse Services for Native American and Alaska Native Women: Phase II Final Report

ABSTRACT NUMBER: 059

ABSTRACT: The major goal of Phase II of this study was to collect and analyze much-needed descriptive, demographic, social, cultural, and clinical information about women receiving treatment in alcohol and substance abuse treatment centers funded in whole or in part by the Indian Health Service. This evaluation provides both qualitative and quantitative information about a group of women who have been traditionally underrepresented in research. The themes and variations that emerged from the evaluation show that the life conditions of the women studied are extreme. For many, adverse or abusive childhood experiences and conditions have carried through into adulthood. The vast majority of women were exposed to physical, sexual, or emotional abuse during their childhood and adult lives. The availability of women-centered, family-focused approaches to alcohol and other drug treatment is severely limited in the United States. The leading obstacle to treatment for women with children is the lack of adequate care for their children while they are in treatment.

AGENCY SPONSOR: Office of Planning, Evaluation and Legislation

FEDERAL CONTACT: Leo Nolan

PHONE NUMBER: (301) 443-4700

PIC ID: 5675

PERFORMER ORGANIZATION: University of California, San Francisco, Institute for Health Policy Studies, San Francisco, CA
 

TITLE: Evaluation of the Indian Health Service (IHS) Adolescent Regional Treatment Centers (RTCs)

ABSTRACT NUMBER: 060

ABSTRACT: This study evaluated the effectiveness and efficiency of, and consumer satisfaction with, the nine regional treatment centers (RTC's) that provide alcoholism and substance abuse rehabilitation treatment to American Indian and Alaska Native youth. The following conclusions are based on the findings of this study. The RTC's have developed effective adolescent alcohol and substance abuse programs, but continuity of care and after-care are the biggest problems in the programs. RTC's need additional mental health staff resources, client-charting improvements, and innovative ways to increase family involvement. Recommendations include improving the continuum of care to adolescent abusers, self-evaluation, and regional treatment center effectiveness and efficacy.

AGENCY SPONSOR: Office of Planning, Evaluation and Legislation

FEDERAL CONTACT: Leo Nolan

PHONE NUMBER: (301) 443-4700

PIC ID: 5941

PERFORMER ORGANIZATION: Support Services, Inc., Silver Spring, MD
 

TITLE: Methodology for Adjusting IHS Mortality Data for Inconsistent Classification of Race-Ethnicity of American Indian and Alaska Natives Between State Death Certificates and IHS Patient Registration Records

ABSTRACT NUMBER: 061

ABSTRACT: This project developed an empirically based method for improving the consistency of Indian Health Service (IHS) mortality statistics when compared with race reported for the same decedents obtained from other sources. The study findings indicate a large variation in the number of deaths reported from 1986 to 1988 across IHS Service Area Offices, as well as extensive variation in the number of deaths reported in the matched IHS National Death Index data across states of occurrence. There is also a great variation in the percentage of consistently reported American Indian and Alaska Native race identification on State death records and IHS patient registration records by State of a resident. Recommendations include replicating the study using data on deaths occurring since 1988, using the adjustment factors developed in the study, and working with States to decrease inconsistent race reporting.

AGENCY SPONSOR: Office of Planning, Evaluation and Legislation

FEDERAL CONTACT: Leo Nolan

PHONE NUMBER: (301) 443-4700

PIC ID: 5946

PERFORMER ORGANIZATION: Support Services, Inc., Silver Spring, MD

National Institutes of Health

Contents

Bridges to the Future: Program Evaluation

Clinical Research Under Support by NIDDK: Fiscal Year 1995 Data

Development of Medications for the Treatment of Opiate and Cocaine Addictions: Issues for the Government and Private Sector

Evaluation of NIH Shared Instrumentation Grant Program: Reports From Users

National Heart, Lung, and Blood Institute Article Citation and Patent Reference Study: An Analysis of Article Citations and Patent References to Journal Articles Supported by the National Heart, Lung, and Blood Institute

Navajo Alcohol Rehabilitation Demonstration Program

Statistics on U.S. Immigration: An Assessment of Data Needs for Future Research
 

TITLE: Bridges to the Future: Program Evaluation

ABSTRACT NUMBER: 062

ABSTRACT: In February 1992, the National Institute of General Medical Sciences (NIGMS) and the National Institutes of Health Office of Research on Minority Health announced the implementation of the Baccalaureate and Doctoral Bridge programs. The objective of the Baccalaureate Bridge program is to facilitate the transition of minority students at 2-year colleges into colleges with baccalaureate degree programs in the sciences. The Doctoral Bridge program seeks to facilitate the transition of students into Ph.D. programs when they complete the M.S. degree. Two program outcomes that are particularly relevant to the goals of the Bridges program were examined in this study: (1) the transition of students from 2-year to 4-year institutions, or from M.S. to Ph.D. programs; and (2) the retention of students in the educational pathway, both before and after making the transition from one institution to another. Although the quantity and quality of the comparison data are not high, rates of transfer among Bridges students appear to be higher than rates among other students. Success in completing the baccalaureate degree among those students who have transferred is high and is comparable to the rate found among other minority students in an evaluation of the NIGMS Minority Access to Research Careers program. At the baccalaureate level, approximately 70 percent of the students are majoring in such fields as biology, chemistry and biochemistry, molecular biology, and microbiology. From the current data, it is difficult to develop good estimates of the eventual outcomes of participants in the Doctoral Bridges program. The early cohorts of students were small. Of the 21 students in the first two cohorts, about half have transferred to a doctoral program. While the rate of Bridges student attrition from graduate programs is below the national average, it is still too soon to predict their rate of completing their doctoral programs.

AGENCY SPONSOR: National Institute of General Medical Sciences

FEDERAL CONTACT: James Onken

PHONE NUMBER: (301) 594-2764

PIC ID: 6093

PERFORMER ORGANIZATION: Turner Consulting Group and TYC Associates, Rockville, MD
 

TITLE: Clinical Research Under Support by NIDDK: Fiscal Year 1995 Data

ABSTRACT NUMBER: 063

ABSTRACT: This study was conducted in response to the National Institute of Diabetes and Digestive and Kidney Disease (NIDDK) Advisory Council for a determination of the level of NIDDK support for clinical research during FY 1995. Two definitions of clinical research are currently under examination within the National Institutes of Health (NIH) and in the scientific community at large. The first, a narrow definition, requires only that patient and physician are alive in the same room; this is the one requested by the Advisory Council for estimating NIDDK support of clinical research. The broader, NIH-wide definition was adopted by the Advisory Committee to the NIH Director on Clinical Research. It includes areas of epidemiology, behavioral studies of patients, outcomes research, and health services research, as well as technology development intended for clinical use and contract studies supporting clinical research. Although the Advisory Council asked that the present study examine only clinical studies fitting the narrow definition, the concurrent need to support the efforts of NIH as a whole prompted the inclusion of NIDDK-supported clinical research fitting both definitions. The report finds that, in FY 1995, NIDDK supported (1) 678 extramural research projects fitting the narrower definition, at a cost of $178,144,000 (31percent of the NIDDK budget) and 68 applicable intramural projects, at a cost of $24,445,000 (29 percent of the budget); (2) 919 extramural projects fitting the broader definition, at a cost of $222,261,000 (37 percent of the budget) and 81 intramural projects ata cost of $27,635,000 (33 percent of the budget); (3) clinical research under the narrower definition took the largest percentage of a division budget (37 percent) in the Division of Kidney, Urologic, and Hematologic Diseases; and (4) at the program level, urology led in the percent of its budget spent on clinical research under the narrower definition.

AGENCY SPONSOR: National Institute of Diabetes and Digestive and Kidney Diseases

FEDERAL CONTACT: Carol Feld

PHONE NUMBER: (301) 496-6623

PIC ID: 6286

PERFORMER ORGANIZATION: National Institute of Diabetes and Digestive and Kidney Diseases, Bethesda, MD
 

TITLE: Development of Medications for the Treatment of Opiate and Cocaine Addictions: Issues for the Government and Private Sector

ABSTRACT NUMBER: 064

ABSTRACT: The Medications Development Division (MDD) at the National Institute on Drug Abuse was established to coordinate and encourage academic, private, and Federal regulatory involvement in developing and bringing to market new medications for treating drug abuse. This report examines MDD's progress and explores the factors that hinder the development of antiaddiction medications. The report finds that pharmacotherapy for the treatment of drug addiction has received little attention. In fact, despite the success of methadone for the treatment of opiate addiction, only two additional drugs have been approved, both of which were developed in the late 1960's and early 1970's. Furthermore, there is still no approved medication for treating cocaine addiction. The report also finds that the major disincentives to pharmaceutical research and development for antiaddiction medications include (1) an inadequate scientific base on the mechanisms of addiction and on the prevention of relapse, particularly for cocaine addiction; (2) a lack of financing for treatment and of trained treatment specialists; (3) narrow Federal and State regulations; (4) the relatively small market for such drugs; (5) pricing issues and the societal stigma against drug abusers; (6) liability issues and difficulties in conducting clinical research; and (7) a lack of sustained Federal leadership. The report provides recommendations to help remove these barriers. (Final report: 199 pages, plus appendices.

AGENCY SPONSOR: National Institute on Drug Abuse

FEDERAL CONTACT: Susan Azeka

PHONE NUMBER: (301) 443-6036

PIC ID: 5831

PERFORMER ORGANIZATION: National Academy of Sciences, Institute of Medicine, Washington, DC
 

TITLE: Evaluation of NIH Shared Instrumentation Grant Program: Reports From Users

ABSTRACT NUMBER: 065

ABSTRACT: The Shared Instrumentation Grant (SIG) Program of the National Institutes of Health (NIH) National Center for Research Resources has provided costly, state-of-the-art instruments for NIH-supported investigators since FY 1982. Between FY 1982 and FY 1995, 1,487 instruments were awarded through the SIG program, for a total program expenditure of $295.6 million. These awards were made to graduate schools, medical and other health professional schools, hospitals, and research organizations. SIG principal investigators and major users defined as NIH-supported investigators who accounted for a significant share of SIG instrument use time were surveyed through two related written questionnaires that addressed user characteristics, instrument usage and sharing arrangements, and the importance of SIG instruments to NIH research. A key finding of the survey was that in 1993, fully 70 percent of the 11-year-old instruments were still in service supporting NIH research. Furthermore, 78 percent of the major users of the oldest SIG instruments evaluated the instruments as adequate for their specific area of research. These findings indicated a greater-than-expected longevity for SIG instruments. See also PIC ID No. 2753. (Finalreport: 41 pages, plus appendices.)

AGENCY SPONSOR: National Center for Research Resources

FEDERAL CONTACT: Carol Brown

PHONE NUMBER: (301) 435-0866

PIC ID: 5083

PERFORMER ORGANIZATION: Abt Associates, Inc., Cambridge, MA
 

TITLE: National Heart, Lung, and Blood Institute Article Citation and Patent Reference Study: An Analysis of Article Citations and Patent References to Journal Articles Supported by the National Heart, Lung, and Blood Institute

ABSTRACT NUMBER: 066

ABSTRACT: The primary objective of all National Institutes of Health (NIH) activities is to improve the health of the American public. However, it is generally impossible to demonstrate the impact of basic research in terms of reductions in disease mortality or morbidity. Therefore, alternative measures must be relied upon. The objectives of this project are to (1) introduce the concept of examining patents as an additional measure for assessing the impact of NIH-supported research; (2) develop a methodology to use patents in evaluation research; and (3) apply this methodology to a National Heart, Lung, and Blood Institute (NHLBI) project in order to compare the outcomes for individual research project grants (R01's) funded under a Request for Applications with the outcomes of individual research project grants funded from unsolicited applications. The methods, data bases, software, and results of this project are applicable well beyond the NHLBI. Other Institutes, Centers, and Divisions will be able to use the methodology developed in this study to evaluate their own funding mechanisms.

AGENCY SPONSOR: National Heart, Lung, and Blood Institute

FEDERAL CONTACT: Carl A. Roth

PHONE NUMBER: (301) 496-3620

PIC ID: 5577

PERFORMER ORGANIZATION: National Heart, Lung, and Blood Institute, Bethesda,MD
 

TITLE: Navajo Alcohol Rehabilitation Demonstration Program

ABSTRACT NUMBER: 067

ABSTRACT: The Navajo Alcohol Rehabilitation Demonstration Program (NARDP) is a federally funded program to provide culturally sensitive treatment for Navajo Native Americans who suffer from alcoholism and the effects of alcohol abuse. NARDP is the only inpatient alcohol treatment program in the Gallup, New Mexico, area and is the only treatment program to use Navajo-speaking counselors. The program was designed exclusively for Navajo Native Americans by Behavioral Health Services (BHS), which has 6.5 beds available to NARDP. NARDP is fully integrated into BHS; the program consists of 28 days of inpatient treatment and rehabilitation based on a modified 12-step philosophy. This report presents findings from a congressionally mandated evaluation of NARDP: (1) Between March 1988 and January 1991, NARDP had a total of 425 client episodes and provided nearly 10,400 total days of treatment; (2) occupancy levels are high; (3) client interviews indicate a higher level of participation in culturally sensitive therapies after completion of an inpatient treatment program; (4) use of aftercare services is low; and (5) data obtained during the evaluation are inadequate to accurately measure client outcomes or program effectiveness, but indirect measures of effectiveness indicate a high program completion rate and a low readmission rate. (Final report: 136 pages, plus appendices.)

AGENCY SPONSOR: National Institute on Alcohol Abuse and Alcoholism

FEDERAL CONTACT: Laura Abbott

PHONE NUMBER: (301) 443-2369

PIC ID: 6018

PERFORMER ORGANIZATION: Abt Associates, Inc., Cambridge, MA
 

TITLE: Statistics on U.S. Immigration: An Assessment of Data Needs for Future Research

ABSTRACT NUMBER: 068

ABSTRACT: In September 1992, the Committee on National Statistics and the Committee on Population of the National Research Council held a workshop to explore data collection and preparation in the area of U.S. immigration. This workshop was held to help the Immigration and Naturalization Service (INS) develop a statistical information system, as required by the Immigration Act of 1990. Furthermore, the workshop provided suggestions for how data collection and analysis efforts of Federal statistical agencies and the social science research community might be improved in this regard. This report presents the conclusions and recommendations of the committees. The conclusions include (1) new large-scale data sets, including longitudinal data, are needed on immigrants by nationality group; (2) many surveys and the decennial census have limited information for distinguishing immigrant origins, making research on immigrants using existing data difficult; (3)information on legal status and visa status for legal immigrants is missing from most data; (4) data sets should focus on studies of the immigrant family; and (5) INS data could be used to study immigrants' use of various social programs, including welfare, Social Security, Medicare, and Medicaid. The report recommends that (1) the INS work with other Federal agencies and the Bureau of the Census to include key immigration questions on future censuses, including questions on nativity and parental nativity; (2) the Bureau of the Census consider ways to add local-area contextual data to the Public Use Microdata Sample files; (3)the Bureau of the Census retain key questions on immigration in the Current Population Survey; (4) Federal statistical agencies meet with counterpart agencies in Mexico to discuss the potential for establishing joint surveys on immigration; and (5) INS establish the design and usefulness of a survey of green card applicants.

AGENCY SPONSOR: National Institute of Child Health and Human Development

FEDERAL CONTACT: Mona Rowe

PHONE NUMBER: (301) 496-1877

PIC ID: 5622

PERFORMER ORGANIZATION: National Academy of Sciences, National Research Council, Washington, DC

Office of the Assistant Secretary for Planning and Evaluation

Contents

Alcohol and Other Drugs Treatment for Parents and Welfare Recipients: Outcomes, Costs, and Benefits

Americans With Disabilities

Assumptions Underlying Insurance Participation Modeling: A Background Report

Changing Numbers, Changing Needs: American Indian Demography and Public Health

Coordinated Community Responses to Domestic Violence in Six Communities: Beyond the Justice System

Cost of Domestic Violence to the Health Care System

Developing Integrated Managed Care Systems for Children With Disabilities: The State of the Art

Family and Community Violence Prevention Program: Technical Assistance in Program Evaluation and Capacity Building Assessment

Health Care in Transition: Technology Assessment in the Private Sector

Information to Guide Physician Practice

Interdisciplinary Education and Training of Professionals Caring for Persons With Disabilities: Current Approaches and Implications for a Changing Health Care System

Report on the Effects of Population on Quality of Care: Analysis of the Effect of Regulation on the Quality of Care in Board and Care Homes

Subacute Care: Policy Synthesis and Market Area Analysis
 

TITLE: Alcohol and Other Drugs Treatment for Parents and Welfare Recipients: Outcomes, Costs, and Benefits

ABSTRACT NUMBER: 069

ABSTRACT: In August 1994, the State of California released a report entitled Evaluating Recovery Services: The California Drug and Alcohol Treatment Assessment (CALDATA). This study found considerable savings to taxpayers associated with alcohol and drug treatment (an average of $7 saved for each dollar spent), primarily in reduced crime and reduced health care costs. While some data were reported on female clients and several statistics were reported regarding welfare receipt, the report included little detail specific to welfare clients and none regarding child welfare issues. This study analyzes the outcomes, benefits, and costs of substance abuse treatment for two partially overlapping groups: (1) parents of children under 18 years of age; and (2) recipients of public income support such as Aid to Families with Dependent Children. The report finds that many persons in substance abuse treatment, especially women, were parents, welfare recipients, or both. For example, (1) 36 percent of the California treatment population had children living with them prior to treatment--about half of these were men; (2) although women in treatment were outnumbered by men, they were much more likely to have children or to express parenting concerns; and (3) about 41 percent of all the women in treatment and 64 percent of women with children in their households received welfare income in the year before treatment. The report also finds that, compared with the year before treatment, the number of drug users after treatment among women with children who received welfare dropped about 39 percent for crack cocaine, 42 percent for cocaine powder, 48 percent for amphetamines, 14 percent for heroin, and 26 percent for alcohol. Finally, the report concludes that measurable treatment benefits exceeded treatment costs for all groups studied. (Final report: 43 pages.)

AGENCY SPONSOR: Office of Human Services Policy

FEDERAL CONTACT: Laura Feig

PHONE NUMBER: (202) 690-5938

PIC ID: 4511

PERFORMER ORGANIZATION: National Opinion Research Center, Chicago, IL
 

TITLE: Americans With Disabilities

ABSTRACT NUMBER: 070

ABSTRACT: This project provided funds to the Census Bureau to analyze the 1990 Census questions on disability. Data were collected for the population aged 16 and over on several types of disability, such as limitations in work, ability to work, mobility limitations, and self-care limitations. The tabulations will be the only source of disability data for States, counties, and large metropolitan areas and the only source that includes persons with disabilities residing both in the community and in all types of institutions. The information will serve as a national baseline. As such, the data are useful for virtually every aspect of disability policy and for portions of health care reform and welfare reform that pertain to people with disabilities. Results from this joint Office of the Assistant Secretary for Planning and Evaluation and Census Bureau project are available on the Internet. Results are presented in tables, graphs, and maps. Tables for States, counties, and metropolitan statistical areas contain information by disability status and type of disability on a variety of characteristics, including employment and labor force status, education, income, poverty, marital status, race, and ethnicity. Separate tables are available for the elderly and nonelderly. In addition, national disability estimates from two other Census surveys, the Survey of Income and Program Participation and the Current Population Survey, are presented. The Internet address for the results is http://www.census.gov/hhes/www/disable/census.html.

AGENCY SPONSOR: Office of Disability, Aging, and Long-Term Care Policy

FEDERAL CONTACT: Michele Adler

PHONE NUMBER: (202) 690-6613

PIC ID: 6174

PERFORMER ORGANIZATION: Bureau of the Census, Washington, DC
 

TITLE: Assumptions Underlying Insurance Participation Modeling: A Background Report

ABSTRACT NUMBER: 071

ABSTRACT: This report provides background material to help analyze policies designed to expand health insurance coverage, especially models that seek to estimate the impact of incremental health care reforms on the size and composition of the uninsured population. The report provides information regarding the "baseline uninsured" and the behavioral response to insurance strategies. The study finds that (1) the number of uninsured persons increased from 30.5 million in 1979 to 39.6 million in 1994--about one-third of this increase was due to increases in the population; (2) disagreement over estimates of the numbers of uninsured persons can be traced to several factors, including differing survey methods and disparity between "point-in-time" and "longitudinal" estimates; (3) 75 percent of the uninsured are in spells that will last longer than 1 year, while only 3.5 percent are in spells that will last less than 5 months; (4) estimates of employee insurance decisions indicate that in 1988 and 1993, about two-thirds of employees were offered insurance and enrolled in the employer's health insurance plan--only 10 percent of employees who were offered insurance did not enroll in their employer's plan; and (5) insurance coverage rates fell slightly between 1988 and 1993, from 68 to 65 percent--the percentage of employees offered insurance during this period also fell from 77.6 to 75.6 percent. The report concludes that the major contributor to the problem of uninsured workers lies in the area of insurance market supply, although it concedes that lack of demand on the part of some workers also contributes to the problem. (Final report: 65 pages, plus appendices.)

AGENCY SPONSOR: Office of Health Policy

FEDERAL CONTACT: Eleanor Dehoney

PHONE NUMBER: (202) 690-6250

PIC ID: 6455

PERFORMER ORGANIZATION: University of Missouri-St. Louis, St. Louis, MO
 

TITLE: Changing Numbers, Changing Needs: American Indian Demography and Public Health

ABSTRACT NUMBER: 072

ABSTRACT: This report summarizes the state of knowledge about the demography of the American Indian and Alaska Native populations, about the major health problems they face, and about their utilization of health care services. The report notes the difficulties in tracing demographic and public health trends in this population. These difficulties are attributable to several factors including (1) the relatively small proportion of the U.S. population that is American Indian; (2) American Indian residences tend to be either highly clustered in a small number of geographic areas or spread lightly over a large number of geographic areas; (3) American Indians have experienced a high rate of marital exogamy; and (4) there have been shifts over time in whether self-identifying or being identified as an American Indian is perceived as an advantage or a disadvantage. The report finds that (1) the rapid growth of the American Indian population since the turn of the century is due to changes in self-identification and relatively high fertility rates; (2) Indian infant mortality rates have improved over the past 15 years, but the death rates for American Indian youths and young adults remain high; (3) over one-half of American Indians live in urban areas; (4) American Indians have experienced a decline in infectious diseases, an increase in chronic diseases, and high levels of mortality due to violence, accidents, and alcohol and drug abuse; (5) during the 1980's, the economic situation of American Indians worsened; (6) the Indian Health Service (IHS) service population numbered about 1.21 million in 1990 (about 62 percent of the total American Indian population); and (7) IHS facilities are well suited to serve rural populations at no cost to patients, but access due to geographic isolation and budgetary issues may be difficult. (Final report: 314 pages.)

AGENCY SPONSOR: Office of Program Systems

FEDERAL CONTACT: Susanne A. Stoiber

PHONE NUMBER: (202) 690-7150

PIC ID: 6216

PERFORMER ORGANIZATION: National Research Council, Washington, DC
 

TITLE: Coordinated Community Responses to Domestic Violence in Six Communities: Beyond the Justice System

ABSTRACT NUMBER: 073

ABSTRACT: Dramatic changes in community response to domestic violence have occurred over the past two decades. Justice systems have implemented changes to laws and practices that hold batterers more accountable and support battered women. Awareness of domestic violence has extended past the justice and domestic violence service systems to health care providers, child welfare agencies, substance abuse services, churches, and businesses. This study examines coordinated responses to domestic violence in six communities: Baltimore, Maryland; Kansas City, Missouri; Carlton and northern St. Louis Counties, Minnesota; and San Diego and San Francisco, California. The report finds that (1) a strong community response to domestic violence requires that each part of the criminal justice system has appropriate policies that are actually followed in practice--many justice agencies in the study communities had developed their policies through discussions with other justice agencies and domestic violence service providers; (2) some study communities have adopted policies that include attention to the victim as a standard part of their response; (3) in the sites with specialized staff, many respondents feel that the specialization had improved the criminal justice response; (4) domestic violence service providers need to build relationships with providers of other services or representatives of other community sectors and to develop ways to translate their extensive experience into policies and procedures that other providers can understand and follow; (5) health care providers and other agencies must develop screening protocols to identify women who are battered, as well as a system for referring victims to the most appropriate agency; and (6) communitywide education and prevention activities are essential in formulating a coordinated response to domestic violence. (Executive summary: 21 pages; Final report: 165 pages.)

AGENCY SPONSOR: Office of Human Services Policy

FEDERAL CONTACT: Jerry Silverman

PHONE NUMBER: (202) 690-5654

PIC ID: 6154

PERFORMER ORGANIZATION: Urban Institute, Washington, DC
 

TITLE: Cost of Domestic Violence to the Health Care System

ABSTRACT NUMBER: 074

ABSTRACT: This study (1) reviews the literature on the prevalence, incidence, and cost of domestic violence; (2) examines the literature on the mental health consequences of domestic violence; (3) determines whether it is feasible to estimate the costs associated with this impact; and (4) develops an economic model of the cost of domestic violence to the health care system. The report finds that (1) domestic violence has extensive economic consequences, affecting the health care system, the mental health system, the social service system, and the criminal justice system; (2) the available data are flawed and necessitate a large number of assumptions in order to develop estimates; (3) the data sets that focus on health care (including mental health services) do not permit identification of services received as a result of domestic violence; and (3) data derived from the criminal justice system do permit estimates of the number of episodes of domestic violence, but do not contain data on service usage or costs. The report also provides an exploratory paper on the mental health consequences of domestic violence, including depression, suicidal thoughts, substance abuse, and posttraumatic stress disorder. The model developed in the estimates direct costs, related costs, and indirect costs (including morbidity and mortality). (Final report: variously paginated, plus appendices.)

AGENCY SPONSOR: Office of Health Policy

FEDERAL CONTACT: Sandra Howard

PHONE NUMBER: (202) 690-7778

PIC ID: 4528

PERFORMER ORGANIZATION: University of California at San Francisco, SanFrancisco, CA
 

TITLE: Developing Integrated Managed Care Systems for Children With Disabilities: The State of the Art

ABSTRACT NUMBER: 075

ABSTRACT: This project represents an initial inquiry into managed care as it affects children with disabilities. Despite the fact that many health care services are moving toward managed or coordinated care, many of those with disabilities express concern with how they will fare under such a system. The project involved a literature review and open-ended telephone interviews to gain an understanding of the state of the art in managed care for children with disabilities. Private providers and insurers, managed care experts, and State and Federal officials involved in Medicaid managed care demonstrations were interviewed. The interviews identified States or sites where managed care is being delivered or contemplated for children with disabilities. It also reviewed the various definitions of managed care used in these programs, services covered, and programs that provide traditional managed care (primarily acute care) or some combination of acute and long-term care. The project described how the programs operate and identified real and perceived barriers to care for these children under managed care systems. The project involved four site visits to States and programs already delivering managed care to these children. Using the material gathered from the interviews and site visits, the contractor developed a list of key policy concerns and barriers to developing and implementing managed care for children with disabilities.

AGENCY SPONSOR: Office of Disability, Aging, and Long-Term Care Policy

FEDERAL CONTACT: Ruth Katz

PHONE NUMBER: (202) 690-6613

PIC ID: 5760

PERFORMER ORGANIZATION: SysteMetrics/McGraw-Hill, Lexington, MA
 

TITLE: Family and Community Violence Prevention Program: Technical Assistance in Program Evaluation and Capacity Building Assessment

ABSTRACT NUMBER: 076

ABSTRACT: Since 1994, the Office of the Assistant Secretary for Planning and Evaluation has provided support for the evaluation of the Family and Community Violence Prevention Program, sponsored by the Office of Minority Health. The program was developed, implemented, and managed by the Minority Male Consortium, a group of 19 historically black colleges and universities (HBCU's), to reduce and prevent violence in both campus and community settings. This report discusses the development of an evaluation design for the program. A needs assessment was conducted to determine technical assistance (TA) requirements of Consortium members and to plan an evaluation TA conference and site visits to deliver individualized TA. The report identifies three broad areas of concern that were addressed at the TA conference. Common evaluation planning issues discussed in the report include (1) agreement among the program director, evaluator, and staff regarding evaluation design and implementation; (2) the challenges involved with data collection and analysis; and (3) the application of models and theories of violence prevention. Some consortium members encountered individual difficulties with evaluation planning. As described in the report, these difficulties included differing perspectives between the program director and evaluator; varying capabilities to implement effective programs; and recruitment and retention of appropriate subjects for the treatment groups. The report concludes that four activities are key in an overall evaluation of the program, and that, with technical assistance, individual HBCU's can make meaningful contributions to each activity. The activities are (1) audience research; (2) environmental profiling; (3) community resources inventories; and (4) assaultive behavior audits. Possible next steps for the evaluation process are recommended, along with suggestions for data collection on various topics.

AGENCY SPONSOR: Office of Program Systems

FEDERAL CONTACT: Patricia S. Hazard

PHONE NUMBER: (202) 690-8291

PIC ID: 5789.1

PERFORMER ORGANIZATION: Macro International, Inc., Calverton, MD
 

TITLE: Health Care in Transition: Technology Assessment in the Private Sector

ABSTRACT NUMBER: 077

ABSTRACT: Historically, the policy discussion on technology assessment (TA) in medicine has focused most of its attention on the role of the Federal Government. This report focuses on TA in the private health care system. Although some attention is given to TA in the hospital and physician sectors, the report is concerned with TA in the managed care sector. The health system environment affecting TA has changed profoundly in the past 5 years (1990-95) compared with the period immediately preceding it (beginning in the mid-1970's). In this context, the evaluation of clinical practice for its clinical and cost-effectiveness has assumed greater market value. The use of TA in the managed care sector is primarily to support coverage decision making. However, a number of innovative developments go beyond this narrow range of issues. Finally, an examination of private-sector TA activity leads back to questions about the appropriate role of the Federal Government in TA and the appropriate division of labor between the public and private sectors.

AGENCY SPONSOR: Office of Health Policy

FEDERAL CONTACT: Cheryl Austein-Casnoff

PHONE NUMBER: (202) 690-6102

PIC ID: 6180

PERFORMER ORGANIZATION: Rand Corporation, Santa Monica, CA
 

TITLE: Information to Guide Physician Practice

ABSTRACT NUMBER: 078

ABSTRACT: In recent years, many information resources have become available to physicians, including profiling, guidelines, and on-line information systems. These changes in the information environment have occurred at the same time that the health care delivery system has experienced radical changes. Thus, while physicians have a wealth of information at their disposal, the environment is chaotic. This report examines the information environment available to physicians through (1) a literature synthesis and data base retrieval system; (2) a focused study of On-Line Medical Networked Information for physicians; and (3) three physician focus groups (attending physicians in an academic medical center, a small rural practice, and a small group of resident physicians). The report finds that (1) the wealth of information available to physicians causes them to aggressively screen information sources--this screening process depends significantly on the characteristics of the physician, on the financial and legal practice environment, and on the source and type of information being provided; (2) technology advances are making it more feasible for rural physicians to communicate with colleagues, but financial constraints prevent them from taking full advantage of resources; (3) potential barriers to on-line resources include the fact that the information resources are not of enough value to justify the time needed to access them, lack of easy access, fear of computers, and concerns about information quality and security. (Final report: variously paginated.)

AGENCY SPONSOR: Office of Health Policy

FEDERAL CONTACT: Cheryl Austein-Casnoff

PHONE NUMBER: (202) 690-6102

PIC ID: 4499

PERFORMER ORGANIZATION: The Lewin Group, Fairfax, VA
 

TITLE: Interdisciplinary Education & Training of Professionals Caring for Persons With Disabilities: Current Approaches & Implications for a Changing Health Care System

ABSTRACT NUMBER: 079

ABSTRACT: Because people with disabilities generally use more health care services than those without disabilities and more often require the use of a continuum of care, which includes long-term care and supportive community services, effective health care for this population requires the services of both health care and human services professionals. At a time when increasing numbers of disabled persons may be served by managed care organizations, the ability of professionals to work in an interdisciplinary manner is important. This project identifies and describes particular programs where professionals and students in the health care and human services professions receive interdisciplinary instruction and practical experience in the provision of care to people with disabilities. The project includes a literature review, site visit reports, and a final report on interdisciplinary training in relation to care for people with disabilities. The report concludes that there is a growing consensus that, in many cases, meeting the clients' needs cannot be accomplished effectively and efficiently by one discipline alone. While not every client with a disability may require assessment and service delivery by a team, it is evident that professionals from a variety of disciplines will be increasingly called upon to work interactively. The report also concludes that interdisciplinary training programs are addressing managed care only peripherally, and managed care organizations have little appreciation for the specific knowledge, attitudes, and skills needed by providers to appropriately serve people with disabilities. Recommendations include (1) promoting opportunities for dialog between all parties concerned with training for and delivery of services to the disabled; (2) providing incentives to encourage preservice and in-service interdisciplinary training; and (3) disseminating information regarding innovative interdisciplinary curricula and approaches.

AGENCY SPONSOR: Office of Disability, Aging, and Long-Term Care Policy

FEDERAL CONTACT: Kathleen Bond

PHONE NUMBER: (202) 690-6443

PIC ID: 6052

PERFORMER ORGANIZATION: George Washington University, Center for Health Policy Research, Washington, DC
 

TITLE: Report on the Effects of Population on Quality of Care: Analysis of the Effect of Regulation on the Quality of Care in Board and Care Homes

ABSTRACT NUMBER: 080

ABSTRACT: There are about 34,000 licensed board and care homes in the United States, containing more than 613,000 beds. Residents in unlicensed homes may number more than 400,000. This study examines the effects of regulation on the quality of care in board and care, or domiciliary care, facilities. Data for the study were gathered in 386 licensed board and care homes and 126 unlicensed homes in 10 States. Five of these States have an extensive regulatory system; five have very limited regulation. The 512 board and care homes had 1,138 facility staff and 3,257 residents. The report finds that (1)States with extensive regulatory systems had a significantly smaller proportion of unlicensed homes; (2)extensive regulation is associated with better quality of care, as measured by several factors, including the better ability to meet the needs of frail and disabled residents, lower use of psychotropic drugs, lower use of inappropriate prescriptions for the elderly, and increased use of assistive devices and supportive services; (3) extensive regulation and licensure also lead to better quality of life and increased safety for residents and do not contribute to an "institutional" atmosphere in board and care homes; (4)regulation also prevents homes from being in the lowest range of performance, primarily through licensure alone; (5) neither extensive regulation nor licensure has a positive effect on some aspects of quality, including a requirement for preservice staff training and staff knowledge of care, monitoring, and medication management--it also has no significant effect on the cleanliness of homes or on the likelihood that homes would have any licensed nurses on staff; and (6) there is little variation among licensed and unlicensed homes on such issues as unmet health need, residents' rights, and indicators of resident satisfaction. See also PIC ID Nos. 4720.1-4720.3. (Final report: 67 pages.)

AGENCY SPONSOR: Office of Disability, Aging, and Long-Term Care Policy

FEDERAL CONTACT: Floyd Brown

PHONE NUMBER: (202) 690-6613

PIC ID: 4720

PERFORMER ORGANIZATION: Research Triangle Institute, Research Triangle Park, NC
 

TITLE: Subacute Care: Policy Synthesis and Market Area Analysis

ABSTRACT NUMBER: 081

ABSTRACT: Medicare expenditures for postacute and acute care are expected to rise substantially as both the number and proportion of persons over age 65 grows. As an effort to reduce costs, "subacute care" is being promoted by many providers as a cost-effective alternative to inpatient hospital care, with potential savings of 40 to 60 percent. While there is no specific definition of subacute care, there is general agreement that it encompasses a wide range of medically complex or technologically advanced services at the lower end of acute care, but at a higher level of care than that traditionally provided in nonacute care settings. Services previously provided in acute care hospitals are now more frequently delivered in nonacute care hospital settings. This project was designed to provide an understanding of subacute care in select market areas. In addition to examining a wide variety of subacute care providers, patients, and payers, the project identifies incentives and barriers to the provision of subacute care and reports on the impact of subacute care as perceived by key stakeholders in targeted markets. It also develops methodologies that can be used to assess the costs, savings, and quality of subacute care. Site visits were conducted in four targeted market areas to 19 state-of-the-art, self-declared institutional subacute care providers. Telephone interviews were conducted with several home care agencies, and discussions were held with key stakeholders in the targeted markets. Findings indicate that subacute care has emerged as a result of Medicare payment policies and the pursuit of low-cost, appropriate alternatives to hospital care by managed care providers. Much of what is labeled as subacute care refers to patients previously identified as "high-end Medicare patients." Finally, most providers were implementing some, but not all, of the ideal characteristics of a subacute care program.

AGENCY SPONSOR: Office of Disability, Aging, and Long-Term Care Policy

FEDERAL CONTACT: Jennie Harvell

PHONE NUMBER: (202) 690-6613

PIC ID: 6140

PERFORMER ORGANIZATION: The Lewin Group, Fairfax, VA

Office of Public Health and Science

Contents

Cost-Effectiveness in Health and Medicine

Evaluation of Health Needs of Newly Arriving Refugees

Measuring Expenditures for Essential Public Health Services

Survey of Accused but Exonerated Individuals in Research Misconduct Cases
 

TITLE: Cost-Effectiveness in Health and Medicine

ABSTRACT NUMBER: 082

ABSTRACT: Cost-effectiveness analysis (CEA) is a methodological tool increasingly considered by health care policymakers to evaluate which strategies will best serve theirprogrammatic or financial objectives. In 1993, the Public Health Service appointed 13scientists and scholars with expertise in CEA to the Panel on Cost-Effectiveness in Health and Medicine. The panel (1) assessed the current state of the science of CEA' methodology; (2) made recommendations for the conduct of CEAs that improve their quality, comparability, and utility; and (3) identified unresolved methodological issues. The report finds that there are several problems that limit the policy relevance of existing CEA's. For example, an individual CEA (1) may not define the perspective of its analysis; (2) may present inadequate data on effectiveness or cost, or data that are difficult to evaluate or to generalize; (3) may use a comparison intervention that distorts the cost-effectiveness ratio; (4) may inadequately represent the effect of time on the future value of health and dollars; or (5) may inadequately represent uncertainty with respect to key variables. Furthermore, CEA's may have problems that limit their comparability, such as the use of different perspectives, measures of outcome, costs, or discount rates. The report concludes that many of these problems could be mitigated if CEA practitioners would include in their studies a "reference case" that conforms to a set of standardized methodological practices. Including such a reference case would indicate that researchers had used a common methodological "toolkit" and that the study could reasonably be compared with reference case analyses of other interventions. The report presents recommendations regarding the structure and content of such a reference case. A November 1996 international workshop sponsored by an intradepartmental discussion group on cost-effectiveness allowed for broad discussion of the panel's work. (Executive summary: 24 pages.)

AGENCY SPONSOR: Office of Disease Prevention and Health Promotion

FEDERAL CONTACT: Kristine McCoy

PHONE NUMBER: (202) 205-8180

PIC ID: 5619

PERFORMER ORGANIZATION: Office of Disease Prevention and Health Promotion, Washington, DC
 

TITLE: Evaluation of Health Needs of Newly Arriving Refugees

ABSTRACT NUMBER: 083

ABSTRACT: This report describes and examines the health assessment and follow-up care currently available to refugees, Cuban and Haitian entrants, and Amerasian immigrants from Vietnam during the first eight months following their arrival in the United States. During this period, they are eligible for federally supported health care which fulfills dual objectives: to prevent the spread of health conditions that could affect the public health and to identify health conditions that could impede effective refugee resettlement. The purpose of the study is to collect information on current state and local refugee health care structures and services. The information gathered may help Federal officials modify and refine the support they provide to State and local jurisdictions. The study, conducted during the first quarter of fiscal year (FY) 1995, examines the seven states most impacted by refugees: Florida, California, Illinois, New York, Pennsylvania, Texas, and Washington. States rely on funding provided through the Refugee Medical Assistance (RMA) program but also utilize other funding sources, such as Preventive Health Refugee grants administered by the Centers for Disease Control and Prevention (CDC), Medicaid and state and local resources. The report finds that (1)funding flexibility enables states to fill in gaps in their refugee health programs; (2)states contracting with local providers to deliver health-screens appear to have developed a state-wide refugee health infrastructure; (3)refugee health coordinators are concerned that health screens conducted outside of refugee specific clinics do not provide accurate or complete data on testing and that some tests may be omitted; and (4)states differ in the extent to which they report back to CDC the outcomes of refugee health screens and in the extent to which follow-up care is accessible to refugees.

AGENCY SPONSOR: Office of International and Refugee Health

FEDERAL CONTACT: David B. Smith

PHONE NUMBER: (301) 443-1774

PIC ID: 6199

PERFORMER ORGANIZATION: Macro International, Inc., Calverton, MD
 

TITLE: Measuring Expenditures for Essential Public Health Services

ABSTRACT NUMBER: 084

ABSTRACT: This project will develop a tool for collecting Federal, State, and local public health expenditures, using the essential services of public health terminology. The goal of this effort is to document the relative investment in public health throughout the United States and to lay the groundwork for a system to track these expenditures over time and across agencies and levels of government. The project has provided the financial data that could be related to public health outcomes and public health infrastructure. Nine States participated in this effort: Arizona, Illinois, Iowa, Louisiana, New York, Oregon, Texas, Rhode Island, and Washington. In June 1995, State, local, and Federal officials met to discuss and plan the collection of information on public health expenditures. Participants from public health, mental health, substance abuse, and environmental agencies met in separate work groups to discuss the tools for estimating expenditures for the essential services of public health. Data collection instruments were distributed to State health officials in September 1995. These officials coordinated the data collection for State and local public health, substance abuse, and environmental agencies. A final report was released in November 1996.

AGENCY SPONSOR: Office of Disease Prevention and Health Promotion

FEDERAL CONTACT: Deborah R. Maiese

PHONE NUMBER: (202) 401-5809

PIC ID: 6194

PERFORMER ORGANIZATION: Public Health Foundation, Washington, DC
 

TITLE: Survey of Accused but Exonerated Individuals in Research Misconduct Cases

ABSTRACT NUMBER: 085

ABSTRACT: Approximately 70 percent of all cases of alleged scientific misconduct that come to the attention of the Office of Research Integrity (ORI) result in exoneration. However, little is known about the extent to which such cases result in adverse consequences for the accused or the extent to which institutions comply with a Public Health Service (PHS) regulation requiring them to protect the confidentiality of the accused or to restore their reputations if the accusations are not confirmed. This study presents the results of a surveysubmitted by 54 individuals who had been accused of scientific misconduct in PHS-supported research prior to 1995, and who were subsequently exonerated. The report finds that (1) 60percent of respondents report experiencing one or more negative consequences of being accused of scientific misconduct--17 percent report severe consequences, such as loss of position, promotion, or salary increase; (2) 90 percent of those who report negative consequences indicate that the negative actions began during the inquiry or investigation, and 65 percent report that these actions continued after they had been exonerated; (3) institutional officials and accusers are the major source of severe negative actions, while complainants are the most frequent source of negative actions; (4) 57 percent of respondents view the impact of the accusation on their careers as neutral, while 39 percent view the impact as negative and 4 percent view it as positive; (5) 94 percent of respondents are still conducting research--71 percent are working in the institution where they were accused of misconduct; (6) 39 percent of respondents believe that there is a continuing stigma attached to having been accused of misconduct, while 54 percent think it unlikely, and 11 percent do not know; and (7) 39 percent of respondents are dissatisfied with the efforts made by their institutions to restore their reputations. (Final report: 95 pages, plus appendix.)

AGENCY SPONSOR: Office of Research Integrity

FEDERAL CONTACT: Lawrence J. Rhoades

PHONE NUMBER: (301) 443-5300

PIC ID: 5621

PERFORMER ORGANIZATION: Research Triangle Institute, Research Triangle Park, NC

Substance Abuse and Mental Health Services Administration

Contents

Employment Outcomes of Indigent Clients Receiving Alcohol and Drug Treatment in Washington State

Overview of Addiction Treatment Effectiveness
 

TITLE: Employment Outcomes of Indigent Clients Receiving Alcohol and Drug Treatment in Washington State

ABSTRACT NUMBER: 086

ABSTRACT: Effective treatment approaches for alcohol and other drug abuse are critically needed, especially in an era of constrained resources. Although significant progress has been made in advancing knowledge about treatment outcomes, one area where relatively little is known remains: treatment outcomes for indigent clients, especially in the area of employment. This report builds upon a Washington State Department of Social and Health Services project to evaluate treatment outcomes for indigent clients in a number of areas (medical care use, employment, use of public assistance, and treatment reentry); the report focuses on employment outcomes. The population studied includes indigent clients who were assessed for treatment during a 4-month period beginning in August 1989. The treatment group (489 clients) received care in the following modalities: intensive inpatient treatment, long-term residential treatment, recovery house treatment, and outpatient treatment. The comparison group of 168 clients included those who were assessed but who did not initiate treatment. The report finds that (1) inpatient care appears to have a greater impact on employment and earnings than outpatient care for indigent clients with serious alcohol and other drug problems; (2) the duration of treatment has an important effect on employment outcomes--clients who stay in treatment longer experience better outcomes, as do clients who complete treatment; and (3) the benefits of treatment in terms of enhanced earnings compare favorably with the costs of treatment. The report concludes that although reducing coverage for chemical dependency treatment for public clients will reduce short-term treatment costs, it may also compromise benefits of treatment, such as enhanced employment.

AGENCY SPONSOR: Office of Applied Studies

FEDERAL CONTACT: Coleen Sanderson

PHONE NUMBER: (301) 443-9372

PIC ID: 6394

PERFORMER ORGANIZATION: State of Washington, Department of Social and Health, Olympia, WA
 

TITLE: Overview of Addiction Treatment Effectiveness

ABSTRACT NUMBER: 087

ABSTRACT: A substantial body of research substantiates the effectiveness of substance abuse treatment. This report presents an overview of much of that research, describing specific treatment approaches, treatment settings, and treatment components and services, and analyzing the effectiveness of each. The report notes that addiction is a progressive, chronic, primary, relapsing disorder that generally involves the compulsion, loss of control, and continued use of alcohol and other drugs despite adverse consequences. The primary goal of addiction treatment is to meet the biological, psychological, and social treatment needs of patients. The report discusses four treatment approaches that differ from one another in terms of their philosophies of treatment: (1) methadone maintenance treatment; (2) therapeutic community treatment; (3) traditional chemical dependency treatment; and (4) outpatient "drug free" nonmethadone treatment. It also discusses treatment settings, describing not only the physical environment in which the treatment takes place but also the level of treatment intensity (the number of treatment components provided, the extent of staff attention received, and the amount of time spent receiving treatment). These treatment settings include (1) inpatient hospitalization; (2) residential treatment; (3) intensive outpatient treatment; and (4) outpatient treatment. Treatment components--or specific clinical interventions, strategies, and procedures--are also addressed, as are other factors influencing treatment outcomes, including program factors, therapist factors, and patient factors. Addiction treatment and its relation with the criminal justice system and special populations are also examined.

AGENCY SPONSOR: Office of Applied Studies

FEDERAL CONTACT: Coleen Sanderson

PHONE NUMBER: (301) 443-9372

PIC ID: 6393

PERFORMER ORGANIZATION: Office of Applied Studies, SAMHSA, Rockville, MD

Appendix B. Inventory of HHS Evaluations in Progress During Fiscal Year 1996

Listed below are evaluations that were in progress during Fiscal Year 1996, organized by HHS agency. The expected date of completion is given for each project. For more information about a particular project, one may search the HHS evaluation data base accessible through the Internet at the URL address http://www.hhs.gov.

Administration for Children and Families

Project Title	Expected Date of Completion	PIC Abstract Number
Evaluation of Nine Comprehensive Community-Based Child Abuse and Neglect Prevention Programs	10/31/96	5851.8
Social and Educational Development of Tribal-Based Communities of the Sonoran and Negev Deserts	11/3/96	5978
Descriptive Study of the Head Start Health Component	11/30/96	5849
Responsible Fatherhood: Theoretical and Empirical Foundations for Policy and Program Development	12/31/96	5981
National Child Welfare Research Center--University of Chicago	1/31/97	4390.1
Arkansas Prenatal and Postnatal Paternity Project	3/31/97	6221
Evaluation of the Head Start Family Service Center Demonstrations	4/1/97	4394
Study of the Characteristics of Families Served by Head Start Migrant Programs	5/30/97	4974
Evaluation of the Impact of Homelessness on Administration for Children, Youth, and Families Programs	6/30/97	4396
National Study of Protective, Preventive Reunification Services Delivered to Children and Their Families	6/30/97	3874
Foster Youth Mentors	9/30/97	4388
National Child Welfare Research Center--University of California at Berkeley	9/30/97	4390.2
Factors Related to Gang Membership Resistance	12/28/97	5042.1
Home Visitor Services Demonstration: Home Visiting for Teen Parents Required To Participate in JOBS	9/30/98	6160
Descriptive Study of Families Served by Head Start	7/18/99	6331
Early Head Start Research and Evaluation	9/30/2000	3570
Avenues (Avenidas)	9/29/2002	6430.6
Bridgeport Artisan Center	9/29/2002	6430.1
Green Institute's JOLI Project	9/29/2002	6430.7
Harlem Railyards Transportation Program	9/29/2002	6430.2
HOMECARE CO-OP	9/29/2002	6430.A
Job Opportunities for Low-Income Individuals (JOLI) Project, Bethel New Life, Inc.	9/29/2002	6430.5
JOLI Project of Black Dollars Days Task Force	9/29/2002	6430.9
JOLI Project of the Women's Self-Employment Project	9/29/2002	6430.4
JOLI FY 1996 Projects	9/29/2002	6430
Project RISE, Yakima Valley (Washington) Opportunities Industrialization Center	9/29/2002	6430.8
Venture, a Highway Construction Program of the Rural Advancement Fund in North Carolina	9/29/2002	6430.3

Agency for Health Care Policy and Research

Project Title	Expected Date of Completion	PIC Abstract Number
Mapping a New Industry: The Performance of Outcomes and Effectiveness Research in the Private Sector	2/3/97	6385
Compiling an Encounter-Level, Managed Care, Research Database: First Stage	3/31/97	6374
Diagnostic Efficiency in Primary Care: What Do We Know, and What Do We Need To Know?	3/31/97	6381
Partners in Research: Identifying Common Interests	4/28/97	6386
Effects of Information Tools and Decision Aids To Support Shared Patient Decision Making About Medical Screening and Treatment	5/1/97	6376
Evaluation Synthesis: Research Related to the Implementation and Use of Guidelines and Other Quality Improvement Research	5/12/97	6382
Evaluation of AHCPR Minority Health Services Research Training Activities	5/30/97	6384
Customer Satisfaction Survey of AHCPR Publication Recipients	6/16/97	6375
Assessment of Data Release Strategies	9/29/97	6383
Health Care Informatics Standards of Selected Federal Agencies: A Revised and Updated Report and Conference	12/1/97	6387

Centers for Disease Control and Prevention

Project Title	Expected Date of Completion	PIC Abstract Number
Evaluation of the Medical Certification Process for Death Certificates	9/30/96	4230
Evaluation Design of the Business Responds to AIDS (BRTA) Program	10/15/96	6277
Case Studies of Prevention Marketing Initiatives (PMI) Local Demonstration Projects	12/15/96	6337
Evaluation of Suicide Prevention Interventions in Four Native American Communities	12/30/96	5504
Building Partnerships With Managed Care Organizations	12/31/96	6273
Evaluation of the Effectiveness of CDC Surveillance for Drug Resistant Pneumococcal Infections	12/31/96	5501
Sexually Transmitted Disease Prevention in the United States	12/31/96	5348
Assessment of the Effectiveness of the National Laboratory Training Network	2/3/97	5507
Evaluation of Tuberculosis Outreach Worker Activities	3/30/97	5502
Evaluation of Dental Treatment/Fluoridation	6/20/97	6334
Community Indicators for Assessing HIV Prevention	6/30/97/TD>	6278
Development and Implementation of Strategic Planning	7/6/97	6275
Assessment of Traumatic Brain Injury/Spinal Cord Injury (TBI/SCI) Program	8/30/97	6338
Evaluating Small Business Interventions	9/30/97	6336
Evaluation of Field Epidemiology Training Program (FETP)--Phase II	9/30/97	5923.1
Evaluation of Teen Pregnancy Interventions	9/30/97	6274
Management and Operational Indicators of HIV Community Planning	9/30/97	5518
Evaluation of Data Use of National Electronic Telecommunications System for Surveillance (NETSS) and Public Health Laboratory Information System (PHLIS)	1/31/98	6332
Early Warning Signs: Intimate Violence Prevention	3/30/98	6272
Evaluation of Group B Strep Guidelines	9/30/98	6333
Evaluation of the Impact of Clinical Laboratory Improvement Act (CLIA)	9/30/98	6335
Urban Health Systems Sentinel Network	9/30/98	6276
Evaluating Guidelines for Preventing Perinatal HIV	9/30/99	6279

Food and Drug Administration

Project Title	Expected Date of Completion	PIC Abstract Number
Assessment of the Mammography Quality Standards Act of 1992	9/20/97	6080
Prescription Drug User Fee Act Management	12/31/97	6079.2
FDA Inquiries Assistance Program for Small Businesses	3/20/98	6462
FDA Penalty-Reduction Program for Small Business	3/20/98	6461
Evaluation of Influenza Virus Vaccines in High-Risk Patient Categories	9/30/98	4355
Food Labeling and Package Survey (FLAPS)	12/20/99	5711
Nationwide Evaluation of X-Ray Trends	12/20/99	4984

Health Care Financing Administration

Project Title	Expected Date of Completion	PIC Abstract Number
Evaluation of the Medicare Case Management Demonstrations	12/30/96	6291
Evaluation of the Essential Access Community Hospital/Rural Primary Care Hospital Program	1/30/97	5200.1
Medicaid Quality of Care Study	1/30/97	6302
Evaluation of the Community Nursing Organization Demonstration	2/28/97	6306
Medicare Participating Heart Bypass Center Demonstration Extended Evaluation	2/28/97	5958.5
Evaluation of Cost Health Maintenance Organizations and Health Care Prepayment Plans	3/30/97	6314
Evaluation of Rural Health Clinics	3/30/97	6299
Evaluation of the Utah Prepaid Mental Health Plan: Coordinated Care Systems as Alternatives to Traditional Fee for Service	3/30/97	6293
Examination of the Medicaid Expansions for Children	3/30/97	6300
Evaluation of the Iowa Implementation of Ambulatory Patient Groups (APGs)	4/30/97	6320
Evaluation of Medicaid-Managed Care Programs With 1915(b) Waivers	5/31/97	6290
Evaluation of the Program of All-Inclusive Care for the Elderly Demonstration	6/30/97	6309
Impact of the Medicare Fee Schedule on Access to Physician Services	6/30/97	6301
Monitoring and Evaluation of the Medicare Cataract Surgery Alternate Payment Demonstration	8/30/97	6295
Community-Supported Living Arrangements Program: Process Evaluation	9/30/97	6304
Comparative Study of the Use of Early and Periodic Screening, Detection, and Treatment and Other Preventive and Curative Health Care Services by Children Enrolled in Medicaid	9/30/97	6236
Evaluation and Technical Assistance of the Medicare Alzheimer's Disease Demonstration	9/30/97	6305
Evaluation of the Demonstration Entitled Delaware Health Care Partnership for Children	9/30/97	6288
Evaluation of the Demonstration for Improving Access to Care for Pregnant Substance Abusers	9/30/97	6297
Evaluation of the Effectiveness of the Operation Restore Trust Demonstrations	9/30/97	6311
Evaluation of the Impact of Health Plan Report Cards on Consumer Knowledge, Attitudes, and Choice in a Managed Competition Setting	9/30/97	6312
Evaluation of the Medicaid Uninsured Demonstrations	9/30/97	6298
Project Demonstrating and Evaluating Alternative Methods To Assure and Enhance the Quality of Long-Term Care Services for Persons With Developmental Disabilities	9/30/97	6310
Medicare HMO Evaluation	10/10/97	6294
Drug Utilization Review Evaluation Contract	2/28/98	6296
Evaluation of the Medicare Choice Demonstration	6/30/98	6292
Evaluation of HMO Outlier Demonstration	9/1/98	6287
Maximizing the Effective Use of Telemedicine: A Study of the Effects, Cost Effectiveness, and Utilization Patterns of Consultation via Telemedicine	9/30/98	6303
Evaluation of Oregon Medicaid Reform Demonstration	9/20/99	6166
Evaluation of Phase II of the Home Health Agency Prospective Payment Demonstration	9/30/99	6308
Evaluation of the Nursing Home Case-Mix and Quality Demonstration	9/30/99	6307
Evaluation of the State Medicaid Reform Demonstrations (HI, RI, TN)	9/30/99	6289
Evaluation of the State Medicaid Reform Demonstrations (OH, MN)	9/30/2000	6289.1

Health Resources and Services Administration

Project Title	Expected Date of Completion	PIC Abstract Number
Rural Applications of Telemedicine	10/23/96	5749
Bilingual Assistance Program: Evaluation of Strategies for Reducing Cultural and Linguistic Barriers to Health Care for Hispanic and Asian Pacific Islander Populations	11/22/96	6356
Analysis of HRSA Participation in the HBCU Initiative: FY 1992-FY1995	11/30/96	6206
Collaborative Models That Work: Examining the Relationship Between HIV/AIDS and TB Programs in Selected Title I-Eligible Metropolitan Areas	11/30/96	6359
Development of a Performance Management System for Ryan White Titles I, II, and SPMS	12/10/96	5954.1
Development of a Performance Measurement System for the Division of Organ Transplantation	12/10/96	5954.2
Sixth National Sample of Registered Nurses	2/27/97	6367
Ability of Community-Based Health Organizations To Integrate Services for Individuals With Multiple Risk Factors	2/28/97	6369
Evaluation of Activities To Reduce Perinatal HIV Transmission: Case Studies and Feasibility Assessment	4/30/97	6202
Institutional Factors Which Influence the Likelihood of Living-Related Kidney Transplant Operations--Part II	4/30/97	4917.1
Person-Place Approximations and Utilization versus Health Status Analysis of the National Health Interview Survey	5/30/97	5614
Reasons African-American and White Waiting List Patients Are Unavailable for an Organ Offer	6/30/97	5601
Development of a Self-Assessment Tool for HIV Health Services Planning Councils and HIV Care Consortia	7/31/97	5735
Health Maintenance Organization Staffing Strategies in Underserved Areas	7/31/97	6366
Casemix Differences in Health Centers and Other Providers and Their Relationship to Cost	8/9/97	6363
Follow-Up To Determine State and Local Utilization of the Maternal and Child Health Programs Model Application--Phase I	8/15/97	5598
Tracking of J-1 Visa Exchange Students	8/23/97	6360
Training and Educational Needs of the Public Health Workforce Under Health Care Reform	8/30/97	5747
Review and Synthesis of HIV-Related Consumer-Level Evaluations	9/4/97	6362
BPHC Uniform Data System: Data Collection and Analysis Plan	9/9/97	6365
Analysis of CHC User Survey and Analysis of Managed Care/Non-Managed Care Users of Community Health Centers	9/11/97	6253
Evaluation of the Response to Medicaid Managed Care Initiatives by Federally Qualified Health Centers	9/17/97	6353
Meeting Industry Standards for Performance Measurement Through Clinical Outcome Measures at Community and Migrant Health Centers	9/23/97	6254
Strategies for the Recruitment, Retention, and Graduation of Hispanics into Baccalaureate Levels of Nursing	9/29/97	6249
Community and Migrant Health Centers' Immunization Efforts--A Joint Study With the Centers for Disease Control and Prevention	9/30/97	4921
Community Health Center User and Visit Survey	9/30/97	5737
Consortia Development for Health Professions Training in Community-Based Settings	9/30/97	6251
Development of Outcome Measures for Critical Ryan White CARE Act Titles I and II Funded Services	9/30/97	6252
Evaluation of the Healthy Schools, Healthy Communities Program	9/30/97	5739
Health Care Status Outcome Measures for the Bureau of Primary Health Care: The Assessment of Ambulatory Care Sensitive Conditions Through State Medicaid Recipient Files	9/30/97	6364
Measuring the Costs of Medical Education in Ambulatory Training Sites	9/30/97	6358
Performance Assessment of Managed Care in Community and Migrant Health Centers	9/30/97	6354
Synthesis of Local Evaluations Sponsored by Ryan White Grantees	9/30/97	6256
Community Care Network Evaluation	10/30/97	6361
Assessment of Bi/Multilingual Services Offered at Selected Community and Migrant Health Centers	12/10/97	6368
Evaluation of the Effectiveness and Impact of Community and Migrant Health Centers: Implementation Phase	12/31/97	4918.1
Impact of Community Health Advisors on Access, Utilization, and Patient Knowledge and Behavior	12/31/97	6355
Selected Aspects of Education in Managed Care	6/30/98	5603
National Evaluation of the Healthy Start Program	9/20/98	5610
Effectiveness of the National Health Service Corps	12/9/98	6357

Indian Health Service

Project Title	Expected Date of Completion	PIC Abstract Number
Development and Implementation of a Strategic Planning Evaluation Model	5/30/97	5548
Development of a Health Services Research Agenda for American Indian and Alaska Native Populations	5/30/97	6153
Resource Requirements Methodology Update	4/29/98	6450
Evaluation of IHS-Supported Substance Abuse Treatment Program for American Indian and Alaska Native Women--Phases III and IV	7/31/98	5675.1
Elderly Wellness	9/1/2001	6452
Prior Trauma Care of Intoxicated Patients as a Predictor of Subsequently Fatal Injury	9/1/2001	6451

National Institutes of Health

Project Title	Expected Date of Completion	PIC Abstract Number
Methodology To Assess the Impact of NHLBI Research	9/20/96	5577
Design and Pilot Test of an Inventory of Cancer Research Across Federal Agencies	12/6/96	6091
Electronic Student Tracking and Reporting System--The Bridges Program	12/20/96	6093
Evaluation of the Regional Primate Research Centers Program--PhaseI--Planning and Methodology Development for a Full-Scale Evaluation	12/20/96	6045
Evaluation of the NHLBI Programs of Excellence in Molecular Biology	1/31/97	5584
Comprehensive School Health Programs in Grades K-12	3/31/97	5586
Development of a Database for NIH-Supported Rare Disease Research	3/31/97	6043
Use of Antimicrobial Agents and Omeprazole in the Treatment of Peptic Ulcer Disease--Analysis of Physician Prescribing Practices	3/31/97	6284
National Institute of Mental Health (NIMH) Training Program for Minority Students Evaluation	6/1/97	6282
Evaluation of the National Institute of Mental Health (NIMH) National Research Service Awards for Predoctoral Individual M.D./Ph.D. Fellows--Phase I	6/20/97	6096
Follow-Up Study of Young Investigators in the Biological and Biomedical Sciences	6/20/97	6098
Characterization of User Population and User Satisfaction of the Physician Data Query (PDQ) Database	6/30/97	6092
Evaluation of the Research Demonstration Projects for Homeless Persons With Alcohol and Other Drug Problems	6/30/97	4509
Phase II: Survey To Assess the Impact of the NICHD Antenatal Steroids Consensus Development Conference	6/30/97	6120
Evaluation of the National Research Service Award Research Training Programs	7/20/97	5571
Chemical Science Roundtable	9/1/97	6281
Survey of Academic Research Equipment	9/20/97	6121
Survey of Biomedical Research Facilities	9/20/97	3562.2
Career Outcomes of a Matched Sample of Men and Women Ph.D.'s in Science and Engineering	9/30/97	5588
Career Status and Satisfaction With NIH Research Grant Application and Award Processes: A Sample Survey of FY 1992 R01 and R29 Applicants	9/30/97	6280
Planning Phase for an Evaluation of the NIDR Center Program	9/30/97	5578
Task Force on Genetic Testing	9/30/97	6090
Vaccine Development: Establishing Priorities for the United States for the 21st Century	9/30/97	5984
NIH AIDS Research Program Evaluation	10/20/97	6122
Evaluation of Knowledge-Based Expert System for Subject Indexing	12/20/97	4663.1
Evaluation of New Technology: Audio-Casi	12/20/97	5569
Evaluation of the Institutional Award (IDeA) Program, Phase I: Evaluation Design	12/20/97	6046
Evaluation of the NCRR Research Centers in Minority Institutions Program Phase I: Planning and Methodology Development for a Full-Scale Evaluation	12/20/97	5580
Evaluation of the Resource Centers Activities With the Biomedical Research Technology Program of the NCRR, Phase I: Planning and Methodology Development	12/20/97	6047
Strategic Plan for Public Health Agencies	12/20/97	6097
Evaluation of Laboratory Animal Use, Facilities and Resources--Phase II	1/20/98	6048
Evaluation of the NHLBI Short-Term Training for Minority Students Program	6/19/98	6118.1
Full-Scale Evaluation of the Regional Primate Research Centers (RPRC) Program	12/20/98	6045.1

Office of the Assistant Secretary for Planning and Evaluation

Project Title	Expected Date of Completion	PIC Abstract Number
Research and Evaluation Framework for the New Pharmaceutical Marketplace	9/20/96	6049
Public Health Performance Monitoring and Evaluation	12/1/96	6177
Augmenting the Evaluation of Iowa's Limited Benefit Plan	12/27/96	6390
Child Welfare and Domestic Violence: An Exploratory Study	12/31/96	6155
Waiver Evaluation Project	12/31/96	6050
Analysis of AFDC Caseload Growth and Decline	1/1/97	5953
Innovative and Interdisciplinary Education and Training Programs for Professionals Caring for Persons With Disabilities	1/31/97	6052
Identifying and Characterizing Distressed Urban Neighborhoods	2/1/97	6421
Strategies for Obtaining Public Health Infrastructure Data at Federal, State, and Local Levels	3/20/97	6179
Administrative Data Guide	3/30/97	6389
Characteristics of Informal Kinship Care	3/31/97	6016
Registry of Projects to Integrate Health Information Systems at the State Level	3/31/97	6178
State Telephone Survey Estimates for Evaluation and Monitoring	3/31/97	6420
Learning From Ongoing Welfare Waiver Evaluations: Lessons From Implementing Work-Focused Employment Programs for Welfare Recipients	4/13/97	6227
Identification and Synthesis of Activities Addressing the Adequacy of Information About the Organization and Delivery of Health Care in the United States	4/30/97	6388
Learning From Ongoing Welfare Waiver Evaluations: Lessons From Programs for Teen Parents	5/3/97	6228
Trends in the Well-Being of America's Children and Youth: 1997	5/20/97	6170.1
Evolving Health Care Delivery Systems: A Workshop	5/31/97	6341
Assessing the Feasibility of Making State-Level Estimates From National Surveys	6/1/97	6380
Medicare Home Health and Medicaid Personal Assistant Service Benefits	6/1/97	6426
Public Health Infrastructure and the Private Sector	6/30/97	6458
State Linked-Data Meeting and Monograph	6/30/97	6346
Exploring the Linking of Supply and Demand Health Utilization Data	7/1/97	6425
Exploratory Study of Health Care Coverage and the Employment of People With Disabilities	7/31/97	6240
Assessment of Access to Liability Insurance for Tribes and Tribal Organizations With Self-Determination Contracts or Compacts	8/15/97	6345
Evaluation of Practice in Care (EPIC)	9/20/97	5888
Consequences and Costs of Intimate Partner Violence	9/30/97	4528.1
Evaluation of the Minority Male Consortium for Violence Prevention	9/30/97	5789.2
Resource Evaluation of the FDA Food Additive Petition Review Process	9/30/97	6342
Role of Home- and Community-Based Services in Meeting the Health Care Needs of People With HIV/AIDS	9/30/97	6418
Statistical Linkage of Data Files for Evaluation: National Health Interview Survey and Survey of Income and Program Participation	9/30/97	6419
Analysis of Data on Use of Nursing Homes in Florida	10/20/97	5759
Analysis of the Health and Mental Health of Immigrant Children and Their Families	10/20/97	6150
Health and Long-Term Care Expenditure Patterns of Children With Disabilities	10/20/97	5758
Evaluating the Effects of Direct-to-Consumer Advertising on Consumers	10/30/97	6343
Teenage Parent Demonstration and Evaluation	12/20/97	2771.9
Panel on Performance Measures and Data To Support Public Health Performance Partnership Grants	12/31/97	6176
Structural Health Care Changes in Maryland: Studies on Access and Public Health Infrastructure	12/31/97	6459
What Works for Youth: An Examination of Youth Development Evaluations	12/31/97	6551
Innovative Coverage of Health Promotion, Clinical Preventive Services, and Other Coverage Issues in the Private Sector	1/30/98	6347
National Study of Assisted Living for the Frail Elderly	6/30/98	4719.5
Study of Private Long-Term Care Insurance Claims	8/1/98	6399
State Development and Use of Insurance Coverage Surveys	9/20/98	6017
Study of Managed Delivery Systems for Medicare Beneficiaries With Disabilities and Chronic Illnesses	9/30/98	6391
National Evaluation of Family Preservation Services	1/20/99	5337.1
Evaluation of the Oregon Medicaid Reform Demonstration	8/31/99	6166.1
Evaluation of the State Health Reform Demonstrations	8/31/99	6289.2
Evaluability Assessment of Responsible Fatherhood Projects	9/20/99	6159
Estimating an Analytical Framework for Assessing Insurer Response to the Health Care Market	9/30/99	6429
Private Payers Serving Disabled Individuals and Research Synthesis on Managed Care for Persons With Disabilities	9/28/2000	6398

Office of Public Health and Science

Project Title	Expected Date of Completion	PIC Abstract Number
Assessing Public Health Agencies' Capacity for Environmental Health Surveillance	3/31/97	6322
Evaluation Synthesis of School Health Programs	5/31/97	6198
Evaluation of the Office of Minority Health Resource Center--Phase I	6/30/97	6244
County-Level Tracking of Public Health Functions and Policy Issues	9/1/97	6325
Evaluation of Models To Integrate HIV Prevention and Treatment Services Within PHS-Supported Programs	9/1/97	6321
Commission on Dietary Supplement Labels	9/30/97	6193
Evaluation of the Cooperative Agreements for Demonstration Project for Capacity Building at Historically Black Colleges and Universities (HBCUs) FY 1992-95	9/30/97	6246
Healthy People 2000 Consortium Group	9/30/97	6491
International Year 2000 Plan Review	9/30/97	6489
Public Health Information Dissemination	9/30/97	6493
States' and Localities' Ability To Measure Their Own Objectives	9/30/97	6488
The Role of Health Plans in Community-Level Health Improvement Activities	10/31/97	6492
Evaluation of Coordinated Public Health Information Releases	1/31/98	6328
Evaluation of the Effectiveness and Efficacy of the Office of Minority Health's Bilingual/Bicultural Service Demonstration Grants Program	3/31/98	6247
Analysis of State and Local Public Health Policy and Program Activities	9/28/98	6326
Integrated Information Action Plan	9/30/98	6324
Science Panel on Interactive Communication Technology and Health	12/31/98	6327
NHIS 1998 Supplement	9/30/99	6490
Scientific Evaluation of Dietary Reference Intakes	9/30/2000	6323

Substance Abuse and Mental Health Services Administration

Project Title	Expected Date of Completion	PIC Abstract Number
Evaluation of Demonstration Treatment Programs in Job Corps	10/1/96	4523
National Treatment Improvement Evaluation Study (NTIES)	2/28/97	5346.1
Evaluation of the Pregnant and Postpartum Women and Their Infants Demonstration Program	9/20/97	5674
Outcome Evaluation for the Community Partnership Program	12/20/97	4997
DC Courts Initiative Evaluation	6/20/98	4998
National Evaluation Data Coordination Center	9/20/98	5994
Evaluation of High-Risk Youth Substance Abuse Prevention Initiatives Funded in 1994	4/20/99	6044
Cross-Site Evaluation of the Community Prevention Coalitions Demonstration Grant Program	5/20/99	6042
Evaluation of the HHS Access to Community Care and Effective Services and Supports Program	12/20/99	4980.1
State Substance Abuse Managed Care Evaluation Program	9/26/2000	6454

Appendix C. Senior Editorial Advisors' Review Criteria for Assessing Program Evaluations

Overall Significance

• The study addresses a significant issue of policy relevance.
• Evaluation findings are likely to be useful.

Conceptual Criteria

Conceptual Foundations

• A literature review is included.
• The project is shown to be logically based on previous findings; the report uses either theory or models, or both.
• The program assumptions are stated.
• The evaluation draws from any previous evaluation.
• The report is linked with a program and describes the program.
• The report presents multiple perspectives.
• Multiple relevant stakeholders are consulted and involved.
• The timing is appropriate because the program is ready for evaluation.

Questions for Evaluation

• The aims of the evaluation are clear, well-specified, and testable.
• The questions are feasible, significant, linked to the program, appropriate for the resources and audience, and derive logically from the conceptual foundations.
• The questions show ingenuity and creativity.

Findings and Interpretation

• The conclusions are justified by the analyses.
• The summary does not go beyond what the data will support.
• The appropriate qualifiers are stated.
• The conclusions fit the entire analysis.
• Equivocal findings are handled appropriately.
• The initial questions are answered.
• The interpretation ties in with the conceptual foundation.
• The report notes that the findings are either consistent with or deviate from the relevant literature.
• The presentation is understandable.
• The results have practical significance.
• The extent of program implication is assessed.

Recommendations

• The recommendations follow from findings, are worth carrying out, and are affordable, timely, feasible, useful, and appropriate.
• The recommendations are shown to be relevant to the questions asked.
• The breadth of specificity of the recommendations is addressed.
• Any recommendations for either future evaluations or improvements or both are clearly presented.

Methods

Evaluation Design

• Design considerations include overall appropriateness, soundness, funding and time constraints, generalizability, applicability for cultural diversity, assessment of the extent of program delivery, validity, feasibility for data collection, reliability of selected measurements, multiple measures of key concepts, and appropriateness of the sample.
• Variables are clearly specified and fit with the questions and concepts.
• The design permits measuring the extent of implementation of the program and answering the evaluation questions.

Data Collection

• Data are collected using appropriate units of measurement for analysis, controls for participant selection and assignment bias, and proper handling of missing data and attrition.
• Data collection is characterized by use of an appropriate comparison group of control; adequate sample size, response rate, and information about the sample; a plan and methods that are faithful to the plan; attention to and cooperation with the relevant community; project confidentiality; and consistency.
• The quality of the data (including the quality of any extant data sets used in the study) and the efficiency of sampling are addressed.
• The data collection is appropriate to the evaluation questions.

Data Analysis

• The data analysis addresses the handling of attrition, the matching of the analysis to the design, the use of appropriate statistical controls, the use of methodology and levels of measurement appropriate to the type of data; and estimation of effect size.
• The analysis shows sensitivity to cultural categories.
• The analysis makes appropriate generalizability of inferences.
• The chosen analysis type is simple and efficient.

Cross-Cutting Factors

The following are cross-cutting factors that are likely to be important at all stages of a report: clarity, presentation, operation at a state-of-the-art level, appropriateness, understandability, innovation, generalizability, efficiency of approach, logical relationships, and discussion of the report's limitations. The report should also address ethical issues, possible perceptual bias, cultural diversity, and any gaps in study execution.

Appendix D. Glossary of Acronyms

ACF	Administration for Children and Families
ACYF	Administration for Children, Youth, and Families
AETC	AIDS Education and Training Center
AFDC	Aid to Families With Dependent Children
AHCCCS	Arizona Health Care Cost-Containment System
AHCPR	Agency for Health Care Policy and Research
ALTCS	Arizona Long-Term Care System
AoA	Administration on Aging
ASC	Ambulatory-Sensitive Care
ASMB	Assistant Secretary for Management and Budget
ASPE	(Office of the) Assistant Secretary for Planning and Evaluation
ATSDR	Agency for Toxic Substances and Disease Registry
BHP	Bureau of Health Professions
BHS	Behavioral Health Services
BPHC	Bureau of Primary Health Care
CARE	Comprehensive AIDS Recovery Emergency Act
CCN	Community Care Network
CDC	Centers for Disease Control and Prevention
CDP	Chronic Disease Prevention
CEA	Cost-Effectiveness Analysis
CMHC	Community and Migrant Health Centers
CONQUEST	Computerized Needs-Oriented Quality Measurement Evaluation System
CPCP	Comprehensive Perinatal Care Program
CPMS	Comprehensive Performance Monitoring System
CPS	Current Population Survey
CTCG	Clinical Telemedicine Cooperative Group
DHF	Dengue Hemorrhagic Fever
DYFS	Division of Youth and Family Services
EHS	Early Head Start
EMS	Emergency Medical Services
EMSC	Emergency Medical Services for Children
ENP	Elderly Nutrition Program
EPOC	Evaluation Policy Oversight Committee
FDA	Food and Drug Administration
FETP	Field Epidemiology Training Program
FLAPS	Food Label and Package Survey
FQHC	Federally Qualified Health Centers
FY	Fiscal Year
GAO	Government Accounting Office
GME	Graduate Medical Education
GPRA	Government Performance and Results Act
HBCU	Historically Black Colleges and Universities
HCFA	Health Care Financing Administration
HCH	Health Care for the Homeless
HEDIS	Health Plan Employer Data and Information Set
HHE	Health Hazard Evaluation
HHS	(Department of) Health and Human Services
HRET	Hospital Research and Educational Trust
HRSA	Health Resources and Services Administration
HRT	Harlem Railyards Transportation
IC's	Institutes and Centers
IHS	Indian Health Service
IMG	International Medical Graduate
INS	Immigration and Naturalization Service
JOBS	Job Opportunities and Basic Skills
JOLI	Job Opportunities for Low-Income Individuals
KDA	Knowledge, Development, and Application
MDD	Medications Development Division
MEDTEP	Medical Treatment Effectiveness Program
MQSA	Mammography Quality Standards Act
NARDP	Navajo Alcohol Rehabilitation Demonstration Program
NCCAN	National Center for Child Abuse and Neglect
NCHS	National Center for Health Statistics
NCVS	National Credential Verification System
NETSS	National Electronic Telecommunications System for Surveillance
NEXT	Nationwide Evaluation of X-Ray Trends
NHLBI	National Heart, Lung, and Blood Institute
NICHD	National Institute of Child Health and Human Development
NIDDK	National Institute of Diabetes, Digestive and Kidney Diseases
NIDR	National Institute of Dental Research
NIGMS	National Institute of General Medical Sciences
NIH	National Institutes of Health
NIOSH	National Institute of Occupational Safety and Health
NIS	National Immunization Survey
NLEA	Nutrition Labeling and Education Act
NLTN	National Laboratory Training Network
NNIS	National Nosocomial Infections Surveillance System
NTIS	National Technical Information Service
OCS	Oregon Consumer Scorecard
OEI	Office of Evaluations and Inspections
OHP	Oregon Health Plan
OHPA	Office of the Health Plan Administrator
OIG	Office of the Inspector General
OPEL	Office of Planning, Evaluation and Legislation
OPHS	Office of Public Health and Science
OPO	Organ Procurement Organization
ORD	Office of Research and Demonstrations
ORI	Office of Research Integrity
PBM	Pharmacy Benefit Management
PDUFA	Prescription Drug User Fee Act
PHLIS	Public Health Laboratory Information System
PHS	Public Health Service
PIC	Policy Information Center
PPIP	Putting Prevention Into Practice
PPR	Physician Payment Reform
PRO	Peer Review Organization
RRM	Resource Requirements Methodology
RTC	Regional Treatment Center
SAMHSA	Substance Abuse and Mental Health Services Administration
SDX	State Data Exchange
SIG	Shared Instrumentation Grant
TA	Technical Assistance
TANF	Temporary Assistance for Needy Families
TB	Tuberculosis
TEFRA	Tax Equity and Fiscal Responsibility Act
TMRC	Technical Merit Review Committee
UCDSS	Uniform Clinical Data Set System
URS	Uniform Reporting System
USMG	U.S. Medical Graduate
VDU	Ventilator-Dependent Unit
WIC	Women, Infants, and Children
WSEP	Women's Self-Employment Program