HHS Announces Review of Evaluation Studies
A total of 18 evaluation studies, representing eight (8) HHS agencies, were submitted for review and ranking by a panel of non-Federal evaluation experts. Six studies were selected as “excellent” and will be featured in the Department’s annual evaluation report to Congress, entitled Performance Improvement 2002. These research studies were selected based upon rigorous evaluation criteria, including the research methodology, the soundness and interpretation of the findings, and a comprehensive analysis of the data. This issue of PIC Highlights contains abstracts of all six studies.
- Nurse Staffing and Patient Outcomes in Hospitals
- Building Their Futures: How Early Head Start Programs Are Enhancing the Lives of Infants and Toddlers in Low-Income Families
- Head Start Children’s Entry into Public School
- Reforming Medicaid: The Experience of Five Pioneering States with Mandatory Managed Care and Eligibility Expansions
- The Family Transition Program: Final Report on Florida’s Initial Time-Limited Welfare Program
- The Final Evaluation Report on the National Home Health Prospective Payment Demonstration: Agencies Reduce Visits While Preserving Quality
The number and type of nurse staffing in a hospital has an effect on patient outcomes, according to a Department of Health and Human Services Study.
The study, based on 1997 data from more than five million patient discharges from 799 hospitals in 11 states, found a strong and consistent relationship between nurse staffing and five outcomes in medical patients: urinary tract infection, pneumonia, shock, upper gastrointestinal bleeding and length-of-stay. A higher number of registered nurses (RNs) were associated with a 3 percent to 12 percent reduction in the rates of these adverse outcomes. Higher staffing levels of all types of nurses (RNs plus Licensed Practical Nurses (LPNs) and aides) were associated with a decrease in these adverse outcomes from 2 percent to 25 percent. Among major surgery patients, there was a reduction in the number of deaths among patients with shock, sepsis, pneumonia, deep vein thrombosis/pulmonary embolism or gastrointestinal bleeding associated with 4 percent to 6 percent higher RN staff and 2 percent to 12 percent for higher nurse staffing overall. This study demonstrated that the strongest relationship overall between outcomes and staffing was associated with an increase in the number of RNs.
This study responded to the HHS’s national research agenda to develop the evidence base of the relationship between patient outcomes and nurse staffing in inpatient units in acute care hospitals. The data collected in this effort provides a foundation on which to build analyses of what is actually happening in hospitals at a time when cost containment and staffing reductions are prevalent.
Although this study suggests higher nurse staffing and especially RNs, is related to fewer adverse patient outcomes, the study does not include evidence of a minimum nurse staffing level. This study does, however, strengthen the evidence from other recently published studies, that there is a relationship between nurse staffing and patient outcomes.
The study was conducted by Harvard University School of Public Health for four HHS agencies: the Health Resources and Services Administration (HRSA), the Agency for Healthcare Research and Quality (AHRQ), the Health Care Financing Administration (HCFA), and the National Institute of Nursing Research (NINR).
The Project Officer, Carole Gassert, can be reached at (301) 443-1272, or you can call the PIC and request PIC ID# 6864.
Building Their Futures:
How Early Head Start Programs Are Enhancing the Lives of Infants and Toddlers in Low-Income Families
The Early Head Start program serves some 45,000 low income pregnant women and families with infants and toddlers (birth to age three) and includes 664 programs nationwide. This two-generation program provides high-quality child and family development services, a focus on staff development, and a commitment to community partnerships.
In 1995 and 1996, the Administration for Children, Youth, and Families (ACYF) funded the first 143 programs, revised the Head Start Performance Standards to bring Early Head Start under the Head Start umbrella, created an ongoing national system of training and technical assistance and began regular program monitoring. Simultaneously, ACYF selected 17 programs to participate in a rigorous, large-scale, random assignment evaluation. Approximately 3000 children and families are participating in the evaluation, with 1500 randomly chosen to receive Early Head Start services and the other 1500 chosen to be in the control group. The control group were not able to receive Early Head Start services although they were able to access other services in the community. This summary report highlights the first main impact findings emerging from the analysis of child and family outcomes through the first two years of the children’s lives.
This national evaluation was conducted by Mathematica Policy Research, Inc. and Columbia University’s Center for Children and Families, in collaboration with the Early Head Start Research Consortium. The methodology included a randomized impact evaluation design in 17 sites and included programs in all regions, both urban and rural locations, all program approaches, and families that reflect the diversity of families in the Early Head Start program. To be eligible for the research, applicants had to be either pregnant women or families with a child 12 months of age or younger. Comprehensive longitudinal data were collected through parent services follow-up interviews (PSIs) completed approximately 6, 15, and 26 months after enrollment and in parent interviews, direct child assessments and videotaped parent-child interactions when children were 14, 24, and 36 months old. Data for this report come from the 6 and 15-month PSIs and the 14- and 24-month assessments.
The findings indicate that after a year or more of program services, when compared with a randomly assigned control group, 2-year-old Early Head Start children performed significantly better on a range of measures of cognitive, language, and social-emotional development. Their parents scored significantly higher than control group parents on many of the measures of the home environment, parenting behavior, and knowledge of infant-toddler development. Early Head Start families were more likely to attend school or job training and experienced reductions in parenting stress and family conflict.
These impacts are promising because the pattern of positive findings is consistent across multiple domains of child and family functioning known to be associated with later child outcomes, including social abilities, literacy, and school readiness. Future reports will assess whether these effects are sustained as children grow and have additional exposure to the program. The final report covering outcomes through age 3, is due to Congress in June, 2002.
This national evaluation was conducted by Mathematica Policy Research, Inc., Columbia University's Center for Children and Families at Teachers College, and the Early Head Start Research Consortium. The Project Officer, Rachel Cohen, can be reached at (202) 205-8810. A copy of the report can be found on the web at http://www.acf.dhhs.gov/programs/core/ongoing_research/ehs/buildingvol_1/bldg_vol1_title.html.
In 1990, the U.S. Congress authorized a new program to test the value of extending comprehensive, Head Start-like supports “upward” through the first four years of elementary school. This project, administered by the Head Start Bureau of the Administration on Children, Youth, and Families, funded 31 Transition Demonstration Programs in 30 states and the Navajo Nation from the 1991-92 school year through the 1997-98 school year and involved more than 450 public schools.
The study design involved random assignment of schools to a Transition Demonstration group, which received additional support and staff funded by this project, or to a Comparison group. Overall, the program implementation data (based on annual site visits, review of program documentation, and reports of participating families, teachers, principals, and Family Service Coordinators) support four major findings:
- All sites encountered multiple obstacles and barriers in their efforts to provide comprehensive and well-coordinated support to children, families and schools.
- Highly competent and stable leadership exerted a powerful influence on the strength and implementation of a local program.
- Only about 20% of the sites implemented very strong programs. The majority of local programs (17, or 55%) showed a combination of strengths and weaknesses in their programs.
- It was evident from the annual site visits and other information that many of the comparison schools had adopted the same reforms and programs that were being instituted in the demonstration schools. Because demonstration and comparison schools were usually within the same system, it was difficult to maintain a separate “treatment” group that was clearly different from the comparison group.
- Collectively, the schools, Head Start programs, and communities strongly endorsed the value of outreach efforts to families and the need to address young children’s needs during their early years of transition to school.
- Former Head Start children in both demonstration and comparison schools were achieving at average levels in both reading and math, compared to national norms, by the time they reached third grade. This was an unanticipated finding, in that children in low income families often are found to fall behind in school in the early elementary years. Instead, these children made rapid growth once they entered school, and continued to make growth throughout the study period. This is in marked contradiction to the “fade out” effect that has been reported for Head Start children.
- There were no substantial differences between the demonstration and comparison groups in terms of child outcomes; the children in both groups were doing well.
This project was prepared for ACF by the University of Alabama at Birmingham. The Project Officer, Mary Bruce Webb, can be reached at (202) 205-8175. A copy of the report can be found on the web at http://www.acf.dhhs.gov/programs/core/pubs_reports/hs/transition_study/trans_study.html.
The Experience of Five Pioneering States with Mandatory Managed Care and Eligibility Expansions
This report was prepared during a six-year evaluation of five state health reform initiatives, which were implemented as Medicaid, Section 1115, research and demonstration projects, initiated between 1994 and 1997. During the early 1990s states were confronted by rapidly rising Medicaid program costs and increasing numbers of uninsured people. States turned to managed care in an attempt to solve these problems and proposed implementation of mandatory, statewide managed care for large parts of their Medicaid populations. To make major changes such as these, states have to request waivers of Medicaid regulations. Thus, in 1993, applications for Section 1115 increased. The states hoped that by contracting with capitated health maintenance organizations (HMOs) to provide services, the quality of care would improve for Medicaid populations. Also, some states hoped to reduce the number of uninsured people by offering Medicaid coverage to low-income families who were not Medicaid-eligible, and planned to cover the costs of the expansions through savings from managed care.
The five state projects in the evaluation were (1) Hawaii’s QUEST, (2) Maryland’s HealthChoice, (3) Oklahoma’s SoonerCare, (4) Rhode Island’s Rite Care, and (5) Tennessee’s TennCare.
The data and methods consisted of information collection through case studies of the demonstration states, focus groups of providers and enrollees, and household surveys of low-income uninsured people and people enrolled in the demonstrations. Also used were vital records and Medicaid claims data. To assess the effects on beneficiaries of the switch to managed care from a fee-for-service delivery system, the researchers compared cross-sectional data on the Medicaid-eligible population from before and during the demonstration.
It was found that each of the five states made major investments of time and resources to accomplish the sweeping changes implemented in their Medicaid programs. The evidence suggests that managed care has not had a detrimental effect on enrollees, although the findings indicate that the experiences of people with disabilities, especially those with mental illnesses, suggest room for improvement in managed care.
This report was prepared by Mathematica Policy Research, Inc. The Project Officer, Penelope Pine, can be reached at (410) 786-7718. The report, PIC ID# 6289.1, can be obtained from the PIC.
The Family Transition Program:
Final Report on Florida’s Initial Time-Limited Welfare Program
Florida’s Family Transition Program (FTP) was the first welfare reform initiative in which families that reached the time limit on their welfare eligibility had their benefits terminated.
FTP, operated in Escambia County until 1999 and, limited most families to 24-months of cash welfare assistance in any 60-month period; and provided a wide array of services to help welfare recipients find work.
To assess the impact FTP made, the study compared the experiences of two groups of people: the Family Transition Program group, and the Aid to Families with Dependent Children (AFDC) group, which was subject to the prior welfare rules.
Relative to families in the AFDC group, FTP families gained more in earnings than they lost in welfare benefits, resulting in a modestly higher average income for the FTP group. These income and earnings gains came in the middle of the study period, however; by the end, the employment and income of the two groups were about equal.
FTP had few impacts, positive or negative on the well-being of elementary school-aged children. Adolescents in the FTP group, however, performed somewhat worse than their AFDC counterparts on a couple of measures of school performance.
FTP was successful in substantially reducing long-term welfare receipt–a central goal of the program. Because many people in FTP left welfare and others were granted exemptions from time limits, only 17 percent of people in the program reached their time limits and thus had their welfare benefits canceled during the study period.
The final results from the FTP evaluation show that, at least under certain circumstances, time limits can be implemented without having widespread, severe consequences for families. Nevertheless, FTP operated in a strong local and national labor market, had plentiful resources for staff and services, and imposed no lifetime limit on welfare receipt. Where these conditions do not hold, the consequences of time limits might differ from those found in this evaluation.
This study was prepared by the Manpower Demonstration Research Corporation. The Project Officer, Alan Yaffe, can be reached at (202) 401-4537. The report can be found on the web at http://www.mdrc.org/Reports2000/Florida-FTP/FTP-Final-FullRpt.pdf.
National Home Health Prospective Payment Demonstration:
Agencies Reduce Visits While Preserving Quality
This study was designed to test whether an alternative payment system could reduce public expenditures by promoting efficiency in the delivery of home health care. Phase I of this demonstration ran from 1990 to 1993. It tested the effects of a predetermined per-visit payment rate for the Medicare program. The results showed that to achieve substantial cost savings, the payment system for home health care had to provide incentives to reduce the volume of services. This report discusses the findings from Phase II, which ran from 1995 to 1998.
Ninety-one Medicare-certified home health agencies in five states (California, Florida, Illinois, Massachusetts, and Tennessee) enrolled in the three-year (Phase II) per-episode demonstration. Forty-seven were randomly assigned to the treatment group and received per-episode payment. The remaining 44 were assigned to the control group and continued to operate under cost reimbursement.
The methodology utilized was structured to determine the effects of a predetermined, per-episode payment rate. Individual and agency level data were utilized to investigate these evaluation issues: participation; profits and losses; costs per-episode; revenues; use of home health; quality of care; access to care; use of other Medicare services; and use of informal care and other, non-Medicare reimbursed services.
All types of agencies investigated in this study substantially reduced their provision of home health services under the demonstration, but this reduction did not cause compensating utilization of other services, regardless of whether Medicare or another source paid for those services. Morever, the study found no evidence that prospectively paid agencies were engaging in patient selection designed to reduce their financial risk. Further, no evidence was found to suggest that prospective payment caused high-cost patients to have poorer functioning, health outcomes, or any other indicator of poor-quality care. Nor was any evidence found that the Medicare service use of high-cost patients was affected any differently under prospective payment, when compared with low-cost patients.
Finally, regarding home health care agency profits and costs, the study found that the typical agency earned very small profits on its Medicare services during the demonstration. However, over time agencies earned larger profits–the median profit rate for those with three years of data rose from 2.57 percent (as a percentage of Medicare revenues) in year 2, to 7.8 percent in year 3.
This report was prepared by Mathematica Policy Research, Inc. The Project Officer, Ann Meadow, can be reached at (410) 786-6602. A copy of the report, PIC ID# 6318, can be obtained from the PIC.
SERVICES AVAILABLE FROM THE PIC
The Policy Information Center (PIC) at the Department of Health and Human Services (HHS), is a centralized source of information on in-process, completed, and on-going evaluations; short-term evaluative research and policy-oriented projects conducted by HHS as well as other Federal departments and agencies. The PIC on-line database provides project descriptions of these studies. Inquiries about PIC services should be directed to Richard Silva, Information Management and Dissemination, (202) 690-6445. Or, you may reach us on the web at: http://www.aspe.hhs.gov/pic.
PIC Highlights is now available as an electronic newsletter. If you wish to sign up to receive the PIC Highlights newsletter, please email firstname.lastname@example.org with “SUBSCRIBE PIC-HIGHLIGHTS” as the message, or sign up here.