Chapter  I

Results in Brief:  Completed Outcome-Oriented Program Evaluations

In this chapter of Performance Improvement 2002, the U.S. Department of Health and Human Services (HHS) presents the “results in brief” of FY 2001 completed evaluations that specifically address the effectiveness of a program or service.  The evaluation may focus either on: (1) broad program strategies, which may encompass a number of authorized programs to address a particular need; (2) a single program that has it own congressional authorization and appropriation; or (3) a specific activity within a program that is essential to its overall effectiveness.  These evaluations may provide important information for Congress in decisions about HHS programs.  The evaluations are organized according to general HHS programs areas:

The “results in brief” presentation for each evaluation provides the title of the final report, followed by the study purpose, major findings, and the recommendations or uses.  The studies listed in Chapter I are not repeated in Chapter II.

Healthcare/Public Health

Outstanding Evaluation

Reforming Medicaid: The Experiences of Five Pioneering States with Mandatory Managed Care and Eligibility Expansions

This report summarizes findings of a six-year evaluation of Medicaid 1115 demonstrations (managed care) in five states—Hawaii, Maryland, Oklahoma, Rhode Island, and Tennessee—that were initiated in 1994-1997. Information was collected from various sources, including case studies of the demonstration states and household surveys of low-income insured people and people enrolled in demonstrations.  Site visits, program documentation, and focus groups were used to assess implementation impacts on states, plans and providers.  To assess impacts on beneficiaries, the researchers examined data on the Medicaid population before and after the demonstration period.  Current population survey data and aggregate Medicaid cost data were also analyzed. 

The evaluation found that:

PIC ID: 6289.1; AGENCY SPONSOR: Centers for Medicare and Medicaid Services; CONTACT: Penny Pine (CMS), Jeff Buck (SAMHSA), Hunter McKay (ASPE); PERFORMER: Mathematica Policy Research Inc., Princeton, NJ

Outstanding Evaluation

Nurse Staffing and Patient Outcomes in Hospitals

In response to Congressional concern about the impact of staffing cuts on quality of care, HRSA, CMS, AHRQ, and NINR jointly funded this study, which was designed to identify outcomes of patient care sensitive to nursing and to analyze the relationship between these outcomes and nurse staffing in inpatient hospital settings.

The analyses were conducted on 799 hospitals from 11 states diversified with respect to size and geographic location.  In addition, data taken from CMS discharge abstract files for a national sample of 3,357 hospitals was used to analyze these relationships for Medicare patients.   A sensitivity analysis was also undertaken with a data set of 256 California hospitals that provided service-level data for nurse staffing (as opposed to data aggregated at the hospital level). 

Results differed for medical and surgical patients.  Staffing (the number and credentials of staff) was consistently linked to urinary tract infections, pneumonia, length of stay, upper gastrointestinal bleeding, and shock/cardiac arrest for medical patients, and to failure to rescue for surgery patients.  Weaker relationships were observed for other outcomes.  The investigators concluded that higher numbers of registered nurses (RN) were associated with a 3 to 12 percent reduction in rates of adverse health outcomes.  Higher staffing levels of all types of nurses (RNs plus Licensed Practical Nurses (LPN) and aides) were associated with a decrease in these adverse outcomes, from 2 to 25 percent.  Specifically, the study demonstrated that the strongest relationship overall between outcomes and staffing was associated with an increase in the number of RNs (i.e., LPN and aide staffing were not consistently related to the outcomes). The evaluators also pointed out the “catch 22”: higher staffing levels may lead to better charting of some adverse outcomes, thereby confounding the relationship between staffing levels and negative outcomes.

PIC ID:  6864; AGENCY SPONSOR: Health Resources and Services Administration; CONTACT:  Carol Gassert; PERFORMER:  Harvard School of Public Health, Boston MA

Do Food Labels Work?  Gauging the Effectiveness of Food Labels Pre- and Post-NLEA

This chapter in the Handbook of Marketing and Society is a synthesis of findings on the use and apparent consequences of food labeling, with special attention to the possible changes associated with implementation of the National Labeling and Education Act (NLEA).  NLEA had many features, including mandatory nutrition labeling for most packaged foods, uniform definitions for terms such as “light”, and limits on allowable health claims.  

The chapter summarizes and integrates findings from investigations that used a variety of methods, including sample surveys (before and after NLEA implementation), experiments, focus groups, and analysis of food consumption trends.  Populations used were generally not explicitly reported (but seemed apparent for sample surveys). 

Findings were summarized for consumer effects (including awareness of labels, label use, changing food decisions based on label information, effects of label claims, and dietary impacts) and marketplace impacts (including prevalence of claims on labels, introduction of healthier brands, and market share).  Key findings included increasing awareness of the new label over time and significant levels of reported label use to make product choices. The food label caused some consumers to stop buying products or to try new products.  It also provided the types of information that consumers are most concerned about (e.g., dietary fats) and presented the information more clearly than previously, according to the authors.

PIC ID:  7735; AGENCY SPONSOR: Food and Drug Administration; CONTACT: Alan Levy; PERFORMER: Food and Drug Administration, Washington, DC

The Efficacy of Interventions to Modify Dietary Behavior Related to Cancer Risk, Vols. I and II

This review seeks to clarify the existing knowledge base on behavioral interventions to alter dietary behavior related to cancer risk and offers directions for future research.  Interventions consisted of behavioral dietary interventions on fruit and vegetable intake and behavioral dietary interventions on decreases in fat intake.

In the studies that were reviewed, dietary interventions were consistently associated with an increase in fruit and vegetable consumption (with greater increase noted for fruit intake).   More than three-quarters of the studies reviewed reported statistically significant increases in fruit and vegetable intake.  Several dietary intervention components appear to be particularly promising in modifying dietary change favorable to cancer risk reduction: social support, goal setting, small groups, food-related activities, and the incorporation of family components. 

Using the differences-in-deltas approach, the reviewers determined that the average increase in fruit and vegetable intake reported was 0.6 servings per day, and consistent decreases were noted in the intake of total fat and saturated fat.  The mean change in total fat intake was estimated as 7.3 percent reduction in the percentage of calories from fat.  Interventions appeared to be more successful at positively changing dietary behavior in populations at risk of (or diagnosed with) disease than in healthy populations.

PIC ID:  7691; AGENCY SPONSOR: Agency for Healthcare Research and Quality; CONTACT: Kathryn Rickard; PERFORMER: Research Triangle Institute, Research Triangle Park, NC

5 A Day for Better Health Program Evaluation Project

The purpose of this project was to review and evaluate (1) the science underlying the 5 A Day program, (2) the implementation and accomplishments, and (3) the degree of achieved goals and objectives.  Twelve scientific experts, as members of the Program Evaluation Committee, reviewed the scientific evidence and program infrastructure (materials, collaborations, partnerships, message delivery, implementation and process measures) and supported research grants.  

During the first 10 years of the 5 A Day Program, public awareness on the importance of vegetable and fruit consumption and knowledge of the 5 A Day Program and its message has increased.   The message has reached more women than men, and more whites than Latinos or African Americans.  Evidence indicates that a diet rich in vegetables and fruits reduces the risk of cancer and other chronic diseases.  For example, the evidence of a relationship between higher vegetable and fruit consumption and lower risk of several types of cancers has been strengthened.

It is recommended that the 5 A Day Program should continue as a multifaceted program to support research and applied public health efforts to promote increased vegetable and fruit consumption.
http://cancercontrol.cancer.gov/5ad_exec.html
PIC ID: 7635; AGENCY SPONSOR:   National Institutes of Health; CONTACT: Kevin Callahan; PERFORMER: SCG: The Scientific Consulting Group, Inc., Gaithersburg, MD 20878-1409

The National Cross-Site Evaluation of High-Risk Youth Programs: Final Report

The purpose of this study was to determine the effectiveness of various programs designed to prevent substance abuse in high-risk youth. 

This rigorous research involved more than 10,500 at-risk 9- to 18-year-olds in communities across the United States.  Of 48 sites in this Cross-Site Evaluation, three were programs associated with universities or medical centers, three were higher education institutions, three were public school districts, and there were three state, county, or tribal agencies.  The majority of grantees were small community-based programs not affiliated with larger institutions.  The programs ranged in delivery method from primarily classroom-style, didactic programming in which youth receive substance abuse prevention-related information through lectures, videos, or other similar means, to interactive and experiential programming where youth are required to actively participate, reflect on the subject at hand, and act upon it.  There were four areas of emphasis for programming: (1) programs that focused on developing life-skills (e.g., refusal skills, communication skills, anger management, and other social skills); (2) programs that primarily provided knowledge of substance abuse, and other attendant problems (e.g., teen pregnancy, AIDS, gangs, and violence); (3) programs focused on developing self-esteem, or “affective-dominant” programs; and (4) recreation-dominant programming. 

Typically, the interactive type of programs included role play activities, team-building projects, outdoor wilderness experiences, or other activities requiring active engagement.  These type programs were found to be more effective in reducing substance abuse than programs that were less interactive.

The research shows that “prevention works”--programs designed to prevent substance abuse produce lasting reductions in substance use and have important positive effects on the lives of youth at risk.  The results represent accomplishments by programs operating in communities where youth face multiple risk factors.  Even within the challenging circumstances of these programs, the entire pool of participants across all programs decreased their use of alcohol and marijuana relative to similar youth who did not participate in the programs.  The study findings have important implications for emphases in prevention design and implementation.  By contributing to understanding the factors that prevent substance abuse in youth, and producing science-based lessons about effective prevention strategies and program practices, this major research effort holds the promise of more effective prevention and more positive futures for America’s youth who are at high risk.
PIC ID: 6044; AGENCY SPONSOR:  Substance Abuse and Mental Health Services Administration; CONTACT: Soledad Sambrano; PERFORMER: EMT Associates, St. Louis, MO

Effectiveness of Women’s Substance Abuse Treatment Programs:  A Meta-Analysis

Recent data suggest that at least 4.5 million women in the United States are currently alcohol abusers, 3.1 million are regular users of illicit drugs, and 3.5 million misuse prescription drugs.  The current demand for effective substance abuse treatment programs for women is, therefore, high.  To date, however, no formal quantitative synthesis or meta-analysis of the women’s treatment literature has been conducted.  In addition, compared to other social services fields, only a handful of meta-analyses have been conducted in the substance abuse treatment field.  

This technical report represents an attempt to fill this key knowledge gap by describing a quantitative, meta-analytic synthesis of the literature on the effectiveness of substance abuse treatment for women.  This study focused on the effectiveness of women-only treatment programs; women-only treatment programs compared to mixed-gender treatment programs; and, enhancements to women-only treatment programs.  The meta-analysis consisted of defining criteria for study inclusion, a literature search, study screening and selection process, the identification of relevant and representative outcome domains, the coding of studies, and quantitative analysis of study outcomes. 

The findings are:  1) different types of treatment were positive on average in all domains; 2) women-only programs provide additional treatment benefits beyond those resulting from standard, mixed-gender programs, i.e. women-only programs are best suited to the treatment needs of women; 3) enriching women’s treatment with additional components specifically oriented toward meeting women’s needs adds value above and beyond the expected effects of standard, women-only programs.
PIC ID: 5994.8; AGENCY SPONSOR:   Substance Abuse and Mental Health Services Administration; CONTACT: Ron Smith; PERFORMER: Battelle Corporation, Arlington, VA

The Effectiveness of Substance Abuse Treatment in Reducing Violent Behavior

This analysis, using data from the National Treatment Improvement Evaluation Study (NTIES), addresses the important policy issue of the association between substance abuse and violent behavior and the manner in which effective substance abuse treatment can reduce the likelihood of interpersonal violence. 

Reductions in violence were analyzed by gender, treatment modality--methadone treatment, non-methadone outpatient treatment, long-term residential treatment, short-term residential treatment, and facilities located in correctional institutions--and whether the client was a victim of violence in addition to being a perpetrator. 

The findings indicate that treatment for substance abuse can be highly effective in reducing specific violent acts among persons with a history of committing violence (i.e., perpetrators).   It was also found that these improvements are consistent across client subgroups such as gender, modalities of treatment, and past status as a victim of violence.
PIC ID: 5994.10; AGENCY SPONSOR:   Substance Abuse and Mental Health Services Administration; CONTACT: Ron Smith; PERFORMER: Caliber Associates, Fairfax, VA

Assessing the Effectiveness of Hazard Analysis Critical Control Point (HAACP) Systems in the Seafood Industry

The Seafood Hazard Analysis Critical Control Point (HACCP) Program mandates the application of HACCP principles to the processing of seafood.  HACCP is a preventive system of hazard control that can be used by processors to evaluate the kinds of hazards that could affect products, institute appropriate controls to keep the hazards from occurring or minimize their occurrence, and monitor performance of those controls.  This program supplants the old FDA seafood inspection system. 

This report provides an overview of the findings from a review of the relevant food safety literature and the advice from HACCP and evaluation experts regarding the feasibility of and alternative methods for evaluating the impacts of the seafood HACCP program.   

The major finding is that although an analysis of the impact of HACCP on the number and severity of seafood-borne illnesses in the United States should be the area of highest priority for an evaluation, data limitations make it virtually impossible to assess.   One reason is that attributing foodborne illness to a particular food is often not possible.  Second, attributing reductions in foodborne illness to a specific regulation may not be possible, since it would be difficult to isolate the impact of the regulations from changes in other factors that may also affect the incidence of foodborne illness. 

Recommendations are that an evaluation: (1) assess the impact of HACCP on the incidence and levels of physical, chemical, and biological hazards; (2) identify the set of factors affecting seafood safety; and (3) assess the impacts of those factors on seafood hazard levels, at least qualitatively.
PIC ID: 7343; AGENCY SPONSOR:   Office of the Assistant Secretary for Planning and Evaluation; CONTACT: Laina Bush; PERFORMER: Research Triangle Institute, Research Triangle Park, NC

Healthcare Financing

Evaluation of the Program of All-Inclusive Care for the Elderly Demonstration: Comparison of the Pace Capitation Rates to Projected Costs in the First Year of Enrollment

The Program of All-Inclusive Care for the Elderly (PACE) is designed to provide a complete range of integrated preventative, acute, and long term care services for the frail elderly by combining Medicaid and Medicare capitation in contracts with community providers of such services.  This evaluation sought to estimate the economic costs and benefits of enrollment during the first 12 months of participation in PACE.

Eleven PACE programs operating under dual capitation were involved in the evaluation, which was based on a comparison group design.  The treatment group included individuals who expressed interest in PACE, had a home visit, decided to enroll in PACE, and were accepted into the PACE program prior to the collection of site visit data. The comparison group consisted of individuals who had the same screening process and submitted initial applications, but decided not to enroll.  During the study enrollment period, 3,009 individuals met the study criteria.  Multivariate statistical procedures were used to control for selection bias and to adjust the impact measures.  Regression models were used to analyze the relationships between background characteristics and subsequent Medicare and Medicaid reimbursement for comparison group members.  Regression coefficients were then combined with information on the baseline characteristics of enrollees to estimate their projected Medicare and Medicaid costs. 

Findings indicated that projected costs at the 11 PACE sites were about 10 percent lower than the combined Medicaid and Medicare capitation payments.  When two high-cost sites were excluded from the analysis, estimated Medicare and Medicaid costs approximated the reimbursement rates.  There was a differential impact on Medicare and Medicaid, with PACE representing savings for the Medicare program, but additional costs for Medicaid.
PIC ID: 6309; AGENCY SPONSOR: Centers for Medicare and Medicaid Services; CONTACT: Fred Thomas, 410-786-6675; PERFORMER: Abt Associates, Inc.

Outstanding Evaluation

The Final Evaluation Report on the National Home Health Prospective Payment Demonstration: Agencies Reduce Visits While Preserving Quality

This evaluation presents findings from Phase II of the National Home Health Prospective Payment Demonstration, which ran from 1995 to 1998 and was designed to test the effects of a predetermined per-episode payment rate.  In that demonstration, 91 Medicare certified home health agencies from five states were randomly assigned to the control (i.e., cost reimbursement) and treatment (i.e., per-episode payment) groups.

The five studies reported here examined impacts on service delivery; patients’ use of Medicare services other than home health; costs per episode, per visit, and profit potential; and quality of care.  Multiple data sources were used, including Medicare cost reports, Medicare claims data (including UB-92 bill records that include patient characteristics), CMS’s Enrollment Data Base, Area Resource File (for agency information), a quality assurance database constructed for the demonstration, a patient survey, and site visits.  

The main finding was that agencies in the treatment group (per-episode payment) dramatically reduced their number of home health visits, primarily due to earlier discharge, without substantial adverse effects on quality of care.  However, the evaluators noted that control agencies also reduced their visits, in response to various changes in the environment of home health.  Other findings indicated that the overwhelming majority of patients in both groups were satisfied with their home health service.  The evaluation also identified methods that agency staff found successful in reducing home health utilization, including more careful supervision of visiting staff and improved patient education.  While treatment group agencies reduced costs per episode, their per-visit costs rose more for skilled nursing and aide visits than did the same costs for control group agencies.  In other words, although the volume of services fell, the treatment agencies’ fixed costs could not adjust as rapidly.  As well, some of the strategies used to reduce visits actually resulted in higher overhead costs.
PIC ID: 7738.4; AGENCY SPONSOR: Centers for Medicare and Medicaid Services; CONTACT: Ann Meadow; PERFORMER: Mathematica Policy Research, Inc., Princeton, NJ

Prospective Pymt (a) Prospective Payment Demonstration for Medicare Home Health:  No Clear Signs that Quality of Care Suffered

This report represents a summary of the quality of home health care throughout the three years demonstration project noted in earlier reports (7738, 7738.1, 7738.2).  It provides an update to the previous report which covered the first two years of the demonstration.  The demonstration tests the extent to which a fixed, lump-sum prospective payment to home health agencies for the first 120 days of each episode of care provided to Medicare beneficiaries increases efficiency in service provision.  By allowing agencies to retain most of any surplus payments over cost, this payment method gives agencies an incentive to provide home health care in a cost-efficient manner.  

The main findings are that, despite prospectively paid agencies’ progressive reduction of services over time, there were no serious adverse effects on major outcomes of care.  There were a few small negative effects in Year 3 on stabilization in functioning, as well as several large positive effects in all three years on improvement in symptoms.  These could be viewed either as subtle signs of negative effects late in the demonstration or, just as easily, as the absence of any true impacts.
PIC ID: 7738.3; AGENCY SPONSOR:   Centers for Medicare and Medicaid Services; CONTACT: Ann Meadow; PERFORMER: Mathematica Policy Research, Inc., Princeton, NJ

Prospective Pymt (b) Per-Episode Prospective Payment for Medicare Home Health Care Sharply Reduces Service Use

As part of its ongoing effort to study methods of providing more cost-effective care, the Centers for Medicare and Medicaid Services (CMS) implemented the Per-Episode Home Health Prospective Payment Demonstration.  Under the demonstration, participating home health agencies are paid a fixed, lump-sum payment for the first 120 days of each episode of care provided to Medicare beneficiaries and a predetermined rate for each visit thereafter.  By allowing agencies to retain most of any surplus payments over cost, this payment method gives agencies an incentive to provide home health care in a cost-efficient manner. 

It was found that prospective payment reduced the average number of visits to a patient in the year following admission by 24 percent compared to their levels under cost-based reimbursement.  Prospectively paid agencies achieved these reductions by shortening the overall length of service and by lowering the frequency of visits provided.  These results strongly suggest that prospective payment is a highly successful method for controlling the costs of Medicare home health care; further analyses must evaluate the consequences of these declines on patient health and access, non-home health expenditures, and other outcomes before final recommendations can be made.
PIC ID: 7738; AGENCY SPONSOR:  Centers for Medicare and Medicaid Services; CONTACT: Ann Meadow; PERFORMER: Mathematica Policy Research, Inc., Princeton, NJ

Prospective Pymt. (c) Prospective Payment for Medicare Home Health: A Promising System to Save Resources

(See Prospective Pymt (b) above for background.)

It was found that the typical agency in the demonstration was able to earn small profits under the prospective payment system.  They were able to do so by reducing their cost per episode by 14 percent--a decrease achieved by reducing service use.  However, efforts also to reduce service use resulted in increases in the agencies cost per visit.  The outcome effectively reduced profits, as approximately half an average agency’s visits were paid according to the per-visit methodology, at a predetermined rate.   Furthermore, the agencies’ Medicare revenues fell by 20 percent, largely because the number of visits rendered after 120 days of care dropped as well.  Although the small number of some types of agencies in the sample makes it difficult to draw firm conclusions, it appears that almost all types of agencies experiences these financial changes.  These results suggest that CMS could save resources relative to cost-reimbursement by implementing a prospective payment system.

PIC ID:  7738.1; AGENCY SPONSOR:  Centers for Medicare and Medicaid Services; CONTACT:  Ann Meadow; PERFORMER:  Mathematica Policy Research, Inc., Princeton, NJ

Study of Pharmaceutical Benefit Management

The purpose of this study was to assist the Centers for Medicare and Medicaid Services in selecting a pharmacy benefit manager (PBM) as it implements a Medicare drug benefit.  As the possibility of extending a Medicare benefit moves through legislation, Congress and the CMS will need to decide how to obtain discounted pricing, the benefits and services it will offer, and whether it will use the PBM’s formulary and clinical services or design its own. 

PBMs have emerged as the national standard for the administration of prescription drug insurance in the United States.  They manage drug benefits of approximately 70% of Americans, including 65% of the country’s seniors.  This is also a very successful industry in that PBMs have the ability to reduce costs, provide national pharmacy access, and administer benefits that are customized to meet the needs of a wide range of clients in a highly automated environment. 

The PBM industry, suggests the report, would not be overwhelmed by providing services to the Medicare population because the industry is relatively insensitive to volume increases.  However, better rebates and administrative fees could result from the creation of a Medicare formulary, which could help control Medicare drug costs.  A PBM could be contracted with for administrative services, such as claims processing, Medicare formulary, management, member services, and clinical control.  This scenario would maximize price discounts and rebates.
PIC ID: 7591; AGENCY SPONSOR:   Centers for Medicare and Medicaid Services; CONTACT: Peri Iz; PERFORMER: Price Waterhouse, Washington, DC

Discontinuous Coverage in Medicaid and the Implications of 12-Month Continuous Coverage for Children

This report is a policy analysis that examines the implication of 12-month Continuous Coverage for children under the Medicaid program by simulating enrollment based on 1994-95 State Medicaid Resource File (SMRF) data from California, Michigan, Missouri, and New Jersey. 

The researchers modeled what the enrollment, payments, and ER use would have been had the states implemented this policy.  Four types of yearly claims files were included: inpatient hospitalization, drugs, long-term care, and other services.  Claims data from California were also used to examine impacts on quality of care as reflected in use and payments for ER care.  Analyses included only children who could be eligible for Continuous Coverage; thus, medically needy, SSI, special refugee status, and other groups were excluded. 

Findings indicated that a policy of 12-month Continuous Coverage would reduce, but not eliminate, the problem of discontinuous coverage; the policy does extend coverage to more than half of children who would otherwise lose it.  In addition, results showed that Continuous Coverage would increase total Medicaid payments, but that over time, this policy could decrease overall health costs as acute episodes are presented.  Continuous Coverage would also reduce estimated administrative costs related to enrollment and disenrollment as children move in and out of plans, but only by a small percentage.  Findings were inconclusive with regard to analyses of costs.  The researchers conclude that Continuous Coverage holds promise as a policy option that could improve continuity of care and costs for children under Medicaid.  They state that future analyses should be conducted to examine the implications of these findings.
PIC ID: 7774; AGENCY SPONSOR: Health Resources and Services Administration; CONTACT: Jacob Tenenbaum; PERFORMER: Mathematica Policy Research, Inc., Princeton, NJ

The Medicaid DSH Program and Providing Health Care Services to the Uninsured: A Look at Five Programs

Perhaps one of the most important health policy issues facing the United States is how to care for the uninsured.   Uninsured individuals often lack access to appropriate care, but they still use health services when they become ill and, in many cases, they do not have the financial resources to fully pay for their care.  Through various public programs--local, state, and federal--many hospitals receive subsidies to help pay for the costs associated with uncompensated care. 

One of the largest subsidy programs is the Medicaid disproportionate share hospital (DSH) program.  This study provides some insights on the experience of five programs that have used Medicaid DSH funds to enhance care for the uninsured.  Those programs were: Denver County (Denver), Colorado; Marion County (Indianapolis), Indiana; Ingham County (Lansing), Michigan; Wayne County (Detroit), Michigan; and Bexar County (San Antonio), Texas.  These programs reflect diversity in target population, organization, financing, delivery system and services provided. 

The findings indicate the Medicaid DSH funds were an important source of funding for all programs.  The programs examined here highlight how DSH funds have been used in a positive way:  To provide health care services to the uninsured.  By emphasizing primary care services, all of these programs, as diverse as they are, aim to reduce hospitals’ uncompensated care burdens, one of the principal goals of the DSH program.  The programs in this study provide models by which states and localities could provide primary care services using Medicaid DSH funding.
PIC ID: 6703; AGENCY SPONSOR:   Office of the Assistant Secretary for Planning and Evaluation; CONTACT: George Greenberg; PERFORMER: Urban Institute, Washington, DC

Human Service Programs

Outstanding Evaluation

The Family Transition Program: Final Report on Florida’s Initial Time-limited Welfare Program

Florida’s Family Transition Program (FTP), initiated in 1994 in Escambia County, was designed to limit the number of months that welfare could be collected.  This fifth and final report examines the implementation, implication, and cost benefits of FTP among 2,817 recipients randomly assigned to receive FTP or standard AFDC between 1994 and 1999.  Excluded from the sample were families with disabled or chronically ill members and/or with children under seven months of age.  At the time of enrollment, the FTP program included a time limit on cash assistance and an array of support services, whereas traditional AFDC had neither (although this was later changed). 

The evaluation utilized information from a variety of sources, including baseline data collected prior to random assignment; administrative records (e.g., AFDC/TANF payments, food stamp benefits); and various stakeholder surveys, such as four-year client surveys in 1998 and 1999 that included questions on employment, household income, and child well-being. 

Findings indicated that most families on AFDC left welfare during the study period; however, FTP substantially reduced long-term welfare receipt, with only 6 percent receiving welfare for more than 36 months versus 17 percent of those in the AFDC group.  In addition, FTP families gained more in earnings than they lost in welfare, resulting in modestly higher incomes; only 17 percent of FTP families reached their time limits during the study period.  While results indicated that FTP had few impacts, positive or negative, on the well being of elementary school children, adolescents in the FTP group performed somewhat worse on some of measures of school performance (e.g., expulsions).   A cost-benefit analysis showed that program expenses were about $8,000 more per client over the four-year period when compared to typical welfare costs.  
PIC ID: 6820; AGENCY SPONSOR: Administration for Children and Families; CONTACT: Alan Yaffe; PERFORMER: Manpower Demonstration Research Corporation

Head Start FACES (Family and Child Experiences Survey): Longitudinal Findings on Program Performance (Third Progress Report)

Head Start FACES is an ongoing national longitudinal study of the cognitive, social, emotional, and physical development of Head Start children; the characteristics, well-being, and accomplishments of their families; the observed quality of Head Start classrooms; and the characteristics and opinions of Head Start teachers and other program staff. 

Using a nationally stratified random sample of 3,200 children and families in 40 Head Start programs, the evaluation addressed four study questions: 1) Does Head Start enhance children’s development and school readiness; 2) Does Head Start strengthen families as the primary nurturers of their children; 3) Does Head Start provide children with high quality educational, health, and nutritional services; and, 4) How is classroom quality related to child outcomes?  Data components (child assessment, parent interview, teacher and staff interviews, and classroom observations) were collected at five different points in time, and 80 percent response rates were achieved.  Multiple measurement instruments and scales were used.  Quantitative information was enhanced with case studies of 120 families, also selected randomly (3 per site). 

Based on Fall 1997 and Spring 1998 data, positive findings indicated that Head Start participation narrowed the gap between disadvantaged children and all children in vocabulary and writing skills, and led to improvements in areas such as social skills, word knowledge, and math skills.  However, there were no improvements on literacy measures (e.g., letter-word identification), and behavior problems were largely unchanged.  With regard to families, the parents of Head Start children reported extremely high levels of satisfaction (over 85 percent) with the program, and a greater sense of control over their lives after one year.  However, participation by fathers in their children’s lives remained low, and family social support and mental health needs did not change.
PIC ID: 6331.2; AGENCY SPONSOR: Administration for Children and Families; CONTACT:  Louisa B. Tarullo; PERFORMER: Westat, Ellsworth Associates, Abt Associates, Inc., and the CDM Group

Outstanding Evaluation

Head Start Children’s Entry into Public School: A Report on the National Head Start/Public School Early Childhood Transition Demonstration Study

This longitudinal study examined the scholastic functioning of 7,515 former Head Start children from 31 diverse sites across America, with approximately half the children receiving extended comprehensive services for grades K through 3.  Modeled after Project Head Start, the National Transition Demonstration Project was passed by Congress in 1991 to provide extended developmentally appropriate and comprehensive school-based services to Head Start graduates. 

The 12-chapter report covers a range of topics, beginning with a review of social policy behind the Head Start Transitional Program and the research questions to be addressed in the study, and ending with analyses of the main predictors of children’s school functioning.  In this regard, growth curve analyses and hierarchical linear modeling (HLM) were utilized to identify influences that promote and detract from children’s achievement.  

Overall, the most impressive finding was the rapid growth demonstrated in math, reading, and social development.  Head Start children from both groups made rapid gains between kindergarten and third grade, often starting out a standard deviation below the mean in kindergarten and reaching the national mean (or above) by grade three.  Results also highlighted the contribution that services and other aspects of the transition program make toward scholastic achievement and social development.  Family gains were noted as well, including economic growth in transition families.
PIC ID: 4393; AGENCY SPONSOR: Administration for Children and Families; CONTACT: Mary Bruce Webb; PERFORMER: Civitan International Research Center, the University of Alabama at Birmingham, Birmingham, AL

Outstanding Evaluation

Building Their Futures: How Early Head Start Programs Are Enhancing the Lives of Infants and Toddlers in Low-income Families

Early Head Start provides services to low-income pregnant women and families with infants and toddlers in order to improve children’s development (cognitive, language, social-emotional, and health); to foster close supportive relationships between parents and their infants and toddlers; and to encourage the development of community partnerships.

This report summarizes the interim results of an evaluation initiated in 1995 that includes about 3,000 children and families across 17 sites selected to represent a variety of milieu (regional, rural, and urban), ethnic and racial compositions, and types of program approaches.  At each site, children and families were randomly assigned to program and control groups; follow-up was carried out over three years of program participation through a child’s third birthday.  Results described in this report reflect program impacts through the second birthday of study participants, with data collected from researcher observations, parent reports, and direct child assessment.

Findings indicate that at two years of age, Early Head Start children demonstrated higher positive impacts on infant and toddler cognitive development, vocabulary and sentence complexity, and aggressive behavior than did the control group (children not in Early Head Start).  However, no differences were found between the two groups in terms of variables such as regulation of emotions, task-oriented behavior, or attention span (videotaped freeplay interaction with mothers).   With regard to the parents of two year-olds, results showed that parents of Early Head Start group children learned more about early childhood development and were more likely to provide experiences and environments that supported development than did control group parents.
PIC ID: 3570; AGENCY SPONSOR: Administration for Children and Families; CONTACT: Rachel Chazan Cohen; PERFORMER: Mathematica Policy Research, Inc., Princeton, NJ

[Previous] [Table of Contents] [Next]