Performance Improvement 2002. Healthcare/Public Health


Outstanding Evaluation

Reforming Medicaid: The Experiences of Five Pioneering States with Mandatory Managed Care and Eligibility Expansions

This report summarizes findings of a six-year evaluation of Medicaid 1115 demonstrations (managed care) in five states - Hawaii, Maryland, Oklahoma, Rhode Island, and Tennessee - that were initiated in 1994-1997. Information was collected from various sources, including case studies of the demonstration states and household surveys of low-income insured people and people enrolled in demonstrations. Site visits, program documentation, and focus groups were used to assess implementation impacts on states, plans and providers. To assess impacts on beneficiaries, the researchers examined data on the Medicaid population before and after the demonstration period. Current population survey data and aggregate Medicaid cost data were also analyzed.

The evaluation found that:

  • Each of the five states made major investments of time and resources to accomplish the sweeping changes they implemented in their Medicaid programs.
  • By introducing managed care, states have enhanced the potential for improving the quality of care delivered to Medicaid beneficiaries.
  • The states were successful in contracting with a relatively stable set of HMOs to provide services to Medicaid beneficiaries.
  • Change in medical costs per beneficiary relative to the pre-demonstration period were fairly similar for the five demonstration states and the nation.
  • Most Medicaid HMOs made money during the early years, but setting rates fairly required ongoing negotiations and adjustments.
  • Comparison of managed care and fee-for-service Medicaid for Tennessee showed no differences in prenatal care or birth outcomes.
  • Beneficiaries’ access to care and satisfaction varied widely across plans and plan characteristics.
  • Rural beneficiaries in Hawaii, Tennessee, and Oklahoma fared nearly as well as did urban enrollees.
  • Experiences of people with disabilities, especially those with mental illnesses, suggest room for improvement in managed care.
  • Coverage expansion in Hawaii and Tennessee yielded much better experiences for the previously uninsured.

PIC ID: 6289.1; AGENCY SPONSOR: Centers for Medicare and Medicaid Services; CONTACT: Penny Pine (CMS), Jeff Buck (SAMHSA), Hunter McKay (ASPE); PERFORMER: Mathematica Policy Research Inc., Princeton, NJ

Outstanding Evaluation

Nurse Staffing and Patient Outcomes in Hospitals

In response to Congressional concern about the impact of staffing cuts on quality of care, HRSA, CMS, AHRQ, and NINR jointly funded this study, which was designed to identify outcomes of patient care sensitive to nursing and to analyze the relationship between these outcomes and nurse staffing in inpatient hospital settings.

The analyses were conducted on 799 hospitals from 11 states diversified with respect to size and geographic location. In addition, data taken from CMS discharge abstract files for a national sample of 3,357 hospitals was used to analyze these relationships for Medicare patients. A sensitivity analysis was also undertaken with a data set of 256 California hospitals that provided service-level data for nurse staffing (as opposed to data aggregated at the hospital level).

Results differed for medical and surgical patients. Staffing (the number and credentials of staff) was consistently linked to urinary tract infections, pneumonia, length of stay, upper gastrointestinal bleeding, and shock/cardiac arrest for medical patients, and to failure to rescue for surgery patients. Weaker relationships were observed for other outcomes. The investigators concluded that higher numbers of registered nurses (RN) were associated with a 3 to 12 percent reduction in rates of adverse health outcomes. Higher staffing levels of all types of nurses (RNs plus Licensed Practical Nurses (LPN) and aides) were associated with a decrease in these adverse outcomes, from 2 to 25 percent. Specifically, the study demonstrated that the strongest relationship overall between outcomes and staffing was associated with an increase in the number of RNs (i.e., LPN and aide staffing were not consistently related to the outcomes). The evaluators also pointed out the “catch 22”: higher staffing levels may lead to better charting of some adverse outcomes, thereby confounding the relationship between staffing levels and negative outcomes.

PIC ID: 6864; AGENCY SPONSOR: Health Resources and Services Administration; CONTACT: Carol Gassert; PERFORMER: Harvard School of Public Health, Boston MA

Do Food Labels Work? Gauging the Effectiveness of Food Labels Pre- and Post-NLEA

This chapter in the Handbook of Marketing and Society is a synthesis of findings on the use and apparent consequences of food labeling, with special attention to the possible changes associated with implementation of the National Labeling and Education Act (NLEA). NLEA had many features, including mandatory nutrition labeling for most packaged foods, uniform definitions for terms such as “light”, and limits on allowable health claims.

The chapter summarizes and integrates findings from investigations that used a variety of methods, including sample surveys (before and after NLEA implementation), experiments, focus groups, and analysis of food consumption trends. Populations used were generally not explicitly reported (but seemed apparent for sample surveys).

Findings were summarized for consumer effects (including awareness of labels, label use, changing food decisions based on label information, effects of label claims, and dietary impacts) and marketplace impacts (including prevalence of claims on labels, introduction of healthier brands, and market share). Key findings included increasing awareness of the new label over time and significant levels of reported label use to make product choices. The food label caused some consumers to stop buying products or to try new products. It also provided the types of information that consumers are most concerned about (e.g., dietary fats) and presented the information more clearly than previously, according to the authors.

PIC ID: 7735; AGENCY SPONSOR: Food and Drug Administration; CONTACT: Alan Levy; PERFORMER: Food and Drug Administration, Washington, DC

The Efficacy of Interventions to Modify Dietary Behavior Related to Cancer Risk, Vols. I and II

This review seeks to clarify the existing knowledge base on behavioral interventions to alter dietary behavior related to cancer risk and offers directions for future research. Interventions consisted of behavioral dietary interventions on fruit and vegetable intake and behavioral dietary interventions on decreases in fat intake.

In the studies that were reviewed, dietary interventions were consistently associated with an increase in fruit and vegetable consumption (with greater increase noted for fruit intake). More than three-quarters of the studies reviewed reported statistically significant increases in fruit and vegetable intake. Several dietary intervention components appear to be particularly promising in modifying dietary change favorable to cancer risk reduction: social support, goal setting, small groups, food-related activities, and the incorporation of family components.

Using the differences-in-deltas approach, the reviewers determined that the average increase in fruit and vegetable intake reported was 0.6 servings per day, and consistent decreases were noted in the intake of total fat and saturated fat. The mean change in total fat intake was estimated as 7.3 percent reduction in the percentage of calories from fat. Interventions appeared to be more successful at positively changing dietary behavior in populations at risk of (or diagnosed with) disease than in healthy populations.

PIC ID: 7691; AGENCY SPONSOR: Agency for Healthcare Research and Quality; CONTACT: Kathryn Rickard; PERFORMER: Research Triangle Institute, Research Triangle Park, NC

5 A Day for Better Health Program Evaluation Project

The purpose of this project was to review and evaluate (1) the science underlying the 5 A Day program, (2) the implementation and accomplishments, and (3) the degree of achieved goals and objectives. Twelve scientific experts, as members of the Program Evaluation Committee, reviewed the scientific evidence and program infrastructure (materials, collaborations, partnerships, message delivery, implementation and process measures) and supported research grants.

During the first 10 years of the 5 A Day Program, public awareness on the importance of vegetable and fruit consumption and knowledge of the 5 A Day Program and its message has increased. The message has reached more women than men, and more whites than Latinos or African Americans. Evidence indicates that a diet rich in vegetables and fruits reduces the risk of cancer and other chronic diseases. For example, the evidence of a relationship between higher vegetable and fruit consumption and lower risk of several types of cancers has been strengthened.

It is recommended that the 5 A Day Program should continue as a multifaceted program to support research and applied public health efforts to promote increased vegetable and fruit consumption.
PIC ID: 7635; AGENCY SPONSOR: National Institutes of Health; CONTACT: Kevin Callahan; PERFORMER: SCG: The Scientific Consulting Group, Inc., Gaithersburg, MD 20878-1409

The National Cross-Site Evaluation of High-Risk Youth Programs: Final Report

The purpose of this study was to determine the effectiveness of various programs designed to prevent substance abuse in high-risk youth.

This rigorous research involved more than 10,500 at-risk 9- to 18-year-olds in communities across the United States. Of 48 sites in this Cross-Site Evaluation, three were programs associated with universities or medical centers, three were higher education institutions, three were public school districts, and there were three state, county, or tribal agencies. The majority of grantees were small community-based programs not affiliated with larger institutions. The programs ranged in delivery method from primarily classroom-style, didactic programming in which youth receive substance abuse prevention-related information through lectures, videos, or other similar means, to interactive and experiential programming where youth are required to actively participate, reflect on the subject at hand, and act upon it. There were four areas of emphasis for programming: (1) programs that focused on developing life-skills (e.g., refusal skills, communication skills, anger management, and other social skills); (2) programs that primarily provided knowledge of substance abuse, and other attendant problems (e.g., teen pregnancy, AIDS, gangs, and violence); (3) programs focused on developing self-esteem, or “affective-dominant” programs; and (4) recreation-dominant programming.

Typically, the interactive type of programs included role play activities, team-building projects, outdoor wilderness experiences, or other activities requiring active engagement. These type programs were found to be more effective in reducing substance abuse than programs that were less interactive.

The research shows that “prevention works”--programs designed to prevent substance abuse produce lasting reductions in substance use and have important positive effects on the lives of youth at risk. The results represent accomplishments by programs operating in communities where youth face multiple risk factors. Even within the challenging circumstances of these programs, the entire pool of participants across all programs decreased their use of alcohol and marijuana relative to similar youth who did not participate in the programs. The study findings have important implications for emphases in prevention design and implementation. By contributing to understanding the factors that prevent substance abuse in youth, and producing science-based lessons about effective prevention strategies and program practices, this major research effort holds the promise of more effective prevention and more positive futures for America’s youth who are at high risk.
PIC ID: 6044; AGENCY SPONSOR: Substance Abuse and Mental Health Services Administration; CONTACT: Soledad Sambrano; PERFORMER: EMT Associates, St. Louis, MO

Effectiveness of Women’s Substance Abuse Treatment Programs: A Meta-Analysis

Recent data suggest that at least 4.5 million women in the United States are currently alcohol abusers, 3.1 million are regular users of illicit drugs, and 3.5 million misuse prescription drugs. The current demand for effective substance abuse treatment programs for women is, therefore, high. To date, however, no formal quantitative synthesis or meta-analysis of the women’s treatment literature has been conducted. In addition, compared to other social services fields, only a handful of meta-analyses have been conducted in the substance abuse treatment field.

This technical report represents an attempt to fill this key knowledge gap by describing a quantitative, meta-analytic synthesis of the literature on the effectiveness of substance abuse treatment for women. This study focused on the effectiveness of women-only treatment programs; women-only treatment programs compared to mixed-gender treatment programs; and, enhancements to women-only treatment programs. The meta-analysis consisted of defining criteria for study inclusion, a literature search, study screening and selection process, the identification of relevant and representative outcome domains, the coding of studies, and quantitative analysis of study outcomes.

The findings are: 1) different types of treatment were positive on average in all domains; 2) women-only programs provide additional treatment benefits beyond those resulting from standard, mixed-gender programs, i.e. women-only programs are best suited to the treatment needs of women; 3) enriching women’s treatment with additional components specifically oriented toward meeting women’s needs adds value above and beyond the expected effects of standard, women-only programs.
PIC ID: 5994.8; AGENCY SPONSOR: Substance Abuse and Mental Health Services Administration; CONTACT: Ron Smith; PERFORMER: Battelle Corporation, Arlington, VA

The Effectiveness of Substance Abuse Treatment in Reducing Violent Behavior

This analysis, using data from the National Treatment Improvement Evaluation Study (NTIES), addresses the important policy issue of the association between substance abuse and violent behavior and the manner in which effective substance abuse treatment can reduce the likelihood of interpersonal violence.

Reductions in violence were analyzed by gender, treatment modality--methadone treatment, non-methadone outpatient treatment, long-term residential treatment, short-term residential treatment, and facilities located in correctional institutions--and whether the client was a victim of violence in addition to being a perpetrator.

The findings indicate that treatment for substance abuse can be highly effective in reducing specific violent acts among persons with a history of committing violence (i.e., perpetrators). It was also found that these improvements are consistent across client subgroups such as gender, modalities of treatment, and past status as a victim of violence.
PIC ID: 5994.10; AGENCY SPONSOR: Substance Abuse and Mental Health Services Administration; CONTACT: Ron Smith; PERFORMER: Caliber Associates, Fairfax, VA

Assessing the Effectiveness of Hazard Analysis Critical Control Point (HAACP) Systems in the Seafood Industry

The Seafood Hazard Analysis Critical Control Point (HACCP) Program mandates the application of HACCP principles to the processing of seafood. HACCP is a preventive system of hazard control that can be used by processors to evaluate the kinds of hazards that could affect products, institute appropriate controls to keep the hazards from occurring or minimize their occurrence, and monitor performance of those controls. This program supplants the old FDA seafood inspection system.

This report provides an overview of the findings from a review of the relevant food safety literature and the advice from HACCP and evaluation experts regarding the feasibility of and alternative methods for evaluating the impacts of the seafood HACCP program.

The major finding is that although an analysis of the impact of HACCP on the number and severity of seafood-borne illnesses in the United States should be the area of highest priority for an evaluation, data limitations make it virtually impossible to assess. One reason is that attributing foodborne illness to a particular food is often not possible. Second, attributing reductions in foodborne illness to a specific regulation may not be possible, since it would be difficult to isolate the impact of the regulations from changes in other factors that may also affect the incidence of foodborne illness.

Recommendations are that an evaluation: (1) assess the impact of HACCP on the incidence and levels of physical, chemical, and biological hazards; (2) identify the set of factors affecting seafood safety; and (3) assess the impacts of those factors on seafood hazard levels, at least qualitatively.
PIC ID: 7343; AGENCY SPONSOR: Office of the Assistant Secretary for Planning and Evaluation; CONTACT: Laina Bush; PERFORMER: Research Triangle Institute, Research Triangle Park, NC