Performance Improvement 2000. Program Effectiveness

01/01/2000

Case Studies of the PMI Local Demonstration Site Projects: Experiences During Transition and Implementation

Highlights

Young people under 26 may be at risk for HIV and other sexually transmitted diseases and need information on how to prevent it. The Prevention Marketing Initiative (PMI) was a 5-year demonstration of social marketing initiated by the Office of HIV/AIDS of the Centers for Disease Control and Prevention in 1993. Prevention marketing involves social marketing, the behavioral sciences, and community participation. Both local and national in scope, PMI has three major components: local demonstration sites, national health communications, and prevention collaborative partners such as health communication firms.

This study evaluates the implementation phase of local demonstration projects in five sites. During the five years of the project, researchers conducted two case studies involving prevention marketing HIV/STD programs for young people under the age of 26. The first, completed in 1996, reported on the interventions designed by the local sites. The second describes the experiences of the local sites and the collaborative national partners who provided technical assistance during the transition and implementation phases of the project.

Purpose

The purpose of the PMI demonstrations was to test the efficacy of combining social marketing with elements of behavioral sciences and community participation in reducing the risk of AIDS in young people. The researchers applied the concept of prevention marketing to the demonstration sites as an experiment in demonstrating the prevention marketing process, including the skills and resources needed to engage the community effectively; measuring the behavioral effects of data-based prevention marketing interventions; and documenting the lessons learned that may inform future prevention-marketing efforts. In addition to describing the experiences of the project participants, information in the second case study also will provide context for further evaluation of both outcomes of site-based skills-building workshops and the community-wide level of exposure to media messages.

Background

During the planning of the project, each site considered intervention components previously shown as effective, with each site tailoring intervention components to its target audience and prevention objectives. Sites chose multifaceted interventions using social marketing techniques to reach the targeted population delivered through workshops and outreach components. An initial task was to hire staff and build community participation among key organizations and individuals. One national partner, the Academy for Educational Development (AED), and its partner, Porter/Novell, provided intensive technical assistance, focusing on developing organizational structures, establishing procedures for managing potential resistance to prevention programs, and building community participation.

In the following year, demonstration sites received technical assistance in conducting formative research, considered a critical step in social marketing. It involves an assessment of local needs and resources, helping to define target audiences and behavioral objectives through focus groups and other research, such as community environment and epidemiological profiles. Some sites added development of community networks or the support of community leaders to their plans.

Local sites chose interventions to reduce levels of selected risk behaviors among their target audiences, adapt programs to their local needs, and plan for implementation during the next two years. AED continued to provide technical and managerial support through the end of the fifth year. In the final year of the project, sites focused on fielding their efforts, evaluating them, and planning for continuity of their efforts.

Methods

This study had two major objectives: to describe changes in organizational structures and processes that had occurred since the first case study was conducted in 1996, and to document experiences with the major components of PMI during its final two years. The components involved social marketing, community collaboration (including media relations), behavioral science, youth participation, and technical assistance. Collecting and analyzing qualitative data was guided by seven research questions designed to meet the major objectives.

The case study used a qualitative approach to capture the experiences and recommendations of the PMI participants. A two-person field team visited each site and conducted interviews with 64 people. They conducted 10 more interviews with the national partners. The team used these sources of information to evaluate the project: (1) interviews with PMI staff, volunteers, and implementation partners; (2) observations of meetings or activities that occurred during the data collection period; (3) interviews with such national partners as technical assistance providers and project officers; (4) a review of current site-based documents; and (5) the final report for the initial case study. The data collected and analyzed then were used to create a thematic database, case summaries, and the final report representing an integrated summary of findings across all sites.

Findings

All five PMI sites contained structural elements that remained the same throughout the project. These included a lead agency, an advisory committee, an on-site staff, a youth committee, and technical-assistance providers. The sites experienced high turnover in organizations and individuals as the PMI project moved from planning to implementation. The lead agency changed at all five sites because of a mismatch of PMI site objectives and those of the lead agencies. The sites selected new agencies to manage implementation of the interventions. The role of the advisory committees changed from planning to project oversight; however, several sites experienced declining attendance and interest when they failed to define a clear role for their advisory committees.

A significant change in the program plan was to engage subcontractors to field the intervention components and evaluate the interventions. One aspect of the intervention implementation seen by respondents as highly successful was the way the various components of programs worked together and complemented each other.

Several forms of evaluation were used to document the effects of PMI, including: (1) community- based telephone interview in Sacramento assessing the scope and effects of the program; (2) a workshop evaluation to assess the effectiveness of the program for participants 3) process evaluation including this case study; (4) monitoring and accountability, and (5) formal and informal evaluations conducted by the sites themselves.

The community-based outcomes study involved a random telephone survey of teenagers inquiring about their attitudes, knowledge gains, and sexual behavior as a result of the PMI project. Sacramento provided convincing evidence that exposure to the social-marketing campaign produced the desired results–increased condom use among sexually active teenagers. More than 60 percent of the teens in the Sacramento target area were exposed to the social-marketing campaign through multiple channels.

Use of Results

The Prevention Marketing Initiative is a model that may be replicated and modified to fit the needs of other communities. The final report lists numerous recommendations based on the findings of the case study and lessons learned from the experiences of the project participants that will be of value to future public health promotion efforts.

AGENCY SPONSOR: Centers for Disease Control and Prevention

FEDERAL CONTACT: Nancy Chalmers

PHONE NUMBER: 404-639-7085

PERFORMER ORGANIZATION: Battelle, Durham, NC

The Impact of Health Claims on Consumer Search and Product Evaluation Outcomes: Results from FDA Experimental Data

Highlights

The relationship between the regulation of health claims on food labels and the actual effects on consumer behavior in searching for and using the information provided is the subject of this study. The researchers interviewed more than 1,400 shoppers to assess and record how consumers use food label information appearing in different areas of the label when evaluating a product and making a purchase decision. The results suggested that consumers are likely to read no further than the health and nutrient-content claims on the front of the package and to give greater weight to the information mentioned in them than that contained in the Nutrition Facts panel.

Health claims also were found to produce a halo effect, extending to other health aspects not mentioned in the claim, and, in the case of one of three food products tested, a magic-bullet effect, attributing inappropriate health benefits to the product. The study acknowledges and extends earlier research and discussion of consumer information search behavior and regulation of health claims on food labels by the Food and Drug Administration.

Purpose

The authors wanted to better understand the extent of consumer reliance on health claims appearing on the front of food packages and their extended search to the back of the package to view the Nutrition Facts panel. The study sought to answer this question: “Do consumers rely only on claims when evaluating a product and making a purchase decision, and if so, how does this affect their nutritional judgments and the quality of their choices?”

Background

The 1990 Nutrition Labeling and Education Act (NLEA) regulates health claims on food labels. NLEA was enacted to “assist consumers in maintaining healthy dietary practices” amid concerns that unregulated health claims might be confusing and deceptive.

Health claims first appeared on Kellogg products in 1984. The number and types of diet and disease linkages used in advertising and on food labels throughout the early 1990s rapidly increased following Kellogg’s initiative. Research indicated that health claims and product advertising since 1984 have increased consumer awareness of diet-disease linkages and given rise to new or reformulated food products with greater nutritional value. Some, however, believed that health claims and product advertising were incomplete or misleading and failed to provide accurate information to consumers about the nutritional content of the product.

The Food and Drug Administration issued implementing rules for NLEA in May 1994, limiting claims to those supported by “publicly available scientific evidence” and the types of products that display health claims to those that meet a set of global nutritional standards. New products following enactment of NLEA tended to be healthier by reducing the level of negative nutrients, such as fat. However, researchers have found it difficult to measure the effects of health claim regulations, given the NLEA requirement that the Nutrition Facts panel appear on food labels. As early as 1973, numerous studies have been conducted using experimental protocols to understand the effects of nutrition information on consumer behavior.

Methods

A total of 1,403 primary food shoppers were interviewed at eight United States metropolitan statistical areas located in the Northeast, Southeast, Midwest, and Southwest. Interviews were conducted with each subject following initial screening on the basis of the relative amount of shopping, age, literacy, and

ability to read small print. Participants were presented three-dimensional mock-ups of well-known products (with brand names removed) under three categories of products, cereals, lasagna, and yogurt. The researchers experimentally manipulated an independent variable, creating ten variations in label conditions. Participants were given time to examine the package of each product prior to being asked a set of prescribed questions. In each question series, the researcher observed whether the respondent looked at (a) only the front panel of the package, (b) only the Nutrition Facts panel, (c) both the front and the Nutrition Facts panels; or (d) neither panel. The researchers tested ten separate outcome predictions relating to the effects of the truncated information search and the presence of health or nutrient-content claims on consumer information search and processing behavior.

Findings

The results of inferential statistical tests suggested that the presence of a health claim and a nutrient claim on a food package has a significant effect on the relative degree to which consumers limit their search for nutrition facts to the front panel of food packages. Consumers who either truncate their information search to the front or view nutrient claims are significantly more positive in their summary judgments of products. These consumers also are more prone to give greater weight to the information cited in the health claims than to that presented in the Nutrition Facts panel.

Information search behavior affected product ratings and health evaluations, independently from the presence of a health claim. For example, those who look only at the front panel give significantly higher purchase intention ratings than those who consult the Nutrition Facts panel. Researchers discovered this effect to be as large as those associated with product type or education level. They also found that the presence of a claim on a food package significantly influenced the halo effect described earlier, and for one product tested, a magic-bullet effect.

Use of Results

The findings of the study are useful and of interest to several audiences. For the FDA, the findings support the rationale for regulation of health claims. Based on the data, the authors suggest that manufacturers may find it “advantageous to rely more heavily on” nutrient-content claims, because the regulations permit a greater number of products to qualify for nutrient-content than for health claims. The effect of the latter on consumer information search and evaluation were found to be similar to that of health claims. Findings from the study may increase public awareness and encourage consumers to read beyond health claims. It extends previous research efforts and also suggests further avenues for explanation of consumer search and product evaluation behavior.

AGENCY SPONSOR: Food and Drug Administration

FEDERAL CONTACT: Alan Levy

PHONE NUMBER: 202-205-9448

PERFORMER ORGANIZATION: Food and Drug Administration, Rockville, MD

The Costs and Benefits of Substance Abuse Treatment: Findings from the National Treatment Improvement Evaluation Study (NTIES)

Highlights

This study investigated the costs and benefits that accrue as the result of substance abuse treatment. Using data from NTIES, the research team constructed and compared estimates of treatment costs, crime-related and health care costs, and the income of 5,264 substance abusers in the periods before and after treatment. The findings of the study indicate dramatic reductions in crime-related costs, modest reductions in health-care costs, and modest increases in the earnings of substance abusers in the period after treatment.

Purpose

Despite federal expenditure of over $2 million and an overall national expenditure in excess of $12 billion, it is estimated that about 1 million substance abusers in need of publicly supported treatment do not receive services. Policymakers often face the difficult task of justifying to taxpayers the use of public funds to support substance abuse treatment. For the public and policymakers, the relevant question relates to the degree to which substance abuse treatment provides benefits for the rest of society. Thus, the economic soundness of relevant policies depends on the cost effectiveness of treatment. The primary purpose of this study was to assess the costs and benefits of alcohol and drug abuse treatment and determine the resulting economic benefits to society.

Background

One of the major goals of the DHHS Center for Substance Abuse Treatment (CSAT) is to improve the lives of those affected by alcohol and other substance abuse, and through treatment, to reduce the adverse effects of substance abuse on individuals, families, communities, and the society at large. Toward this goal, CSAT has invested and continues to invest significant resources in the development and provision of effective substance abuse treatment and recovery services. In 1996, Federal expenditure accounted for almost $2.7 billion of the about $12.7 billion national expenditures for the diagnosis and treatment of alcohol and drug abuse. Issues related to the investment in or continuation of existing programs and the development of new programs generally center on their efficacy and the extent to which they are cost effective. The collection and systemic analysis of high-quality data about substance abuse treatment services, clients, and outcomes provide sound scientific evidence upon which to address such questions as which treatment approaches are cost-effective methods for curbing addiction and addiction-related behaviors and what kinds of treatment work for what groups of clients. The National Evaluation Data Services (NEDS) project was established to provide CSAT with data management and scientific support services across an array of programmatic and evaluation activities. One of the specific objectives of the NEDS project is to provide CSAT with the capability to use existing data to address policy-relevant questions about substance abuse treatment. The investigation of the costs and benefits of alcohol and drug abuse treatment and the economic benefits to society was undertaken in pursuit of this objective.

Method

The research task for this study was accomplished by analyzing NTIES “client-level” data and service delivery (SDU) administrative data collected between FY1992 and FY1995 on 5,264 clients who were in treatment demonstration programs supported by CSAT. Using SDU’s administrative/ service data, the research team constructed estimates of treatment costs per client for each of the modalities and across all modalities. To measure benefits, the research team estimated and compared the average costs per client, in terms of crime-related and health-care cost associated with substance abusers, and the income and social welfare benefits received by substance abusers in the 12 months prior to and after treatment. The difference between pre-treatment and post-treatment costs was used to estimate the economic impact of substance abuse treatment.

Findings

The analysis of data revealed: 1) approximately 75 percent reductions in crime-related costs; 2) 11 percent reduction in health-care costs per client; and 3) 9 percent increase in the earnings of substance abusers in the period after treatment. No change was found in welfare payments and Supplemental Social Security Income. Overall, the results obtained indicate that treatment created an average benefit to society of $9,177 per client and an average benefit to the non-treated population of $12,477 per client. Deducting the cost of treatment per client resulted in average treatment benefits of $6,236 and $9,536 per client for society and the non-treated population, respectively.

The research team concluded that, based on these results, the total benefits of substance abuse treatment, in terms of avoided costs and increased earning, far exceeded the costs of treatment. As a recommendation, the researchers indicated that the next steps should include analyses to identify what kinds of treatment works for whom. The team intends, in future work, to link intensity and cost of services to outcomes to identify the cost effectiveness of different treatment services.

Use of Results

Because of the large numbers of incomplete responses to cost questions in the baseline and administrative survey questionnaires, and discrepancies between total revenues and sum of details reported by SDU's the researchers choose to proxy treatment costs with information on revenues. Nevertheless, there is no evident reason to believe that the outcomes of the study would be any different, especially in light of the special steps the researchers took to ensure the reliability of the information derived.

AGENCY SPONSOR: Substance Abuse and Mental Health Services Administration

FEDERAL CONTACT: Ron Smith

PHONE NUMBER: 301-443-7730

PERFORMER ORGANIZATION: Caliber Associates, Fairfax, VA