Positive Youth Development in the United States: Research Findings on Evaluations of Positive Youth Development Programs. Effective Competence Promotion Programs in Schools

11/13/1998

Life Skills Training (LST)

This review analyzed two evaluations of the Life Skills Training (LST) in 56 New York State public schools, one covering immediate posttest results of a three-year intervention (Botvin, Baker, Dusenbury, Tortu & Botvin, 1990) and the other, follow-up data collected three years after the end of the intervention (Botvin, Baker, Dusenbury, Botvin & Diaz, 1995).  Both evaluations reflected studies conducted in the school domain, although we note that later replications of this intervention model also target the community domain.  The three-year intervention addressed six positive youth development constructs, including social, emotional, cognitive, and behavioral competencies, self-efficacy, and prosocial norms.

Program Description

The curriculum emphasized the practice of personal and social skills, including decision making, refusal and resistance, anxiety management, communication, and assertiveness (competencies).  Other goals were to increase youth's information and awareness of substance use and abuse, media and advertising influences, and the capacity for self-directed behavior change.  The program was 15 sessions for two sessions per week in the first year, 10 booster sessions in the second year, and five booster sessions in the third year.  Teachers implementing the program received one day of instruction and a detailed program manual.

Research Design

The study used an experimental pre-post-follow-up design that divided the schools into three levels based upon a survey of existing smoking levels (high, medium and low).  Within these levels, schools were divided by geographic region (eastern New York state, central New York state and Long Island), and then randomly assigned to one of three groups: a prevention program with one-day formal teacher training and implementation feedback (E1), a prevention program with videotaped provider training and no feedback (E2), or a no intervention control group C.  There were an uneven number of schools (the first prevention condition had 18 schools, the second prevention condition had 16 schools, and the control condition had 22 schools), which evaluators noted was due to an uneven number of schools in the original assignment blocks, loss of one school after randomization but before intervention, and intentional oversampling of control schools.  The attrition analysis showed greater overall losses in the control group.  The evaluation established equivalence among the groups at pretest on the variables associated with the evaluation's behavioral outcomes Of those students in all groups whose pretest measures indicated substance use, smoking, or marijuana use, those in the control group who used marijuana at pretest had higher attrition rates.  Over the six years of the long-term study, 40% of the original sample were unavailable for follow-up due to absenteeism, transfers, and drop-outs.  There were 5954 students in the original sample drawn from a universal population of students in 56 New York State public schools in grades seven to nine.  The samples used for the posttest analyses included 4466 students who took both pretests and posttests (referred to as the full sample), and 3684 students who were identified as having received a nearly complete implementation of the program (referred to as the high fidelity sample).  The samples used for the long-term analyses included 3597 students who completed both pretests, posttests, and follow-up measures (full sample), and 2752 who received higher fidelity implementation (high fidelity sample).  Across studies, most participating youth were white (91%) and lived with both parents (83%).

Implementation

The study addressed implementation through teacher training and manuals, student guides, staff observation, feedback, and reinforcement.  Implementation was measured by dividing objectives covered during each session by the total number of actual curriculum objectives for the particular session observed.

Immediate Results of the Intervention

The evaluation of immediate posttest results for Life Skills Training (three years after baseline) reported significant changes in youth's knowledge, attitudes, and behavior.  Both intervention groups showed significantly lower cigarette (F(2,3678) = 5.72, p<.003) and marijuana (F(2,3678) = 4.04, p<.01) smoking rates compared to the control group.  The second intervention group (taped teacher training) showed a lower frequency of alcohol intoxication (F(2,3678) = 3.25, p<.04) compared with controls.  Both prevention conditions showed improvements for knowledge of substance use.  The first intervention group (workshop teacher training) showed significantly higher scores than the second intervention group on knowledge of smoking consequences, and both intervention groups had significantly lower expectations than the control group concerning adult smoking norms.  The first intervention group had significantly lower expectations about norms for adult and peer marijuana use.  Both prevention groups had significantly higher interpersonal skills knowledge than the control group.

Results of Long-Term Follow-Up

The long-term follow-up (Botvin, Baker, Dusenbury, Botvin & Diaz, 1995) measured the participants six years after baseline when they averaged 18 years of age, and reported results separately for the full sample and the high fidelity sample.  This evaluation reported only long-term behavioral outcomes associated with substance use.  The authors noted "Carbon monoxide levels in expired air were significantly correlated with self-reported cigarette smoking (r=.35, p<.001) across groups, providing presumptive evidence for the validity of the self-report data" (Botvin et al., 1995:1109).  The long-term follow-up results for the full sample (n=3597) showed significant decreases in monthly smoking rates (E1 = .27, p<.05, E2 = .26, p<.01, C= .33) for both prevention groups compared to the control, and lower weekly rates for both intervention groups as well (E1 = .23, p<.01, E2 = .21, p<.05, C = .27).  The prevalence of heavy cigarette smoking (full sample) was lower for the second prevention group (E2 = .09 vs C = .12, p<.05) compared to the control.  Problem drinking rates (full sample) decreased for both prevention groups compared to the control group youth (E1 = .57, p<.05, E2 = .55, p<.01, C = .59, p<.05).  Polydrug use rates showed lower combined monthly cigarette smoking and alcohol use for both prevention groups (E1 = .21, E2 = .21, C= .29, p<.01), lower combined weekly tobacco and alcohol use for the second prevention group (E2 = .10, C = .14, p<.05), lower combined weekly tobacco and marijuana use for both prevention groups (E1 = .04, E2 = .04, Cl = .08, p<.01), and lower weekly levels of combined alcohol, tobacco and marijuana use for both prevention groups (E1 = .03, E2 = .03, C = .06, p<.05).

The long-term follow-up results for the high fidelity sample (n=2752) of youth also demonstrated continued effects of the intervention.  Six years after baseline, the sample who had received a relatively complete implementation of the program showed significantly lower rates for both intervention groups compared to the control group on measures of monthly (E1 = .24, E2 = .23, C = .33, p<.01), weekly (E1 = .20, E2 = .19, C = .27, p<.05), and heavy cigarette smoking (E1 = .09, E2 = .08, C = .12, p<.05).  Both intervention groups had lower levels of weekly (E1 = .24, E2 = .20, C = .29, p<.01) and heavy (E1 = .53, E2 = .52, C = .59, p<.01) drinking, as well as intoxication (E1 = .31, E2 = .28, C = .40, p<.01).  The second intervention group had lower levels of monthly drinking (E2 = .54, C = .60, p<.01) and the first intervention group had lower monthly marijuana rates (E1 = .10, C = .14, p<.05).  Both groups had lower weekly marijuana rates (E1 = .05, E2 = .05, C = .09, p<.05) compared with controls.  Polydrug use measures showed lower rates for both intervention groups for monthly combined cigarettes and alcohol (E1 = .19, E2 = .1, C = .27, p<.01), monthly combined cigarettes and marijuana (E1 = .05, E2 = .07, C = .12, p<.01), weekly combined alcohol and marijuana (E1 = .03, E2 = .04, C = .07, p<.01), weekly combined cigarettes and marijuana (E1 = .02, E2 = .03, C = .08, p<.01), and weekly combined cigarettes, alcohol, and marijuana (E1 = .02, E2 = .02, C = .06, p<.01).  The first group showed lower polydrug rates of combined monthly alcohol and marijuana (E1 = .09, C = .13, p<.05) and combined monthly cigarettes, alcohol, and marijuana (E1 = .05, C = .10, p<.01). The second intervention group showed lower rates of combined weekly cigarettes and alcohol (E2 = .09, C = .13, p<.01).

The PATHS Program

Greenberg (1996) and Greenberg & Kusche (1997) evaluated the impact of The Providing Alternative Thinking Strategies (PATHS) program, a school-based skills training program thataddressed 10 positive youth development constructs, including social, emotional, cognitive, behavioral and moral competencies, bonding, self-efficacy, resiliency, recognition for positive behavior, and prosocial norms.

Program Description

The PATHS curriculum (Kusche & Greenberg, 1994) included strategies for self-control and coping; self-management of feelings; and interpersonal problem-solving.  The three-step self-management process centered on the use of red, yellow and green stoplights as techniques for the child to guide herself in monitoring and managing behavioral and emotional impulses.  Program principles presume that recognition, labeling, and understanding of emotional states are necessary for effective behavioral self-control, positive peer relations, and good problem-solving abilities.  The self-control unit is a modified version of the stoplight model used in the Yale-New Haven Middle School Social Problem-Solving Program (Weissberg, Caplan & Bennetto, 1988).

Research Design

The study used a pre-post-follow-up experimental design comparing the intervention vs control groups by normally adjusted vs.high risk/special needs students.  For the regular needs subsample, four schools were randomized to either the intervention or control group.  For the special needs subsample, 14 special education classrooms from three school districts were randomly assigned to intervention or control condition.  Posttesting and two follow-up assessments were conducted over the next four years.  At pretest, the participants were in the first and second grades.  At posttest they were in the third and fourth grades.  The program was conducted with an original sample of 426 six- to eleven-year-olds.  The final group of 286 subjects were those who completed all of the individual tests as well as both teacher interviews.  This sample was comprised of all ethnic groups: 165 Caucasians, 91 African Americans, 11 Asians, seven Filipinos, seven Native Americans, one Hispanic, and four children of unknown ethnic identity.  Of the 286, 130 received the intervention (83 regular education; 47 special education) and 156 were in control classrooms (109 regular education; 47 special education.) The evaluation measured the program over one school year, with classes taught three times a week for 20-30 minutes by the regular teachers.  Teachers underwent a three-day training workshop and received weekly consultation and observation from project staff. The original intention of the study was to assess the effects of one versus two years of intervention.  However since only about 30% of children received two years of intervention, there was insufficient power to detect differences in those analyses.

Immediate Results of the Intervention

The posttest results for the normally adjusted subsample showed significant improvements in the intervention group children's abilities to make important discriminations among internal emotional states (the ability to generate positive and negative feeling words (p<.01), to define five complex feelings (p<.01), and to provide appropriate personal examples of different feelings (p<.001); in children's sense of self-efficacy and beliefs that they can manage their feelings (p<.01); improved reasoning with respect to the feelings of others (p<.01); and how feelings change (p<.001).  There were general group improvements in solutions to interpersonal problem-solving, including a higher percentage of prosocial solutions, higher scores for total effectiveness of solutions, less likelihood to show aggressive solutions, and a greater likelihood to show prosocial solutions in second and third choices for solutions (no p values were provided).  On cognitive competence measures there were two significant improvements for the general group: non-verbal reasoning on a test of cognitive skills (p<.01), and visual-spatial reasoning (p<.03).  Improvements in social competence were found only for the boys in the intervention group compared to the control group boys.

Among the special needs youth at posttest, there were self-reported improvements for the intervention group in the ability to generate positive and negative feeling words (p<.01), to provide appropriate personal examples of different feelings (p<.001), in their efficacy/belief that they can hide, manage, and change their feelings (p<.01), in improved reasoning about others' feelings (p<.01),and how feelings change (p<.001).  There was a higher percentage of non-confrontational solutions and a lower percentage of aggressive solutions.  Significant changes in social competence were noted, including specific improvements in internalized symptoms (p<.01), frustration tolerance (p<.01), assertiveness (p<.01), peer social skills (p<.01), and on a combined measure of social competence (p<.001).  Teacher ratings of these children showed improvements in empathy, the ability to stop and calm down, to resolve peer conflicts, to define feelings, and to identify problems.

Results of the Long-term-Follow-Up

The follow-up evaluation of the general group showed continued effects for the intervention on measures of emotional understanding and interpersonal problem-solving skills.  At one year, there were lasting effects for greater self-control and prosocial solutions among the intervention group children, plus sustained improvement in the quality of planning on a task of social planning (p<.05), and a test of concentration and focus (p<.05).  Lower aggression and passivity levels were also sustained (no p values reported). At the second year, there were significant improvements on measures of externalizing behaviors and social/school functioning (p<.05), and lower rates of student-reported conduct problems (year 2).

At one-year follow-up, the special needs group showed continued effects for the quality of social planning (p<.05), focus and concentration (p<.05), less aggression and passivity, more self-control and prosocial solutions, lower rates of student-reported conduct problems and somatic complaints (p<.05), and lower rates of depressive symptoms (p<.01).

Project ALERT

Ellickson, Bell ,and Harrison (1993), Ellickson, Bell ,and McGuigan (1993), and Ellickson and Bell (1990) evaluated the immediate posttest and long-term impact of Project ALERT, a school curriculum that addressed five positive youth development constructs, including social, cognitive and behavioral competencies, self-efficacy, and prosocial norms.

Program Description

This multi-year, school-based program was provided to students in grades seven to eight and used refusal and resistance skills development strategies (competence, self-efficacy) seeking to motivate young people to resist pro-drug influences (prosocial norms) and to give them the capacity for doing so (competence, self-efficacy).

Research Design

The evaluation of Project ALERT used an experimental pre-post-follow-up design in which 30 schools were chosen from eight school districts in California and Oregon.  These districts represented a range of community environments, racial and ethnic groups, and socioeconomic levels.  The schools were blocked by district, and assignment was restricted to a subset balanced among experimental conditions on characteristics such as school test scores, language at home, drug use among the schools' eighth graders, and the ethnic and income composition of school districts.  Schools were then randomly assigned to one of three conditions: control, experimental-teacher-led or experimental-teacher-led with assistance from teen leaders.  The posttest evaluation was conducted 15 months after baseline after the students had received both the 10-session program in the seventh grade and the three-session booster program in the eighth grade.  The baseline sample consisted of 6527 students.  The final sample (n= 3852) at the end of the eighth grade intervention represented 60-64% of the original sample.  Of the missing students, about one half moved after baseline and one half were absent or failed to supply the relevant information at one or more surveys.  Attrition analyses showed the experiment's internal validity was not affected and attrition rates did not differ significantly across experimental conditions.  However, external validity was affected due to students omitted from the end of eighth grade survey being more likely to have before-treatment characteristics often cited as risk factors (e.g.  family disruption and early drug use).  Posttest analyses used three sub-groups based on baseline risk levels, with risk level 1 being students who had tried neither cigarettes or marijuana, risk level 2 students who had tried cigarettes but not marijuana, and risk level 3 were those students who had previously tried marijuana.  The demographic composition of the final sample was 49% male, 71% White, 9% Hispanic, 8% African-American, 10% Asian, and 3% Native American and another ethnic identity.  Twenty-five percent of the sample had a C average or lower grades, 20% had a father who wasn't a high school graduate, 33% were not living with both birth parents, 48% had prior use of cigarettes, 75% had prior use of alcohol and 14% had prior use of marijuana.  Nine of the schools had minority populations of at least 50% and 18 drew from neighborhoods with household incomes below their state median.

Immediate Results of the Intervention

The program showed short-term positive effects through the end of eighth grade on behavioral measures and measures of cognitive risk factors associated with substance use.  Sub-group analyses based on baseline risk level showed that cigarette use reduced significantly for baseline experimenters in measures of past month use (E2 = 23.6, C=32.3, p<.01), in monthly use (E1=16.5, C=22.4, p<.05), in weekly (E1=5.7, C=11.1, p<.01) and daily (E1=2.3, C=5.1, p<.05) use.  For smoking marijuana, there were significant improvements in prevalence estimates (29% vs 36%, p<.01), expectations of using, and perceived consequences.  The impact on marijuana beliefs was slightly more effective among those who had not tried either marijuana or cigarettes (risk level 1) than among those who had tried cigarettes but not marijuana (risk level 2), particularly for students in the teen leader group; this latter effect, however, diminished over time.  For cigarette and marijuana knowledge and attitudes, changes were particularly positive for baseline non-users of cigarettes and marijuana.  The program also had a significant impact on each item making up the social consequences scale.  Program effects were smaller for alcohol; effects on youth beliefs were limited to positive changes in youth prevalence estimates at 15 months (p<.01), and the perceived ability to resist an offer of alcohol on a date (p<.05).  Expectations of future use changed more for marijuana than for the other target substances.  Fewer students in both intervention conditions anticipated using marijuana in the future, but the program had no impact on cigarette and alcohol expectations during the second year.  Compared to the control group, intervention students were more likely to believe that using cigarettes and marijuana can bring immediate and negative social consequences and can result in becoming dependent on the substance.  The program reduced the intervention group estimates of how many students use each target drug and helped those in the teen-leader schools believe that resisting cigarettes and marijuana can bring greater respect from one's friends.

Ellickson and Bell (1990) reported negative outcomes for baseline smokers at the end of the intervention.  Characterizing these as "boomerang effects," they noted these negative results were stronger in the teen leader schools (where many of the other robust positive effects were measured).  By 12 months, baseline users had increased smoking by 20% in these schools (p<.05), which grew to 30% after exposure to the booster program (p=.004).  The authors noted that this boomerang effect for baseline smokers has been found in other anti-smoking programs (e.g., Biglan et al, 1987).  They offered this interpretation (p.1304):  "The results...suggest that early cigarette smokers need a more aggressive program than that offered by the social influence model alone...  For these more confirmed smokers, being told that most of their peers do not smoke and exposing them to nonsmoking teens appears to be irrelevant at best and counterproductive at worst."

Results of Long-Term Follow-Up

Results of long-term follow-up measured at grades 10 and 12 showed that all intervention effects diminished by the end of high school.  Once the lessons stopped, the program's effects on drug use stopped.  Significant effects on cognitive risk factors (belief that drug use has negative personal consequences and that resistance brings respect from one's friends) persisted for a longer time (many through grade 10) in the teen leader schools (p<.01), but did not produce corresponding reductions in use.  By the end of high school, the intervention no longer had a significant effect on behavior.  Its earlier positive impact on cigarette and marijuana use had completely disappeared as had the negative boomerang effect for baseline cigarette users.