Positive Youth Development in the United States: Research Findings on Evaluations of Positive Youth Development Programs. Evaluations Excluded from the Effective Interventions


Seventy-seven evaluations were identified by this review for analysis; however, eight were sufficiently limited by missing information in key parts of the evaluation that they had to be removed from the summary analyses of programs. Thus 69 programs  25 well-evaluated programs, and 44 that did not have adequate evaluations  are analyzed. Further, of the excluded programs, although 44 were included in the summary, not all contained complete information that permitted comparisons with the group of effective programs on each dimension. Therefore, depending on the dimension in question, the number of excluded programs used as the basis for comparison will vary slightly (see Appendix J).

Generally, programs were excluded from the effective category based on weaknesses in the evaluation that made it impossible to draw conclusions about the intervention's effects on youth behavior. Or, there was a strong design and no effects were shown. Thus, four types of problems caused placement of programs in this category:  (1) evaluation design weaknesses; (2) insufficient behavioral outcome measures; (3) outcomes showing no impact for the intervention, or limited to only measurement of knowledge or attitude changes; and (4) lack of methodological information needed to draw conclusions about program effectiveness. More specifically, some programs (n= 12) had evaluations which received a "medium confidence" designation. This group fell into two sub-sets:  one set (n=8) had a reasonable design but provided insufficient data in the report to conclude that the comparison groups were solidly equivalent; the second sub-set (n= 4) was excluded because there was a stronger evaluation of the same program. Another group of programs (n=19) received the designation of "low confidence" for one of two reasons:  either the description of the comparison group did not establish that the intervention and control groups were equivalent (n=10), or there was no comparison group at all (n=9). In addition to these programs with design or methodological issues, some interventions with very strong designs (n=5) were excluded because they had no significant outcomes (n=3), or because their evaluation measured only attitude and knowledge changes, not behavioral outcomes (n=2) (see Appendix K).