The first step of the evaluation entailed selecting abstinence programs for study. The evaluation team first called and met with numerous state officials and experts across the country to identify promising programs for inclusion in the evaluation. Grant applications and program documents then provided additional detail on program goals, target population, activities, size, and curricula. The evaluation team visited and observed 28 abstinence education programs across the nation. After extensive communication with abstinence experts and DHHS staff, 11 programs were invited and agreed to participate in the evaluation (Table 1). Although not a representative set of Section 510 abstinence education programs, these 11 programs are judged to offer a rich range of program strategies and implementation settings for study.
Five of the programs included in the evaluation are referred to as “targeted” programs, as they target their services to specific, identifiable groups of youth. In addition to providing valuable information about program implementation, each of these programs offers the potential to provide rigorous evidence on the effectiveness of abstinence education program strategies in reducing teen sexual activity and other risk behavior. The other six programs represent a range of multifaceted, community-wide initiatives that aim to alter youth behavior through stimulating systemic change. These programs are using their abstinence education funding in a variety of ways to increase public awareness of the problems of teen sexual activity, to change community norms and attitudes, to engage parents and encourage stronger parent-child communications, and to engage youth in abstinence education and support services. Including these community-wide initiatives in the evaluation adds breadth to an understanding of strategies for changing youth behavior.
For several reasons, though, rigorous impact studies of these community-wide abstinence program initiatives are not possible. First, these programs often use their abstinence funding to form or become part of a larger network of services for teens, making it impossible to separate the effects of the abstinence program from those of other programs or providers. Second, the target population often is not easily identifiable, since, by intent, these programs aim to change the norms and behavior of an entire community. Finally, systemic change is a long-range goal, and the time frame for the impact analysis and federally funded evaluation is too short to allow such change to be measured.
In selecting programs for the evaluation, the goal was to maximize the overall knowledge that would be generated, including providing operational lessons and impact estimates based on a range of program approaches and implementation settings. Moreover, all 11 programs met two additional criteria: (1) each conformed to, and in some cases was based on, a theoretical framework that links its services to changes in youth knowledge, attitudes, intentions, and behavior; and (2) each appeared to be reasonably well implemented, which included having its core services operational, committed staff in place, and key partnerships established.
For the five programs targeting services on particular groups of youth, three additional site selection criteria were applied in order to ensure the feasibility of conducting a rigorous impact analysis of each program:
- The program’s services and activities differ from what participants otherwise would receive. The evaluation contrasts outcomes of youth receiving program services with those of control group youth who do not. If participants would receive similar services without the program, the evaluation would essentially compare a program to itself, leading to negligible estimated impacts.
- The program can readily adapt to evaluation procedures. It was important that a program be able and willing to adapt to critical evaluation procedures without adversely affecting the basic program services. It was essential that the evaluation be able to address programmatic concerns while not compromising the evaluation’s credibility.
- Both the programs and the target population need to be large enough to support the sample size requirements for a rigorous study. It was important that a program be able and willing to adapt to critical evaluation procedures without adversely affecting the basic program services. It was essential that the evaluation be able to address programmatic concerns while not compromising the evaluation’s credibility.
While the abstinence education programs selected are considered interesting and well-implemented programs, they are not necessarily better than or representative of the more than 700 abstinence education programs funded under Section 510 and operating nationwide. Many of the other Section 510 abstinence education programs are being examined in evaluations that states and localities have funded themselves, and these other studies will provide rich and important detail on the range of abstinence strategies and their effects (Mathematica Policy Research, Inc. 2001).