The Evaluation of Abstinence Education Programs Funded Under Title V Section 510: Interim Report

Submitted by:
Barbara Devaney
Amy Johnson
Rebecca Maynard
Chris Trenholm
Mathematica Policy Research, Inc

Submitted to:
Meredith Kelsey
Office of the Assistant Secretary for Planning and Evaluation
U.S. Department of Health and Human Services

"

Acknowledgements

Many people contributed in important ways to this report. First and foremost, we thank the many abstinence education program grantees who have generously allowed us to visit their programs, meet with staff, and observe their operations. We are especially grateful to those who created and/or are directing the 11 programs that are the focus of the study: Jacqueline Del Rosario with ReCapturing the Vision in Miami, Florida; Vicki Hearns with Teens in Control in Clarksdale, Mississippi; Anne Badgley and Susan Swanson with Heritage Community Services in South Carolina; Gale Grant, Ginelle Ampy-Thornhill, and Kathy Douglas with My Choice, My Future! in Powhatan, Virginia; Marty Kerrigan, Rosemary Fisher, and Lyn Hildenbrand with Families United to Prevent Teen Pregnancy in Milwaukee, Wisconsin; Ruth Ann Eccles with the Abstinence Education Initiative (AEI) Coalition of Equipping Youth in Cedar Rapids, Iowa; Marilyn Ammon with McClennan County Collaborative Abstinence Project in McClennan County, Texas; Sherrie Ahlstrom with the Youth Abstinence Education Program in Tooele, Utah; Lori Kuykendall with Fort Bend Abstinence Leadership and Resources for Teens in Fort Bend, Texas; and Andy Doniger with the Not Me Not Now program from the Monroe County Health Department, Rochester, New York. We also would like to thank the many school district staff in these communities who have supported the evaluation.

We have consistently received outstanding guidance and support from members of the project's technical workgroup, most of whom have been with us continuously from the start of the project: Marilyn Benoit, Sarah Brown, Jim Jaccard, David Larson (1998-2000), Joe McIlhaney, Robert Michael, Kristin Moore, Susan Philliber, Robert Rector, David Rowberry, Freya Sonenstein, Marta Tienda (1998-2000), John Vessey, Stan Weed (1998-2000) and Brian Wilcox. Many members of the research and policy community — especially Christine Bachrach, Leanna Benn, Cassie Bevan, Peter Brandt, Ron Haskins, Michele Lawler, Onalee McGraw, Susan Newcomer, Karen Thiel Raykovich, Lisa Rue, Shepherd Smith, Matthew Stagner, Amy Stephens, Kathleen Sullivan, Leslee Unruh, Peter van Dyck, and Pat Funderburk Ware — have been extremely generous in answering questions and offering advice. Gary Burtless, Judy Gueron, Rob Hollister, and David Myers provided an extremely valuable review of the study design.

John Homrighausen, Anne Self, Linda Mendenko, Jason Markesich, and Linda Bandeh shared various responsibilities for the design and oversight of the survey and school records data collection. Jean Knab and Licia Gaber provided database management and programming support for the project. Walt Brower provided editorial support for the report, which was produced by Jennifer Chiaramonti and Monica Capizzi.

Finally, we are deeply grateful for the support of staff at the U.S. Department of Health and Human Services who oversee this project, especially to Meredith Kelsey, the project officer for this study, and Barbara Broman. They have accompanied us on site visits, worked with us to stay abreast of and respond quickly to emerging issues and concerns related to the study, and provided consistent support and guidance for the project.

Although we gratefully acknowledge the inputs of these people and many others, we alone are responsible for any errors or omissions in the report. Any opinions expressed in this report are those of the authors and do not necessarily reflect the views of the U.S. Department of Health and Human Services or of others.

Executive Summary

In 1996, Congress authorized $50 million annually for five years to promote abstinence education.  This funding was established through a new formula grant program authorized under the Personal Responsibility and Work Opportunity Reconciliation Act (PRWORA) of 1996.  The funding became available to states in 1998 and is administered by the Maternal and Child Health Bureau.  States must provide $3 in matching funds for every $4 in federal funds, resulting in a total of up to $87.5 million available annually for such state programs.  Abstinence education programs funded through this new grant program teach an unambiguous abstinence message to youth.  Programs receiving these abstinence education funds may not endorse or promote contraceptive use.

This report presents interim findings from an independent, federally funded evaluation of the abstinence education programs authorized under PRWORA and defined under Title V, Section 510 (b)(2)(A-H) of the Social Security Act.  This report draws most heavily on four years of implementation experiences in a selected group of abstinence education programs funded under Section 510.  Later reports from the evaluation will present estimates of short- and longer-term program impacts, as well as studies on special topic areas.

Evaluation of Section 510 Abstinence Education Programs

The evaluation addresses three important questions: 

1. What are the nature and underlying theories of the abstinence education programs?
2. What are the implementation and operational experiences of local communities and schools that have received abstinence education funding? and
3. What are the impacts of abstinence education programs on the attitudes and intentions of youth to remain abstinent, on their sexual activity, and on their risks of pregnancy and sexually transmitted diseases (STDs)?

The first stage of the evaluation entailed selecting programs for study.  The evaluation team first visited and observed numerous abstinence education programs across the nation.  Eleven of these, representing a range of program models and serving different target populations, were then selected for in-depth analysis.  Five of the 11 are referred to as “targeted” programs:  they target services to specific, identifiable groups of youth.  The remaining six are community-wide, systemic-change initiatives, which use the abstinence education funding to increase public awareness of the problems of teen sexual activity, to change community norms and attitudes, to encourage stronger parent-child communications, and to engage youth in abstinence education and support services.

The evaluation includes an extensive implementation and process analysis and an impact analysis.  The implementation and process analysis uses program documents, program observations, focus groups with program participants and parents, and interviews with program staff and community leaders to document and understand the abstinence education programs implemented in the 11 sites.  The impact analysis uses longitudinal survey data for groups of youth randomly assigned to program and control groups in the 5 targeted program sites.  Enrollment in the impact evaluation study samples spanned three school years and was just completed in fall 2001.  Therefore, the follow-up data that are necessary for the impact analysis are not available for inclusion in this report.

Early Implementation and Operational Lessons

The first four years of Section 510 funding for abstinence education have generated a wealth of experience on how local grantees have designed and implemented abstinence education programs.  Among the early lessons are the following:

1. Section 510 abstinence funds are changing the local landscape of approaches to teenage pregnancy prevention and youth risk avoidance.  Despite an initial debate in some states over whether and how to spend abstinence education block grant funds, all states applied for funding at some point, and most are using monies in innovative ways to promote abstinence from sexual activity as the healthiest choice for youth.
2. Most abstinence education programs offer more than a single message of abstinence.  Examples of curricula and program components from sites participating in the federally funded evaluation indicate the diverse, creative, and often complex nature of many initiatives.

Common Curriculum Topics of Abstinence Programs Participating in the Impact Evaluation
Building Self-Esteem	Preventing STDs
Developing Values/Character Traits	Withstanding Social and Peer Pressure
Formulating Goals	Addressing Consequences/Self-Control
Making Decisions	Resolving Sexual Conflicts
Avoiding Risky Behavior	Learning Etiquette and Manners
Maximizing Communication	Aspiring to Marriage
Strengthening Relationships	Understanding Parenthood
Understanding Development and Anatomy

3. Most participants report favorable feelings about their program experience.  Youth respond especially positively to staff who show strong and unambiguous commitment to the program message.  They also like programs that deliver an intensive set of youth development services to enhance and support the abstinence message.
4. Abstinence education programs face real challenges addressing peer pressure and the communication gulf between parents and children.  Sexual activity often elicits only casual mention among youth, and is tolerated and even promoted by their peer culture.  Many programs attempt to address peer pressure through parents.  Yet, engaging parents has proven to be extremely challenging.
5. Local schools are valuable program partners, but establishing these partnerships is sometimes difficult.  Their broad access to youth makes schools logical and important partners for many programs, but some schools resist collaboration with abstinence programs.  Sometimes schools resist because of competing priorities; at other times, resistance stems from debate about health and sex education policies.

More Lessons to Come

Congress has authorized a scientifically rigorous, independent evaluation of the abstinence education programs funded under Title V Section 510 to determine the extent to which abstinence programs achieve six specific goals:

1. Strengthen knowledge and attitudes supportive of abstinence
2. Induce more youth to embrace abstinence from sexual activity as a personal goal
3. Reduce sexual activity among youth
4. Persuade sexually experienced youth to become and remain abstinent
5. Lower the risk of STDs
6. Lower the risk of nonmarital pregnancies

Obtaining clear and definitive evidence on the success of abstinence education programs in achieving these goals is a difficult task that requires time.  Over the past four years, the evaluation effort has laid the foundation for a careful, comprehensive, and rigorous impact study and has successfully implemented the research design in the five targeted program sites.  Study enrollment is completed, and longitudinal tracking of youth through surveys and school records is ongoing.

Critical features of the impact study design now under way are the following:

• The impact evaluation uses an experimental design.  In each site, program effectiveness will be measured by comparing outcomes of eligible youth who were randomly assigned to the program or to a control group.  The experimental design offers the only means of measuring, with a known degree of certainty, how successful the programs are overall and how well they serve key subgroups of youth.  Other evaluation designs are vulnerable to “selection bias,” which can seriously undermine the credibility of their results.
• The impact evaluation has large sample sizes of between 400 and 700 youth per site.  Large sample sizes protect against failing to detect true program impacts simply because the study lacked statistical power.  Three years of study enrollment (fall 1999 through fall 2001) were necessary to achieve adequate sample sizes.
• The study sample is being followed for up to 36 months.  Because so few youth engage in sexual activity before entering high school, outcome estimates based on middle school youth will miss program impacts on behaviors that could emerge later.  The follow-up period for the evaluation is such that almost two-thirds of the study sample will be 14 to 18 years of age by the time of the final survey.

Analysis and Reporting Plans

Study enrollment began in fall 1999 and continued through fall 2001.  Youth are surveyed at or close to study enrollment (wave 1), between 6 and 12 months following enrollment (wave 2), and then 18 to 36 months later (wave 3).  In sites where program participation might be expected to affect school performance, school records will be collected through spring 2004.  Throughout, data on program operations are being collected through observations, executive interviews, program documents, and focus groups.

Interview Schedule
Sample Enrollment	Wave 1	Wave 2	Wave 3
Fall ’99/Spring ’00	Fall ’99/Spring ’00	Fall ’00	Spring/Fall ’02
Fall ’00	Fall ’00	Spring ’01	Fall ’03
Fall ’01	Fall ’01	Spring ’02	Fall ’03

A report on the effectiveness of the programs in achieving their short-term goals of changing knowledge, attitudes, and near-term behavioral choices will be completed in early 2003, after Wave 2 survey data are available for the full study sample.  The final study evaluation report will be completed in summer 2005.  During intervening periods, the team will prepare a limited number of special-focus reports that address particular questions of interest to Congress or the U.S. Department of Health and Human Services.

Chapter I: Background

In 1996, Congress authorized $50 million annually for five years in funding to states for programs that teach abstinence from sexual activity outside of marriage as the expected standard for school-age children.  This funding was established through a new formula grant program created under Title V, Section 510 of the Social Security Act, authorized under the Personal Responsibility and Work Opportunity Reconciliation Act (PRWORA) of 1996.  The funds became available to states in 1998 and are administered by the Maternal and Child Health Bureau.  Currently, Title V Section 510 abstinence education is in its last funding cycle, and deliberations regarding reauthorization will begin shortly.

Four years into the Section 510 abstinence education funding, the percentage of teens reporting that they have had sex has decreased, continuing a decline that started in 1991.  At this time, however, no definitive research has linked the abstinence education legislation with these downward trends.  Most people acknowledge that “abstinence works.”  It is certain to prevent unwanted pregnancy, sexually transmitted diseases (STDs), abortions, and out-of-wedlock childbearing.  However, an important question is:  To what extent are abstinence education programs effective in persuading youth to be sexually abstinent and in changing teen sexual behavior?

Empirical evidence on the effectiveness of abstinence education is limited.  Moreover, most studies of abstinence education programs have methodological flaws that prevent them from generating reliable estimates of program impacts.  Even the features of abstinence programs implemented, the curricula used, and the experiences of program staff and program participants are not well documented in a readily accessible way.  To address this gap in information, Congress authorized a federally funded, independent evaluation of Section 510 abstinence education programs in the Balanced Budget Act of 1997 (Public Law 105-33).  Through extensive implementation, process, and impact analyses, the evaluation will strengthen the research base and knowledge about strategies for promoting sexual abstinence among youth and the benefits of various approaches to abstinence education.

This report presents interim findings from the congressionally authorized evaluation. The first in a series of reports from the evaluation, this report draws most heavily on four years of implementation experiences in a selected group of abstinence education programs funded under Title V Section 510.  This report also uses information from federal program monitoring reports, efforts by state and local evaluators, and policy and issue statements by various constituent groups and policy organizations.  Later reports from the evaluation will present estimates of short- and longer-term program impacts, as well as studies on special topical areas.

What Is the Context for the Title V Section 510 Abstinence Education Program?

Despite a steady decline in the teen birthrate between 1991 and the present — from a high in 1991 of 62 births per 1,000 females age 15 to 19, to 49 such births in 2000 — concerns about teen sexual activity persist:

• In 1999, half of all high school students and nearly two-thirds of graduating seniors reported having had sexual intercourse (Centers for Disease Control and Prevention 2000).
• In 1999, one in five high school seniors reported having had sex with four or more partners (Centers for Disease Control and Prevention 2000).
• Nearly 480,000 babies were born to teens in 2000, and 79 percent of these births were out of wedlock (National Center for Health Statistics 2002).

The consequences of teenage sexual activity and out-of-wedlock childbearing are many and serious for teens, their families, their communities, and society.  Over three-fifths of teen mothers live in poverty at the time of their child’s birth, and over four-fifths eventually live below poverty (Maynard 1996).  Children born to teen mothers often fare badly during infancy, early childhood, and their adolescent and adult lives.  Compared with children born to mothers who delay childbearing until age 21 or older, children of teen mothers are more likely to grow up in homes that are not emotionally supportive or cognitively stimulating, to suffer from abuse and neglect, to repeat a grade in school, and to drop out of high school (Moore et al. 1997; Goerge and Lee 1997; and Haveman et al. 1997).

In addition to its social and economic consequences, teen sexual activity also brings increased risks of STDs.  In fact, teenage females have the highest rates of STDs of any age group.  In the United States, more than 65 million people have an STD, and most are incurable viral infections (National Institute of Allergy and Infectious Diseases 2000).  STDs may cause such lifelong complications as infertility, ectopic pregnancies, miscarriages, stillbirths, intrauterine growth retardation, and perinatal infections.  One STD, human papillomavirus, is the primary cause of cervical cancer.  Moreover, because of limitations in study design, the scientific evidence on the effectiveness of condoms in preventing STDs is inconclusive (National Institute of Allergy and Infectious Diseases 2000).

What Is the Title V Section 510 Abstinence Education Program?

In this context of high rates of teen sexual activity and their serious public health and socioeconomic consequences, interest in abstinence education has increased over the past decade.  As a result, in 1998 the federal government provided $50 million annually for five years for block grants to states to support abstinence education programs.  States must provide $3 in matching funds for every $4 in federal funds, which results in a total of up to $87.5 million available annually for such programs.

These abstinence education grants are allotted to states through a formula based on the proportion of low-income children in the state relative to the total number of low-income children for all the states.  States then decide which programs to fund and at what level.  Most states have disbursed their funding to numerous local agencies and organizations.  However, a few states, such as Massachusetts, have retained their entire funding allocation for a single statewide initiative, such as a media campaign.

Although abstinence education programs have been around for decades, the new investment raised the profile of programs that teach an unambiguous abstinence message to youth.  The main factor that distinguishes the Section 510 abstinence education funding from the previous generation of federally funded abstinence education programs is the “A-H definition” (Title V Section 510 (b)(2)(A-H) of the Social Security Act), which specifies that an abstinence education program funded under the block grant must:

The Maternal and Child Health Bureau guidelines for these abstinence education programs offer the following interpretation of the A-H definition:  “It is not necessary to place equal emphasis on each element of the definition.  However, a project may not be inconsistent with any aspect of the abstinence education definition” (Maternal and Child Health Bureau 1997; Haskins and Bevan 1997).  Providing instruction in or promoting the use of birth control would be inconsistent with the A-H definition.

The A-H definition of abstinence education has generated controversy.  Some abstinence educators and policymakers are critical of Section 510 abstinence education programs that do not emphasize all elements of the A-H definition.  These criticisms have been especially strong for some preexisting health and teenage risk reduction programs that are perceived to have simply added limited abstinence education modules to their services specifically to gain access to Section 510 funding.  On the other hand, some policymakers and health educators object to the Section 510 abstinence education programs primarily because of their restrictive definition of abstinence education.

Still, since the inception of the Section 510 abstinence education funding, the number of abstinence education providers has increased dramatically.  The funding has stimulated considerable discussion at the state and local level on the problem of teen sexual activity and raised local awareness and consideration of the role of abstinence education in local programs and policies.  Supporters of abstinence education contend that such programs are effective because they are consistent with the developmental needs of adolescents for clear, consistent messages and boundaries.  Their concern with sex education programs that teach about sexuality, contraceptives, and abstinence is that they send the mixed message that (1) teens should be abstinent, but (2), if they are going to have sex, they need information about and access to contraceptives.

On the other side, proponents of comprehensive sex education programs contend that, while abstinence is preferred, broader sex education is essential because most teens are sexually active by the time they finish high school.  Comprehensive sex educators contend that teens need to be taught about and provided access to contraceptives to reduce their risk of pregnancies and STDs.

Despite these different perspectives on how best to address high rates of teen sexual activity, a common thread in the ongoing policy debate is an underlying interest in learning about effective program strategies that help youth make good choices that avoid risk-taking behaviors and promote healthy future lives.  The evaluation of Section 510 abstinence education programs is designed to contribute much-needed knowledge on approaches to, and the effectiveness of, selected abstinence education programs.

What Is the Evaluation of Abstinence Education Programs Funded Under Title V Section 510?

In fall 1998, the Assistant Secretary for Planning and Evaluation, U.S. Department of Health and Human Services, awarded a competitive contract to Mathematica Policy Research to conduct an independent evaluation of Section 510 abstinence education programs.  The evaluation addresses three important questions:

• What are the nature and underlying theories of the abstinence education programs supported with Section 510 funding?
• What are the implementation and operational experiences of local communities and schools that have received Section 510 abstinence education funding?
• What are the impacts of abstinence education programs?
  • How successful are they in changing the knowledge, attitudes, and intentions of youth?
  • How successful are they in reducing teen sexual activity among youth?
  • How do they change the risk of pregnancy and STDs?

To address these questions, the evaluation includes an extensive implementation and process analysis, focused on 11 abstinence education programs, as well as rigorously designed impact studies of 5 of these programs.  The implementation and process analysis uses program documents; program observations; focus groups with program participants, parents, and other area youth; and interviews with program staff and community leaders to document and understand important features of the range of programs that have been implemented.  The impact study uses longitudinal survey data for groups of youth randomly assigned to the abstinence program in the community or to a control group.

The evaluation design was developed and implemented with guidance from a technical workgroup composed of individuals with demonstrated expertise in the myriad critical aspects of this complex research agenda (see Appendix A).  In addition, the evaluation team held meetings with numerous interest groups to solicit their input regarding the evaluation questions, site selection criteria, and data collection strategies.

Site Selection

The first step of the evaluation entailed selecting abstinence programs for study.  The evaluation team first called and met with numerous state officials and experts across the country to identify promising programs for inclusion in the evaluation.  Grant applications and program documents then provided additional detail on program goals, target population, activities, size, and curricula.  The evaluation team visited and observed 28 abstinence education programs across the nation.  After extensive communication with abstinence experts and DHHS staff, 11 programs were invited and agreed to participate in the evaluation (Table 1).  Although not a representative set of Section 510 abstinence education programs, these 11 programs are judged to offer a rich range of program strategies and implementation settings for study.

Five of the programs included in the evaluation are referred to as “targeted” programs, as they target their services to specific, identifiable groups of youth.  In addition to providing valuable information about program implementation, each of these programs offers the potential to provide rigorous evidence on the effectiveness of abstinence education program strategies in reducing teen sexual activity and other risk behavior.  The other six programs represent a range of multifaceted, community-wide initiatives that aim to alter youth behavior through stimulating systemic change.  These programs are using their abstinence education funding in a variety of ways to increase public awareness of the problems of teen sexual activity, to change community norms and attitudes, to engage parents and encourage stronger parent-child communications, and to engage youth in abstinence education and support services.  Including these community-wide initiatives in the evaluation adds breadth to an understanding of strategies for changing youth behavior.

For several reasons, though, rigorous impact studies of these community-wide abstinence program initiatives are not possible.  First, these programs often use their abstinence funding to form or become part of a larger network of services for teens, making it impossible to separate the effects of the abstinence program from those of other programs or providers.  Second, the target population often is not easily identifiable, since, by intent, these programs aim to change the norms and behavior of an entire community.  Finally, systemic change is a long-range goal, and the time frame for the impact analysis and federally funded evaluation is too short to allow such change to be measured.

In selecting programs for the evaluation, the goal was to maximize the overall knowledge that would be generated, including providing operational lessons and impact estimates based on a range of program approaches and implementation settings.  Moreover, all 11 programs met two additional criteria:  (1) each conformed to, and in some cases was based on, a theoretical framework that links its services to changes in youth knowledge, attitudes, intentions, and behavior; and (2) each appeared to be reasonably well implemented, which included having its core services operational, committed staff in place, and key partnerships established.

Table 1: Abstinence Education Programs Participating in the Evaluation

Location and Sponsoring Agency	Principal Program Components	Target Population
Targeted Programs (Impact, Implementation, and Process Analysis)
FL (Miami) Youth service agency	Elective class offered daily, all year to girls in middle schools.  Urban setting; diverse student population.  Curriculum:  ReCapturing the Vision and Vessels of Honor	Grades 6–8
MS (Clarksdale) Community health agency	Mandatory weekly year-long curriculum.  Rural community; extremely poor population.  Curriculum:  Revised Postponing Sexual Involvement and Sex Can Wait	Grades 5–6
SC (Edgefield) Youth service agency	Five-session mandatory curriculum with voluntary enrollment in weekly or biweekly character clubs.  Middle-income community.  Curriculum:  Heritage Keepers	Grades 6–9
VA (Powhatan) County health department	36-session mandatory curriculum.  Middle-income community.  Curriculum:  Reasonable Reasons to Wait; The Art of Loving Well; and Choosing the Best	Grades 8 and 10, with 9th and 11th grade boosters
WI (Milwaukee) Social service agency	Voluntary after-school program; two hours daily all year for multiple years.  Seven-week summer program.  Poor, inner-city community.  Curriculum:  Families United to Prevent Teen Pregnancy	Grades 3–8
Community-Wide Initiatives (Implementation and Process Analysis)
IA (Cedar Rapids) Not-for-profit coalition	Abstinence curriculum; community resource library; classroom presentations; workshops for parents and educators; Baby Think It Over dolls; speakers; mentoring program; teen panels	Emphasis on middle school youth
NY (Monroe County) County health department	Not Me Not Now media campaign; abstinence curriculum; parent outreach program; interactive web site; mentoring program	Emphasis on 9–14 year olds
SC (statewide) Youth service agency	Abstinence curriculum; character clubs; school assemblies; training/information for medical providers, faith workers, parents, and media personnel; numerous collaborations and partnerships	Middle and high school youth
TX (Fort Bend County) Community-based organization	Abstinence curriculum; separate youth development programs for girls and boys; peer education program; school assemblies; community training; parent education programs; parent resource center; community events; medical provider involvement	9–17 year olds, with a heavy focus on middle school youth
TX (McLennan County) Community-based organization	Abstinence curriculum; school assemblies; character education in elementary schools; mentors; media spots; medical provider training; faith-based partners; resource library	Emphasis on 10–14 year olds
UT (Tooele County) County health department	Abstinence curriculum; parenting class; self-esteem days for 5th–8th graders; self-esteem classes for high-risk youth; Baby Think It Over dolls; peer educators; school fairs; billboards and newsletters; merchant involvement; faith-based linkages	9–14 year olds

For the five programs targeting services on particular groups of youth, three additional site selection criteria were applied in order to ensure the feasibility of conducting a rigorous impact analysis of each program:

• The program’s services and activities differ from what participants otherwise would receive.  The evaluation contrasts outcomes of youth receiving program services with those of control group youth who do not.  If participants would receive similar services without the program, the evaluation would essentially compare a program to itself, leading to negligible estimated impacts.
• The program can readily adapt to evaluation procedures.  It was important that a program be able and willing to adapt to critical evaluation procedures without adversely affecting the basic program services.  It was essential that the evaluation be able to address programmatic concerns while not compromising the evaluation’s credibility.
• Both the programs and the target population need to be large enough to support the sample size requirements for a rigorous study.  It was important that a program be able and willing to adapt to critical evaluation procedures without adversely affecting the basic program services.  It was essential that the evaluation be able to address programmatic concerns while not compromising the evaluation’s credibility.

While the abstinence education programs selected are considered interesting and well-implemented programs, they are not necessarily better than or representative of the more than 700 abstinence education programs funded under Section 510 and operating nationwide.  Many of the other Section 510 abstinence education programs are being examined in evaluations that states and localities have funded themselves, and these other studies will provide rich and important detail on the range of abstinence strategies and their effects (Mathematica Policy Research, Inc. 2001).

Implementation and Process Evaluation

The implementation and process analysis documents the experience of organizations and communities applying for and receiving abstinence education funding in both the targeted and community-wide program sites.  It describes the abstinence interventions implemented — the program goals, the underlying theoretical framework, and the specific curriculum elements covered by the program.  It examines the target population and community context, and reports the participants’ experiences with the program.  It also details the organizational structure of the abstinence education program models, identifying those models that have been used and the factors associated with successful implementation.

The implementation and process analysis uses three primary data sources:  (1) review of program documents and records; (2) interviews and focus groups with program staff, school staff, community leaders, parents, and program participants; and (3) on-site program observations.  Program documents and records provide important background information on the program’s objectives and message, as well as data on youth served.  Most programs selected for the evaluation have promotional materials that are distributed to youth and, in some cases, to the community at large.  Some have web sites or use the media to deliver the abstinence message and to identify community resources available for youth.  Program documents also often describe staff qualifications and background, present staff training materials, and include written communications among abstinence providers.  Program records provide valuable information on youth served, resource requirements, and costs.

Interviews with program and school staff also convey important information on program goals and implementation, as well as on the more intangible aspects of commitment to the program message and the importance of helping youth.  Interviews and focus groups with program participants provide insights into the experiences of youth, their perceptions of the program, and the intangible factors that lead to program success.  Focus groups with parents yield additional insights on the needs of youth and how programs can best engage parents in helping their children make good choices.  Finally, firsthand program observations are invaluable for assessing how the program messages and services are delivered and received.

The data collection efforts to support the implementation and process analysis were extensive.  Evaluation team members made multiple rounds of site visits to all programs included in the evaluation, as well as to many others ultimately not included in it.  During these site visits, the evaluators conducted executive interviews with numerous staff members, facilitated the focus groups, and observed program delivery.  In addition, phone calls and ongoing review of program records provided further detail on program implementation.  Finally, an extensive communications effort and outreach to constituent groups and policy organizations yielded rich information on the context of abstinence funding and the types of programs supported.

Impact Evaluation

To date, the evaluation team has successfully implemented a scientifically rigorous impact study design in the five targeted program sites.  Study enrollment is completed, and longitudinal tracking of youth is ongoing.  Because of the importance of having large sample sizes and sufficient followup of program and control youth, no impact estimates are available for inclusion in this report.  This report does, however, provide detail on the key features of the study design, the characteristics of the youth served by these programs, and plans for the impact analysis and reporting.

As discussed in greater detail later, critical features of the impact analysis study design are:

• The Impact Study Uses Experimental Designs in All Sites.  In each site, program effectiveness will be measured by comparing outcomes of eligible youth who were randomly assigned to the program or to a control group.
• Sample Sizes Are Between 400 and 700 Youth per Site.  Large sample sizes protect against the possibility of failing to detect true program impacts simply because the study lacks statistical power.  To achieve adequate sample sizes, youth were enrolled in the study from fall 1999 through fall 2001.
• Followup of the Sample Will Continue for Up to Three Years After Study Enrollment.  The first wave of data collection occurred at “baseline” as participants were first enrolled in the program or the control group (fall 1999 through fall 2001), the second wave occurs 6 to 12 months later (fall 2000 through spring 2002), and the third wave will occur 18 to 36 months after initial sample enrollment (spring 2002 through fall 2003).
• Data Collection Procedures Respect the Rights of Students and Parents and Protect the Privacy of Respondents.  The evaluation uses an active parental consent process whereby the parent or guardian must provide written consent before a student is eligible to participate.  The evaluation also uses confidentiality protections designed to ensure that no one from local schools — including teachers, administrators, and program staff — has access to the student surveys.
• Student Surveys and Data Collection Procedures Are Designed to Maximize the Accuracy and Reliability of Student Responses.  Research indicates that youth may underreport socially undesirable behaviors, such as sexual activity and other risk-taking activity.  Furthermore, abstinence means different things to different people.  The surveys for the study measure specific, clearly defined behaviors.  Students provide their answers on self-administered forms under the supervision of trained, independent interviewers.

The first report on short-term program impacts will be released early in 2003, once the second wave of survey data is available for the full study sample.  The final impact analysis report will be released in summer 2005.

Chapter II: Early Implementation Experiences of Abstinence Education Programs

The first four years of Title V Section 510 funding for abstinence education have generated a wealth of experience on how local communities and schools have designed and implemented abstinence programs and how youth have responded to them.  Interviews with program staff, parents, and students, as well as observations of what and how the abstinence message is conveyed, have yielded rich information on program operations, program models, youth response, and local agency partnership opportunities and challenges.  This chapter discusses the experiences of communities, schools, and youth with abstinence education programs, focusing primarily on the 11 programs selected for the evaluation.  The analysis also draws on observations of the uses of abstinence education funding more generally.  Federal monitoring efforts document the funding allocations and the types of programs supported nationwide.  State and local evaluations provide additional detail on how the state grants are dispersed to local communities and the range of programs and activities supported.

The experience of these early years of funding has produced five important conclusions that can guide future fiscal and programmatic decisions:

1. Section 510 abstinence funds are changing the local landscape of approaches to teenage pregnancy prevention and youth risk avoidance.  Despite an initial debate in some states over whether and how to spend abstinence education funds, all states applied for funding at some point and are using the monies in innovative ways to promote abstinence from sexual activity as the healthiest choice for youth.
2. Most abstinence education programs offer more than a single message of abstinence.  Examples of curricula and program components from sites participating in the evaluation indicate the diverse, creative, and often complex nature of many initiatives.
3. Most participants report favorable feelings about their program experience.  Youth respond especially positively to staff who show strong and unambiguous commitment to the program message.  They also seem to like programs that deliver an intensive set of youth development services to enhance and support the unambiguous abstinence message.
4. Abstinence programs face real challenges addressing peer pressure and the communication gulf between parents and children.  Sexual activity often elicits only casual mention among youth, and is tolerated and even promoted by their peer culture.  Many programs attempt to address peer pressure, in part, through parent involvement.  Yet, for many programs, engaging parents has proven to be extremely challenging.
5. Local schools are valuable program partners, but establishing these partnerships is sometimes difficult.  Their broad access to youth makes schools logical and important partners for many programs, but some schools resist collaboration with abstinence education programs.  Sometimes schools resist because of competing priorities; at other times, resistance stems from debate about health and sex education policies.

Title V Section 510 Abstinence Funds Are Changing the Local Landscape of Approaches to Preventing Teen Pregnancy

Funding for abstinence education has contributed to the evolving national struggle to address the social and economic consequences of teenage sexual activity, teenage childbearing, and out-of-wedlock births.  Following considerable and sometimes rancorous controversy over whether and how to spend the $50 million in annual abstinence education block grant funds, every state ultimately applied for the money.  In each of the successive funding years, nearly all of the states and territories took advantage of the funds available to them.

In 1988, only 2 percent of teachers responsible for sexuality education in public secondary schools reported teaching abstinence as the sole way to prevent pregnancy and STDs; by 1999, this figure had risen to 23 percent of secondary school sexuality education teachers (Darroch, et al. 2000).

Three factors help explain the current extent of acceptance of abstinence education.  First, the Section 510 abstinence education programs implemented across the country have gained support because they are more than “Just Say No” programs.  They offer a breadth of services and activities designed to support youth, equip them with knowledge and decision-making skills to help them make good choices, and provide them with constructive activities that are fun and widely perceived as good for kids.  Second, many programs focus on middle school students, where there is general agreement about the appropriateness of a strong abstinence approach to sexual education.  Finally, the coalitions formed at the local level to deliver the abstinence message often bridge a gap in ideological perspectives, allowing abstinence education programs to coexist with other programs that respond in a variety of ways to the needs of teens and their communities.

The Section 510 abstinence education funding supports more than 700 programs nationwide and has resulted in a tremendous range of new programmatic approaches to preventing teen sexual activity and out-of-wedlock pregnancy (Maternal and Child Health Bureau 2000).  The amounts awarded to local grantees by states vary widely.  For example, among the programs included in the evaluation, annual award amounts range from $50,000 to over $800,000. States have awarded abstinence education grants to community-based organizations, local school districts, local health departments, faith-based organizations, and universities, among others.  The funding guidelines encourage states to fund grantees’ efforts directed at local priority needs, and the diversity of uses of funding reflects this intent (Maternal and Child Health Bureau 1997).  Local grantees use funds for community-based projects, as well as for evaluation and program monitoring, technical assistance and training, media campaigns, advisory councils, resource and communication networks, toll-free hotlines, and satellite conferences.

Number of States	Grantees
38	Community-based organizations
29	Local Boards of Education/School Districts/Schools
27	Youth Serving Organizations
23	Local Health Departments
22	Faith-Based Organizations
19	Universities
17	Local Coalitions/Partnerships/Advocacy Groups
17	Consultants/Contractors
16	Media/Research Firms
15	Health Care Organizations
14	Non-Profit Organizations
SOURCE:  Maternal and Child Health Bureau (200).

The core of all these efforts is a message about the benefits of abstinence from sexual activity, which most often is delivered through a curriculum-based program in a school setting.  This approach, which often has a youth development component, frequently is referred to as character-based education or “assets building.”  Other common efforts include adult mentoring, peer mentoring, parent education, before- and after-school programs, and recreational-based activities.  More broadly based initiatives include curriculum development, public awareness campaigns, and community partnership development (Maternal and Child Health Bureau 2000).

The majority of programs aim their abstinence message at middle school students.  However, some target a wider age spectrum, starting younger and persisting longer.  Many also target high school youth, and a few target out-of-school youth.  Resource constraints lead many programs to limit their selection of a target population.

Focusing on youth of middle school age or younger has helped some communities resolve the debate between those who favor an “abstinence-only” approach and those who favor an “abstinence-plus-contraception” approach.  The emerging consensus that the middle school years are an appropriate time to offer these interventions suggests some agreement that a message of abstinence is an important foundation for all efforts at youth risk avoidance and pregnancy prevention.

Abstinence Education Programs Offer More than a Single Message of Abstinence

A common perception is that abstinence education programs focus narrowly on teaching youth the single message of abstaining from sexual activity before marriage.  Observations of the programs participating in the evaluation suggest that many of the programs include multiple components designed to reinforce and support their abstinence message.  For example, the program curricula used by the five targeted programs address a broad range of issues, from building self-esteem to understanding and aspiring to healthy marriages and parenthood, and  to teaching skills that will help youth make — and follow through on — good decisions (Table 2).

Table 2: Curriculum Topics of Abstinence Education Programs Participating in the Impact Evaluation

Topics	FL	MS	SC	VA	WI
Building Self-Esteem	X	X	X		X
Developing Values/Character Traits	X	X	X	X	X
Formulating Goals	X	X	X	X	X
Making Decisions	X	X	X		X
Avoiding Risky Behavior	X	X	X	X	X
Maximizing Communication	X	X	X	X	X
Strengthening Relationships	X	X	X	X	X
Understanding Development and Anatomy	X	X	X	X	X
Understanding STDs	X	X	X	X	X
Withstanding Social and Peer Pressure	X	X	X	X	X
Addressing Consequences/Self-Control	X	X	X	X
Resolving Sexual Conflicts	X	X	X	X
Learning Etiquette and Manners	X				X
Aspiring to Marriage	X		X	X	X
Understanding Parenthood				X
Source:  Program curricula manuals.

In addition to the abstinence education curricula, program services illustrate the breadth of activities offered to youth.  Weekend summits, community roundtable discussions, lending libraries and websites, essay contests, door prizes at school dances, “abstinence coupon books” for local businesses, summer programs, family retreats, and program recognition ceremonies are examples of the range of activities offered to program participants.

In general, the program curricula, activities, and opportunities provided to youth reflect, either implicitly or explicitly, various underlying theories of adolescent behavior and the implied logic models that explain the knowledge, attitudes, intentions, and behavior of youth.  The most influential theories of adolescent behavior incorporate multiple factors believed to shape youth behavior, including:

• Youth background and personality characteristics.  Youth background  characteristics include demographic factors, biological factors such as age and physical development, religion, and school and community characteristics.  Important personality characteristics include impulsivity, risk-taking proclivities, sense of efficacy, and youth temperament, which also can influence youth attitudes and behavior (Costa et al. 1995; and Jessor and Jessor 1977).
• Family attitudes and relationships.  This includes parental attitudes, values, and communication, as well as the influence of siblings.  The quality of relationships and extent of interaction with parents are related to the degree of engagement in risky behaviors (Feldman and Brown 1993; Blum et al. 1987; and Whitbeck et al. 1993).  Siblings, too, influence youth behaviors, including the number and birth order (Rodgers and Rowe 1988).
• Youth attitudes, values, and knowledge.  Attitudes and values strongly influence adolescent behavior.  Knowledge about risks of certain behavior affect the decision-making and behavior of adolescents.  Youth who perceive their vulnerability as high, the consequences as serious, and the costs greater than the benefits are most likely to avoid risk (Bandura 1977 and 1986; and Rosenstock 1988).
• Peer relationships and social influences.  Attitudes and values of peers are powerful predictors of adolescent attitudes and behavior (Evans 1976; McGuire 1964; Schinke et al. 1985; and Fishbein and Ajzen 1980).  Avoidance of risky behavior requires that youth understand social pressures and that they have the skills to resist those that are negative.  Interactions of individuals and their environment influence youth behavior (Hawkins and Catalano 1992; and Klitzner 1993).  Negative attitudes, behaviors, and interactions can be risk factors, while positive bonds formed with a school, peer group, or community can be protective factors.

With varying emphasis, abstinence education programs recognize these social, developmental, and community antecedents and mediators of youth behavior and draw on one or more of four complementary strategies for promoting abstinence and other healthy behavior:  (1) Helping youth learn skills to deal effectively with social influences and peer pressure; (2) providing them with information to better assess the benefits and costs of their actions; (3) altering family and community norms and supports; and (4) promoting healthy development through age-appropriate, healthy-activity choices.

Social Influences and Peer Pressures.  All 11 abstinence education programs in the evaluation address social and peer pressures.  Lessons on decision-making and communication, and active-learning exercises (such as role-playing) often are used to help youth develop and apply critical skills needed in situations that involve peer pressure.  Discussions of attitudes, beliefs, and values help youth distinguish themselves from perceived peer norms.  The Teens in Control program in Clarksdale, Mississippi, for example, uses videos to depict teens in relevant situations and then engages program youth in role-playing exercises so that they can apply decision-making and communication skills.

Benefits Assessment.  Many abstinence education programs seek to reduce the motivation to engage in risky behaviors by teaching youth to recognize the consequences of such behaviors and the benefits of avoiding them.  They use various strategies to alter motivation, including exercises to build confidence, self-esteem, problem-solving abilities, and conflict negotiation skills.

The Not Me Not Now program in Monroe County, New York, has as its cornerstone a media campaign that makes adolescents, parents, and the community more aware of the consequences of teenage sexual activity and stresses positive future options for teens to motivate them to remain abstinent.  The media campaign includes paid television and radio advertising, billboards, 5,000 posters in schools, mouse pads for public school students ages 9 to 14, t-shirts, educational materials for parents and schools, and a quarterly newsletter mailed to youth ages 9 to 14.  Parents are targeted through workshops, as well as through a widely distributed pamphlet and video.  The advertisements convey the program’s message by drawing on local youth to act in the commercials.  Local youth also serve on an advisory panel to help shape the media messages and gauge response.

Several of the programs work to alter benefits assessments through teaching about the values in and what constitutes a good marriage.  The very heavy emphasis on the institution of marriage in some of the programs reflects a belief that the lack of understanding of, or role models for, marriage results in its being undervalued.  This undervaluing of marriage is believed to contribute to casual and early sexual relationships.  The ReCapturing the Vision program in Miami, Florida, is an example of a program that attempts to change knowledge about and the perceived value of marriage.  The topic of marriage is covered over a period of more than a month of daily classes in which program participants paint their own small “hope chests,” discuss extensively what makes for a good partner in life, “plan” for their own weddings, and hold a mock wedding at a local hotel.  The selected “bride” draws on the lessons on relationships and partner qualities in selecting the “groom.”  Parents of the bride and groom, as well as program participants from all schools, attend the mock “wedding,” which culminates in vows of chastity until a real wedding.

Family and Community Norms and Support.  Particularly the community-wide programs and the more intensive targeted programs often attempt to enhance youth’s involvement with their families, peer groups, schools, and community through offering a multifaceted set of services, activities, and educational and training opportunities.  They may be designed to mobilize broad, interrelated factors within the larger community to strengthen positive influences on individual behavior.

The Families United to Prevent Teen Pregnancy (FUPTP) program in Milwaukee, Wisconsin, and the ReCapturing the Vision program in Miami, Florida, for example, both work hard to improve parent-child communication and to enhance participants’ involvement with their families through means such as monthly parent meetings, a weekend family retreat, and home visits.  The ReCapturing the Vision program seeks to develop positive peer relationships by running a class-appointed “court system” for students who cause problems and an annual Teen Talk Symposium in which teen and celebrity panelists address questions on relationships and sexual issues.

Both FUPTP and the ReCapturing the Vision programs aim to strengthen participants’ commitment to school through a heavy emphasis on school performance, with report card checks and dedicated homework/tutoring time.  Both programs are intensive; they meet daily throughout the school year and offer program participants the opportunity to enroll for more than one year.  They address skills needed to support community engagement, with opportunities for community service and lessons on social etiquette through dining at local restaurants.  These programs aim to provide youth with a value system that will help them develop their decision-making skills, communication skills and relationships, and goal setting.  In addition to participants’ attitudes and values, they also focus on self-esteem.  For example, one strategy used by ReCapturing the Vision, an all-girls program, is to provide participants with “makeovers” to improve their self-image.

Developmental Needs.  Many programs offer age-appropriate activities and supports designed to fill unmet psychological and emotional needs, develop psychosocial competence, and ease teenagers’ transition to positive, independent, and productive adulthood.  For both the ReCapturing the Vision and the FUPTP programs, a primary vehicle for engaging youth is giving them a strong sense of identity with a group that embraces positive values, such as community, responsibility, leadership, trust, and respect for others.  This group identity is achieved in a number of ways.  In the ReCapturing the Vision program, a local business pays to have suits designed and tailor-made for each program participant.  Both programs help foster a sense of identity in the public’s eyes through highly publicized public rallies to support the choice of abstinence.  In the FUPTP program, participants write their own “raps” that reflect what they’ve learned from the program and each day recite a program pledge.

Most Participants Feel Favorably About Their Program Experience

Youth tend to respond especially positively to programs when the staff are unambiguously committed to abstinence until marriage and when the program incorporates the broader goal of youth development.  Young teachers who are public about their own commitment to abstinence appear to be very successful in engaging program youth.  The Heritage Keepers Program in South Carolina, for example, uses teachers who demonstrate this unqualified endorsement of abstinence until marriage.  The program trains these teachers to be direct and to communicate their commitment to abstinence.  Observations during site visits suggest that committed and outspoken teachers are effective in capturing the attention of students and getting them to listen and question.

Most programs have limited resources and so must make trade-offs between the intensity and duration of services they provide each participant and the overall number of youth they serve.  As observed in classrooms and reported during focus groups, youth seem to respond especially favorably to the intensive programs because they are tailored to the developmental needs of youth and provide services and activities that go far beyond the classroom curricula.  These programs often include field trips, weekend activities, end-of-the-year celebrations, and local and national motivational speakers, all of which are geared to helping youth make informed choices about their behaviors.  During focus groups, students in one program reported that they are learning about goals; values; high and low self-esteem; high- and low-risk behaviors; good and bad consequences; responsibility; social skills; and abstinence from sex, drugs, and alcohol — and learning not to become a teen parent.

Not all programs have met with enthusiasm, however.  The less intensive programs, in particular, more often fail to engage students fully and encounter dissatisfaction among youth with program services.  Participants in one such program complained that the class was boring and was “just another class” that “didn’t offer much benefit.”  Students in another site acknowledged that some students make fun of the program’s slogan.

Challenges Addressing Peer Pressure and the Communication Gulf Between Parents and Adolescents

Abstinence programs face real challenges in addressing peer pressure and the communication gulf between parents and adolescents.  Testimony from youth about their perceptions of what is going on among their peers reflects, among other things, the extent to which they have been inundated with media messages, images, and thoughts about sexual activity at a very young age.  Youth are observing, thinking about, and using sexual activity as a system for peer classification.

Constructive activities, particularly during after-school hours, can be an antidote to peer pressure, but such activities are not always available to youth.  During focus group sessions, youth acknowledged that sexual activity takes place during unsupervised hours after school, as well as at large parties and on “dates” or in small gatherings of friends on weekends.  When asked about what they usually do after school, many said they are bored.  Many go home and watch television, talk on the phone, do homework, or baby-sit siblings.  In one community, parents and their children both said, “We just don’t have anything here,” indicating few options or places for youth to go after school.  In another community, there is a youth recreation center, but not within walking distance of the school.

Good communication between parents and adolescents can also counter peer pressure (Miller 2001; and Blum and Reinhart 2001).  Recognizing this, many programs try to bridge the gap in parent-child communication in any of three ways.  First, they often try to help youth feel more comfortable discussing with their parents issues related to sex.  Second, they may try to engage parents actively in the programs, inviting them to program events with their children.  Third, many have special parent-focused components that aim to strengthen the parents’ ability to interact more effectively with their children.

Both the Abstinence Education Initiative (AEI) Coalition of Equipping Youth in Cedar Rapids, Iowa, and the Not Me Not Now program in Monroe County, New York, report increases in parent-child communication as a result of homework assignments requiring parent participation.  Testimony from parents during focus groups confirms that programs are experiencing some success at improving parent-child communication.  In focus groups, parents report that youth are becoming more comfortable talking about sensitive topics, “asking questions that they didn’t ask before” and “opening up conversations.”  One parent commented that she is “embarrassed to talk about some of these things, but [her son] talks and makes [her] more comfortable.”  Another parent said, “My son has calmed down a lot; we talk to each other more.”  Particularly in the intensive programs, parents generally agree that the program is having a positive effect on their children by giving them some important skills, or “building blocks,” with which to have positive interactions and communication with others, including themselves.

Program efforts to involve parents in special program events with their children often succeed in bringing parents to the events.  For example, parents attend weekend rallies held by the FUPTPprogramin Milwaukee, Wisconsin, by Heritage Community Services program in South Carolina, and by ReCapturing the Vision in Miami, Florida.

On the other hand, promoting and sustaining active involvement in parent education and enrichment programs has been difficult.  Despite widespread parent enthusiasm for programs, getting more than a small fraction actively involved has proven to be a major challenge for virtually all programs.  In the Not Me Not Now program in Monroe County, New York, and the Youth Abstinence Education Program in Tooele, Utah, workshops on parent-child communication have been widely advertised, but attendance has not measured up to the extensive outreach campaign.  Free pamphlets and videos have been made available to parents in local supermarkets, but these too have not been taken at the rate expected.  The Not Me Not Now program is now considering ways to work within existing parent groups, such as those convened by local churches or local adult education programs.

Even when a program includes a focus on the whole family, engaging parents can be a struggle.  In FUPTP, staff members often register students for the program through a home visit, during which they explain the importance of parental involvement.  The program holds monthly meetings for parents for which they provide food, transportation, and child care.  They send home a regular newsletter and provide parents with progress reports.  The curriculum uses take-home handouts for parents.  Despite these efforts, parent involvement remains low.

Partnerships with Local Schools Are Valuable but Often Difficult to Establish

Local schools are usually important partners in abstinence education programs.  Schools provide unparalleled access to youth as a captive audience.  Other than through the media, there is really no other way to reach so many youth on a regular basis.  Furthermore, although the media can reach extraordinary numbers of youth, they do not have the targeted focus on youth development that many educators feel is critical to decisions regarding sexual attitudes and behaviors.  However, establishing partnerships with schools is sometimes difficult.

Based on observations and reported experiences of the programs visited during site selection and more in-depth examination of the 11 sites that are the focus of the evaluation effort, it appears that schools generally become partners in abstinence education funding through one of four models of organizational structure.  Three involve cooperation between the grantee (the local agency that received Section 510 funding from the state) and others (Figure 1):

Figure 1: Models of Organization Among Section 510 Abstinence Education Programs

• The one-to-one program model refers to an organizational structure in which a community-based agency is awarded a grant to provide abstinence education to youth in a school-based setting, and establishes a partnership with the local schools or school district.  The Powhatan County Health Department in Powhatan, Virginia, for example, is providing an abstinence education curriculum to students in the county’s middle and high schools.  Program staff, hired by the health department, have a cooperative agreement with the school district to teach these classes.
• The wheel program model depicts an organizational structure in which the Section 510 grant recipient spearheads an abstinence education initiative in several different settings, often drawing on other community resources such as schools, local businesses, health care providers, or other social service organizations to assist with newly developed efforts.  The Fort Bend Abstinence Leadership and Resources for Teens initiative in Fort Bend, Texas, as well as the McLennan County Collaborative Abstinence Project in McLennan County, Texas, have spearheaded numerous and diverse abstinence initiatives in their communities.  In both cases, these new initiatives include involvement of the medical community and development of a local information and resource center.  The Youth Abstinence Education Program of the Tooele, Utah, County Health Department has launched school, extracurricular, parent, and community awareness activities and programs.  Examples are a “self-esteem day” for fifth through eighth graders, games and door prizes at program-sponsored or supported parties and dances, parent-teen conferences, and abstinence messages contained in discount coupons for local stores.
• The coalition program model is an organizational structure in which multiple participating organizations contribute staff to a newly formed entity and jointly oversee program initiatives and the Section 510 funding.  For example, coalition members in the AEI Coalition of Equipping Youth program in Cedar Rapids, Iowa, include a county school district, a corps of high school students who present skits in other schools, a volunteer organization operated by physicians’ spouses, a group that focuses on substance abuse, and a program staffed by parenting teens.  Funding is coordinated through the coalition, and monthly meetings of coalition members provide an opportunity for members to coordinate and build on one another’s efforts.

The fourth model, though not used by any of the programs selected for the evaluation, is a single-agency model.  For example, a school that receives funds to redesign its own sex education curriculum may operate independently rather than in a cooperative relationship with any other organization.

Regardless of the organizational structure, creating and sustaining partnerships between abstinence education programs and local schools often requires enormous persistence and resources.  All the programs in the evaluation have established such partnerships, but with varying degrees of challenge and success.  Some programs have been welcomed and given extensive support.  Some have had to pursue partnership agreements on numerous fronts over long periods of time, conducting community discussions, seminars with parents, and repeated conversations with principals, district superintendents, and local school boards.  Still others have received little ongoing support or have been denied access and forced to seek partnerships in other districts or with other organizations.

Where districts and schools are reluctant to support abstinence education programs, it is often because of conflicts over the sex education curriculum.  In an effort to appear even-handed, the Cedar Rapids, Iowa, school district excluded from its classrooms both the abstinence educators and educators from Planned Parenthood.  Still, the AEI Coalition of Equipping Youth program steadily expanded its access to county schools as the positive reputation of its school-based program in one school district spread.  The public schools in Waco, Texas, rejected the abstinence education funds.  As a result, the McLennan County Abstinence Education Project focuses its school-based services in surrounding school districts and works with faith-based and other community service organizations to reach high-risk youth in the city.  And in South Carolina, the Heritage Keepers program was rejected in some communities that did not want abstinence education as the sex education curriculum in their schools.

Lack of support from school staff is often a factor jeopardizing a school-based abstinence education program.  Based on the reported experiences of leaders of the 11 programs in the evaluation, as well as a number of other abstinence education programs visited during site selection, skepticism can emanate from the principal or from classroom teachers.  Even when principals invite a program into a school, unless they work to underscore its merit, classroom teachers may withhold support.  Teachers’ resistance to a program’s mission or unwillingness to coordinate with program staff, as many sites have reported, can undermine the effectiveness of program operations.

Waning support can result from the emergence of new priorities.  For example, in Powhatan, Virginia, an increase in school violence usurped much of the principal’s time and resources, so that the abstinence program received less attention.  The current national emphasis on school accountability for student achievement is increasing the priority given to “core” rather than “non-core” courses, and abstinence education programs often struggle in this environment.

Lack of support from a principal also can emerge as a result of staff turnover.  As in several schools participating in the evaluation, the principal who invited the program into the school leaves, and the new principal’s agenda and priorities do not include strong support for the abstinence education program or the organization that runs it.  The Teens in Control program in Mississippi, for example, worked very hard for an entire year to gain the full cooperation and welcome from two of the three districts in which it planned to operate.  In one of these districts, a state takeover resulted in a new principal being hired to “turn the school around.”  As a result, the new principal had extremely limited opportunity or incentive to give any priority to the abstinence program.

Lack of dedicated space for the abstinence program can be a symptom of weak school support.  Unless an abstinence program is replacing an existing school offering, space availability is often an issue.  Several of the programs involved in the evaluation face space constraints and often get shuffled around.  This instability in physical location can further undermine support for the program, even among program participants.  Uncertainty regarding program location and the inability to establish a secure “home” (either to leave materials/resources or to create a physical identity for the program) can create frustration and ultimately jeopardize the interest and commitment of students and teachers.

Programs can strengthen their partnerships with schools through visibility.  They can do this by making sure that the principal and the teachers understand the program’s mission and curriculum, and that they are regularly informed on issues related to the program and its participants.  For programs with time and flexibility, providing a direct link to the needs of classroom teachers — such as through the provision of time for homework or tutoring assistance, as is done in the FUPTP program in Milwaukee — can help integrate the program into the school’s existing agenda.  Finally, most schools will welcome programs that achieve visibility through popularity with participants and parents; those that make a real investment in youth will be rewarded.  An example is the success of the grassroots efforts of parents from the Iowa College Community School District in convincing the school board to adopt the abstinence education program offered through the AEI Coalition of Equipping Youth.

Among the partners of programs participating in the evaluation, some principals and schools have been deeply committed to the abstinence education initiative.  In such cases, positive, mutually beneficial relationships for all — the school, the program, the participants, and the parents — have generally emerged.

More Lessons to Come

The collective state and local abstinence education program experiences from across the country now constitute a sizable body of information to inform the dialogue over approaches to reducing teen sexual activity.  The level of attention now focused on teen sexual attitudes, behaviors, and consequences should help determine how best to assist communities in selecting programs to meet their local needs.  This attention can, as noted in the Charleston Post and Courier, create “a healthy dialogue among teachers, clergy, parents, health professionals and students to share experiences from each perspective and come to a consensus of what’s best for the children” (Lawrence 2001).

Getting the most out of these experiences depends critically on learning what impacts various approaches to abstinence education have for the youth they serve.  The findings from the impact evaluation component of this evaluation will be critical to shaping the future policies and programs to best meet the needs and interests of youth.

Chapter III: The Foundation for Assessing the Impacts of Abstinence Education Programs

In 1997, Congress authorized, and its staff requested, a scientifically rigorous impact evaluation of the abstinence education programs funded under Title V Section 510 of the Social Security Act.  Policymakers, school officials, community leaders, program staff, and parents all want to know the extent to which particular program strategies succeed.  They want to know for whom these strategies work and to understand the ingredients of that success.  They also want to gather information that will guide program improvement for any groups identified as not responding well to particular strategies.

The early operational findings discussed in Chapter II provide a critical foundation for subsequent reports to address these questions of program effectiveness.  Much has been learned about school and community responses to the abstinence education funding, the range and nature of coalitions formed, the design and operational experiences of the programs, and the responses of youth and their parents.  Evidence on the impacts of the abstinence education programs, however, is not currently available, because obtaining definitive and rigorous evidence on program impacts is a complicated, long-term process.

Over the past four years, the evaluation effort has laid the foundation for a careful, comprehensive, and rigorous assessment of program impacts.  The research team has selected five targeted programs for the impact evaluation, built the partnerships needed to conduct the evaluation, enrolled samples large enough to support reliable estimates of program impacts of each program, and designed and implemented a rich and multipronged data collection strategy to support the evaluation of each program.  The impact evaluation will build on this foundation to determine the extent to which the abstinence programs in the evaluation achieve six specific goals:

1. Strengthen knowledge and attitudes supportive of abstinence
2. Induce more youth to embrace abstinence as a personal goal
3. Reduce sexual activity among youth
4. Persuade sexually experienced youth to become or remain abstinent
5. Lower the risk of STDs
6. Lower the risk of nonmarital pregnancies

Evidence on attainment of these goals is being developed through a scientifically rigorous impact evaluation design, careful and comprehensive data collection, and detailed and deliberate analysis and reporting.  The impact evaluation design avoids the limitations of most prior research on abstinence education programs.  Few previous studies, for example, used rigorous experimental research designs to generate program and control groups.  Those that did use experimental designs usually randomized entire classrooms or schools rather than individual students, which severely reduced their effective sample sizes.¹  Few were able to use independent professional data collectors.  Finally, few were able to track outcomes of their sample members over an extended follow-up period.  Consequently, results usually pertain to outcomes of youth before they reached the age when many were engaging in sexual activity.

Scientific Rigor in the Study Design

The scientific rigor of the impact study design rests on four key elements.  It begins with the selection of strong, well-implemented, replicable program models.  Second, the impact study uses a rigorous experimental design to create program and control groups within each site.  Third, the sample enrollment period was long enough to generate adequate sample sizes to support reliable impact estimates.  Finally, the impact evaluation includes a reasonable length follow-up period to ensure that relevant changes in behavioral outcomes can be measured.

The impact evaluation examines five programmatic strategies geared to the needs of the local communities (Table 3).  Measuring impacts for a range of program models promotes the goal of identifying and documenting abstinence education strategies appropriate to varied local needs and contexts.  For example, the Florida and Wisconsin programs serve mainly youth from single-parent households; these programs are intensive and include strong components on relationship development and maintenance, as well as understanding and appreciation for the institution of marriage.  In the Mississippi program site, many youth live in large, multigenerational households isolated from the broader community.  The program in this community is delivered through the schools and emphasizes both basic knowledge development and components focused on managing peer pressure.  Youth in the South Carolina and Virginia programs live in communities that mirror “middle America.”  The program in Virginia is a low-cost, school-based intervention, while the one in South Carolina is a more comprehensive and intensive youth development initiative.  These choices of program strategies reflect community characteristics and perceptions of how best to serve youth, given local needs and the resources and constraints of the partner schools.

One implication of the variation in program interventions and services is that it is not possible to reach a single judgment about the efficacy of abstinence education.  Such a judgment would only be possible if there were a single, well-defined intervention, one that could vary in its “dosage” across sites but is similar in nature across all sites.  In the case of the Section 510 abstinence education programs, however, the interventions and services vary considerably across program sites and sometimes even within a program site.  In the absence of definitive evidence on the efficacy of a specific abstinence education approach, this variation is a natural result of the funding opportunities available through Title V Section 510.  In addition, the variation in the abstinence education programs provides the opportunity to learn about the effectiveness of different programmatic strategies.

Table 3: Program Interventions and Services Received by the Control Groups

Program Location	Program Intervention	Control Group Services
FL (Miami)	Elective class offered daily, all year to girls in middle schools (ReCapturing the Vision and Vessels of Honor)	Other elective class
MS (Clarksdale)	Mandatory weekly year-long abstinence education curriculum (Revised Postponing Sexual Involvement and Sex Can Wait)	Regular health class
SC (Edgefield)	Five-session mandatory curriculum with voluntary enrollment in weekly or biweekly character clubs (Heritage Keepers)	Five-session mandatory abstinence curriculum without character clubs
VA (Powhatan)	36-session mandatory curriculum (Reasonable Reasons to Wait; The Art of Loving Well; and Choosing the Best)	Regular health class
WI (Milwaukee)	Voluntary after-school program; two hours daily all year for multiple years (Families United to Prevent Teen Pregnancy)	Regular after school programs; no special services

The impact evaluation uses an experimental design.  In an experimental design study, program slots are filled by youth who are selected at random from a larger pool of eligible and appropriate youth (Figure 2).  Random assignment procedures divide youth into a program group that has access to the abstinence education program and a control group that does not receive the program, but may receive regular or alternative services.  The contrast in services being studied varies depending both on the nature and intensity of the program services and the experiences of the control group (see Table 3).

Figure 2. Study Sample Enrollment and Tracking

Longitudinal tracking of both the program and control group youth begins at the time of sample enrollment and continues for 18 to 36 months, depending on the time of initial enrollment.  The comparison of outcomes for these two groups over time provides the basis for judging impacts of the program.

The experimental design offers the best means of measuring, with a known degree of certainty, how successful the programs are overall and how well they serve key subgroups of youth within a site.  This is because, with careful implementation, the only systematic difference between the program and control youth should be their access to the program.  As a result of the random assignment, the program and control groups have similar demographic and background characteristics within any study site (Figure 3) and they are exposed to a common school and community context.

Figure 3. Demographic and Background Characteristics are Similar for Program and Control Youth Within Each Site.

However, the characteristics of sample youth vary across study sites due to a combination of factors, including program targeting practices and differences in the program and community characteristics.  For example, the average age of youth at the time of sample enrollment ranges between 10 in the Wisconsin program site, which delivers its services through an after-school program, to 13 in the Virginia program site, which serves exclusively eighth graders.  The proportion of sample youth who are nonHispanic black ranges from a low of 12 percent in the Virginia program site to over 80 percent in two other programs, one of which operates in a rural southern community, the other in an inner-city setting.  The proportion living in two-parent families ranges from 37 percent to more than 75 percent.

Random assignment generates, in each study site, program and control groups consisting of youth who, on average, are subject to similar family rules and express similar attitudes and values about abstinence before the program group is exposed to abstinence education services (Figure 4).  For example, the proportion of youth who say their parents have strict rules about companions they spend time with varies across sites between 15 and 45 percent, but is similar for program and control youth within each site.  Between 62 and 83 percent of sample youth in each study site reported believing that “having sex as an unmarried teen would make it harder to subsequently have a good marriage,” and between 16 and 35 percent hold the view that “having sex is a way to tell someone you love them.”  In all cases, however, the views of program and control youth are nearly identical within each site.

Figure 4. Family Rules and Attitudes about Teen Sex are Similar for Program and Control Youth at Sample Enrollment.

A major advantage of the random assignment design is that it protects against selection bias in the impact estimates for the individual programs studied.  Other evaluation designs are vulnerable to selection bias, which can seriously undermine the credibility of their results.  Some evaluations, for example, have relied on comparisons of outcomes for participants in “elective” programs and youth at the same site who, for some reason, do not participate.  Others compare outcomes for program youth with youth who responded to local or national surveys.  In both cases, there is a strong possibility that the participants differ in some preexisting but unobservable way from the comparison group.  These preexisting differences may lead to biased estimates of program impacts.

Pre-post comparison designs have other defects.  Comparisons of measures for participant groups before and after their involvement in a program can be affected not only by the program but also by natural maturation effects.  For example, data from the National Longitudinal Survey of Adolescent Health show that the percentage of teens who have ever had sex increases from 9.6 percent at age 13 to 19.6 percent at age 14.  Thus, using a pre-post design to measure program impacts on abstinence would seriously bias the results toward estimates of no impacts or possibly even adverse impacts.

Studies that rely on comparison samples drawn from existing survey databases can be weakened by both bias and unreliability.  Some studies, for example, compare program participants with respondents to the Youth Risk Behavior Survey or the National Longitudinal Survey of Adolescent Health.  Such study designs have the added complications arising from noncomparability of survey instruments, data collection methods, and timing of the data collection (Santelli et al. 2000).

Carefully designed and implemented experimental design studies can both overcome these weaknesses and offer unanticipated bonuses for programs and policymakers.  When program resources are not sufficient to serve everyone, many youth will not receive the abstinence education program services, regardless of whether there is an experimental-design evaluation or not.  Random assignment is often fairer than commonly used practices such as “first come, first served” or referral systems to allocate scarce program resources.  Random assignment designs also can provide valuable information about the magnitude of “unmet” demand for the program services.  Assuming that the evaluation design is implemented so that programs operate at capacity, the size of the control group provides a lower-bound estimate of unmet demand.  At the same time, the operational experience with outreach and recruitment provides qualitative information regarding how thorough and successful the outreach efforts are and may provide tips on how to strengthen future outreach efforts.

One limitation of a random assignment design for measuring program impacts arises if any of the programs has major spillover effects.  If, for example, youth who are assigned to the program group interact with youth in the control group in ways that transfer the benefits of the program intervention to peers in the control group, the random assignment study design will underestimate program impacts.  Similarly, if the presence of an intervention in the school or community significantly alters the overall school or community climate in important ways, this could lead to underestimates of program impacts.  The overall judgment of the evaluation team is that, for each of the five sites included in the impact evaluation, spillover effects are expected to be very small in relation to the direct effects on those who participate in the program.  Nonetheless, this is an issue that has received ongoing attention by the evaluation team and that is addressed in the follow-up surveys with students.²

The impact evaluation has large sample sizes of between 400 and 700 youth per site.  Large sample sizes protect against the possibility of failing to detect true program impacts simply because the study lacks statistical power.  It is important that, if no statistically significant program-related impacts are detected on sexual activity or on risks of STDs or pregnancy, for example, one of two conditions holds:  (1) there really was no impact of the program at all, or (2) any program impact was sufficiently small as to be of no importance to policymakers or practitioners.

What constitutes a sample size large enough to detect true impacts depends in large part on the nature of the program.  Generally, low-intensity or short programs have smaller impacts and, thus, require larger sample sizes to ensure that true impacts are picked up in the analysis.  The opposite is generally true of programs that are longer or more intensive.

The originally planned one-year period of sample enrollment for the evaluation was extended to three years in order to generate samples large enough to ensure detecting meaningful program effects and to avoid false claims of no effects.  Final sample sizes per site are expected to vary between 443 (280 program/163 control) and 700 (371 program/329 control) students.  Table 4 presents estimates of changes in outcomes the study will be able to detect using reasonable standards of statistical power and precision, given these sample sizes and given national estimates of the prevalence for selected outcomes.  For example, the study will be able to detect true program impacts on the percentage of students who are sexually experienced of 7.2 percentage points or larger in the site with 700 youth in the study sample and of 11.2 percentage points or larger in the site with 443 youth in the sample.

Table 4: Minimum Detectable Changes in Outcomes

Outcome Measure (Wave 3)	Estimated Prevalence of Outcome(a)	Minimum Change Detectable(b)
		Largest Sample	Smallest Sample
Taken Virginity Pledge	14.9%	±6.0%	±9.3
Sexually Experienced	24.1%	±7.2%	±11.2
Abstinent at Follow-up(c)	86.5%	±5.8%	±8.9
At Risk of Pregnancy(d)	17.3%	±6.4%	±9.8
Sample Sizes		700	443
Program Group		371	280
Control Group		329	163
Notes: a.  These estimates are based on computations from the National Longitudinal Survey of Adolescent Health data.  National prevalence estimates for youth at different ages have been weighted by the age distribution of the Title V Section 510 abstinence education program evaluation sample in the construction of these estimates. b.  Minimum detectable differences are calculated based on the actual sample sizes, adjusted for anticipated nonresponse to follow-up surveys.  A 95 percent confidence interval and an 80 percent power requirement were used. c.  Defined as never had sexual intercourse or not sexually active in past 90 days. d.  Defined as sexually experienced and did not use a highly effective method of contraception at last intercourse.

To guard against errors that might arise based on findings from small sample sizes with low statistical power, no impact evaluation results will be released until data for the full study sample are available.  Results based on just the first one or two years of sample enrollment would run a risk of missing true impacts simply because of small sample sizes.

The study sample is being followed for up to 36 months.  The data collection schedule balances the need to release study findings at the earliest point possible with the importance of ensuring that study findings offer reliable guidance for policy and practice decisions.  Two waves of follow-up surveys are planned.  The wave 2 follow-up survey is being administered 6 to 12 months after initial study enrollment (when the wave 1 baseline survey was administered), and the wave 3 follow-up survey will be administered between 18 and 36 months after enrollment.  The interval between sample enrollment and the wave 3 survey depends on the age of youth at enrollment and the latest calendar date when surveys can be administered given the reporting schedule.  Under this plan, it is possible to analyze both short-term impacts on knowledge, attitudes, and intentions of youth related to abstinence and longer-term impacts on behavior.

Because so few youth engage in sexual activity before entering high school, outcome estimates based on wave 2 outcome data from middle-school years would miss program impacts on behaviors that most often would emerge at later ages.  Indeed, a shortcoming of previous abstinence education evaluations has been a follow-up period that does not extend beyond the middle school years.  Nationally, only 12 percent of males and 8 percent of females under age 13 have ever had sex (tabulations of the National Longitudinal Survey of Adolescent Health).  It is important to have the data collection period extend as long as possible in order to measure behavioral outcomes at ages where the prevalence of the behavior is high enough that changes in behavior will be observed.

The follow-up period for this evaluation is such that almost two-thirds of the study sample will be 14 to 18 years of age by the time of wave 3 followup and no youth will be younger than age 12.  Even with the extended follow-up period, however, only six percent of the study sample will have reached ages 18 and 19, when over half their peers are expected to be sexually active.  To address the potential need for even longer followup, the data collection procedures and plans for the evaluation are designed to accommodate longer followup, if resources were to become available.

Careful and Thorough Data Collection Plans and Procedures

Plans and procedures for the data collection in the impact evaluation are designed to capture the high-quality data needed for a thorough evaluation.  A conceptual framework for the program intervention strategies, which is consistent with the main theories of adolescent behavior discussed earlier, dictates the data collected and the timing of those data (Figure 5).

Figure 5. Conceptual Framework for Evaluating Abstinence Education Programs.

This framework acknowledges that the decisions youth make regarding sexual activity and other risk-taking behavior (Column IV) depend critically on a range of antecedent factors (Column I), including demographic and background characteristics of the youth, characteristics of their parents and their families, and the school and community context in which they have been raised.  For all youth, these antecedent factors are mediated by current parental attitudes, values, and supports; the attitudes, knowledge and relationships of the youth; and the current school and community context in which youth live (Column III).

There are two means by which the abstinence education programs (or any other intervention) operate to potentially alter the key outcomes of interest.  One is by directly altering youth behavior.  The other is through affecting the natural mediating factors, for example, by providing parents with knowledge and tools to better guide their children in sound decisionmaking; by changing the attitudes, knowledge and relationships of youth in ways that reduce their inclination to engage in risk-taking behaviors; or by changing the school and community climate in ways that are more expectant and supportive of abstinence.

The first wave of student surveys administered near the time of enrollment in the evaluation study gathers information on the antecedents of teen sexual activity and baseline values of the natural mediating factors (Columns I and III).  Wave 2 and wave 3 surveys gather information to mark changes in the natural mediating factors and the key outcomes (Columns III and IV).

A number of critical issues relate to the design and administration of these surveys to support the rigor of the impact study.  These include:

• Protecting the privacy of sample members
• Using questions that will generate valid and reliable measures of the constructs of interest
• Dealing with normal reluctance to report sensitive and socially undesirable information
• Addressing the fact that youth may have different definitions of what “abstinence” means
• Ensuring that questions are age appropriate
• Avoiding contamination of the programs’ abstinence message by the data collection itself

The rights and privacy of sample members and their parents are paramount.  Only youth whose parents have given active parental consent for their child to participate in the study are included in the study sample.  Moreover, youth themselves must actively consent to each wave of data collection.  The privacy of student responses is protected through a rigorous system that relies on professional, independent data collectors; that permits no personal identifying information on any survey form or data file containing survey responses; that maintains secure data files; and that has the protection of a Federal Certificate of Confidentiality (HRSA-00-15).

Survey questions were selected with attention to issues of the validity and reliability of the core constructs for the evaluation.  Each question included in any of the three surveys has been mapped to one of the core constructs in the conceptual framework (Figure 5 above).  Moreover, in determining the particular questions that would be asked to address each construct, careful attention was paid to the experience of prior studies with similar populations, including the validity and reliability of measures for different target populations and when questions were administered through different data collection modes.  For example, questions about school and family draw heavily on the National Longitudinal Survey of Youth and the National Education Longitudinal Study of 1988; questions on youth attitudes about sexual activity draw heavily on questions used in prior studies of abstinence education programs, such as Values and Choices (Olsen et al. 1991), Teen Aid (Weed et al. 1998), Responsible Social Values Program (Adamek 1993), Best Friends (Best Friends Foundation 1997), and Sex Respect (Weed and Olsen, no date); questions about other risk-taking behaviors, such as drinking and using drugs, draw heavily on questions in the Youth Risk Behavior Surveillance Survey (Centers for Disease Control and Prevention 1993) and the National Longitudinal Survey of Youth (Card 1993);  and questions about romantic relationships and actual sexual experiences draw on the National Longitudinal Survey of Adolescent Health (Udry and Bearman 1998), the National Survey of Family Growth (Card 1993), and the Youth Risk Behavior Surveillance Survey (Centers for Disease Control and Prevention 1993).

Each of the survey questionnaires was pretested with small groups of youth.  After revisions, they were then reviewed by key staff in the five programs participating in the impact evaluation, by the Federal Office of Management and Budget, and by the University of Pennsylvania’s Institutional Review Board.  In addition, staff from various constituent groups and policy organizations reviewed the survey questionnaires, provided useful insights, and made helpful suggestions.

Youth may not want to report sensitive and socially undesirable information.  Some respondents may feel uncomfortable reporting accurate information on questions about sexual intercourse and may distort their responses in the direction that they perceive as socially desirable.  Moreover, the problem of underreporting behavior that is considered socially undesirable may be exacerbated for youth who participate in abstinence programs, given the strong and unequivocal message of these programs.

To minimize the underreporting of sensitive behaviors, as well as to protect the privacy of the study sample, the evaluation uses self-administered survey data collection, maintains the strictest standards of confidentiality, and informs the survey respondents about them.³  The data collection procedures ensure that no one from the local schools — including teachers, administrators, and counselors — has access to students’ survey responses.  School and program staff are not allowed to participate in the data collection; trained interviewers conduct all survey data collection and focus groups.  As soon as the student surveys are completed, the interviewers immediately separate student contact information from the surveys and remove them from the school grounds.

Before the students complete the surveys, the interviewers assure all respondents that their answers will be kept confidential and will not be shared with anyone.  The consent forms sent home to parents, as well as the assent forms given to students, make it clear that no individual-level data from the surveys will be reported.  Rather, information on individual students will be combined into groups for analysis and reporting purposes.

Youth may have different definitions of abstinence.  The primary goal of the Section 510 abstinence education programs is to persuade youth to abstain from sexual activity.  Thus, it is very important that survey questions accurately measure this outcome.

Survey questions on abstinence from sex are difficult to design, since abstinence means different things to different people.  Some consider abstinence to mean refraining from all intimacy except for kissing and holding hands, while others consider abstinence as anything except sexual intercourse.  Participation in abstinence education programs may lead some youth to change their definitions of what constitutes sexual activity and abstinence.  Failure to address such program-induced changes in definitions could result in a downward bias in the reporting of abstinence by program youth relative to control youth and thereby limit the detection of true program impacts.

It is essential to ask in the clearest way possible about specific behaviors of greatest interest.  To have reliable measures of sexual activity, the evaluation survey instruments measure whether study youth have ever had sexual intercourse.  Since program and control youth are likely to have the same understanding, on average, of what sexual intercourse is, this measure has greater reliability than survey questions that ask simply about abstinence from sexual activity.

Outcome measures must be age appropriate.  The survey and administration methods for the study are sensitive to the social and emotional development of sample youth.  The abstinence education programs target youth in their preadolescent and adolescent years, and measurement of outcomes must reflect that age span.  Some programs serve youth as young as third or fourth grade.  Measures of program impacts for preadolescent youth may be quite different from those for adolescent youth.  For example, questions related to sexual intercourse are not appropriate for preadolescent youth, given the low prevalence of the behavior and, more importantly, the age inappropriateness.

The survey was designed to avoid contamination of the abstinence message.  The Section 510 abstinence education programs promote a strong message that teenagers should postpone sexual activity until marriage.  The programs do not promote use of contraception, on the premise that such information is inconsistent with program goals and sends a mixed message to youth.

However, a careful evaluation must measure the main outcomes of an abstinence education program.  The evaluation must be able to measure whether program participants do or do not abstain from sex and whether program participants do or do not engage in behaviors that risk pregnancy and exposure to STDs.  This requires that the survey questions about sex measure similar behaviors for the program and control youth and be detailed enough to measure exposure to risks of pregnancy and STDs.  Moreover, it is critical that the study’s informed consent procedures are consistent with asking youth these sensitive questions.

Accurate assessment of whether programs affect risk of STDs and pregnancy must take into account the behaviors of those youth who become sexually active.  Among youth who are sexually active, exposure to unwanted pregnancy and STDs depends, among other factors, on the use of condoms or other contraceptives.  Therefore, the evaluation survey instruments ask a limited number of questions about use of condoms and other contraceptives.  These questions are seen by and are asked only of youth who have already stated that they have had sex, and they are designed so that they do not provide information that the abstinence programs themselves avoid communicating.

Future Analysis and Reporting Plans

Over the three-year sample enrollment period, which ended in fall 2001, the evaluation team secured cooperation from 3,300 youth and their parents to participate in the impact evaluation.  To date, 3,081 of these youth have completed the wave 1 survey.  The wave 2 survey has been administered to those youth enrolled during the first two study years, with 1,791 completing this survey thus far.  In spring 2002, the wave 2 survey will be administered to the remaining sample, and the wave 3 survey will be administered to those who enrolled in the study sample during the first year of sample enrollment.  Sample youth will continue to be tracked through surveys and, in some cases, school records through fall 2003.  Furthermore, program operations and community context will be monitored continuously throughout the remainder of the study period to support the evaluation.

Table 5: Interview Schedule and Sample Sizes, by Time of Sample Enrollment

Sample Enrollment	Total N	Wave 1	Wave 2	Wave 3
Fall ’99/Spring ’00	1,040	Fall ’99/Spring ’00	Fall ’00	Spring/Fall ’02
Fall ’00	901	Fall ’00	Spring ’01	Fall ’03
Fall ’01	1,359	Fall ’01	Spring ’02	Fall ’03
Total Number	3,300	3,081	2,970a	2,805a
a.  Estimated number of completed surveys.

A report on the effects of the programs in achieving their short-term goals of changing knowledge, attitudes, and near-term behavioral choices will be completed in early 2003 once wave 2 survey data are available for the full study sample.  The final study evaluation report will be completed in summer 2005.  During intervening periods, the study team will prepare a limited number of special-focus reports that address particular questions of interest to Congress or the U.S. Department of Health and Human Services.

References

Adamek, Raymond J.  “Responsible Social Values Program:  Five-Year Project Final Report.”  Submitted to the Office of Adolescent Pregnancy Programs, U.S. Department of Health and Human Services, Kent, OH:  Kent State University, March 17, 1993.

Bandura, A.  Social Foundations of Thought and Action.  Englewood Cliffs, NJ:  Prentice-Hall, 1986.

Bandura, A.  Social Learning Theory.  Englewood Cliffs, NJ:  Prentice-Hall, 1977.

Best Friends Foundation.  “Adolescent Behavior Survey:  Pre-Survey for New Best Friends Girls and Pre/Pre Survey for Continuing Best Friends Girls.”  Washington, DC:  Best Friends Foundation, 1997.

Blum, R.W., M.D. Resnick, and T. Strack.  “The Impact of Parental Notification Law on Adolescent Abortion Decision-Making.”  American Journal of Public Health, vol. 77, no. 5, 1987, pp. 619-620.

Blum, Robert, and Peggy Mann Rinehart.  “Reducing the Risk:  Connections that Make a Difference in the Lives of Youth.”  Bethesda, MD:  Add Health c/o Burness Communications, 2001.

Card, J.J. (ed).  Handbook of Adolescent Sexuality and Pregnancy:  Research and Evaluation Instruments.  Newbury Park, CA:  Sage Publications, 1993.

Centers for Disease Control and Prevention.  Youth Risk Behavior Surveillance Survey — United States, 1993.

Centers for Disease Control and Prevention.  “Youth Risk Behavior Surveillance — United States, 1999.”  CDC Mortality and Morbidity Weekly Report Summaries, vol. 49, SS05, June 9, 2000 and vol. 47, no. 36, September 18, 1998 [www.cdc.gov/mmwr/PDF/SS/ S4905.pdf] and [www.cdc.gov/mmwr/PDF/wek/m4736pf], respectively.

Costa, F., R. Jessor, J. Fortenberry, and J. Donvan.  “Early Initiation of Sexual Intercourse:  The Influence of Psychosocial Unconventionality.”  Journal of Research on Adolescence, vol. 5, 1995, pp. 91-121.

Darroch, Jacqueline, David Landry, and Susheela Singh.  “Changing Emphases in Sexuality Education in U.S. Public Secondary Schools, 1988-1999.”  Family Planning Perspectives, vol. 32, no. 5, September/October 2000, pp. 204-211.

Evans, R.  “Smoking in Children:  Developing a Social Psychological Strategy of Deterrence.”  Preventive Medicine, vol. 5, 1976, pp. 122-127.

Feldman, S., and N. Brown.  “Family Influences on Adolescent Male Sexuality:  The Mediational Role of Self-Restraint.”  Social Development, vol. 2, no.1, 1993, pp. 15-35.

Fishbein, M., and I. Ajzen.  Understanding Attitudes and Predicting Social Behavior.  Englewood Cliffs, NJ: Prentice-Hall, 1980.

Goerge, Robert, and Bong Joo Lee.  “Abuse and Neglect of the Children.”  In Kids Having Kids:  Economic Costs and Social Consequences of Teen Pregnancy, edited by Rebecca Maynard.  Washington, DC:  The Urban Institute Press, 1997.

Haskins, R. and C.S. Bevan.  “Abstinence Education Under Welfare Reform:  Children and Youth Services Review, vol. 19, no. 5/6, 1997, pp. 465-484.

Haveman, Robert, Barbara Wolfe, and Elaine Peterson.  “Children of Early Childbearers as Young Adults.”  In Kids Having Kids:  Economic Costs and Social Consequences of Teen Pregnancy, edited by Rebecca Maynard.  Washington, DC:  The Urban Institute Press, 1997.

Hawkins, J.D, and R.F. Catalano, Jr. Communities that Care:  Action for Drug Abuse Prevention. San Francisco:  Jossey-Bass Publishers, 1992.

Jessor, R., and S. Jessor.  Problem Behavior and Psychosocial Development.  New York:  Academic Press, 1977.

Kish, Leslie.  Survey Sampling.  New York, NY:  John Wiley and Sons, 1965.

Klitzner, M.  “A Public Health/Dynamics Systems Approach to Community-Wide Alcohol and Other Drug Initiatives. ”  In Drugs and the Community, edited by R. Davis, A. Lurigio, and D. Rosenbaum.  Springfield, IL:  Charles Thomas Publishers, 1993.

Lawrence, C. “School Board to Vote on Sex Ed Materials.”  Post and Courier, Charleston, SC, July 22, 2001, p. 1B.

Maternal and Child Health Bureau.  1999 Annual Summary for the Abstinence Education Provision of the 1996 Welfare Law, P.L. 104-193, Section 510 of Title V of the Social Security Act, Rockville, MD:  Maternal and Child Health Bureau, 2000.

Maternal and Child Health Bureau.  Application Guidance for “The Abstinence Education Provision of the 1996 Welfare Law, P.L. 104-193, New Section 510 of Title V of the Social Security Act,” Rockville, MD:  Maternal and Child Health Bureau, May 1997.

Mathematica Policy Research, Inc.  “Evaluating Title V Abstinence Education Programs:  Summary of a July 2000 Workshop for State and Local Program Evaluators.”  Princeton, NJ:  Mathematica Policy Research, Inc., February 2001.

Maynard, Rebecca (ed.)  Kids Having Kids:  A Robin Hood Foundation Special Report on the Costs of Adolescent Childbearing.  New York:  The Robin Hood Foundation, 1996.

McGuire, W.J.  “Inducing Resistance to Persuasion.”  In Advances in Experimental Social Psychology, vol. 1, edited by L. Berkowitz.  New York:  Academic Press, 1964, pp. 191-229.

Miller, Brent.  “Family Matters:  A Research Syntheses of Family Influences on Adolescent Pregnancy.”  Washington, DC:  National Campaign to Prevent Teen Pregnancy, 2001.

Moore, Kristen, Donna Morrison, and Angela Greene.  “Effects on the Children Born to Adolescent Mothers.”  In Kids Having Kids:  Economic Costs and Social Consequences of Teen Pregnancy, edited by Rebecca Maynard.  Washington, DC:  The Urban Institute Press, 1997.

National Center for Health Statistics.  “National Vital Statistics Reports.”  Washington, DC: U.S. Department of Health and Human Services, vol. 50, no. 5, February 12, 2002.

National Institute of Allergy and Infectious Diseases, National Institutes of Health, U.S. Department of Health and Human Services.  “Workshop Summary:  Scientific Evidence on Condom Effectiveness for Sexually Transmitted Diseases (STD) Prevention.”  June 12-13, 2000.

Olsen, J.A., S.E. Weed, G.M. Ritz, and L.C. Jensen.  “The Effects of Three Abstinence Sex Education Programs on Student Attitudes Toward Sexual Activity.”  Adolescence, vol. 26, no. 103, 1991, pp. 631-641.

Public Law 105-33.  Balanced Budget Act of 1997, Subtitle 5001, Section a.  August 5, 1997.

Rodgers, J.L., and D.C. Rowe.  “Influences of Siblings on Adolescent Sexual Behavior.”  Developmental Psychology, vol. 24, no. 5, 1988, pp. 722-728.

Rosenstock, I., V. Strecher, and M. Becker.  “Social Learning Theory and the Health Belief Model.”  Health Education Quarterly, vol. 15, 1988, pp. 175-183.

Santelli, John, Laura Duberstein Lindberg, Joyce Abma, Clea Sucoff McNeely, and Michael Resnick.  “Adolescent Sexual Behavior:  Estimates and Trends From Four Nationally Representative Surveys.”  Family Planning Perspectives, vol. 32, no. 4, July/August 2000, pp. 156-165.

Schinke, S., L. Gilchrist, and R. Small.  “Cognitive-Behavioral Prevention of Adolescent Pregnancy.”  Journal of Counseling Psychology, vol. 28, no. 5, 1981, pp. 451-454.

Udry, J.R. and P.S. Bearman.  National Longitudinal Study of Adolescent Health:  Adolescent In-School Questionnaire and the In-Home Interview.  Funded by the National Institute of Child Health and Human Development with cooperative funding from 17 other federal agencies.  1998.

Weed, Stan E. and Joseph A. Olsen.  “Evaluation Report of The Sex Respect Program:  Results for the 1988-1989 School Year.”  Submitted to The Office of Adolescent Pregnancy Programs.  Office of Population Affairs, Department of Health and Human Services, Salt Lake City, UT:  Institiute for Research and Evaluation (no date).

Weed, Stan E., Joseph A. Olsen, and Raja Tanas.  “The Teen-Aid Family Life Education Project.”  An evaluation report prepared for the Office of Adolescent Pregnancy Program (OAPP).  Salt Lake City, UT:  The Institiute for Research and Evaluation, December 28, 1998.

Whitbeck, L., R. Conger, and M. Kao.  “The Influence of Parental Support, Depressed Affect, and Peer on the Sexual Intercourse of Adolescent Girls.”  Journal of Family Issues, vol. 14, no. 2, 1993, pp. 261-278.

Appendix A: Technical Workgroup for the Evaluation of Title V Section 510 Abstinence Education Programs

Marilyn Benoit, MD (1998–    )
3033 New Mexico Avenue, NW
Apt. 201
Washington, DC  20016
Voice:      (202) 607-3032
Fax:         (202) 363-4621
Email:     bartolom@aol.com

Sarah Brown, M.P.H. (1998–    )
Executive Director
The National Campaign to Prevent Teenage Pregnancy
1700 Massachusetts Avenue, NW
Suite 200
Washington, DC  20036
Voice:      (202) 478-8578 (direct);
                (202) 478-8500 (main #)
Fax:         (202) 478-8588 or 8580
Email:     ssbrown@teenpregnancy.org

James Jaccard, Ph.D. (1998–    )
Professor of Psychology
State University of New York at Albany
Department of Psychology
Social Sciences, 248C
1400 Washington Avenue
Albany, NY  12222-0001
Voice:      (518) 442-4684
Fax:         (518) 452-5947
Email:     jjj20@cnsunix.albany.edu

David Larson, MD (1998–2000)
President
National Institute for Healthcare Research
8110 Executive Boulevard, Suite 908
Bethesda, MD  20852
Voice:      (301) 984-3135 ext. 377
Fax:         (301) 984-8143
Email:    

Joe S. McIlhaney, Jr., M.D. (1998–    )
President
The Medical Institute for Sexual Health
P.O. Box 162306
Austin, TX  78716-2306
Voice:      (512) 328-6268
Fax:         (512) 328-6269
Email:     jmcilhaney@medinstitute.org

Robert Michael, Ph.D. (1998–    )
Dean
The University of Chicago
Harris School of Public Policy Studies
1155 East 60th Street
Chicago, IL  60637
Voice:      (773) 702-9623
Fax:         (773) 702-0926
Email:     r-michael@uchicago.edu

Kristin Moore, Ph.D. (1998–    )
Child Trends, Inc.
4301 Connecticut Avenue, NW
Suite 100
Washington, DC  20008
Voice:      (202) 362-5580
Fax:         (202) 362-5533
Email:     kmoore@childtrends.org

Susan Philliber, Ph.D. (1988–    )
President
Philliber Research Associates
16 Main Street
Accord, NY  12404
Voice:      (845) 626-2126
Fax:         (845) 626-3206
Email:     sphilliber@compuserve.com

Robert Rector, M.A. (1998–    )
The Heritage Foundation
214 Massachusetts Avenue, NE
Washington, DC  20002-4999
Voice:      (202) 608-6213
Fax:         (202) 544-0961
Email:     rectorr@heritage.org

David Rowberry, Ph.D. (1998–    )
LDS Institute
1095 University Road
Las Vegas, NV  89119
Voice:      (702) 736-7045
Fax:         (702) 736-8695
Email:     drowberry@hotmail.com

Freya Sonenstein, Ph.D. (1998–    )
The Urban Institute
2100 M Street, NW
Suite 500
Washington, DC  20037
Voice:      (202) 261-5546
Fax:         (202) 452-1840
Email:     fsonenst@ui.urban.org

Marta Tienda, Ph.D. (1998–2000)
Princeton University
Office of Population Research
21 Prospect Avenue
Princeton, NJ  08544
Voice:      (609) 258-5808
Fax:         (609) 258-1039
Email:     tienda@opr.princeton.edu

John Vessey, Ph.D. (2000–    )
Professor
Department of Psychology
Wheaton College
Wheaton, IL  60187
Phone:    (630) 752-5761
Fax:         (630) 784-9897
Email:     John.T.Vessey@wheaton.edu

Stan Weed, Ph.D. (1998–2000)
Institute for Research and Evaluation
6068 S. Jordan Canal Road
Salt Lake City, UT  84118
Voice:      (801) 966-5644
Fax:         (801) 967-8288
Email:     WeedStan@aol.com

Brian Wilcox, Ph.D. (1998–    )
University of Nebraska, Lincoln
Center on Children, Families and the Law
P.O. Box 880227
121 South 13th Street, Suite 302
Lincoln, NE  68508-1906
Voice:      (402) 472-3479
Fax:         (402) 472-8412
Email:     bwilcox@unl.edu

Endnotes

1.  When classrooms or schools are the unit of randomization, the “effective sample size” is substantially lower than would be if students were the unit of randomization.  This is because of the high within-class or within-school correlation (Kish 1965).

2.  This is an issue that was of sufficient concern during the study design that an external review of the study design was commissioned to ensure that there was strong professional support for the random assignment design adopted for the study.

3.  A methodological experiment was conducted to assess whether using personal data-recording devices increased reporting of sensitive behaviors.  It did not have any such effect for the evaluation sample.