Plans and procedures for the data collection in the impact evaluation are designed to capture the high-quality data needed for a thorough evaluation. A conceptual framework for the program intervention strategies, which is consistent with the main theories of adolescent behavior discussed earlier, dictates the data collected and the timing of those data (Figure 5).
Figure 5. Conceptual Framework for Evaluating Abstinence Education Programs.
This framework acknowledges that the decisions youth make regarding sexual activity and other risk-taking behavior (Column IV) depend critically on a range of antecedent factors (Column I), including demographic and background characteristics of the youth, characteristics of their parents and their families, and the school and community context in which they have been raised. For all youth, these antecedent factors are mediated by current parental attitudes, values, and supports; the attitudes, knowledge and relationships of the youth; and the current school and community context in which youth live (Column III).
There are two means by which the abstinence education programs (or any other intervention) operate to potentially alter the key outcomes of interest. One is by directly altering youth behavior. The other is through affecting the natural mediating factors, for example, by providing parents with knowledge and tools to better guide their children in sound decisionmaking; by changing the attitudes, knowledge and relationships of youth in ways that reduce their inclination to engage in risk-taking behaviors; or by changing the school and community climate in ways that are more expectant and supportive of abstinence.
The first wave of student surveys administered near the time of enrollment in the evaluation study gathers information on the antecedents of teen sexual activity and baseline values of the natural mediating factors (Columns I and III). Wave 2 and wave 3 surveys gather information to mark changes in the natural mediating factors and the key outcomes (Columns III and IV).
A number of critical issues relate to the design and administration of these surveys to support the rigor of the impact study. These include:
- Protecting the privacy of sample members
- Using questions that will generate valid and reliable measures of the constructs of interest
- Dealing with normal reluctance to report sensitive and socially undesirable information
- Addressing the fact that youth may have different definitions of what “abstinence” means
- Ensuring that questions are age appropriate
- Avoiding contamination of the programs’ abstinence message by the data collection itself
The rights and privacy of sample members and their parents are paramount. Only youth whose parents have given active parental consent for their child to participate in the study are included in the study sample. Moreover, youth themselves must actively consent to each wave of data collection. The privacy of student responses is protected through a rigorous system that relies on professional, independent data collectors; that permits no personal identifying information on any survey form or data file containing survey responses; that maintains secure data files; and that has the protection of a Federal Certificate of Confidentiality (HRSA-00-15).
Survey questions were selected with attention to issues of the validity and reliability of the core constructs for the evaluation. Each question included in any of the three surveys has been mapped to one of the core constructs in the conceptual framework (Figure 5 above). Moreover, in determining the particular questions that would be asked to address each construct, careful attention was paid to the experience of prior studies with similar populations, including the validity and reliability of measures for different target populations and when questions were administered through different data collection modes. For example, questions about school and family draw heavily on the National Longitudinal Survey of Youth and the National Education Longitudinal Study of 1988; questions on youth attitudes about sexual activity draw heavily on questions used in prior studies of abstinence education programs, such as Values and Choices (Olsen et al. 1991), Teen Aid (Weed et al. 1998), Responsible Social Values Program (Adamek 1993), Best Friends (Best Friends Foundation 1997), and Sex Respect (Weed and Olsen, no date); questions about other risk-taking behaviors, such as drinking and using drugs, draw heavily on questions in the Youth Risk Behavior Surveillance Survey (Centers for Disease Control and Prevention 1993) and the National Longitudinal Survey of Youth (Card 1993); and questions about romantic relationships and actual sexual experiences draw on the National Longitudinal Survey of Adolescent Health (Udry and Bearman 1998), the National Survey of Family Growth (Card 1993), and the Youth Risk Behavior Surveillance Survey (Centers for Disease Control and Prevention 1993).
Each of the survey questionnaires was pretested with small groups of youth. After revisions, they were then reviewed by key staff in the five programs participating in the impact evaluation, by the Federal Office of Management and Budget, and by the University of Pennsylvania’s Institutional Review Board. In addition, staff from various constituent groups and policy organizations reviewed the survey questionnaires, provided useful insights, and made helpful suggestions.
Youth may not want to report sensitive and socially undesirable information. Some respondents may feel uncomfortable reporting accurate information on questions about sexual intercourse and may distort their responses in the direction that they perceive as socially desirable. Moreover, the problem of underreporting behavior that is considered socially undesirable may be exacerbated for youth who participate in abstinence programs, given the strong and unequivocal message of these programs.
To minimize the underreporting of sensitive behaviors, as well as to protect the privacy of the study sample, the evaluation uses self-administered survey data collection, maintains the strictest standards of confidentiality, and informs the survey respondents about them.3 The data collection procedures ensure that no one from the local schools — including teachers, administrators, and counselors — has access to students’ survey responses. School and program staff are not allowed to participate in the data collection; trained interviewers conduct all survey data collection and focus groups. As soon as the student surveys are completed, the interviewers immediately separate student contact information from the surveys and remove them from the school grounds.
Before the students complete the surveys, the interviewers assure all respondents that their answers will be kept confidential and will not be shared with anyone. The consent forms sent home to parents, as well as the assent forms given to students, make it clear that no individual-level data from the surveys will be reported. Rather, information on individual students will be combined into groups for analysis and reporting purposes.
Survey administration methods protect student privacy
- Most students complete the surveys by themselves in the presence of trained interviewers who can answer questions about the survey administration. Younger sample members and those with poor reading skills have the survey read to them, but they mark their own responses.
- Trained, professional interviewers employed by Mathematica Policy Research, Inc., conduct all survey data collection.
- All surveys are removed from the school premises immediately upon completion.
- No personal identifying information is included on the survey instruments.
- A Federal Certificate of Confidentiality protects the student data.
Youth may have different definitions of abstinence. The primary goal of the Section 510 abstinence education programs is to persuade youth to abstain from sexual activity. Thus, it is very important that survey questions accurately measure this outcome.
Survey questions on abstinence from sex are difficult to design, since abstinence means different things to different people. Some consider abstinence to mean refraining from all intimacy except for kissing and holding hands, while others consider abstinence as anything except sexual intercourse. Participation in abstinence education programs may lead some youth to change their definitions of what constitutes sexual activity and abstinence. Failure to address such program-induced changes in definitions could result in a downward bias in the reporting of abstinence by program youth relative to control youth and thereby limit the detection of true program impacts.
It is essential to ask in the clearest way possible about specific behaviors of greatest interest. To have reliable measures of sexual activity, the evaluation survey instruments measure whether study youth have ever had sexual intercourse. Since program and control youth are likely to have the same understanding, on average, of what sexual intercourse is, this measure has greater reliability than survey questions that ask simply about abstinence from sexual activity.
Outcome measures must be age appropriate. The survey and administration methods for the study are sensitive to the social and emotional development of sample youth. The abstinence education programs target youth in their preadolescent and adolescent years, and measurement of outcomes must reflect that age span. Some programs serve youth as young as third or fourth grade. Measures of program impacts for preadolescent youth may be quite different from those for adolescent youth. For example, questions related to sexual intercourse are not appropriate for preadolescent youth, given the low prevalence of the behavior and, more importantly, the age inappropriateness.
The evaluation survey instruments for youth below grade seven do not ask whether the respondent has had sexual intercourse.
The survey was designed to avoid contamination of the abstinence message. The Section 510 abstinence education programs promote a strong message that teenagers should postpone sexual activity until marriage. The programs do not promote use of contraception, on the premise that such information is inconsistent with program goals and sends a mixed message to youth.
However, a careful evaluation must measure the main outcomes of an abstinence education program. The evaluation must be able to measure whether program participants do or do not abstain from sex and whether program participants do or do not engage in behaviors that risk pregnancy and exposure to STDs. This requires that the survey questions about sex measure similar behaviors for the program and control youth and be detailed enough to measure exposure to risks of pregnancy and STDs. Moreover, it is critical that the study’s informed consent procedures are consistent with asking youth these sensitive questions.
Accurate assessment of whether programs affect risk of STDs and pregnancy must take into account the behaviors of those youth who become sexually active. Among youth who are sexually active, exposure to unwanted pregnancy and STDs depends, among other factors, on the use of condoms or other contraceptives. Therefore, the evaluation survey instruments ask a limited number of questions about use of condoms and other contraceptives. These questions are seen by and are asked only of youth who have already stated that they have had sex, and they are designed so that they do not provide information that the abstinence programs themselves avoid communicating.