Impacts of Four Title V, Section 510 Abstinence Education Programs

Design and Methods for the Final Impact Evaluation

[ Main Page of Report | Contents of Report ]

Contents

• Impact Study Design
  • Sample Intake and Random Assignment
  • Sample and Data Collection
  • Sample Characteristics
• Outcome Variables
• Analytic Methods
  • Multivariate Estimation
  • Missing Outcomes Data
  • Nonparticipation and Crossover
  • Statistical Power
  • Hypothesis and Sensitivity Testing

Central to the evaluation of Title V, Section 510 abstinence education programs is a rigorous analysis of the programs' impact on teen sexual abstinence and teen sexual activity. To this end, the impact analysis for the evaluation relies on an experimental design. Under the experimental design, youth in the study sample are assigned to either a program group that receives the services provided by a selected group of Title V, Section 510 programs or a control group that receives only the usual services available in the absence of these programs. When coupled with sufficiently large sample sizes, longitudinal surveys conducted by independent data collectors, and appropriate statistical methods, this design is able to produce highly credible estimates of the impacts of the focal programs.

[ Go to Contents ]

Impact Study Design

Obtaining an unbiased impact estimate requires determining what the outcomes of program group youth would be in the absence of that program (known as the counterfactual). Because the counterfactual cannot be observed directly, it must be estimated. Randomly assigning members of the study sample to either a program group or a control group is considered the most valid approach for estimating the counterfactual. Because of random assignment, program and control groups are similar in all respects except their access to the program services, making the control group a highly credible counterfactual. As a result, unbiased estimates of program impacts can be generated by simply comparing mean values of outcome measures (such as sexual abstinence) for the program group with those for the control group. The evaluation of Title V, Section 510 program uses this type of experimental design.

Sample Intake and Random Assignment

Sample intake took place near the beginning of three school years-1999-2000, 2000-2001, and 2001-2002. In each of these years, either program or school staff identified those youth who were eligible to participate in the programs. In the two non-elective programs — My Choice, My Future! and Teens in Control — eligible youth included all those in the targeted grade level (8th and 5th grades, respectively). In the elective programs — ReCapturing the Vision and FUPTP — eligible youth included all those in the targeted grade levels who had been identified by school or program staff as potential candidates. Once identified, eligible students were given a study consent form that notified parents of their child's eligibility for the program, explained the program and the evaluation, and described how selection for the program would take place through a lottery (random assignment).

In order for a student to be eligible both for the lottery and to participate in the evaluation, parents had to provide signed, active consent. In the two non-elective programs, parent consent rates were high, in excess of 90 percent (youth with parents not agreeing to participate were automatically placed in the classes that control group youth were to take). In the two other programs, parental consent rates could not be estimated because of their elective nature; however, there were few known cases of parents who wanted their child to participate in the program but opted out of the lottery because of the requirement to participate in the study.

In order to conduct the random assignment, lists of the eligible students who had active parental consent were sent to the evaluation team near the beginning of each school year and a random number generator was used to order the applicant pool. Once programs informed the evaluation team of the number of program slots available, the evaluation team released the names of students with that rank order or less in the assignment hierarchy. For example, in a program with 200 applicants and a capacity to serve 100 youth, the evaluation team released the names of the first 100 youth in the randomly ordered list of eligible applicants for inclusion in the program group. All students not selected for the program group formed an ordered "waitlist" and control group. In cases where it was necessary to maintain a minimum program enrollment, program vacancies were filled by releasing youth on the waitlist in the order of their original random number. Along with the original set of youth selected for the programs, all students selected from the waitlist to fill program vacancies became members of the study's program group regardless of whether they actually participated. All remaining students on the ordered waitlist formed the control group for the study.

In some instances, lists of eligible students were sent to the evaluation team in batches, leading to multiple rounds of random assignment within a school year. These multiple rounds of assignment, coupled with sample enrollment taking place over multiple school years, led to modest variation in the likelihood of students being selected for the program or control group. This variation in the selection probability was addressed in the analysis by using sample weights, as described below.

Sample and Data Collection

The resulting study sample includes 2,501 youth, enrolled over a three-year period from fall 1999 through fall 2001 (Table III.1; top panel). Within each program site, sample sizes ranged from 504 for FUPTP to 849 for Teens in Control. Just less than 60 percent of the study sample was assigned to the program group (1,461); the remainder was assigned to the control group (1,040).

Data were collected from the study sample through a series of four surveys.⁽¹⁾ They included a baseline survey, administered near the time that youth began participating in the study, and three follow-up surveys. The surveys were administered either in school using a pen-and-paper instrument or by phone.

The impact findings presented in this report are based on data collected from the final follow-up survey, which was administered to study youth between spring 2005 and winter 2006. This reflects a follow-up period of roughly 42 to 78 months after youth began participating in the study, depending on the year in which they began participating and the exact timing of the survey. The response rate on this survey ranged from 80 to 84 percent across the four study sites, leading to an 82 percent rate overall (Table III.1, middle panel).

In each site, the sample size available for this report is given by the product of the number of youth in the study sample (upper panel of Table III.1) and their corresponding response rate on the final follow-up survey (middle panel of Table III.1). As seen in the lower panel of Table III.1, the resulting sample size for this report ranges from 414 youth for FUPTP to 715 for Teens in Control. The total sample size across the four sites totals 2,057 youth.

Evidence suggests that the program and control groups are well matched, as would be expected given an experimental design. Across a wide range of baseline measures, only a minimal number of differences between the program and control groups were statistically significant-no more than expected by random chance. For example, of over 40 measures based on baseline data (see Appendix Table A.1), no more than seven in each site show a statistically significant difference between the program group and control group.

Sample Characteristics

The sample characteristics of youth in each site reflected both the targeting of the programs and the communities in which youth lived (Table III.2). In the two program sites serving middle schoolers, My Choice, My Future! and ReCapturing the Vision, sample youth averaged over 18 years of age by the time of the final follow-up survey. This is considerably older than the two program sites serving upper elementary school youth, FUPTP and Teens in Control, in which sample youth averaged only 15.5 years of age. While the ReCapturing the Vision sample included only girls, reflecting the targeting of the program, the gender mix in the other three sites was fairly close to even. The race/ethnicity of youth in the study samples largely reflected their communities' composition. More than 80 percent of the youth in the My Choice, My Future! sample were white, non-Hispanic, while high proportions of youth in the other three sites were African American or Hispanic.

Youth in the study sample come from backgrounds that put them at relatively high risk of having sexual intercourse at an early age. With the exception of My Choice, My Future!, one third or fewer of the sample youth in each site reported at baseline having parents who were married. They also reported relatively high rates of life stressors, such as sisters getting pregnant or siblings dropping out of school. Moreover, although almost all youth reported that they had a mother figure (95 percent), only four out of every five youth in the Recapturing the Vision and FUPTP samples reported having a father figure.

[ Go to Contents ]

Outcome Variables

All outcome measures were based on data from the final follow-up survey. These measures fall into two categories (Table III.3):

1. Measures of Risk Behavior and Behavioral Consequences. These include the measures that are most central to the evaluation, including whether youth remained abstinent, expected to abstain in the future, and engaged in unprotected sex (Table III.3; top panel). They also include measures of potential consequences of teen sexual activity, such as pregnancy or reported STDs, and important behavioral correlates of teen sexual activity, such as alcohol and drug use.
2. Measures of Potential Mediators (Knowledge and Perceptions). Several potentially important mediators of teen sexual behavior, shown in the lower panel of Table III.3, were not available until the final follow-up survey. Therefore, along with the measures of risk behavior and behavioral consequences, they are also a main focus of this final impacts report. This new set of measures spans two broad categories: knowledge and perceptions. Measures of knowledge include the ability of youth to identify STDs, their understanding of potential risks of unprotected sex, and their knowledge of potential health consequences of STDs. Measures of perceptions focus on whether youth believe condoms or birth control pills are effective for preventing pregnancy and STDs.

[ Go to Contents ]

Analytic Methods

For each outcome measure, program impacts were estimated as the difference in regression adjusted mean values between the program and control groups. These impacts were estimated both overall and for each site individually. The overall estimate was obtained simply by averaging the estimated impacts for each of the four individual sites. This approach was preferred to weighting each site according to the size of its sample, which would have arbitrarily given some sites (most notably Teens in Control) more importance when computing a pooled estimate.

Multivariate Estimation

The regression analysis used weighted least squares models and pooled data across all four sites. Each regression model included a series of binary variables reflecting the interaction between program site and program status (program or control group). The site-specific estimate is obtained from the regression simply from the difference between the binary variables corresponding to that site's program and control groups. The pooled impact estimate for a given outcome is obtained from the average of these four program-control differences. The weights used in the regressions accounted for the variability in the probability of selection to the program or control groups as well as for youth who did not complete the final follow-up survey.⁽²⁾ Standard errors from the models were calculated taking into account the variability associated with these weights.

In addition to these variables, the regression models included a large number of variables to control for individual demographic and background characteristics measured from the baseline survey (Table III.4). For the small fraction of the sample who did not complete a baseline survey (fewer than 5 percent), a supplemental survey was administered at the next survey to collect key demographic information such as age, gender, and race/ethnicity. For other covariates, missing data were imputed using the mean for the sample in a given program site.

Along with site-level results, the report presents estimated impacts on behavioral outcomes for several subgroups of potential interest.⁽³⁾ Among these are subgroups defined by gender and several measures that might be linked to eventual behavior, such as baseline support for abstinence, religiosity, marital status of parents, and television viewing. All of these subgroups were defined from survey data collected at baseline, prior to any potential influence of the programs. A final subgroup, enrollment cohort, is also investigated because of important variation found across cohorts in an earlier DHHS study report (Maynard et al. 2005). The first of these subgroups includes youth enrolled in the 1999-2000 or the 2000-2001 cohorts; the second includes youth enrolled in the final, 2001-2002 cohort.

Impacts were estimated for one subgroup at time, following nearly the same methods as described above for the full sample. The only difference with these methods is that explanatory terms were added to the regression models reflecting the interaction between a given subgroup of interest (for example, gender) and each of the site dummies and the "site by treatment" interaction terms. Estimates for a given subgroup were then computed using the coefficients on these terms, following the same procedure described above.

Missing Outcomes Data

Although non-response on the individual survey questions was generally very low, typically just one or two percent, for certain outcomes it could still result in slightly biased estimates of outcome measures if left unaddressed. The first set of these questions pertain to knowledge questions — for example, "can you get pregnant if you have sexual intercourse only once?" — where there is a single correct answer. For these questions, it is likely that youth who completed most of the survey section on knowledge, but skipped an individual question or two, did so because they did not know the correct answer. Thus, in order not to understate the proportion of youth who were unsure of a correct answer, the response on individually-skipped knowledge questions was categorized as "don't know/unsure." In contrast, youth who skipped an entire section are excluded from the analysis for that set of outcomes.

A more serious form of missing data pertains to conditional questions, meaning that they are answered by youth only if they provide a particular response on a prior question or questions. For example, in order to answer the question on the number of sexual partners, the respondent must first indicate on the survey that s/he has had sexual intercourse. Since youth who have not had sexual intercourse can correctly be assigned a value of zero partners, this conditional wording means that all missing values for the question will pertain to youth who have had sexual intercourse. In turn, unless there are no missing data, the reported mean value for the full sample will be incorrect — in this case understating the mean number of sexual partners. To correct for this conditional item non-response, missing values were imputed following a commonly used "hotdeck procedure." This procedure assigns a value on the item that was missed based on the reported values of youth with characteristics similar to those of the item non-respondents. Through this method, the estimates for the program and control groups preserve the natural variability of the sample.

Nonparticipation and Crossover

As noted in Chapter II, a sizeable proportion of youth assigned to the program group in the two sites with elective programs, ReCapturing the Vision and FUPTP, did not participate in any program classes or other services (35 percent and 43 percent, respectively). To address this program nonparticipation, impact estimates are presented two ways in the report. The first is for the full program group. This estimate reflects the average effect of having the opportunity to participate in the program, whether or not the youth actually chose to participate. These estimates are featured throughout the report since it generalizes to the youth who were made eligible for the programs. The second is for only those youth in the program group who actually participated. These estimates are derived following the procedure developed by Bloom (1984), which divides the full-sample estimate by the participation rate. Because the standard errors and significance levels associated with the participant-only estimates are roughly similar to those for the full program group, impact estimates found not statistically significant for the full program group are typically not statistically significant for the participants either. As a result, the conclusions from the study do not differ substantively when based on one set of measures or the other.

Crossover of control group youth into the program group was rare, including at most 5 percent of the sample. For this reason, the report does not present estimates that account for crossover. To the extent that youth who did cross over experienced positive benefits from participating in the programs, the impact estimates reported are understated slightly.

Statistical Power

For the full sample, the statistical power of the study to detect impacts is high. Based on the observed explanatory power of the regression models, the study sample supports detection of true overall program impacts of roughly 0.08 standard deviations. (This is based on standard assumptions of 80 percent statistical power and 90 percent statistical confidence, two-tailed.) For a proportional outcome with a mean of 50 percent, this reflects an estimated impact of roughly 4 percentage points. Program impacts that are smaller in size may also be detected from the study sample, but the likelihood of doing so is below the 80 percent probability (power level) that is commonly preferred.

For the individual program sites, statistical power is naturally lower. This is particularly true in the two sites that experienced program nonparticipation, ReCapturing the Vision and FUPTP. For example, in the absence of nonparticipation, the size and allocation of the study sample would support detection of true site-specific impacts on the order of 0.16 standard deviations or larger for ReCapturing the Vision and 0.18 standard deviations for FUPTP. However, in light of the existing nonparticipation, the impacts on participants would need to be considerably larger — about 0.25 standard deviations for ReCapturing the Vision and 0.32 standard deviations for FUPTP — given equivalent levels of statistical power and confidence. This means the available samples in these two sites provide a high likelihood of detecting (that is, stating as statistically significant) true participant impacts only if they are fairly large; for example, for a proportional outcome with a mean of 50 percent, the minimum detectable impacts for participants are about 13 and 16 percentage points in the two respective sites. For the remaining two sites, My Choice, My Future! and Teens in Control, detectable impacts (at 80 percent power) are better — roughly 0.17 and 0.13 standard deviations, respectively.

Hypothesis and Sensitivity Testing

For each impact estimate, a two tailed t statistic tests the null hypothesis that there is no difference between the regression adjusted means for the program and control groups. The associated p value, which reflects the probability of obtaining the observed impact estimate when the null hypothesis of no effect is true, is used to judge the likelihood that a program had a measurable (statistically significant) impact. For categorical outcome variables, a t-test is conducted on the mean (proportion) for each response. In addition, an F-statistic tests the null hypothesis that there is no difference between the distributions of responses for the two experimental groups. This statistic is computed from a site-specific multinomial logistic regression of the categorical outcome variable on an indicator for program status and the covariates listed in Table III.4. The findings based on the F-statistics are consistent with those based on the individual t-test statistics.

Impact estimates with p-values less than 0.10, on two-tailed tests, are denoted in the report by asterisks and referred to in the text as statistically significant (Table III.5). While researchers sometimes use a lower p-value, 0.05 or less, to determine significance, this higher threshold allows a careful assessment of the findings across the range of outcomes being examined. The adoption of this threshold, however, does raise the likelihood of detecting significant impacts that have resulted merely by chance. Therefore, when interpreting the findings, attention is paid to whether significant impact estimates are isolated or whether they are part of a pattern of significant estimates that would point more strongly to a true program effect.

Additional analyses were conducted to examine the robustness of the impact estimates presented in the report. These included estimating impacts through logistic regression models (for binary outcomes) rather than linear probability models, and estimating impacts dropping various combinations of regression adjustment, data imputation, and sample weights. Across all these alternative estimates, findings were consistent with those presented in the report.

[ Go to Contents ]

Endnotes

1. Copies of these surveys are available online at [http://www.mathematica-mpr.com].

2. Selection weights were calculated as the inverse probability of selection to the group of assignment. Non-response weights were calculated using standard modeling techniques to estimate the probability of survey non-response as a function of baseline covariates.

3. Subgroups defined by race/ethnicity could not be investigated because of the very high correlation between program site and a given racial/ethnic group.

Where to?

Top of Page | Contents
Main Page of Report | Contents of Report

Home Pages:
Human Services Policy (HSP)
Assistant Secretary for Planning and Evaluation (ASPE)
U.S. Department of Health and Human Services (HHS)