Evaluating Two Welfare-to-Work Program Approaches: Two-Year Findings on the Labor Force Attachment and Human Capital Development Programs in Three Sites. The Random Assignment Process and Resulting Research Groups

12/01/1997

As noted in Chapter 1, a fundamental question of the National Evaluation of Welfare-to-Work Strategies is to determine the relative effectiveness of two different program approaches for promoting self-sufficiency. Many evaluations utilize cross-site comparisons of alternative program designs, but must overcome the difficulty of isolating the effects of these approaches from other factors, such as local economic conditions and welfare grant levels. To avoid these difficulties, this evaluation took an innovative approach to comparing program strategies. In the three sites examined in this report (Atlanta, Grand Rapids, and Riverside), a three-way random assignment design was used and two different types of welfare-to-work JOBS programs were operated side by side in each site.(6)

In each of the three sites, JOBS orientation attenders were randomly assigned to one of three groups: a Labor Force Attachment (LFA) group, a Human Capital Development (HCD) group, or a control group. Control group members were free to seek out, on their own initiative, training and education programs available in their communities. In addition, since the Family Support Act created a guarantee that child care would be available to welfare recipients participating in JOBS-approved activities, a decision was made by HHS early in the evaluation that control group members, as long as they were participating in an approved activity, should be eligible for this assistance. Finally, in Grand Rapids, control group members in approved activities were eligible for transportation assistance as well.

Using the three-way random assignment design, three sets of comparisons can be made in each site. First, comparisons can be made between outcomes for individuals assigned to each of the program groups and outcomes for those assigned to the control group (LFA versus control; HCD versus control), enabling one to estimate the added benefit of either of these approaches above what the individuals would achieve in the absence of a welfare-to-work program. Additionally, a direct comparison can be made between outcomes for participants in the two program groups (LFA versus HCD), to assess the relative effectiveness of each of these approaches. Thus, impacts (for example, on participation in employment-related activities or on employment, earnings, or welfare receipt) and net costs presented for each of the two program groups represent the difference between outcomes for control group members, that is, what people would do without a welfare-to-work program, and the outcomes for those assigned to each of the two program approaches. Similarly, impacts and net costs presented in the last chapter of the report on a direct comparison of the LFA and HCD approaches represent the added benefit of one approach vis-à-vis the other.

In Atlanta and Grand Rapids, JOBS orientation attenders were equally likely to have been assigned to one of the two program groups or to the control group, as shown on the left side of Figure 2.3. Riverside, however, had pre-existing program regulations governing participation in adult basic education, following regulations in California's Greater Avenues for Independence (GAIN) program, the state's JOBS program. As a result, there were, in effect, two different random assignment evaluations in Riverside. Prior to the National Evaluation of Welfare-to-Work Strategies, GAIN program regulations dictated that only individuals determined to be in need of basic education would be assigned to educational activities as a first step toward self-sufficiency. Thus, all JOBS enrollees were evaluated at orientation to determine whether, according to program regulations, they required basic education: Those who had a high school diploma or GED, or scored 215 or above on both the math and the literacy sections of the GAIN Appraisal test,(7) and were proficient in English were determined not to need basic education. As seen on the right side of the Riverside part of Figure 2.3, this group could be randomly assigned only to the LFA or control group. Those without a high school diploma or GED, who scored below 215 on either section of the GAIN Appraisal test, or who required English remediation were determined by the program to be in need of basic education and, according to program regulations, were eligible for assignment to an education activity. As a result, individuals with these characteristics were eligible to be randomly assigned to any of the three evaluation research groups, including the HCD group.

Figure 2.3: Schematic depiction of random assignment model in three sites.

The situation in Riverside has several implications for the research group comparisons made in this report. First, since only those without a high school diploma or with low reading and math skills were eligible for random assignment to the HCD group in Riverside, any comparisons between the LFA and the HCD groups in Riverside must include only those individuals determined to be in need of basic education as of random assignment. In other words, when the effects, in Riverside, of the LFA approach vis-à-vis the HCD approach are examined, individuals in the LFA group who are not in need of basic education must be dropped from the analysis. Second, Riverside's design also affects the comparability of the HCD research groups across the three evaluation sites. Compared with the HCDs in Atlanta and Grand Rapids, HCDs in Riverside have lower education levels than those in other sites.

In order to present information that can be accurately and easily used to make within-site LFA-HCD comparisons in Riverside and to make cross-site HCD comparisons, subgroup (as well as full-sample) participation, cost, and impact estimates are presented throughout the report. The subgroup estimates always divide the full LFA and HCD samples in each site into those determined to be not in need and in need of basic education in Riverside and into those with and without a high school diploma or GED in Atlanta and Grand Rapids. Those determined to be not in need of basic education in Riverside and those with high school diplomas or GEDs in the other two sites appear under the high school diploma/GED subgroup heading throughout the report; those determined to be in need of basic education in Riverside and those without high school diplomas or GEDs in the other two sites appear under the no high school diploma/GED subgroup heading.

While those determined to be in need of basic education in Riverside appear under the no high school diploma/GED heading, scores on the GAIN Appraisal test are also taken into account in determining if an AFDC recipient in California is in need of basic education, in line with GAIN regulations. As a result, 23 percent of the HCDs in Riverside who appear under the no high school diploma/GED heading actually did have such a credential but scored low on either the reading or math portion of the CASAS test.(8)