How Effective Are Different Welfare-to-Work Approaches? Five-Year Adult and Child Impacts for Eleven Programs. Control Group Treatment Over Time

12/01/2001

Another development that has direct consequences for assessing the impacts of the NEWWS programs studied concerns changes in the treatment of control group members over time. As discussed in Chapter 1, in five sites some control group members who were still receiving welfare after year 3 became eligible for and were required to participate in welfare-to-work program services. In other words, in these sites the control embargo was no longer in effect in year 4 and/or year 5. (In Riverside and Portland, the control embargo was in force for the entire five-year follow-up period for the control group samples analyzed in this report.) This section discusses the treatment of controls in years 4 and 5 of the follow-up period in detail, to assess the extent to which the early lifting of the control embargo in five sites affected the impact estimates in those sites, if it affected them at all.

Figure 2.4 presents a time line of control group members' eligibility for welfare-to-work program services in each site, by quarter of random assignment. In Atlanta, Columbus, and Oklahoma City, there was a fixed date on which the control group embargo on welfare-to-work program services was lifted. In these three sites, then, controls randomly assigned early in the sample intake period would have had little opportunity for exposure to the programs under study while those randomly assigned later in this period would have had more opportunity. In Grand Rapids, as noted above, controls randomly assigned in 1993 retained their embargo on welfare-to-work program services for a full five years; those randomly assigned prior to 1993 would have had their embargo lifted when they reached the end of their particular three-year follow-up period. In this site, as in Atlanta, Columbus, and Oklahoma City, control group members thus would have had differing amounts of time during the five-year follow-up in which to possibly be exposed to program services. In Detroit, the control group embargo on welfare-to-work program services was lifted when controls reached the end of their three-year follow-up period. Consequently, in this site, all controls would have had an equal amount of time during the full follow-up period for possible exposure to the programs under study.(27) All control group members in Riverside and a randomly selected group of control group members in Portland had a five-year embargo on welfare-to-work program services. (For ease of presentation, the randomly selected group of Portland controls rather than all control group members in the site is shown in Figure 2.4.)

Firgure 2.4
Time Line of Changes in Control Group Members' Eligibility for Welfare-to-Work Program Services over the Five-Year Follow-Up Period, by Quarter of Random Assignment and Site

Firgure 2.4 Time Line of Changes in Control Group Members' Eligibility for Welfare-to-Work Program Services over the Five-Year Follow-Up Period, by Quarter of Random Assignment and Site

Firgure 2.4 Time Line of Changes in Control Group Members' Eligibility for Welfare-to-Work Program Services over the Five-Year Follow-Up Period, by Quarter of Random Assignment and Site

Key:  Random assignment quarter Not eligible for welfare-to-work program services Eligible for welfare-to-work program services

NOTE:  Control group members shown in Portland are the randomly selected group of 499 for whom a full five-year embargo on welfare-to-work program services was in effect.

The extent to which control group members actually received welfare-to-work program services once their embargo ended is not known. However, for several reasons it is likely that relatively few control group members did so. First, only control group members still receiving welfare when their embargo on welfare-to-work program services was lifted in follow-up year 4 or 5 would have been informed of and required to participate in such services. (Once the control embargo was lifted, controls would have been considered to be mandatory for participation in a welfare-to-work program, as long as they did not meet any of the exemption criteria established under the Family Support Act.) The following proportions of all control group members in each site had four years of "no treatment" follow-up, that is, years with no months in which they were receiving welfare and were eligible for welfare-to-work program services: 86 percent in Atlanta, 67 percent in Grand Rapids, 91 percent in Columbus, and 37 percent in Detroit. Similarly, the following percentages of controls had five years of follow-up with no months in which they were receiving welfare and were eligible for welfare-to-work program services: 54 percent in Atlanta, 66 percent in Grand Rapids, 76 percent in Columbus, and 36 percent in Detroit. Additionally, for the samples used in this report, 100 percent of controls in Riverside and Portland had five years of follow-up with no months in which they were receiving welfare and were eligible for welfare-to-work program services. (Some of the control group members receiving welfare when the embargo was lifted may no longer have been considered mandatory for welfare-to-work program participation, as all would have been as of random assignment, and probably would not have been told about the program or a requirement to participate in it because of changes since study entry in their personal situations, such as illness or disability, employment of more than 20 hours per week, or the birth of child.)(28)

Second, welfare-to-work programs in most sites had a phase-in schedule for assigning people to the programs if they were required to participate in them. It is likely that control group members who were required to participate in the programs would have waited up to six months before being assigned to a program.

Finally, once assigned to a program, control group members would have needed to attend a program orientation, be assigned to a specific program activity, and participate in that activity in order to actually receive welfare-to-work program services. Typically, less than one-half of those assigned to welfare-to-work programs as mandatory participants end up actually participating in welfare-to-work program services.(29)

It should be kept in mind, however, that any encounters with welfare-to-work program staff would have represented control group members' first exposure to a mandatory welfare-to-work program after random assignment. These encounters and possible participation in welfare-to-work program activities may have given control group members an unanticipated boost  to look for work, to pursue education or training, or to change other behaviors. While this boost would have occurred for program group members shortly after random assignment, and their exposure to program staff and participation in program activities would have continued for several years, the possible effects of this boost for control group members toward the end of the follow-up period cannot be ignored.

As discussed more extensively in Chapter 3, available data suggest that the proportion of control group members who actually received welfare-to-work program services in follow-up years 4 and 5 is likely to have been low. According to five-year survey data, which are available for four of the seven NEWWS sites, less than 6 percent of control group members subject to a five-year embargo received any welfare payments during the fifth year after random assignment and reported participating, at some point during the same year, in activities usually uniquely provided by welfare-to-work programs (job search workshops or work experience).(30) In comparison, participation rates measured in the same way were only slightly higher (between 7 and 12 percent, depending on the site) for control group members with a shorter embargo on program services.(31) Measured in a slightly different way, as the proportion of controls who ever received welfare in a "non-embargoed" month and participated in a job club or work experience activity in the same year, at most 15 percent of the Atlanta controls and 7 percent of Grand Rapids controls were likely to have received welfare-to-work program services during the five-year follow-up period.

While the level of participation of the control groups in welfare-to-work program services appears to be low, the behavior of control group members might have been affected by contact with program staff and by the messages they received about the advantages of working as opposed to receiving welfare.(32) In Portland, where a random sample of 499 control group members had an extension of the welfare-to-work program services embargo, it is possible to directly compare the employment and welfare behavior of control group members who had a full five-year embargo on welfare-to-work program services with those who had only a three-year embargo. As discussed more fully in Chapters 4 and 5, there were some differences in the behavior of these two groups. In follow-up year 3, employment levels were almost identical for the two Portland control groups. In years 4 and 5, employment increased much faster for Portland control group members whose embargo ended at the close of year 3 than for those whose embargo lasted the full five years. The employment rates of the former group were 6 percentage points higher in year 4 and 3 percentage points higher in year 5 than those of the latter group. Similarly, average earnings of the former group were $500 higher per year in these two follow-up years. Average amounts of welfare payments received by the two groups of controls in follow-up years 4 and 5 differed by year in direction and by a small amount ($57 less for the longer-term embargo group in year 4 and $186 more in year 5).

The Portland control group comparisons suggest that in this site the lifting of the embargo on welfare-to-work program services after year 3 did affect the behavior of some control group members. For this reason, the randomly selected 499 control group members, who had a five-year embargo on welfare-to-work program services, are used as the control group in all Portland analyses throughout the report.(33)

Unfortunately, a randomly selected alternative control group, similar to the one in the Portland site, does not exist in the other five sites where some control group members were eligible for welfare-to-work program services earlier than the end of the five-year follow-up period. In four of the sites (Atlanta, Grand Rapids, Columbus, and Oklahoma City) certain cohorts of control group members, that is, individuals who were randomly assigned in certain months, had much longer control service embargoes than did other cohorts. (See Figure 2.4.) In these four sites, there is no easy way to determine what effect the lifting of the embargo had on the behavior of controls toward the end of the five-year follow-up period, since the baseline demographics of sample members often differed between cohorts and the employment and welfare behavior observed for the various cohorts was often different even before the control embargo was lifted. In the fifth site, Detroit, all cohorts of controls had an equally long welfare-to-work program service embargo.

Program impacts on employment and earnings and other outcomes in the last two years of follow-up in a few of the five sites probably would have been somewhat larger had some control group members not been exposed to welfare-to-work programs. Impacts would likely have been affected more in follow-up year 5 than in year 4. However, as discussed in Chapter 4 of this report, many programs can continue to have effects long after control embargoes are lifted. Owing to the program group's early exposure to the programs, early gains in employment and earnings can continue in the later years of follow-up, reflecting a "head start" experienced by program group members.(34) This factor, combined with the findings of low year 5 control group welfare receipt and low year 5 control group use of program services, strongly suggests that ending the control group embargo earlier than the end of the five-year follow-up period did not change the impact findings very much. As a result, in this report all control group members in these five sites are included in the estimates of program impacts. Where appropriate, however, impact analyses for follow-up years 1 to 3 are separated from those for years 4 and 5.

Notably, the control group situations described above do not affect the assessments in this report of the relative merits of the Labor Force Attachment and Human Capital Development approaches in welfare-to-work programs. Owing to the research design in the three-way sites, the fact that the control group embargo ended after year 3 in several of these sites does not affect the estimates of the relative effectiveness of the LFA and HCD approaches over five years. Random assignment ensured that the background characteristics of LFA and HCD program group members did not differ systematically at the time of random assignment, which means that the outcomes for the two program groups can be compared directly with one another without taking the control group into account (Figures 2.1 and 2.2).(35)

Endnotes

1.  As will be discussed in more detail later in this chapter, in some sites control group members became eligible for program services before the end of the five-year follow-up period.

2.  The following hypothetical example of a side-by-side evaluation of two program approaches illustrates these points. Control group members earned a total of $40,000 on average over five years, compared with $40,000 for program group 1 and $35, 000 for program group 2. Direct comparisons of earnings for the two program groups suggest that the first program was relatively more effective than the second, because its members earned $5,000 more on average over five years. However, comparisons with the control group show that neither program was effective because neither raised average earnings above the control group level.

3.  The Riverside design has implications for calculating the LFA program impacts. Whereas the outcomes for sample members in the other six sites are unweighted, in Riverside the outcomes are weighted averages of the outcomes for LFA group members found to need or not to need basic education at random assignment. This weighting scheme compensates for the overrepresentation of those determined not to need basic education in the LFA and control groups.

Owing to the Riverside program design, impacts cannot be correctly calculated in an unweighted regression model (that is, one that includes all the sample members in Riverside and gives all observations equal weight). Instead, the LFA impact is calculated as (Wneed * BLFAneed) + (Wnot * BLFAnot). In this equation, BLFAneed represents the impact for the "in-need" LFA group members and BLFAnot the impact for "not-in-need" LFA group members. Wneed, the weight for the in-need sample, equals the fraction of LFA group members, HCD group members, and control group members who were classified by program staff to be in need of basic education at random assignment, and Wnot, the weight for the not-in-need sample, equals 1 - Wneed.

The Riverside LFA impacts were generated using a regression model that included all Riverside sample members, whereas the Riverside HCD impacts were estimated using a regression model that included only LFA, HCD, and control group members determined to need basic education.

For many outcome measures, the report presents the range of control group averages across the seven sites. For Riverside, the average for the entire control group will be included in the range, and not the separate average for control group members in need of basic education that is used to estimate the impacts of the HCD program.

4.  Nearly one-quarter of the people in the Riverside in-need subgroup actually had a high school diploma or GED. These people were determined to be in need of basic education because they scored low on the math or reading portion of the appraisal test or were judged by program staff to need English remediation. See also Hamilton et al., 1997.

5.  See Hamilton and Brock, 1994, for a more detailed description of the research designs in the seven sites.

6.  For a discussion of enrollment practices in the sites, see Chapter 1. See also Hamilton and Brock, 1994, pp. 51-55.

7.  See Hamilton and Brock, 1994, for a discussion of the implications of orientation attendance. A separate experimental analysis of the deterrence effects of a participation mandate and reasons for nonattendance was conducted in Riverside and Grand Rapids for the NEWWS Evaluation. For this study people who attended a meeting at income maintenance to determine their eligibility for welfare benefits were randomly assigned when income maintenance workers determined they were subject to the participation mandate. They entered either a "pre-orientation program group" and were assigned to attend a program orientation  or a "pre-orientation control group" and were not assigned. Members of the pre-orientation program group who showed up for their orientation during the sample intake period for this study were randomly assigned a second time  to either a program or control group. Only those who were randomly assigned to a program or control group at program orientation in Riverside and Grand Rapids are included in the analyses presented in this report. See Knab et al., 2001, for estimates of the deterrence effects of assignment to a mandatory welfare-to-work program.

8.  Brock and Harknett, 1998; Scrivener and Walter, 2001; and Knab et al., 2001.

9.  Although Oklahoma City included nonapplicants in its participation mandate, recipients were not included in the evaluation because including them would have required significant alterations to existing welfare department procedures.

10.  Storto et al., 2000.

11.  Friedlander, 1988.

12.  The sample includes only the 499 control group members in Portland who had a full five-year embargo on the receipt of program services (more information on the control group embargo is included at the end of this chapter). Also, the sample includes only sample members in Atlanta who were randomly assigned between January 1992 and June 1993, excluding those randomly assigned after June 1993.

13.  Approximately 15,000 more people were randomly assigned than are in the full impact sample. Excluded from this report's analysis are people randomly assigned before they attended a program orientation as part of the deterrence study, two-parent (AFDC-UP) families, and teen parents in Riverside (who faced different program requirements than older sample members).

14.  See Freedman et al., 2000a.

15.  The Two-Year Client Survey was conducted in all seven NEWWS Evaluation sites and included 9,675 respondents. For more information, see Freedman et al., 2000a.

16.  For the two-year results of the former, see McGroder et al., 2000; for the results of the latter, see Bos et al., 2001.

17.  For specific response rates by site and research group, see Appendix Table G.1.

18.  Mothers and focal children in 2,594 families responded to the Five-Year Client Survey. A total of 262 of these families were later dropped from the analysis sample. Of these, 203 families had moved out of the survey area by the time of the five-year survey and therefore were not administered the special in-person COS survey sections (a phone interview was conducted to obtain information for the sections of the survey that were administered to all five-year survey sample members; these sample members remain in the five-year survey sample). Fifty-seven families were dropped because the focal child was not the mother's biological child; one duplicate case was dropped; and one family was dropped because the focal child was deceased at the five-year follow-up point.

19.  For specific response rates by site and research group, see Appendix Table G.1.

20.  A total of 1,489 teachers responded to the teacher survey. Seventeen teacher respondents were dropped from the final analysis sample because they taught focal children who were among the 262 respondents dropped from the COS sample.

21.  For specific response rates by site and research group, see Appendix Table G.1.

22.  As shown in Table 2.3, single fathers, or the husbands of disabled spouses, make up from 3 to 11 percent of the full impact sample, depending on site. Female pronouns will be used hereafter to describe sample members because most of them are women.

23.  Of those who did not take the tests, about one-third did not speak English; the rest were unable to remain on site to be tested, spoke English but were unable to read or write it, or did not take the test for other reasons.

24.  However, inferring that there is no impact when an impact really exists is another error of concern. In an effort to guard against this type of error, impacts with a probability between 10 and 20 percent of having arisen by chance are also occasionally discussed in the report, though these findings are referred to as program-control differences rather than impacts. These program-control differences are discussed if they are comparable in magnitude to a statistically significant impact of another program on the same outcome or if the impact appears to be part of a pattern of increases or decreases relative to the control group.

25.  See, for example, the discussion of two-year earnings impacts for Riverside LFA in Freedman et al., 2000a, pp. 61-63.

26.  See Chapter 3 for a discussion of education-focused program group members' participation in job search activities.

27.  While the research design in Detroit specified a full three-year embargo on welfare-to-work program services, 8 percent of all Detroit controls ended up participating in the new Work First program in follow-up year 3. See Farrell, 2000, for details.

28.  For example, as discussed in Chapter 9, between 12 and 23 percent of control group members, depending on the site, reported at the five-year follow-up point that a new baby had been added to their household since random assignment; the Family Support Act exempted women with children under age 3 from a mandatory welfare-to-work program participation requirement (or, at state option, women with children under age 1).

29.  For a complete description of this process in welfare-to-work programs, see Hamilton, 1995.

30.  Several situations could account for this 6 percent. While it is unusual for community college or other non-welfare programs to offer such activities, control group members may have found these programs on their own and voluntarily enrolled in these activities, a practice permitted under the NEWWS research design. NEWWS field research suggests that, over time, more non-welfare agencies in the evaluation sites began to offer job search assistance, particularly in Portland. In addition, while site welfare-to-work program staff were very diligent in screening for control group members at points of welfare application or program enrollment, some of these control group members could represent exceptions, or a few cases where controls "slipped through" the screening process. Field research, however, as well as periodic welfare case file reviews, indicated that the screening procedures in almost all sites were tight and that outside of Detroit very few control group members slipped through them.

31.  Expressing these numbers as the proportion of control group members who received any welfare in follow-up year 5 rather than as a proportion of all control group members, among sites or groups of controls with a full five-year embargo on welfare-to-work program services, the proportion participating at some point during year 5 in activities usually uniquely provided by welfare-to-work programs was 10 percent in Grand Rapids, 4 percent in Riverside, and 24 percent in Portland (but the denominator, or the number of Portland controls receiving any welfare in year 5, was small). In contrast, among sites or groups of controls where the embargo on welfare-to-work program services was lifted at some point during the last two years of the five-year follow-up period, the proportion of those receiving any welfare in year 5 who reported similar participation was 18 percent in Grand Rapids and 22 percent in Atlanta. While these two sets of percentages are much higher than those mentioned in the text, the difference between them again suggests that the level of contamination with welfare-to-work program services was only somewhat higher in sites or among groups of controls where the control embargo was lifted during follow-up year 4 or 5 than where the embargo was in effect for the full five-year follow-up.

32.  Both control and program group members probably would have been affected by publicity about the 1996 welfare law and, in the three sites where the count toward a welfare time limit began during the five-year follow-up, by the messages conveyed by welfare staff about the urgent need to find a job and leave welfare. In the three sites where the count toward a welfare time limit began during the five-year follow-up (Atlanta, Columbus, and Oklahoma City), the count began at the same time that the control embargo on welfare-to-work program services was lifted.

33.  Data that would show direct evidence that the differences in employment behavior between the two control groups in Portland are due to exposure to welfare-to-work programs (evidence such as differences in measured program participation rates for the two groups) are not available. Using the smaller, five-year embargoed control group to calculate impacts, however, provides the "safest" estimates of the true effects of the Portland program.

34.  Prior studies of the long-term effects of welfare-to-work programs have demonstrated that even after a control embargo on welfare-to-work program services is lifted programs can continue to have impacts, though perhaps diminishing ones, stemming from a labor market "boost" received by program group members early in the follow-up period. (See, for example, the five-year effects of the 1980s SWIM program in San Diego, presented in Friedlander and Hamilton, 1993.) Many of the NEWWS Evaluation programs examined here provide similar examples.

35.  As noted above, direct comparisons between the LFA and HCD programs in Riverside can be made only by comparing the HCD group with those members of the LFA group who lacked a high school diploma or basic skills at random assignment.