How Effective Are Different Welfare-to-Work Approaches? Five-Year Adult and Child Impacts for Eleven Programs. Control Group Treatment Over Time

12/01/2001

Part of the reliability of the findings from a random assignment social experiment rests on the assumption that control group members are not exposed to the specific program being evaluated. If control group members are exposed to the program, measured impacts could represent an underestimate of the true effects of the program. Rarely, however, is it possible to maintain this lack of exposure for a long period of time. In this regard, the NEWWS Evaluation was no exception.(29)

Agreements reached with the NEWWS Evaluation sites at the beginning of the study specified that all control group members were to be kept out of the welfare-to-work programs being studied in each site for at least three years from their random assignment date. (As noted earlier, control group members were free to enroll in other employment-related activities offered in their communities during this period and, as will be discussed below and in Chapter 3 of this report, many controls did, in fact, enroll in such activities.) Midway through the evaluation, program operators in Riverside and MDRC agreed to extend the embargo on providing welfare-to-work program services to control group members to five years. Program operators in Portland and Grand Rapids agreed to a similar extension, but only for a subsample of control group members. In Portland, about one-quarter of the control group was randomly selected (from every month of the random assignment period) to remain ineligible for welfare-to-work program services until the end of year 5; in Grand Rapids, the embargo on welfare-to-work program services was extended for all control group members who were randomly assigned during 1993, the last year of sample intake in the site. However, Atlanta, Columbus, and Oklahoma City could not continue barring control group members from welfare-to-work program services longer than three years because of the implementation of welfare time limits in their states; if NEWWS control group members could not be exempted from welfare time limits, then both HHS and MDRC felt it necessary to allow control group members to be eligible for some type of welfare-to-work program services once their time limit clock "starting ticking." Finally, in Detroit, it was not feasible to continue the control services embargo, after a different agency took over administration of the site's welfare-to-work program and entry points into the new program were so numerous and widespread that screening to identify NEWWS control group members would have been close to impossible.

Most probably program impacts on employment and earnings and other outcomes in the last two years of follow-up in a few of the five sites above would have been somewhat larger had some control group members not been exposed to welfare-to-work programs. For several reasons, however, lifting the control group embargo on services prior to the end of the five-year follow-up period in these five sites most likely had only a small effect on measured program impacts. As discussed in detail in Chapter 2, most control group members were not eligible to receive program services when the control group embargo ended, often because they had already left welfare. From one-quarter to one-half of control group members in these five sites were receiving welfare when their embargo on program services was lifted. In addition, after the embargo was lifted, some control group members were not contacted about enrolling in the program until after the end of the follow-up period, and others were assigned to a program orientation but did not show up. Moreover, the likely effects of ending the control group embargo were estimated for the sites where the embargo was lifted by calculating impacts for a subsample of control group members who were precluded from program services for four to five years. It was found that the patterns of impacts in years 4 and 5 resembled those for all sample members in those sites.

It should be emphasized that the control group situations described above do not affect the assessments in this report of the relative merits of the Labor Force Attachment and Human Capital Development approaches in welfare-to-work programs. As described in Chapter 2, the three-group random assignment designs in the three sites in which these two types of programs were simultaneously operated permit a direct comparison of these two approaches, that is, a comparison that does not need to take into account the services received by and the behavior of control group members.