This section considers whether the NEWWS Evaluation underestimated the longer-term effects on employment and earnings in sites where the embargo on control group services ended after year 3. It concludes that the effect of removing the embargo was likely to be small except in Portland, but that most programs' effects on employment and earnings would likely have diminished in years 4 and 5 even if the control group embargo had remained in place. Because the embargo did appear to have an effect in Portland, results in this report were limited in Portland to 499 control group members who were prohibited from receiving program services for five years.
The likely effect of ending the control group embargo was examined in four sites where some control group members gained access to program services in year 4 and some did not gain access until later. In those four sites Atlanta, Columbus, Grand Rapids, and Portland control group members were divided into a "short-term embargo" group whose members became eligible for program services in year 4 and a "longer-term embargo" group, whose members remained ineligible for program services for at least four years.
These comparisons were not made in Riverside, Oklahoma City, and Detroit. In Riverside, the test was unnecessary because all control group members remained ineligible for services for five years after random assignment. In Oklahoma City, the fact that the program did not significantly affect employment or earnings during years 1 to 3 when all control group members remained ineligible for services suggests that ending the embargo did not alter the program's effects. In Detroit, the test could not be performed because all control group members became eligible for services after three years of follow-up. It is possible that the ending the embargo caused impacts in the program to decline after year 3, but there is no way to know whether that was true.
In comparing the short-term and longer-term embargo groups, the rate of decline of impacts for the two groups in year 4 and the size of impacts for the longer-term embargo group in year 4 were examined. If impacts declined faster for the short-term embargo group than for the longer-term embargo group in year 4, it is evidence that ending the embargo did reduce impacts. In contrast, if a program did not significantly affect outcomes for the longer-term embargo group in year 4 even though the group was still embargoed from receiving services it is evidence that ending the embargo did not have a large effect because the effectiveness of the program ended before the embargo.
Portland provided the most reliable comparison. Several years into the Portland evaluation, 499 control group members were chosen at random from the control group and remained ineligible for services for five years after random assignment. This group of 499 constituted the longer-term embargo group while the short-term embargo group consisted of all other control group members. Because the groups were chosen at random, they did not differ systematically from each other in background characteristics, and both groups resembled the program group.
A comparison of the short-term and longer-term embargo groups in Portland suggested that ending the control group embargo did contribute to a decline in impacts there. The effect of the program on employment was about the same in year 3 whether it was calculated with the short-term or longer-term embargo group (12 versus 11 percentage points). In year 4, however, the effect was only 2.6 percentage points when measured with the short-term embargo group but 8.5 percentage points when measured with the longer-term embargo group. In year 5, the effect had largely disappeared when measured with the short-term embargo group but was 3.8 percentage points when measured with the longer-term embargo group. Differences in the trends in earnings impacts were similar to differences in the trends in employment impacts during years 3 to 5. Because of these differences, MDRC decided to include only members of the longer-term embargo control group in estimating impacts of the Portland program.
Comparisons were conducted less rigorously in the other three sites. Control group members in the short-term and longer-term embargo groups were randomly assigned at different times and therefore might have differed systematically from each other at the time of random assignment. Consequently, one cannot rule out the possibility that variation in impacts between the two groups resulted from differences in background characteristics, labor markets, or other external factors that may have affected employment.
Despite these measurement problems, it appears that ending the control group embargo was not responsible for the small effect on employment and earnings in Grand Rapids at the end of the follow-up period. For both programs, the effect on employment and/or earnings declined during year 3, when all control group members were ineligible for services. This downward trend in impacts continued in years 4 and 5 and was seen in both the short-term and longer-term cohorts for both the LFA and the HCD programs.
In Atlanta, the trend in impacts after year 3 suggests that the end of the control group embargo had only a limited effect. For the longer-term embargo group, the effect of the LFA program on employment and earnings was no longer statistically significant by the second half of year 4 even though control group members were not yet eligible for program services. In other words, it appears that the effectiveness of the program had largely disappeared before control group members were allowed to receive program services. A similar pattern was seen for the HCD program in Atlanta.(13)
The evidence is similar for the Columbus Integrated program. During year 3, the estimated effect of the program on employment was 4.3 percentage points and statistically significant for the longer-term embargo group. In year 4, however, the effect on employment had decreased to 2.4 percentage points and was not statistically significant. The program's effect on earnings also began to decline during year 3 and continued downward slightly during year 4.
Ending the embargo may have made more of a difference for the Columbus Traditional program. Impacts on earnings for the first three years after random assignment were similar whether calculated for the short-term or the longer-term embargo groups. In years 4 and 5, however, the estimated impact on earnings was about $950 higher when calculated with the longer-term embargo group. On the other hand, Columbus Traditional had only a negligible effect on employment for either sample, either before or after year 3.
Whether or not allowing the control group to receive program services diminished the effect of the programs at the end of the follow-up period, the essential story is clear from the first three years of follow-up: The Portland program had the largest and most consistent effects by far, employment-focused job-search-first programs had the next largest effects, and education-focused programs had the smallest effects. Moreover, issues surrounding the control group embargo do not affect the comparison of LFA and HCD programs in Atlanta, Grand Rapids, and Riverside. That comparison indicates that the two program approaches were equally effective for graduates, that the LFA programs were more effective for nongraduates, and that there was no evidence that the cumulative effect of the HCD programs would surpass the cumulative effect of the LFA programs with longer follow-up.
1. As discussed elsewhere, results for Portland use only the 499 control group members who were prohibited from receiving program services for five years.
2. Program impacts on average number of quarters employed can be greatly affected by the total number of quarters of follow-up during which employment is measured. For this reason, some researchers prefer to convert this measure to a standard format by dividing average number of quarters employed by the total number of follow-up quarters. The result (usually expressed as a percentage) is called the "average quarterly employment rate." For example, in Portland, program group members worked for pay for an average of 9.4 of the 20 quarters of follow-up, whereas control group members were employed for 7.8 quarters on average. These averages are equivalent to quarterly employment rates of 47 percent (9.4 / 20 x 100) and 39 percent (7.8 / 20 x 100), respectively. Similarly, in Portland the impact on average number of quarters employed (of 1.6 quarters) is equivalent to an impact of 8 percentage points in the average quarterly employment rate (1.6 / 20 x 100; or 47 percent - 39 percent).
3. Riccio, Friedlander, and Freedman, 1994; Friedlander and Burtless, 1995; Bloom and Michalopoulos, 2001. The Riverside GAIN program increased employment by 9.9 percentage points and earnings by about $1,000 per year (Freedman et al., 1996; Hotz, Imbens, and Klerman, 2000). Two other programs came close to Portland over a three-year period: the Canadian Self-Sufficiency Project (Michalopoulos and Berlin, 2001), which increased employment by 7.1 percentage points and earnings by about $725 per year, and the Connecticut Jobs First program, which increased employment by 7.8 percentage points but increased earnings by only about $425 per year (Bloom et al., 2000b).
4. Bloom and Michalopoulos, 2001; Michalopoulos and Schwartz, 2001.
5. This finding is based on field research. On the other hand, results in Chapter 7 demonstrate that Portland's approach led to large earnings impacts for sample members who faced at least one serious barrier to employment.
6. Unlike Oklahoma City, Detroit operated a traditional case management system, in which separate Income Maintenance staff determined recipients' eligibility for public assistance and calculated their monthly welfare and Food Stamp grants. However, the Detroit welfare-to-work program was not fully staffed during the first years of the evaluation, and its case managers reported spending most of their limited time working with clients who were participating on their own initiative and who had requested child care and other supportive services.
7. Results over time are also shown in Appendix Tables C.2 (impacts on total earnings in each year of follow-up), C.3 (impacts on cumulative measures of employment and earnings in years 1 to 3), and C.4 (impacts on employment and earnings in year 5). Appendix Table C.5 shows employment and full-time employment at the end of year 5 according to responses to the Five-Year Client Survey. Results in those tables confirm that few programs continued to have significant effects in year 5.
8. The two Atlanta programs are discussed first because they appear side by side in Figure 4.1 and because they used different self-sufficiency approaches. Section IV of this chapter presents a statistical comparison of employment and earnings in the LFA and HCD programs in Atlanta, Grand Rapids, and Riverside.
9. In addition, Freedman, 2000, describes results on employment stability and earnings growth through four years in much greater detail, and Michalopoulos, 2001, compares the effects of the NEWWS programs with effects from programs in Minnesota and Canada that supplemented earnings to encourage work.
10. The few high school graduates who were assigned to the Riverside HCD program were considered in need of basic education and included in the group of nongraduates shown.
11. Michalopoulos and Schwartz, 2001, report that impacts over a three-year follow-up period were larger for high school graduates than for nongraduates. Among the programs they studied, three of the four with the largest differences in earnings impacts between high school graduates and nongraduates were in the Alameda, Riverside, and San Diego GAIN programs, all of which used a mix of initial activities and none of which were in the NEWWS Evaluation. Even among the 20 programs that they studied, however, impacts were larger for high school graduates in 11 programs but larger for nongraduates in 9 programs.
12. Recall that people not in need of basic education were not randomly assigned to the Riverside LFA program, so that a comparison of the two approaches could not be made for the full sample in Riverside.
13. For unknown reasons, the effects of the HCD program were larger in years 4 and 5 when calculated with the short-term embargo group than the longer-term embargo group.