[ Main Page of Report | Contents of Report ]
This chapter describes the integrated and traditional programs' three-year impacts on employment, earnings, AFDC receipt and payments, and combined income from earnings, AFDC, and Food Stamps.(1) The impact estimates in the chapter are based on quarterly unemployment insurance (UI) records and monthly AFDC and Food Stamp payment records.(2) As mentioned in Chapter 1, sample members were randomly assigned to either the integrated group, the traditional group, or the control group. This research design allows for three different experimental comparisons: integrated-control, traditional-control, and integrated-traditional. The first two comparisons provide estimates of the effects of each program (averages for control group members represent outcomes that are expected to occur in the absence of the programs); the third comparison provides estimates of the relative effectiveness of the two programs. Unless otherwise stated, the impacts discussed in this chapter are statistically significant.(3)
[ Go To Contents ]
Over three years, the integrated and traditional programs produced similar employment and earnings gains. Researchers had hypothesized that the higher participation rate in the integrated program would lead to larger impacts on employment and earnings, but this was not the case. Quarterly impact patterns suggest, however, that the integrated program may prove more successful in the fourth year of follow-up than the traditional program.
The integrated program produced somewhat larger decreases in months of AFDC receipt and AFDC payments measured over three years, probably because integrated case managers could more quickly respond to changes in sample members' employment and welfare eligibility status, and because they had more knowledge about status changes than staff in the traditional program.
Neither of the programs increased average "combined income" from earnings, AFDC, and Food Stamps. On average, people in the programs replaced some public assistance dollars with earnings.
Among sample members who had a high school diploma or GED at random assignment (graduates), the two programs produced roughly similar effects. Among nongraduates, however, the integrated program was more successful than the traditional program in increasing earnings and decreasing cash assistance payments.
[ Go To Contents ]
As discussed in prior chapters, both programs aimed to increase welfare recipients' skills levels before they looked for work. Employment gains and welfare reductions in programs such as these may be delayed while recipients participate in education and training activities. After an initial period of investment in skills-building, integrated and traditional group members may make up for forgone earnings by obtaining more jobs or higher-paying jobs than control group members.
The evaluation designers expected that the programs would affect employment and welfare receipt to different degrees. Specifically, they hypothesized that the integrated program would be more effective than the traditional program in increasing employment and in decreasing welfare receipt.
The hypothesis that the integrated program would produce larger employment and earnings gains was based primarily on two expectations. First, as discussed in Chapter 3, the integrated approach was expected to engage more people in the program than the traditional approach, and it did. It was expected that exposing more people to the program's messages and services, would, in turn, result in larger effects on employment and earnings. Second, as discussed in Chapter 2, the integrated program was expected to more effectively deliver program services and monitor welfare recipients' situations than the traditional program, which could lead to larger employment and earnings effects. In fact, the implementation data suggested some differences between the programs: namely, integrated case managers provided more personalized attention than did traditional case managers and more closely monitored participation in program activities.
The hypothesis that the integrated program would produce larger decreases in welfare receipt and payments than the traditional program was predicated on two expectations. First, if the integrated program increased employment and earnings more than the traditional program (as discussed above), that, in turn, likely would result in larger welfare reductions. Second, it was expected that the integrated structure would engender more effective eligibility case management than the traditional structure by giving case managers more knowledge about the client more quickly, and allowing them to close ineligible cases more quickly. For example, the closer contact between integrated case managers and recipients might allow integrated staff to learn about eligibility changes that traditional staff might not. Also, if a sample member became employed, an integrated case manager might find out about this change more quickly because the integrated staff see their clients more frequently. Once they had this knowledge, integrated staff would also be able to respond more quickly because they could reduce a grant amount or close a grant themselves, rather than having to ask another staff member do so.
As Chapter 1 described, random assignment in Columbus occurred at the point of referral to the JOBS program. The impacts presented in this chapter, therefore, reflect the effects not only of the program services and mandates but also of the referral to the program and any related follow-up, such as sanctioning for orientation nonattendance. Telling someone she must participate in a welfare-to-work program could affect her labor market and welfare behavior in at least two ways: She could be motivated to quickly find a job and leave welfare to avoid the program mandate or, alternatively, to delay employment to gain access to the services offered by the program. Because random assignment occurred only at the point of referral to the program, it is impossible to isolate the effects of either the referral to the program or the program services and mandates.(4) The impacts presented in this chapter, therefore, represent estimates of the combined or average effect of the program services and mandates and the referral to the program.
As mentioned in Chapter 3, some people in the integrated and traditional groups never attended JOBS orientation and thus had no chance to attend program activities. The outcomes for these sample members are averaged together with the impacts for orientation attendees. This may "dilute" the estimate of the effects of the welfare-to-work program services and mandates, especially for the traditional program in which even fewer people attended orientation.
[ Go To Contents ]
Table 5.1 shows the two programs' impacts on employment and earnings. The first set of columns shows the impacts of the integrated program (integrated-control comparison), and the second set shows the impacts of the traditional program (traditional-control comparison). The last column shows the difference between outcomes of the integrated and traditional programs (integrated-traditional difference).
| Integrated-Control Comparison | Traditional-Control Comparison | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Outcome | Integrated Group | Control Group | Difference (Impact) | Percentage Change | Traditional Group | Control Group | Difference (Impact) | Percentage Change | Integrated- Traditional Difference (Impact) |
| Ever employed, years 1-3 (%) | 81.1 | 78.5 | 2.6** | 3.3 | 80.7 | 78.5 | 2.2** | 2.8 | 0.4 |
| Year1 | 60.0 | 60.1 | -0.1 | -0.2 | 59.9 | 60.1 | -0.1 | -0.2 | 0.0 |
| Year2 | 65.2 | 62.9 | 2.3* | 3.7 | 64.5 | 62.9 | 1.6 | 2.6 | 0.7 |
| Year3 | 68.9 | 65.3 | 3.6*** | 5.5 | 67.9 | 65.3 | 2.6** | 3.9 | 1.0 |
| Quarters employed, years 1-3 | 5.75 | 5.46 | 0.29*** | 5.3 | 5.69 | 5.46 | 0.23** | 4.1 | 0.06 |
| Year1 | 1.64 | 1.62 | 0.02 | 1.0 | 1.66 | 1.62 | 0.04 | 2.7 | -0.03 |
| Year2 | 1.97 | 1.82 | 0.15*** | 8.5 | 1.94 | 1.82 | 0.13*** | 7.0 | 0.03 |
| Year3 | 2.14 | 2.02 | 0.12** | 5.8 | 2.08 | 2.02 | 0.06 | 2.8 | 0.06 |
| Earnings, years 1-3 ($) | 13,208 | 12,027 | 1,181*** | 9.8 | 13,027 | 12,027 | 1,000** | 8.3 | 181 |
| Year1 | 2,994 | 2,914 | 80 | 2.8 | 3,099 | 2,914 | 185 | 6.4 | -105 |
| Year2 | 4,578 | 3,982 | 596*** | 15.0 | 4,472 | 3,982 | 490*** | 12.3 | 106 |
| Year3 | 5,635 | 5,131 | 505*** | 9.8 | 5,456 | 5,131 | 325* | 6.3 | 180 |
| Sample size (total = 7,242) | 2,513 | 2,159 | 2,570 | 2,159 | |||||
| Sources: MDRC calculations from
Ohio unemployment insurance (UI) earnings records. Notes: Estimates were regression-ajusted using ordinary least squares, controlling for pre-random assignment characteristics of sample members. "Percentage change" equals 100 times "difference"divided by "control group". Rounding may cause slight discrepancies in calculating sums and differences. A two-tailed t-test was applied to differences between outcomes for the program and control groups and to differences between outcomes for the integrated and traditional program groups. Statistical significance levels are indicated as: * = 10 percent; ** = 5 percent; *** = 1 percent. Year 1 refers to quarters 2 to 5; year 2 refers to quarters 6 to 9; year 3 refers 10 to 13. Because quarter 1, the quarter of random assignment, may contain some earnings and AFDC payments from the period prior to random assignment, it is excluded from follow-up measures. |
|||||||||
In the context of Columbus's strong labor market, employment rates were high even without the programs' intervention: As the table shows, 78.5 percent of control group members were employed at some point during the three years after random assignment. They were employed for an average of 5.46 quarters (just over 16 months) and earned an average of $12,027 over the three-year period (this average includes zeros for people with no earnings).
Both programs produced small increases in employment rates and the length of time employed. Over three years, 81.1 percent of the integrated group worked for pay, a 2.6 percentage-point increase, and 80.7 percent of the traditional group worked for pay, a 2.2 percentage-point increase. Integrated group members worked an average of 5.75 quarters, an increase of 0.29 of a quarter (almost a month), and traditional group members worked an average of 5.69 quarters, an increase of 0.23 of a quarter (about two-thirds of a month).
Integrated group members earned on average $13,208 over the three-year period, a $1,181, or 10 percent, increase above the control group. Traditional group members earned on average $13,027, a $1,000, or 8 percent, increase above the control mean. (The $181 difference between the program groups' average earnings is not statistically significant.) These gains are similar to the earnings impacts of the other education-focused programs studied as part of the NEWWS Evaluation.(5) The earnings gains in the Columbus programs were primarily the result of longer duration of employment and higher earnings on the job.(6) In other words, the programs raised total earnings by enabling integrated and traditional group members who would have been employed anyway to obtain better jobs.
As is often found for programs that emphasize building skills prior to finding a job, neither program increased employment levels or earnings during the first year of follow-up. (This indicates that the referral to a mandatory welfare-to-work program did not, on average, spur people to quickly begin a job to avoid the program.) Employment and earnings gains began in the second year of follow-up. By the end of the third year of follow-up, the integrated program's impacts had decreased but remained statistically significant. The traditional program's impacts, in contrast, were less consistent during the third year. (See Appendix Table C.1 for the programs' impacts displayed for each quarter of the follow-up period.) These patterns suggest that the integrated program will likely continue to increase employment and earnings during the fourth year of follow-up, but the traditional program may not.
Contrary to researchers' expectations, more personalized attention, closer monitoring, and the higher rate of participation in program activities in the integrated program did not translate into larger employment and earnings impacts (although quarterly patterns suggest that the integrated program may have more positive results than the traditional program during the fourth year of follow-up). A recently published MDRC analysis of participation in welfare-to-work programs found that although a minimum level of participation is necessary to produce employment impacts, above that threshold there is no linear relationship between participation levels and impacts.(7) In light of this new information, one should not expect that higher participation rates would necessarily yield larger employment and earnings impacts.
[ Go To Contents ]
The employment gains in Columbus were accompanied by cash assistance reductions. Over a three-year period, control group members received AFDC for an average of about 21 1/2 months (see Table 5.2). The integrated program reduced AFDC receipt by more than 2 1/2 months, a decrease of 12 percent relative to the control group mean. The traditional program reduced receipt to a lesser extent — by about 1 2/3 months, or 8 percent. The integrated program's reduction in months of welfare receipt was the largest among the education-focused programs in the NEWWS Evaluation.(8) The Columbus program impacts on cash assistance receipt grew throughout the follow-up period. In the last quarter of year 3, 40.3 percent of the control group received AFDC benefits compared with 33.2 percent of the integrated group and 34.9 percent of the traditional group (see Appendix Table C.1).
| Integrated-Control Comparison | Traditional-Control Comparison | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Outcome | Integrated Group | Control Group | Difference (Impact) | Percentage Change | Traditional Group | Control Group | Difference (Impact) | Percentage Change | Integrated- Traditional Difference (Impact) |
| Ever received AFDC, years 1-3 (%) | 96.4 | 96.9 | -0.5 | -0.6 | 96.3 | 96.9 | -0.6 | -0.7 | 0.1 |
| Year1 | 95.8 | 96.6 | -0.8 | -0.8 | 96.0 | 96.6 | -0.6 | -0.6 | -0.2 |
| Year2 | 65.1 | 69.1 | -4.0*** | -5.7 | 65.9 | 69.1 | -3.2** | -4.6 | -0.8 |
| Year3 | 47.0 | 54.4 | -7.4*** | -13.6 | 49.0 | 54.4 | -5.4*** | -10.0 | -2.0 |
| Months received AFDC, years 1-3 | 18.87 | 21.48 | -2.61*** | -12.2 | 19.77 | 21.48 | -1.71*** | -8.0 | -0.90*** |
| Year1 | 8.91 | 9.62 | -0.71*** | -7.3 | 9.16 | 9.62 | -0.46*** | -4.8 | -0.25** |
| Year2 | 5.91 | 6.79 | -0.87*** | -12.9 | 6.22 | 6.79 | -0.57*** | -8.4 | -0.30** |
| Year3 | 4.04 | 5.08 | -1.03*** | -20.4 | 4.39 | 5.08 | -0.68*** | -13.5 | -0.35** |
| AFDC amount, years 1-3 ($) | 6,071 | 7,151 | -1,079*** | -15.1 | 6,335 | 7,151 | -816*** | -11.4 | -264** |
| Year1 | 2,880 | 3,199 | -318*** | -10.0 | 2,950 | 3,199 | -249*** | -7.8 | -70* |
| Year2 | 1,895 | 2,270 | -375*** | -16.5 | 1,989 | 2,270 | -281*** | -12.4 | -95** |
| Year3 | 1,297 | 1,682 | -386*** | -22.9 | 1,396 | 1,682 | -286*** | -17.0 | -99** |
| Sample size (total = 7,242) | 2,513 | 2,159 | 2,570 | 2,159 | |||||
| Sources: MDRC calculations from
Ohio AFDC records. Notes: Estimates were regression-ajusted using ordinary least squares, controlling for pre-random assignment characteristics of sample members. "Percentage change" equals 100 times "difference"divided by "control group". Rounding may cause slight discrepancies in calculating sums and difference. A two-tailed t-test was applied to differences between outcomes for the program and control groups and to differences between outcomes for the integrated and traditional program groups. Statistical significance levels are indicated as:*= 10 percent; ** = 5 percent; *** = 1 percent. Year 1 refers to quarters 2 to 5; year 2 refers to quarters 6 to 9; year 3 refers 10 to 13. Because quarter 1, the quarter of random assignment, may contain some earnings and AFDC payments from the period prior to random assignment, it is excluded from follow-up measures. |
|||||||||
Over three years, control group members received an average of $7,151 in AFDC payments. Both programs reduced welfare payments, but the integrated program's impacts were larger. Integrated group members received an average of $6,071 in AFDC payments over the three-year period, a reduction of $1,079, or 15 percent, compared with the control mean, and traditional group members received an average of $6,335, a reduction of $816, or 11 percent. The percentage reduction in the integrated program is the largest reduction among the NEWWS Evaluation education-focused programs.(9) Most of the decrease in AFDC payments occurred because integrated and traditional group members spent less time on welfare than their control group counterparts, rather than receiving lower grant amounts.(10)
The programs reduced AFDC payments during each year of follow-up; the effects grew over time and remained substantial at the end of year 3 (see Appendix Table C.1). This suggests that the reductions are very likely to persist during the fourth year of follow-up. The fact that during year 1 the programs reduced welfare receipt and payments but did not increase employment and earnings suggests that some people may have left the welfare rolls to avoid the participation mandate.
As hypothesized, the integrated program generated larger reductions in welfare receipt and payments than the traditional program. This difference occurred because integrated group members spent less time on welfare, on average, than their traditional group counterparts.(11) In other words, the integrated case management structure facilitated case closures. Specifically, integrated case managers closed cases more quickly, on average, than traditional staff. They also closed cases that would have remained open in the traditional program, likely because they were better able to detect individuals who should not be receiving welfare.
[ Go To Contents ]
The earnings gains produced by the Columbus programs did not exceed the public assistance losses, thus providing no gain in average "combined income." As discussed in a previous NEWWS Evaluation report, there are several ways to measure a program's effect on sample members' economic self-sufficiency; one way is to examine sample members' average combined income from earnings, AFDC, and Food Stamps.(12) This income measure does not include estimates of the Earned Income Credit, a credit against federal income taxes for low-income taxpayers. Over three years, the Columbus integrated program reduced Food Stamp payments by $697, and the traditional program reduced Food Stamp payments by $483 (these numbers are not presented in a table).(13) During the three years following random assignment, control group members received on average $25,490 from earnings, AFDC, and Food Stamps. Integrated group members received $24,895 ($595, or 2 percent, less), and traditional group members received $25,192 ($298, or 1 percent, less). These small decreases in average combined income are not statistically significant.
[ Go To Contents ]
Employment and earnings impacts for people entering the programs with a high school diploma or GED (graduates) are presented in Table 5.3. Neither program increased three-year employment levels for graduates, but the integrated program produced small increases in employment levels in years 2 and 3. Measured over the three-year follow-up period, the traditional program increased graduates' average earnings by $1,105, or 7 percent; the $633 increase for the integrated program is not statistically significant. (The difference between the earnings of the integrated group and the traditional group is not statistically significant.) Table 5.4 shows that the two programs decreased the number of months that graduate sample members received welfare and their average welfare payments.
| Integrated-Control Comparison | Traditional-Control Comparison | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Outcome | Integrated Group | Control Group | Difference (Impact) | Percentage Change | Traditional Group | Control Group | Difference (Impact) | Percentage Change | Integrated- Traditional Difference (Impact) |
| Ever employed, years 1-3 (%) | 85.5 | 83.0 | 2.0 | 2.4 | 84.2 | 83.0 | 1.2 | 1.4 | 0.8 |
| Year1 | 65.8 | 65.3 | 0.5 | 0.8 | 64.3 | 65.3 | -0.9 | -1.5 | 1.5 |
| Year2 | 70.8 | 68.0 | 2.8* | 4.2 | 69.1 | 68.0 | 1.2 | 1.7 | 1.7 |
| Year3 | 72.5 | 69.7 | 2.8* | 4.0 | 71.5 | 69.7 | 1.8 | 2.6 | 1.0 |
| Quarters employed, years 1-3 | 6.37 | 6.12 | 0.25* | 4.2 | 6.34 | 6.12 | 0.22 | 3.6 | 0.03 |
| Year1 | 1.85 | 1.84 | 0.01 | 0.5 | 1.89 | 1.84 | 0.05 | 2.8 | -0.04 |
| Year2 | 2.20 | 2.05 | 0.16** | 7.7 | 2.17 | 2.05 | 0.12* | 5.9 | 0.04 |
| Year3 | 2.32 | 2.23 | 0.09 | 3.9 | 2.28 | 2.23 | 0.05 | 2.3 | 0.04 |
| Earnings, years 1-3 ($) | 15,544 | 14,911 | 633 | 4.2 | 16,016 | 14,911 | 1,105* | 7.4 | -473 |
| Year1 | 3,558 | 3,617 | -59 | -1.6 | 3,854 | 3,617 | 237 | 6.6 | -296* |
| Year2 | 5,404 | 5,014 | 390 | 7.8 | 5,525 | 5,014 | 511** | 10.2 | -121 |
| Year3 | 6,582 | 6,280 | 302 | 4.8 | 6,637 | 6,280 | 357 | 5.7 | -56 |
| Sample size (total = 4,135) | 1,428 | 1,230 | 1,477 | 1,230 | |||||
| Sources: MDRC calculations from
Ohio unemploment insurance (UI) earnings records. Notes: Estimates were regression-ajusted using ordinary least squares, controlling for pre-random assignment characteristics of sample members. "Percentage change" equals 100 times "difference"divided by "control group". Rounding may cause slight discrepancies in calculating sums and difference. A two-tailed t-test was applied to differences between outcomes for the program and control groups and to differences between outcomes for the integrated and traditional program groups. Statistical significance levels are indicated as:* = 10 percent; ** = 5 percent; *** = 1 percent. Year 1 refers to quarters 2 to 5; year 2 refers to quarters 6 to 9; year 3 refers 10 to 13. Because quarter 1, the quarter of random assignment, may contain some earnings and AFDC payments from the period prior to random assignment, it is excluded from follow-up measures. |
|||||||||
| Integrated-Control Comparison | Traditional-Control Comparison | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Outcome | Integrated Group | Control Group | Difference (Impact) | Percentage Change | Traditional Group | Control Group | Difference (Impact) | Percentage Change | Integrated- Traditional Difference (Impact) |
| Ever received AFDC, years 1-3 (%) | 96.6 | 96.6 | 0.0 | 0.0 | 96.2 | 96.6 | -0.4 | -0.4 | 0.4 |
| Year1 | 96.1 | 96.2 | -0.1 | -0.2 | 95.7 | 96.2 | -0.5 | -0.5 | 0.4 |
| Year2 | 62.7 | 65.4 | -2.7 | -4.1 | 61.7 | 65.4 | -3.7** | -5.7 | 1.0 |
| Year3 | 44.0 | 49.7 | -5.7*** | -11.5 | 44.0 | 49.7 | -5.7*** | -11.5 | 0.0 |
| Months received AFDC, years 1-3 | 17.84 | 19.94 | -2.10*** | -10.5 | 18.23 | 19.94 | -1.72*** | -8.6 | -0.38 |
| Year1 | 8.66 | 9.30 | -0.65*** | -7.0 | 8.82 | 9.30 | -0.49*** | -5.2 | -0.16 |
| Year2 | 5.48 | 6.21 | -0.72*** | -11.6 | 5.59 | 6.21 | -0.62*** | -10.0 | -0.10 |
| Year3 | 3.70 | 4.43 | -0.73*** | -16.4 | 3.82 | 4.43 | -0.61*** | -13.8 | -0.12 |
| AFDC amount, years 1-3 ($) | 5,633 | 6,486 | -853*** | -13.2 | 5,720 | 6,486 | -766*** | -11.8 | -88 |
| Year1 | 2,740 | 3,011 | -271*** | -9.0 | 2,778 | 3,011 | -233*** | -7.7 | -38 |
| Year2 | 1,723 | 2,028 | -304*** | -15.0 | 1,742 | 2,028 | -286*** | -14.1 | -19 |
| Year3 | 1,169 | 1,447 | -278*** | -19.2 | 1,201 | 1,447 | -246*** | -17.0 | -31 |
| Sample size (total = 4,135) | 1,428 | 1,230 | 1,477 | 1,230 | |||||
| Sources: MDRC calculations from Ohio AFDC
records. Notes: Estimates were regression-ajusted using ordinary least squares, controlling for pre-random assignment characteristics of sample members. "Percentage change" equals 100 times "difference"divided by "control group". Rounding may cause slight discrepancies in calculating sums and difference. A two-tailed t-test was applied to differences between outcomes for the program and control groups and to differences between outcomes for the integrated and traditional program groups. Statistical significance levels are indicated as:* = 10 percent; ** = 5 percent; *** = 1 percent. Year 1 refers to quarters 2 to 5; year 2 refers to quarters 6 to 9; year 3 refers 10 to 13. Because quarter 1, the quarter of random assignment, may contain some earnings and AFDC payments from the period prior to random assignment, it is excluded from follow-up measures. |
|||||||||
As Table 5.5 shows, the integrated program was much more successful than the traditional program in generating earnings gains for sample members who entered the programs without a high school diploma or GED (nongraduates). The integrated program increased nongraduates' average three-year earnings by $1,730, or 21 percent; the gain of $734, or 9 percent, in the traditional program is not statistically significant. Both programs decreased time on welfare and welfare payments for nongraduates, although the integrated program did so to a greater extent (see Table 5.6). The integrated program decreased months of welfare receipt by 14 percent, compared with 7 percent for the traditional program, and decreased welfare payments by $1,404, or 17 percent, compared with $874, or 11 percent, for the traditional program. Both programs produced small, not statistically significant reductions in average combined income from earnings, AFDC, and Food Stamps for graduates and nongraduates, as they did for the full sample.
| Integrated-Control Comparison | Traditional-Control Comparison | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Outcome | Integrated Group | Control Group | Difference (Impact) | Percentage Change | Traditional Group | Control Group | Difference (Impact) | Percentage Change | Integrated- Traditional Difference (Impact) |
| Ever employed, years 1-3 (%) | 75.9 | 72.9 | 2.9 | 4.0 | 76.1 | 72.9 | 3.2* | 4.4 | -0.2 |
| Year1 | 52.0 | 53.4 | -1.4 | -2.6 | 54.0 | 53.4 | 0.6 | 1.1 | -1.9 |
| Year2 | 57.7 | 56.3 | 1.3 | 2.3 | 58.1 | 56.3 | 1.8 | 3.2 | -0.5 |
| Year3 | 63.9 | 59.9 | 3.9* | 6.5 | 63.0 | 59.9 | 3.0 | 5.1 | 0.9 |
| Quarters employed, years 1-3 | 4.89 | 4.6 | 0.29* | 6.3 | 4.79 | 4.60 | 0.19 | 4.1 | 0.10 |
| Year1 | 1.35 | 1.33 | 0.02 | 1.5 | 1.35 | 1.33 | 0.02 | 1.7 | 0.00 |
| Year2 | 1.66 | 1.52 | 0.14** | 9.1 | 1.64 | 1.52 | 0.12* | 7.8 | 0.02 |
| Year3 | 1.89 | 1.76 | 0.13* | 7.5 | 1.80 | 1.76 | 0.05 | 2.6 | 0.09 |
| Earnings, years 1-3 ($) | 9,938 | 8,208 | 1,730*** | 21.1 | 8,942 | 8,208 | 734 | 8.9 | 996** |
| Year1 | 2,201 | 1,986 | 215 | 10.8 | 2,079 | 1,986 | 93 | 4.7 | 121 |
| Year2 | 3,409 | 2,632 | 777*** | 29.5 | 3,042 | 2,632 | 410** | 15.6 | 367* |
| Year3 | 4,328 | 3,590 | 738*** | 20.5 | 3,821 | 3,590 | 231 | 6.4 | 507** |
| Sample size (total = 3,073) | 1,072 | 915 | 1,086 | 915 | |||||
| Sources: MDRC calculations from Ohio unemploment
insurance(UI) earnings records. Notes: Estimates were regression-ajusted using ordinary least squares, controlling for pre-random assignment characteristics of sample members. "Percentage Change" equals 100 times "difference"divided by "control group". Rounding may cause slight discrepancies in calculating sums and difference. A two-tailed t-test was applied to differences between outcomes for the program and control groups and to differences between outcomes for the integrated and traditional program groups. Statistical significance levels are indicated as:* = 10 percent; ** = 5 percent; *** = 1 percent. Year 1 refers to quarters 2 to 5; year 2 refers to quarters 6 to 9; year 3 refers 10 to 13. Because quarter 1, the quarter of random assignment, may contain some earnings and AFDC payments from the period prior to random assignment, it is excluded from follow-up measures. |
|||||||||
| Integrated-Control Comparison | Traditional-Control Comparison | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Outcome | Integrated Group | Control Group | Difference (Impact) | Percentage Change | Traditional Group | Control Group | Difference (Impact) | Percentage Change | Integrated- Traditional Difference (Impact) |
| Ever received AFDC, years 1-3 (%) | 96.0 | 97.4 | -1.3* | -1.4 | 96.8 | 97.4 | -0.6 | -0.6 | -0.8 |
| Year1 | 95.4 | 97.1 | -1.7** | -1.7 | 96.7 | 97.1 | -0.4 | -0.4 | -1.2 |
| Year2 | 68.5 | 74.2 | -5.6*** | -7.6 | 72.0 | 74.2 | -2.2 | -2.9 | -3.4* |
| Year3 | 51.0 | 60.8 | -9.8*** | -16.1 | 55.8 | 60.8 | -5.1** | -8.3 | -4.7** |
| Months received AFDC, years 1-3 | 20.25 | 23.58 | -3.33*** | -14.1 | 21.93 | 23.58 | -1.64*** | -7.0 | -1.68*** |
| Year1 | 9.26 | 10.04 | -0.78*** | -7.8 | 9.65 | 10.04 | -0.40*** | -3.9 | -0.39*** |
| Year2 | 6.49 | 7.59 | -1.09*** | -14.4 | 7.10 | 7.59 | -0.48** | -6.4 | -0.61*** |
| Year3 | 4.50 | 5.95 | -1.45*** | -24.4 | 5.18 | 5.95 | -0.77*** | -12.9 | -0.69*** |
| AFDC amount, years 1-3 ($) | 6,661 | 8,065 | -1,404*** | -17.4 | 7,191 | 8,065 | -874*** | -10.8 | -530*** |
| Year1 | 3,071 | 3,462 | -392*** | -11.3 | 3,190 | 3,462 | -272*** | -7.9 | -120** |
| Year2 | 2,124 | 2,603 | -479*** | -18.4 | 2,335 | 2,603 | -268*** | -10.3 | -211*** |
| Year3 | 1,467 | 1,999 | -532*** | -26.6 | 1,666 | 1,999 | -334*** | -16.7 | -199*** |
| Sample size (total = 3,073) | 1,072 | 915 | 1,086 | 915 | |||||
| Sources: MDRC calculations from Ohio AFDC
records. Notes: Estimates were regression-ajusted using ordinary least squares, controlling for pre-random assignment characteristics of sample members. "Percentage change" equals 100 times "difference"divided by "control group". Rounding may cause slight discrepancies in calculating sums and differences. A two-tailed t-test was applied to differences between outcomes for the program and control groups and to differences between outcomes for the integrated and traditional program groups. Statistical significance levels are indicated as:* = 10 percent; ** = 5 percent; *** = 1 percent. Year 1 refers to quarters 2 to 5; year 2 refers to quarters 6 to 9; year 3 refers 10 to 13. Because quarter 1, the quarter of random assignment, may contain some earnings and AFDC payments from the period prior to random assignment, it is excluded from follow-up measures. |
|||||||||
[ Go To Contents ]
The final report in the NEWWS Evaluation will track outcomes for sample members in Columbus for up to five years following random assignment. This longer follow-up period is important when evaluating programs that engage many people in education because it can take some time for sample members to put their newly acquired skills to work in the job market. As noted in Chapter 1, however, in October 1997 control group members began to receive program services, and all sample members — from the control group, the integrated group, and the traditional group — began receiving integrated case management. These two changes, which occurred during the fourth or fifth year of follow-up for most sample members (random assignment occurred from 1992 to 1994), may have diminished the differences between the research groups' outcomes.
[ Go To Contents ]
1. As noted in Chapter 1, this report refers to cash assistance as AFDC; although the AFDC program has been converted into a block grant to states, AFDC existed throughout the study period for the report.
2. UI earnings data are collected by calendar quarter (January through March, April through June, and so on). For the research, the quarter during which a sample member was randomly assigned was designated quarter 1. The first follow-up year (called year 1), covers quarters 2 through 5, the second year (year 2) covers quarters 6 through 9, and so on. Monthly AFDC and Food Stamp payments were grouped into quarters and years covering the same periods as earnings quarters and years. See Freedman et al., 2000, for more detail on the methods used in this analysis and on the impacts of the Columbus programs (estimated using two years of follow-up data), and for a more comprehensive comparison of the effects of the Columbus programs with the effects of the other NEWWS Evaluation programs.
3. Differences in outcomes are considered statistically significant if there is less than a 10 percent probability that they occurred by chance.
4. In two sites in the NEWWS Evaluation — Grand Rapids and Riverside — random assignment occurred at two separate points: at the point of referral to the welfare-to-work program and at the point of entry into the welfare-to-work program (program orientation). This design allows researchers to calculate separately the impacts of the referral itself and the effects of the program services and mandates. See Knab et al., 2001, for a presentation of findings from this special study.
5. Three-year earnings impacts for the other education-focused programs were: Atlanta HCD, $1,003, or 11 percent; Grand Rapids HCD, $892, or 10 percent; Riverside HCD, $740, or 14 percent; and Detroit, $848, or 11 percent. Oklahoma City's impact of $12 is not statistically significant. Impacts for the employment-focused programs in the evaluation ranged from $1,292 to $3,152. (These impact findings are from an unpublished MDRC analysis of NEWWS Evaluation data.)
6. In both programs, longer duration of employment and higher earnings on the job represent about two-thirds of the earnings gain, and an increase in the number of jobs found represents one-third. This decomposition is not exact. It is based on the approximate mathematical equivalence of the "percentage difference" in average total earnings to the sum of the percentage differences in "total quarters employed if employed," "average earnings per quarter employed," and "ever employed." The contribution of each effect is obtained by dividing its percentage difference by the percentage difference in average total earnings. The sum of all three contributions does not equal 100 percent because a small portion of the earnings impact is attributable to interactions among the components. (The integrated program increased "total quarters employed if employed" by 0.13 of a quarter, or 1.9 percent, and increased "average earnings per quarter employed" by $95, or 4.3 percent. Corresponding gains in the traditional program were 0.09 of a quarter, or 1.3 percent, and $88, or 4.0 percent.)
7. Hamilton and Scrivener, 1999.
8. Impacts of the other programs ranged from .58 to 1.94 months (from an unpublished MDRC analysis of NEWWS Evaluation data).
9. Decreases in average three-year AFDC payments for the other education-focused programs were: Atlanta HCD, 6 percent; Grand Rapids, 13 percent; Riverside HCD, 12 percent; Detroit, 3 percent; and Oklahoma City, 4 percent. Decreases for the employment-focused programs in the evaluation range from 8 to 21 percent. (These findings are from an unpublished MDRC analysis of NEWWS Evaluation data.)
10. The average monthly payment amount for control group members ($333) multiplied by the reduction in number of months of AFDC receipt indicates what the AFDC savings would have been if average monthly payment amounts were the same for program and control group members who remained on welfare. In the integrated program, for example, this calculation ($333 times 2.61 months) yields $869, which represents 81 percent of the $1,079 three-year AFDC savings. The calculation for the traditional program ($333 times 1.71 months) yields $569, which is 70 percent of the $816 three-year AFDC payment impact. The remainder of the impact on three-year AFDC payments may have come from reductions in grant amounts resulting from sanctions or from increased earnings while still on welfare. Alternatively, the overall reduction in months of receipt may have fallen primarily on cases with above-average monthly grant amounts. This decomposition is not exact, since it ignores interactions between grant level and case closure.
11. The decomposition of the cash assistance payment impact discussed in footnote 10 indicates that $869 of the integrated program's impact on payments was generated because integrated group members spent less time on welfare than their control group counterparts; this figure for the traditional program was $569, $300 less than the figure for the integrated program. The $300 difference exceeds the $264 difference between the two programs' impacts on payments.
12. Freedman et al., 2000.
13. Over three years, the three research groups received Food Stamp benefits valued at the following amounts: control group, $6,312; integrated group, $5,616; traditional group, $5,830. Both programs' impact is statistically significant at the .01 level.
Main Page of Report | Contents of Report
Home Pages:
National Evaluation of Welfare-to-Work Strategies
(NEWWS)
Assistant Secretary for Planning and Evaluation
(ASPE)
U.S. Department of Health and Human Services
(HHS)