[ Main page of Report | Contents of Report ]
The differences in overall characteristics of RSC and TWC participants, economic conditions at the time of participation, and other observable factors discussed in the previous chapter are likely to account — at least in part — for the differences in employment, earnings, and TANF receipt outcomes found for participants in these two programs. We use multivariate statistical analysis techniques to help explain the differences in the outcomes of RSC and TWC participants. Only an experimental design evaluation could determine conclusively the extent to which the RSC and TWC programs contribute to participant outcomes. Nevertheless, given the evidence available from our analyses, we also discuss possible implications of our study findings for future programs.
We have organized the discussions in this chapter around two broad sets of questions:
[ Go to Contents ]
To identify the factors that contributed to differences in RSC-TWC outcomes, we regressed key outcomes on participants’ demographic characteristics, prior work experience, prior TANF receipt, economic conditions after program entry, and an indicator of RSC/TWC status. The primary goal of this analysis was to assess the extent to which the parameter estimate on the RSC/TWC indicator variable could be reduced when the observable participant characteristics and other factors were included as explanatory variables in the models. We used three multivariate statistical analysis techniques: ordinary least squares (OLS) regression, fixed-effects regression, and propensity scoring. Using these techniques, we examined participant outcomes six quarters after the quarter of program enrollment — the latest point for which we have consistent follow-up data for most RSC and TWC participants.
The results from these three techniques were fairly consistent (Table IV.1). For example, the predicted difference in TWC-RSC employment rates six quarters after enrollment was –4.9 percent and insignificant in the OLS model, –6.7 percent and only marginally significant in the fixed-effects model, and –3.8 percent and insignificant in the propensity scoring model. This consistency across techniques suggests that the results are robust. To simplify the discussion, Chapter IV focuses on the OLS regression results. Our main results can be summarized as follows:
Observable factors account for most of the difference in TWC-RSC employment rates. The simple difference (that is, before taking into account differences in observable factors) in employment rates a year and a half after program enrollment for TWC and RSC participants was a statistically significant 14 percentage points. After demographics, prior employment, and economic conditions are taken into account in the OLS model, however, the predicted difference in TWC-RSC employment rates becomes smaller (–4.9 percent) and statistically insignificant (Table IV.1).
| Statistical Method | Employment | Earningsa | TANF Receipt |
|---|---|---|---|
| Difference in the Percentage Employed (TWC-RSC) | Difference in Dollars (TWC-RSC) | Difference in the Percentage Receiving TANF (TWC-RSC) | |
| Simple Difference in Means | –14.1*** | –598.24*** | 16.7*** |
| OLS Regression | –4.9 | –367.62*** | 11.6*** |
| Fixed-Effects | –6.7* | -320.31*** | 11.7*** |
| Propensity Scoring | –3.8 | –347.93*** | 13.8*** |
| Source: Baseline information forms of Welfare-to-Work participants, Mathematica Policy Research, Inc.; state administrative records data; and RSC and TWC Management Information Systems data. | |||
| Note: All models include demographics, prior work or prior TANF receipt, and unemployment rate as explanatory variables. Prior work is included in employment and earnings models; prior TANF receipt is included in TANF receipt model. | |||
| a The earnings models include participants who were not employed and had zero earnings. | |||
| */**/*** Significantly different from zero at the .05/.01/.001 level, two-tailed test. | |||
Even after observable factors are controlled for, however, about half the TWC-RSC differences in earnings and TANF receipt remain. Six quarters after program enrollment, TWC participants earned, on average, about $600 less than RSC participants (Table IV.1). After observable factors are controlled for, the predicted difference in postprogram earnings between TWC and RSC participants declines to $368 and remains statistically significant. Similarly, the difference in TANF receipt rates six quarters after enrollment between TWC and RSC participants is reduced from 17 to 12 percent and remains statistically significant.
Thus, participation in TWC does not appear to lead to a full “catching up” to the outcomes of RSC participants. Despite being about equally likely to be employed six quarters after program enrollment, TWC participants had lower earnings and were more likely to receive TANF than comparable RSC participants. These differences could be due to unobserved factors. Another possible interpretation of this finding is that subsequent employers do not value the time TWC participants spent in transitional work as highly as time spent in unsubsidized employment. Thus, when TWC participants finally moved into unsubsidized jobs, they still entered jobs comparable to those first entered by RSC participants.[1] To the extent that TWC participation delayed participants’ entry into unsubsidized employment, this would mean that, compared with RSC participants, TWC participants may have been at an earlier point in the development of their employment capabilities. They may have had less time to achieve gains in earnings due to advancement within jobs or to progress to better-paying jobs.[2] The lower earnings of TWC participants, in turn, could have contributed to their higher rates of TANF receipt.
Educational attainment, prior earnings, and prior TANF receipt were key factors in explaining outcomes. Both educational attainment and prior earnings provide a good indication of people’s skills and prior workplace performance, and thus their ability to succeed in the labor market. Not surprisingly, having a high school diploma or a GED was a highly significant factor related to employment, earnings, and TANF receipt (Table IV.2). Average earnings in the four quarters before program entry were also significantly related to both postprogram employment and earnings. Similarly, TANF receipt in all four quarters before program enrollment was significantly related to TANF receipt six quarters after program enrollment.
| Factor Associated with Outcomes | Employment | Earnings | TANF Receipt |
|---|---|---|---|
| Program Status | |||
| Participated in TWC | –.05 | –367.62*** | .116*** |
| Baseline Characteristics | |||
| Age Is Less than 30 | .028 | 45.46 | .073* |
| Age Is Greater than 30 and Less than 40 | .015 | 98.77 | .016 |
| Female | –.038 | –605.51* | .261*** |
| Hispanic | –.049 | –57.80 | –.099** |
| White | .007 | 131.72 | –.105* |
| Other Race/Ethnicity | .043 | 159.08 | –.120 |
| Married or Cohabiting | –.021 | –19.17 | .040 |
| Number of Children | .002 | 20.01 | .027*** |
| Age of Youngest Child Less than 5 | .033 | 67.57 | .048* |
| Has High School Diploma or GED | .085*** | 513.64*** | –.130*** |
| Own Health Problem Limits Ability to Work | –.066* | –120.50 | .030 |
| Family Member’s Health Problem Limits Ability to Work | .019 | –70.00 | –.010 |
| Economic Conditions | |||
| Unemployment Rate in Quarter 6 After Program Entry | –.101*** | –191.50* | –.016 |
| Prior Employment | |||
| Never Employed Before Program Entry | –.051 | –123.31 | |
| Employed in All Four Quarters Before Program Entry | –.029 | –29.50 | |
| Average Earnings in Four Quarters Before Program Entry | .273*** | 763.01*** | |
| Prior TANF Receipt | |||
| Received TANF in All Four Quarters Before Program Entry | .121*** | ||
| Received TANF Two to Five Years | –.000 | ||
| Received TANF Five or More Years | –.020 | ||
| Sources: Baseline information forms of Welfare-to-Work participants, Mathematica Policy Research, Inc.; state administrative records data; and RSC and TWC Management Information Systems data. | |||
| Note: Missing values for race/ethnicity, health problem, family member’s health problem, and length of TANF receipt were imputed. Dummy variables for imputed cases, included in the model, were not significant. | |||
| */**/*** Significant at the .05/.01/.001 level, two-tailed test. | |||
Job placement success also was an important factor in explaining later employment, earnings, and TANF receipt. In some analyses, we included as an explanatory variable an indicator of program completion — that is, whether the RSC or TWC participant had successfully reached the point of unsubsidized job placement through the program — to capture unmeasured characteristics, such as greater motivation or a positive attitude, likely to have made participants more job ready. When included in the final OLS regression model, program completion further reduces the predicted difference in TWC-RSC participant outcomes. For example, the predicted difference in earnings declines from –$368 to –$248. Similarly, the predicted difference in rates of TANF receipt declines from 10.4 to 8.9 percent. Hence, differences between TWC and RSC participants along the unmeasured characteristics captured by program completion may be another factor contributing to their differences in outcomes. At the same time, program completion may also measure the programs’ ability to engage participants in activities and help them find unsubsidized jobs. Thus, its inclusion in our regression models may make the remaining difference between RSC and TWC participant outcomes an understatement of real differences in program effects.
RSC noncompleters offer a further comparison group for TWC participants. Failure to complete the program was one way the RSCs identified participants likely to need the more intensive services offered by the TWC program. It is unclear why some RSC noncompleters were not referred to TWC — in theory, all of them should have been — but, as discussed in Chapter III, we do know that, consistent with the programs’ targeting, RSC noncompleters were very similar in their overall characteristics to TWC participants. To the extent that RSC noncompleters truly resembled TWC participants (that is, along both observed and unobserved characteristics), their outcomes offer some suggestion of the outcomes TWC participants might have achieved without this intervention.[3]
On average, TWC participants had outcomes similar to those of RSC noncompleters. After we control for observable factors, there are only small, insignificant differences in employment, earnings, and TANF receipt between TWC participants (both completers and noncompleters) and RSC noncompleters (Table IV.3). Therefore, regardless of which program they were involved in, TWC participants (in general) and RSC noncompleters fared similarly over time. As discussed in Chapter III, there were important differences in the outcomes of TWC completers and TWC noncompleters (although their baseline characteristics are similar). As a result, comparing the average outcomes of TWC completers and noncompleters to the outcomes of RSC noncompleters is likely to mask important relationships. Thus, we compare the RSC noncompleters to TWC completers and TWC noncompleters separately.
| Employment | Earningsa | TANF Receipt | |
|---|---|---|---|
| Statistical Method | Difference in Percentage Employed | Difference in Dollars | Difference in the Percentage Receiving TANF |
| All TWC Participants vs. RSC Noncompleters | |||
| Simple Difference in Means | 0.4 | –93.23 | 6.6* |
| OLS Regression | 5.0 | 20.1 | 1.8 |
| TWC Noncompleters vs. RSC Noncompleters | |||
| Simple Difference in Means | 11.3*** | –487.27*** | 12.4*** |
| OLS Regression | –1.8 | –395.98*** | 10.9** |
| TWC Completers vs. RSC Noncompleters | |||
| Simple Difference in Means | 11.7*** | 460.30*** | –3.8 |
| OLS Regression | 11.1** | 471.27*** | –7.9* |
| Source: Baseline information forms of Welfare-to-Work participants, Mathematica Policy Research, Inc.; state administrative records data; and RSC and TWC Management Information Systems data. | |||
| Note: All models include demographics, prior work or prior TANF receipt, and unemployment rate as explanatory variables. Prior work is included in the employment and earnings models. Prior TANF receipt is included in the TANF receipt models. | |||
| aThe earnings models include participants who were not employed and had zero earnings. | |||
The postprogram outcomes of TWC completers are significantly better than the outcomes of RSC noncompleters. Six quarters after program enrollment, TWC completers were 11 percent more likely to be employed, earned about $470 more, and were 8 percent less likely to receive TANF than comparable RSC noncompleters Table IV.3). To the extent that TWC served people unlikely to succeed in the RSC program, this suggests that the program may have helped these participants achieve better outcomes. Because the original RSC-TWC referral process eventually broke down, it is also possible that TWC completers include people who enrolled directly in this program but could have succeeded in securing unsubsidized employment through the RSCs. To the extent this happened, apparent differences between the outcomes of TWC completers and RSC noncompleters would be an overstatement of TWC’s success. Given the lack of an experimental design, we cannot determine which TWC completers may have succeeded in getting unsubsidized jobs with the less intensive help of the RSC programs, nor the extent to which TWC may have helped “convert” actual or potential RSC “failures” into “successes.”
However, TWC noncompleters fared much worse than RSC noncompleters. TWC noncompleters were as likely as RSC noncompleters to be employed six quarters after program referral, but they earned about $400 less and were 11 percent more likely to receive TANF (Table IV.3). This suggests that TWC noncompleters may have been the most disadvantaged of the WtW population — unable to complete either the TWC or the RSC program. Their poor outcomes highlight the importance of identifying and addressing factors contributing to participants’ lack of success in these types of programs. The marked differences in outcomes between TWC noncompleters and RSC noncompleters further suggest that important, unobserved differences among TWC and RSC participants remain unaccounted for in our study.
[ Go to Contents ]
We used multiple methods — OLS regression, fixed-effects regression, and propensity scoring — to attempt to control for differences in the characteristics of TWC and RSC participants(both observed and unobserved) and other factors likely to have contributed to their differences in outcomes. These regression adjustments reduced the TWC-RSC differences in employment, earnings, and TANF receipt but did not erase them completely. The inclusion of a program completion term — to capture additional unobserved participant characteristics — further reduced the difference in outcomes. In a strategy analogous to propensity scoring, we also restricted our analysis to RSC noncompleters, who were very similar to TWC participants along observable characteristics and who, in theory, should have been referred to the TWC. This analysis revealed that, regardless of which program they were involved in, TWC participants (in general) and RSC noncompleters fared similarly over time. Marked differences in the regression-adjusted outcomes of RSC noncompleters, TWC completers, and TWC noncompleters suggest, however, that important, unmeasured differences remain unaccounted for in our study. Hence, we are unable to reach definitive conclusions about the effects of these programs. The potential benefits of subsidized work experience relative to direct placement in unsubsidized employment for the hard to employ can be assessed only through a randomized trial of such programs.
[ Go to Contents ]
Although our study of outcomes of RSC and TWC participants cannot offer definitive conclusions, it suggests themes that could contribute to the further development of programs aimed at helping the hard to employ succeed in their transition from welfare to work. This section assembles these themes and presents our broad conclusions.
Intensive services can be targeted to the most disadvantaged. The original design of the RSC and TWC programs and, in particular, their sequencing within the larger GPW initiative represented an innovative, commendable approach to program development. Believing that “failure” at the RSCs should not be the only way to secure referral to TWC, program developers specified that staff would have discretion to identify people likely to need more intensive services and refer them directly to the TWC program. This would accomplish several important objectives. It would avert the costly waste of resources in delivering RSC services to participants unlikely to succeed with their basic assistance, minimize the time participants spent in services before successfully transitioning off time-limited TANF, and avoid the potential discouragement of participants required to fail at one program before gaining access to more appropriate services. We can reasonably assume that most of the TWC participants in our study enrolled directly in the program, since enrollments increased markedly after direct TWC outreach was allowed and RSC referrals had been limited to that point. Thus, our finding that TWC participants, in general, were similar along observable characteristics to RSC noncompleters suggests that the intended targeting was both feasible and successful in these programs.
Programs targeting the hard to employ may be more effective if they devote attention to identifying and addressing factors that contribute to participants’ lack of success. As discussed, TWC participants who did not achieve job placement through the TWC program fared worse than any other RSC or TWC participants. These people may have been the most disadvantaged among the WtW-eligible population. Intensive programs aimed at serving this population need to identify and address the barriers they face. One clue that our study offers regarding factors that may contribute to lack of success is that TWC participants without a high school diploma were less likely to complete the program.[4] However, simply focusing on education is unlikely to lead to improved outcomes for these people, since earlier studies have shown that providing education services alone does not generally lead to improved employment outcomes (Michalopoulos and Schwartz 2000; and Burghardt et al. 1992). Those who did not succeed at TWC are likely to have a variety of complex barriers that, unfortunately, remain unmeasured in our study.[5]
The hardest-to-employ participants in intensive programs like TWC may be especially vulnerable during periods of high unemployment. The only other highly significant factor in predicting program completion among TWC participants was economic conditions, as measured by the local unemployment rate.[6] This suggests that programs like TWC may need to offer even more intensive placement help to participants in times when there is more competition for available unsubsidized jobs. Because experience in transitional work may not be as highly valued as unsubsidized work experience, transitional jobs may need to include more skill building and training (to make participants more attractive to prospective employers), and placements may need to be longer. In addition, program staff may need to take on an even more active role in unsubsidized job placement than in a time of more favorable economic conditions.
Services related to retention and advancement remain important in helping participants build on their employment experience and achieve further gains. Our study confirms that those who maintain employment continue to build on these experiences and increase their earnings over time. In addition, the RSC and TWC participants who switched jobs tended to move to jobs with better wages, hours, and benefits. Thus, both job retention and advancement services, including ongoing job search and placement services, are potentially important components to help participants build a strong employment history leading them to further employment success.
Further research is needed to clarify how programs like the RSCs and the TWC contribute to participant outcomes. Shortfalls in program enrollment made it impossible to implement the original random-assignment design planned for this evaluation. Our results hint that the intensive TWC intervention may have partially, but not completely, made up for the greater employment challenges TWC participants faced. Nevertheless, our study leaves unanswered questions that only a more rigorous evaluation can answer. Large scale experiments provide evidence that programs promoting rapid attachment while allowing for some education and training are particularly effective in helping welfare recipients increase earnings and reduce welfare receipt (Hamilton 2002). Transitional work programs, like the TWC, have a similar approach in that they promote rapid entry to work while incorporating ongoing skill-building. Further study is needed to determine the actual effects of transitional work on participants’ outcomes and the most appropriate targeting and sequencing of programs like the TWC and the RSCs.
[ Go to Contents ]
[1] As supporting evidence for this explanation, note that the average earnings of TWC participants three quarters after program entry — when they would have transitioned fully out of unsubsidized employment — are comparable to the earnings of RSC participants one quarter after program entry (Figure III.6).
[2] This interpretation would be consistent with findings from our follow-up survey of WtW participants. As discussed in Chapter II, survey results show that RSC and TWC participants who changed jobs during the first year after program enrollment had higher earnings and worked more hours, or both, and that these gains played a role in their increased earnings.
[3] There are several possible explanations for why RSC noncompleters were not referred to the TWC program. As we noted in Chapter I, Pennsylvania is a client choice state. Hence, when referred to the Philadelphia welfare agencies for referral to another program, RSC noncompleters could have opted out of TWC (for example, because of location or other preferences) and chosen a different employment program. They could also have been exempted from work requirements or could have left TANF altogether. If RSC noncompleters tend to be people who were systematically excluded from, or opted out of, participation in TWC, their outcomes would not necessarily provide a good representation of the likely outcomes of TWC participants in the absence of this intervention.
[4] For further information on factors associated with program success, see Appendix Table A.6.
[5] For example, a study of barriers to completion in Philadelphia’s Single Point of Contact program — one of the other employment assistance programs available to work-mandatory TANF recipients in Philadelphia — suggests that noncompleters often faced many barriers, including child care concerns, domestic violence, and low self-efficacy (Kinnevy et al. 2003).
[6] Since most RSC participants enrolled before the economic downturn in 2000, they had little variation in economic conditions. Therefore, an association with economic conditions was less likely to emerge for these participants.
Main Page of Report | Contents of Report
Home Pages:
Human Services Policy (HSP)
Assistant Secretary for Planning and Evaluation
ASPE)
U.S. Department of Health and Human Services
(HHS)