Targeted Help for the Hard-to-Employ: Outcomes of Two Philadelphia Welfare-to-Work Programs. What Factors Help Explain RSC-TWC Differences in Outcomes

09/02/2004

To identify the factors that contributed to differences in RSC-TWC outcomes, we regressed key outcomes on participants demographic characteristics, prior work experience, prior TANF receipt, economic conditions after program entry, and an indicator of RSC/TWC status.  The primary goal of this analysis was to assess the extent to which the parameter estimate on the RSC/TWC indicator variable could be reduced when the observable participant characteristics and other factors were included as explanatory variables in the models.  We used three multivariate statistical analysis techniques:  ordinary least squares (OLS) regression, fixed-effects regression, and propensity scoring.  Using these techniques, we examined participant outcomes six quarters after the quarter of program enrollment  the latest point for which we have consistent follow-up data for most RSC and TWC participants. 

The results from these three techniques were fairly consistent (Table IV.1).  For example, the predicted difference in TWC-RSC employment rates six quarters after enrollment was 4.9 percent and insignificant in the OLS model, 6.7 percent and only marginally significant in the fixed-effects model, and 3.8 percent and insignificant in the propensity scoring model.  This consistency across techniques suggests that the results are robust.  To simplify the discussion, Chapter IV focuses on the OLS regression results.  Our main results can be summarized as follows:

Observable factors account for most of the difference in TWC-RSC employment rates.  The simple difference (that is, before taking into account differences in observable factors) in employment rates a year and a half after program enrollment for TWC and RSC participants was a statistically significant 14 percentage points.  After demographics, prior employment, and economic conditions are taken into account in the OLS model, however, the predicted difference in TWC-RSC employment rates becomes smaller (4.9 percent) and statistically insignificant (Table IV.1).

 

Table IV.1
Philadelphia WTW Outcomes Study:
Estimated Difference Between TWC and RSC Outcomes Six Quarters after Program Entry
Statistical Method Employment Earningsa TANF Receipt
Difference in the Percentage Employed (TWC-RSC) Difference in Dollars (TWC-RSC) Difference in the Percentage Receiving TANF (TWC-RSC)
Simple Difference in Means 14.1*** 598.24*** 16.7***
OLS Regression 4.9 367.62*** 11.6***
Fixed-Effects 6.7* -320.31*** 11.7***
Propensity Scoring 3.8 347.93*** 13.8***
Source:  Baseline information forms of Welfare-to-Work participants, Mathematica Policy Research, Inc.; state administrative records data; and RSC and TWC Management Information Systems data.
Note:  All models include demographics, prior work or prior TANF receipt, and unemployment rate as explanatory variables.  Prior work is included in employment and earnings models; prior TANF receipt is included in TANF receipt model.
a The earnings models include participants who were not employed and had zero earnings.
*/**/*** Significantly different from zero at the .05/.01/.001 level, two-tailed test.

Even after observable factors are controlled for, however, about half the TWC-RSC differences in earnings and TANF receipt remain. Six quarters after program enrollment, TWC participants earned, on average, about $600 less than RSC participants (Table IV.1).  After observable factors are controlled for, the predicted difference in postprogram earnings between TWC and RSC participants declines to $368 and remains statistically significant.  Similarly, the difference in TANF receipt rates six quarters after enrollment between TWC and RSC participants is reduced from 17 to 12 percent and remains statistically significant.

Thus, participation in TWC does not appear to lead to a full catching up to the outcomes of RSC participants.  Despite being about equally likely to be employed six quarters after program enrollment, TWC participants had lower earnings and were more likely to receive TANF than comparable RSC participants.  These differences could be due to unobserved factors.  Another possible interpretation of this finding is that subsequent employers do not value the time TWC participants spent in transitional work as highly as time spent in unsubsidized employment.  Thus, when TWC participants finally moved into unsubsidized jobs, they still entered jobs comparable to those first entered by RSC participants.[1]  To the extent that TWC participation delayed participants entry into unsubsidized employment, this would mean that, compared with RSC participants, TWC participants may have been at an earlier point in the development of their employment capabilities.  They may have had less time to achieve gains in earnings due to advancement within jobs or to progress to better-paying jobs.[2]  The lower earnings of TWC participants, in turn, could have contributed to their higher rates of TANF receipt.

Educational attainment, prior earnings, and prior TANF receipt were key factors in explaining outcomes. Both educational attainment and prior earnings provide a good indication of peoples skills and prior workplace performance, and thus their ability to succeed in the labor market.  Not surprisingly, having a high school diploma or a GED was a highly significant factor related to employment, earnings, and TANF receipt (Table IV.2).  Average earnings in the four quarters before program entry were also significantly related to both postprogram employment and earnings.  Similarly, TANF receipt in all four quarters before program enrollment was significantly related to TANF receipt six quarters after program enrollment.

 

Table IV.2
Philadelphia WTW Outcomes Study:
Factors Related to Outcomes Six Quarters
After Program Entry, Based on OLS Regression Results
Factor Associated with Outcomes Employment Earnings TANF Receipt
Program Status
Participated in TWC .05 367.62*** .116***
Baseline Characteristics
Age Is Less than 30 .028 45.46 .073*
Age Is Greater than 30 and Less than 40 .015 98.77 .016
Female .038 605.51* .261***
Hispanic .049 57.80 .099**
White .007 131.72 .105*
Other Race/Ethnicity .043 159.08 .120
Married or Cohabiting .021 19.17 .040
Number of Children .002 20.01 .027***
Age of Youngest Child Less than 5 .033 67.57 .048*
Has High School Diploma or GED .085*** 513.64*** .130***
Own Health Problem Limits Ability to Work .066* 120.50 .030
Family Members Health Problem Limits Ability to Work .019 70.00 .010
Economic Conditions
Unemployment Rate in Quarter 6 After Program Entry .101*** 191.50* .016
Prior Employment
Never Employed Before Program Entry .051 123.31  
Employed in All Four Quarters Before Program Entry .029 29.50  
Average Earnings in Four Quarters Before Program Entry .273*** 763.01***  
Prior TANF Receipt
Received TANF in All Four Quarters Before Program Entry .121***
Received TANF Two to Five Years .000
Received TANF Five or More Years .020
Sources:  Baseline information forms of Welfare-to-Work participants, Mathematica Policy Research, Inc.; state administrative records data; and RSC and TWC Management Information Systems data.
Note:  Missing values for race/ethnicity, health problem, family members health problem, and length of TANF receipt were imputed.  Dummy variables for imputed cases, included in the model, were not significant.
*/**/*** Significant at the .05/.01/.001 level, two-tailed test.

Job placement success also was an important factor in explaining later employment, earnings, and TANF receipt. In some analyses, we included as an explanatory variable an indicator of program completion  that is, whether the RSC or TWC participant had successfully reached the point of unsubsidized job placement through the program  to capture unmeasured characteristics, such as greater motivation or a positive attitude, likely to have made participants more job ready.  When included in the final OLS regression model, program completion further reduces the predicted difference in TWC-RSC participant outcomes.  For example, the predicted difference in earnings declines from $368 to $248.  Similarly, the predicted difference in rates of TANF receipt declines from 10.4 to 8.9 percent.  Hence, differences between TWC and RSC participants along the unmeasured characteristics captured by program completion may be another factor contributing to their differences in outcomes.  At the same time, program completion may also measure the programs ability to engage participants in activities and help them find unsubsidized jobs.  Thus, its inclusion in our regression models may make the remaining difference between RSC and TWC participant outcomes an understatement of real differences in program effects.

RSC noncompleters offer a further comparison group for TWC participants.  Failure to complete the program was one way the RSCs identified participants likely to need the more intensive services offered by the TWC program.  It is unclear why some RSC noncompleters were not referred to TWC  in theory, all of them should have been  but, as discussed in Chapter III, we do know that, consistent with the programs targeting, RSC noncompleters were very similar in their overall characteristics to TWC participants.  To the extent that RSC noncompleters truly resembled TWC participants (that is, along both observed and unobserved characteristics), their outcomes offer some suggestion of the outcomes TWC participants might have achieved without this intervention.[3]

On average, TWC participants had outcomes similar to those of RSC noncompleters. After we control for observable factors, there are only small, insignificant differences in employment, earnings, and TANF receipt between TWC participants (both completers and noncompleters) and RSC noncompleters (Table IV.3).  Therefore, regardless of which program they were involved in, TWC participants (in general) and RSC noncompleters fared similarly over time.  As discussed in Chapter III, there were important differences in the outcomes of TWC completers and TWC noncompleters (although their baseline characteristics are similar).  As a result, comparing the average outcomes of TWC completers and noncompleters to the outcomes of RSC noncompleters is likely to mask important relationships.  Thus, we compare the RSC noncompleters to TWC completers and TWC noncompleters separately.

 

Table IV.3
Philadelphia WTW Outcomes Study:
Estimated Difference in Outcomes Between TWC Participants
and RSC Noncompleters Six Quaters after Program Entry
  Employment Earningsa TANF Receipt
Statistical Method Difference in Percentage Employed Difference in Dollars Difference in the Percentage Receiving TANF
All TWC Participants vs. RSC Noncompleters
Simple Difference in Means 0.4 93.23 6.6*
OLS Regression 5.0 20.1 1.8
TWC Noncompleters vs. RSC Noncompleters
Simple Difference in Means 11.3*** 487.27*** 12.4***
OLS Regression 1.8 395.98*** 10.9**
TWC Completers vs. RSC Noncompleters
Simple Difference in Means 11.7*** 460.30*** 3.8
OLS Regression 11.1** 471.27*** 7.9*
Source: Baseline information forms of Welfare-to-Work participants, Mathematica Policy Research, Inc.; state administrative records data; and RSC and TWC Management Information Systems data.
Note:   All models include demographics, prior work or prior TANF receipt, and unemployment rate as explanatory variables.  Prior work is included in the employment and earnings models.  Prior TANF receipt is included in the TANF receipt models.
aThe earnings models include participants who were not employed and had zero earnings.

The postprogram outcomes of TWC completers are significantly better than the outcomes of RSC noncompleters.  Six quarters after program enrollment, TWC completers were 11 percent more likely to be employed, earned about $470 more, and were 8 percent less likely to receive TANF than comparable RSC noncompleters Table IV.3).  To the extent that TWC served people unlikely to succeed in the RSC program, this suggests that the program may have helped these participants achieve better outcomes.  Because the original RSC-TWC referral process eventually broke down, it is also possible that TWC completers include people who enrolled directly in this program but could have succeeded in securing unsubsidized employment through the RSCs.  To the extent this happened, apparent differences between the outcomes of TWC completers and RSC noncompleters would be an overstatement of TWCs success.  Given the lack of an experimental design, we cannot determine which TWC completers may have succeeded in getting unsubsidized jobs with the less intensive help of the RSC programs, nor the extent to which TWC may have helped convert actual or potential RSC failures into successes.

However, TWC noncompleters fared much worse than RSC noncompleters.  TWC noncompleters were as likely as RSC noncompleters to be employed six quarters after program referral, but they earned about $400 less and were 11 percent more likely to receive TANF (Table IV.3).  This suggests that TWC noncompleters may have been the most disadvantaged of the WtW population  unable to complete either the TWC or the RSC program.  Their poor outcomes highlight the importance of identifying and addressing factors contributing to participants lack of success in these types of programs.  The marked differences in outcomes between TWC noncompleters and RSC noncompleters further suggest that important, unobserved differences among TWC and RSC participants remain unaccounted for in our study.

View full report

Preview
Download

"report.pdf" (pdf, 387.86Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®