Targeted Help for the Hard-to-Employ: Outcomes of Two Philadelphia Welfare-to-Work Programs. Appendix B: Methodology


We employed multiple analytic methods to estimate the regression models in order to test the robustness of study findings. In particular, we used three statistical techniques to control for differences between RSC and TWC participants when modeling their postprogram outcomes:

1. Ordinary Least Squares (OLS) Regressions.В  This method estimated the relationship of each observable factor to outcomes while holding all other factors constant. The basic form of the OLS model is: y = Alpha0 + Alpha1*TWC + xBeta + Residual or error term, where y is the outcome measure, TWC is an indicator variable that equals 1 for TWC participants and 0 for RSC participants, x is a vector of observable participant characteristics measured at baseline, the Greek letters are parameters to be estimated, and Residual or error term is a mean-zero error term. In this formulation, the estimate of Alpha1 represents the regression-adjusted TWC-RSC difference. We then compare these estimates to the simple differences in mean outcomes between the two participant groups.[1]

2. Fixed-Effects Regressions.В  This approach used longitudinal data on outcomes over time to examine whether changes in outcomes between the post- and preintervention periods differed across the TWC and RSC groups. This difference-in-difference method attempts to correct for unobserved differences between the two participant groups that remain constant over time and are captured by the preintervention outcome measures. We estimated these models by stacking quarterly outcome data and including time indicators and time*TWC interaction terms as explanatory variables.

3. Propensity Scoring.В  This method matched RSC participants to TWC participants using observable characteristics. The matching was performed in three stages. First, we estimated a logit model where the dependent variable (equaling 1 for TWC participants and 0 for RSC participants) was regressed on the full set of explanatory variables. Second, using the logit results, we calculated a predicted probability of being in the TWC group (that is, a propensity score) for each sample member. Finally, we matched to each TWC sample member that RSC participant with the closest propensity score. The matching was done with replacement, so that an RSC participant could match to more than one TWC participant. We then compared mean outcomes of TWC participants to those of their matched comparison group. This method yielded a comparison group (from among RSC participants) that is very similar to the program group (in this case, TWC participants) on a wide range of characteristics. Thus, our hope is that the two groups also match on unobservables that are correlated with outcomes. Some evidence suggests that this method may be able to replicate experimental findings, but results can be biased to the extent that participants motivation and interest in the program are not measured (Agodini and Dynarski 2001).


[1] Note that this simple difference is the estimate of Alpha1 when no explanatory variables are included in the models.

View full report


"report.pdf" (pdf, 387.86Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®