ESTIMATING PROGRAM IMPACTS WITH 2-YEAR CLIENT SURVEY DATA As with administrative records data, MDRC used OLS regression (PROC REG in SAS) to estimate program effects on survey outcomes. The same covariates were used with each data source to control for differences among research groups in background characteristics. Before running procedures to estimate impacts, researchers should merge the survey data by IDNUMBER with the research group dummy variables and covariates from the Full Impact Sample file. Use SRV2RESP to select survey respondents. With two exceptions, MDRC used the same procedures for estimating program impacts with survey data as were used for estimating impacts with administrative records (see IMP_MEMO.TXT). The exceptions are: 1) In Atlanta, Grand Rapids, Portland, and Riverside, certain subgroups were oversampled for research purposes when choosing the survey sample. Therefore, it is necessary to weight the survey sample to make impact estimates generalizable to the larger "survey eligible" sample from which respondents were drawn. The variable called FIELDWGT is used to weight the survey sample (see N2PSSAMP.TXT for details). FIELDWGT should also be used when calculating adjusted means (see IMP_MEMO.TXT for details). For example, in Atlanta and Grand Rapids, if WtBJC=All sample members weighted by FIELDWGT WtB=HCDs weighted by FIELDWGT WtJ=LFAs weighted by FIELDWGT a) Adjusted mean for control group= ADJMEANC= WtBJC(GDEPVAR) - WtB(COEFOFB * MEANOFB) - WtJ(COEFOFJ * MEANOFJ) Weighted Weighted Weighted Weighted Weighted site mean HCD impact= proportion LFA impact= proportion of dependent coefficient of HCDs in coefficient of LFAs in variable of B from the sample= of J from the sample= from regression Mean of B regression Mean of J MEANS output from output from output MEANS MEANS output output b) Adjusted mean for HCD group= ADJMEANB = ADJMEANC + WtB(COEFOFB) c) Adjusted mean for LFA group= ADJMEANJ = ADJMEANC + WtJ(COEFOFJ) IMPORTANT !!!: FIELDWGT takes on values of 1 or higher (see N2PSSAMP.TXT), so the weighted sample sizes will be larger than the actual sample sizes. When estimating program impacts in SAS (used by MDRC and Child Trends), the WEIGHT variable does not change degrees of freedom or number of observations in calculations of statistical significance. However, researchers using a different software package or running a different procedure in SAS (such as a crosstabulation with chi square) should first check whether weighting by FIELDWGT will inappropriately increase the chances of finding a statistically significant program-control group difference. If so, researchers should multiply FIELDWGT by a second weight that returns the sample sizes to the unweighted number. The weight is equal to the unweighted sample size for a site divided by the weighted sample size. When all members of the 2-Year Client Survey sample are included, the 2nd weight shown below can be applied. However, the 2nd weight should be recalculated when estimating program impacts for subgroups, because the weighted and unweighted sample sizes will be different. (2) (3) (1) SAMPLE SIZE 2ND UNWEIGHTED WEIGHTED BY WEIGHT SAMPLE SIZE FIELDWGT (1) / (2) ATLANTA 3003 4271.77 0.70299 COLUMBUS 1094 1094.00 1.00000 DETROIT 426 426.00 1.00000 GRAND RAPIDS 1732 2171.62 0.79756 OKLAHOMA CITY 511 511.00 1.00000 PORTLAND 610 3023.44 0.20176 RIVERSIDE 2299 4839.09 0.47509 NOTE: 1) To calculate impacts for Riverside LFA, weight survey respondents by FIELDWGT, then follow the procedures for weighting the results again that are outlined in IMP_MEMO.TXT. This additional step is needed because FIELDWGT makes the background characteristics of the survey sample similar to the characteristics of all members of the Full Impact Sample who were eligible to be surveyed. In the Full Impact Sample, however, sample members determined not to need basic education are overrepresented among LFAs and control group members (see RES_MEMO.TXT). An additional weight must be applied to make the results generalizable to the welfare population who were required to participate in a welfare-to-work program in Riverside during the early to mid-1990s. 2) Some outcomes on the survey have missing values. Researchers should be careful about grouping together outcomes with different sample sizes when calculating program impacts. Statistical packages like SAS typically use LISTWISE deletion as the default for regression or GLM, in which case only sample members with no missing values on all dependent variables will be included in the calculations. 3) Researchers will obtain slightly different impact results from those displayed in report tables. (The tables on this CD were copied from Evaluating Alternative Welfare-to-Work Approaches). These differences result from small changes to some background characteristics measures used as covariates in the impact regression model. MDRC implemented these changes after work for the 2-year impact report was completed. Changes include (1) updating of pre-random assignment earnings and welfare payment data (used as covariates) for some sample members, following receipt of more recent source files; and (2) small changes made to values of these and other covariates to protect sample members' confidentiality.