Skip to main content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

National Evaluation of Welfare-to-Work Strategies: 2-Year Client Survey Files: Estimating program impacts with 2-year client survey data

        ESTIMATING PROGRAM IMPACTS WITH 2-YEAR CLIENT SURVEY DATA  As with administrative records data, MDRC used OLS regression (PROC REG in SAS) to estimate program effects on survey outcomes.  The same covariates were used with each data source to control for differences among research groups in background characteristics.  Before running procedures to estimate impacts, researchers should merge the survey data by IDNUMBER with the research group dummy variables and covariates from the Full Impact Sample file. Use SRV2RESP to select survey respondents.  With two exceptions, MDRC used the same procedures for estimating program impacts with survey data as were used for estimating impacts with administrative records (see IMP_MEMO.TXT).  The exceptions are:  1) In Atlanta, Grand Rapids, Portland, and Riverside, certain subgroups were oversampled for research purposes when choosing the survey sample. Therefore, it is necessary to weight the survey sample to make impact estimates generalizable to the larger "survey eligible" sample from which respondents were drawn.  The variable called FIELDWGT is used to weight the survey sample (see N2PSSAMP.TXT for details). FIELDWGT should also be used when calculating adjusted means (see IMP_MEMO.TXT for details).   For example, in Atlanta and Grand Rapids, if  WtBJC=All sample members weighted by FIELDWGT  WtB=HCDs weighted by FIELDWGT  WtJ=LFAs weighted by FIELDWGT  a) Adjusted mean for control group=  ADJMEANC=  WtBJC(GDEPVAR) -  WtB(COEFOFB  * MEANOFB) -   WtJ(COEFOFJ  * MEANOFJ)  Weighted          Weighted        Weighted      Weighted      Weighted site mean         HCD impact=     proportion    LFA impact=   proportion of dependent      coefficient     of HCDs in    coefficient   of LFAs in variable          of B from       the sample=   of J from     the sample= from              regression      Mean of B     regression    Mean of J MEANS             output          from          output        from output                            MEANS                       MEANS                                   output                      output   b) Adjusted mean for HCD group=  ADJMEANB        =       ADJMEANC      +     WtB(COEFOFB)     c) Adjusted mean for LFA group=  ADJMEANJ        =       ADJMEANC      +      WtJ(COEFOFJ)   IMPORTANT !!!: FIELDWGT takes on values of 1 or higher (see N2PSSAMP.TXT), so the weighted sample sizes will be larger than the actual sample sizes.  When estimating program impacts in SAS (used by MDRC and Child Trends), the WEIGHT variable does not change degrees of freedom or number of observations in calculations of statistical significance. However, researchers using a different software package or running a different procedure in SAS (such as a crosstabulation with chi square) should first check whether weighting by FIELDWGT will inappropriately increase the chances of finding a statistically significant program-control group difference.  If so, researchers should multiply FIELDWGT by a second weight that returns the sample sizes to the unweighted number. The weight is equal to the unweighted sample size for a site divided by the weighted sample size.  When all members of the 2-Year Client Survey sample are included, the 2nd weight shown below can be applied.  However, the 2nd weight should be recalculated when estimating program impacts for subgroups, because the weighted and unweighted sample sizes will be different.                                          (2)              (3)                        (1)          SAMPLE SIZE         2ND                     UNWEIGHTED      WEIGHTED BY         WEIGHT                     SAMPLE SIZE     FIELDWGT            (1) / (2)        ATLANTA          3003           4271.77           0.70299       COLUMBUS         1094           1094.00           1.00000       DETROIT           426            426.00           1.00000       GRAND RAPIDS     1732           2171.62           0.79756       OKLAHOMA CITY     511            511.00           1.00000       PORTLAND          610           3023.44           0.20176       RIVERSIDE        2299           4839.09           0.47509   NOTE:  1) To calculate impacts for Riverside LFA, weight survey respondents by FIELDWGT, then follow the procedures for weighting the results again that are outlined in IMP_MEMO.TXT.  This additional step is needed because FIELDWGT makes the background characteristics of the survey sample similar to the characteristics of all members of the Full Impact Sample who were eligible to be surveyed.  In the Full Impact Sample, however, sample members determined not to need basic education are overrepresented among LFAs and control group members (see RES_MEMO.TXT). An additional weight must be applied to make the results generalizable to the welfare population who were required to participate in a welfare-to-work program in Riverside during the early to mid-1990s.  2) Some outcomes on the survey have missing values. Researchers should be careful about grouping together outcomes with different sample sizes when calculating program impacts.  Statistical packages like SAS typically use LISTWISE deletion as the default for regression or GLM, in which case only sample members with no missing values on all dependent variables will be included in the calculations.   3) Researchers will obtain slightly different impact results from those displayed in report tables. (The tables on this CD were copied from Evaluating Alternative Welfare-to-Work Approaches). These differences result from small changes to some background characteristics measures used as covariates in the impact regression model. MDRC implemented these changes after work for the 2-year impact report was completed. Changes include (1) updating of pre-random assignment earnings and welfare payment data (used as covariates) for some sample members, following receipt of more recent source files; and (2) small changes made to values of these and other covariates to protect sample members' confidentiality.