National Evaluation of Welfare-to-Work Strategies: 2-Year Client Survey Files: Background information on estimating program impacts with survey data

09/10/2001

       ESTIMATING PROGRAM IMPACTS WITH 2-YEAR CLIENT SURVEY DATA

As with administrative records data, MDRC used OLS regression (PROC REG in SAS)
to estimate program effects on survey outcomes.  The same covariates were
used with each data source to control for differences among research groups
in background characteristics.

Before running procedures to estimate impacts, researchers should merge the
survey data BY IDNUMBER with the research group dummy variables and
covariates from the Full Impact Sample file. Use SRV2RESP to select survey
respondents.

With two exceptions, MDRC used the same procedures for estimating program
impacts with survey data as were used for estimating impacts with
administrative records  (See IMP_MEMO.TXT).

The exceptions are:

1) In Atlanta, Grand Rapids, Portland, and Riverside, certain subgroups
were oversampled for research purposes when choosing the survey sample.
Therefore, it is necessary to weight the survey sample to make impact estimates
generalizable to the larger "survey eligible" sample from which respondents were
drawn.

The variable called FIELDWGT is used to weight the survey sample. (See
S2SAMPLE.TXT for details).  FIELDWGT should also be used when calculating
adjusted means.  (See IMP_MEMO.TXT for details)


For example, in Atlanta and Grand Rapids, if

WtBJC=All sample members weighted by FIELDWGT

WtB=HCDs weighted by FIELDWGT

WtJ=LFAs weighted by FIELDWGT

a) Adjusted mean for control group=

ADJMEANC=

WtBJC(GDEPVAR) -  WtB(COEFOFB  * MEANOFB) -   WtJ(COEFOFJ  * MEANOFJ)

Weighted          Weighted        Weighted      Weighted      Weighted
site mean         HCD impact=     proportion    LFA impact=   proportion
of dependent      coefficient     of HCDs in    coefficient   of LFAs in
variable          of B from       the sample=   of J from     the sample=
from              regression      Mean of B     regression    Mean of J
MEANS             output          from          output        from
output                            MEANS                       MEANS
                                  output                      output


b) Adjusted mean for HCD group=

ADJMEANB        =       ADJMEANC      +     WtB(COEFOFB)




c) Adjusted mean for LFA group=

ADJMEANJ        =       ADJMEANC      +      WtJ(COEFOFJ)


IMPORTANT !!!: FIELDWGT takes on values of 1 or higher (See S2SAMPLE.TXT),
so the weighted sample sizes will be larger than the actual sample
sizes.  When estimating program impacts in SAS (used by MDRC and Child
Trends), the WEIGHT variable does not change degrees of freedom or number
of observations in calculations of statistical significance.   However,
researchers using a different software package or running a
different procedure in SAS (such as a crosstabulation with chi
square) should first check whether weighting by FIELDWGT
will inappropriately increase the chances of finding a statistically
significant program-control group difference.  If so, researchers should
multiply FIELDWGT by a second weight that returns the sample sizes to the
unweighted number. The weight is equal to the unweighted sample size for a
site divided by the weighted sample size.

When all members of the 2-Year Client Survey sample are included, the 2nd weight
shown below can be applied.  However, the 2nd weight should be recalculated
when estimating program impacts for subgroups, because the weighted and
unweighted sample sizes will be different.


                                       (2)              (3)
                       (1)          SAMPLE SIZE         2ND
                    UNWEIGHTED      WEIGHTED BY         WEIGHT
                    SAMPLE SIZE     FIELDWGT            (1) / (2)

      ATLANTA          3003           4271.77           0.70299
      COLUMBUS         1094           1094.00           1.00000
      DETROIT           426            426.00           1.00000
      GRAND RAPIDS     1732           2171.62           0.79756
      OKLAHOMA CITY     511            511.00           1.00000
      PORTLAND          610           3023.44           0.20176
      RIVERSIDE        2299           4839.09           0.47509


NOTE:  To calculate impacts for Riverside LFA, weight survey respondents by
FIELDWGT, then follow the procedures for weighting the results again that are
outlined in IMP_MEMO.TXT.  This additional step is needed because FIELDWGT
makes the background characteristics of the survey sample similar to the
characteristics of all members of the Full Impact Sample who were eligible to
be surveyed.  In the Full Impact Sample, however, sample members determined
not to need basic education are overrepresented among LFAs and control group
members.  (See RES_MEMO.TXT) An additional weight must be applied to make
the results generalizable to the welfare population who were required to
participate in a welfare-to-work program in Riverside during the early to
mid-1990s.



2) Some outcomes on the survey have missing values.  Researchers should be careful
about grouping together outcomes with different sample sizes when calculating
program impacts.  Statistical packages like SAS typically use LISTWISE deletion
as the default for regression or GLM, in which case only sample members with
no missing values on all dependent variables will be included in the calculations.