National Evaluation of Welfare-to-Work Strategies: Guide to Using the Two-Year Full Impacts Restricted Access Files

09/14/2001

This memo briefly describes the restricted access analysis files and accompanying  documentation for research on the two-year effects of 11 welfare-to-work programs that were operated during the early- to mid-1990s in seven sites: Atlanta, Georgia; Columbus, Ohio; Detroit and Grand Rapids, Michigan; Oklahoma City, Oklahoma; Portland, Oregon; and Riverside, California.

The files contain the sample and outcome measures analyzed for three reports published in 2000 by the U.S. Department of Health and Human Services, Office of the Assistant Secretary for Planning and Evaluation and Administration for Children and Families; and U.S. Department of Education, Office of the Under Secretary and Office of Vocational and Adult Education:

  1. Evaluating Alternative Welfare-to-Work Approaches: Two-Year Impacts for Eleven Programs.
  2. Impacts on Young Children and Their Families Two Years After Enrollment: Findings from the Child Outcomes Study.
  3. Do Mandatory Welfare-to-Work Programs Affect the Well-Being of Children? A Synthesis of Child Research Conducted as Part of the National Evaluation of Welfare-to-Work Strategies.

Please see the NEWWS web site (http://aspe.hhs.gov/hsp/NEWWS/index.htm) for copies of the executive summaries . For copies of the full reports of Evaluating Alternative Welfare-to-Work Approaches: Two Year Impacts for Eleven Program and Impacts on Young Children and Their Families Two Years After Enrollment: Findings from the Child Outcomes Studay, please contact Ethel Norris at ethel.norris@hhs.gov.

The reports and restricted access files were prepared by the Manpower Demonstration Research Corporation (MDRC) and Child Trends as part of the National Evaluation of Welfare-to-Work Strategies (formerly known as the JOBS Evaluation). MDRC conducted the NEWWS Evaluation under a contract with the U.S. Department of Health and Human Services (HHS), funded by HHS under a competitive award, Contract No. HHS-100-89-0030. HHS also received funding for the evaluation from the U.S. Department of Education. The study of one of the sites in the evaluation, Riverside County (California), is also conducted under a contract from the California Department of Social Services (CDSS). CDSS, in turn, is receiving funding from the California State Job Training Coordinating Council, the California Department of Education, HHS, and the Ford Foundation. As part of the NEWWS Evaluation, Child Trends conducted the Child Outcomes Study under subcontract to MDRC.

The data are stored in 3 ASCII files, corresponding to the three main research samples:

N2RI1213.TXT is the Full Sample Impact file. It contains background characteristics data and outcomes calculated with administrative records for all members of the impact sample in 7 sites (N = 44,569). It is organized into 9 records:

  1. DEMOGRAPHIC CHARACTERISTICS
  2. SAMPLES
  3. PRIVATE OPINION SURVEY
  4. BASELINE TEST SCORES
  5. WELFARE PAYMENTS
  6. FOOD STAMPS
  7. INCOME
  8. EARNINGS
  9. COVARIATES

Note: some data on this file are only available for subsamples. See codebooks and other documentation for details. Nonetheless, all 9 records are included for all sample members, even if data were not collected for them. These records will have blanks in fields when data were not recorded.

N2RS1221.TXT contains outcome measures from the 2-Year Client Survey in 7 sites (N = 9,675). It is organized into 18 records:

  1. INTERVIEW DATE, LENGTH, MODE
  2. PARTICIPATION: JOB SEARCH, BASIC EDUCATION, AND HIGH SCHOOL
  3. PARTICIPATION: POST-SECONDARY EDUCATION, VOCATIONAL TRAINING, WORK EXPERIENCE, OJT, ANY ACTIVITY
  4. PARTICIPATION: DATE AND HOURS MISSING DATA FLAGS
  5. ADDITIONAL BASIC EDUCATION VARIABLES
  6. DEGREE RECEIPT
  7. EMPLOYMENT: SOURCE VARIABLES
  8. EMPLOYMENT: CREATED VARIABLES
  9. EMPLOYMENT: DATA PROBLEM FLAGS
  10. CHILD CARE FOR EMPLOYMENT
  11. HOUSEHOLD COMPOSITION
  12. INCOME FOR HOUSEHOLD MEMBERS 1-9 Y/N VARIABLES
  13. INCOME AMOUNTS FOR HOUSEHOLD MEMBERS 1-9 $ VARIABLES
  14. INCOME SUMMARY VARIABLES
  15. INCOME PROBLEM FLAGS
  16. TRANSITIONAL AND NON-CASH BENEFITS AND HEALTH CARE COVERAGE
  17. CHILD OUTCOMES (ALL CHILDREN)
  18. ATTITUDES TOWARD WORK AND WELFARE AND INFORMATION ON SANCTIONS

Note: the Client Survey sample is nested within the Full Impact sample.

N2RC1326.TXT contains outcome measures from the 2-Year Child Outcomes Study (COS) Survey in 3 sites (N = 3,018). It is organized into 14 records:

  1. OUTCOMES
  2. TARGETED INTERVENING MECHANISMS
  3. NON-TARGETED INTERVENING MECHANISMS
  4. HISTORY OF CHILD CARE ARRANGMENTS FOR FOCAL CHILD (SECTION AA)
  5. CURRENT CHILD CARE ARRANGEMENTS FOR FOCAL CHILD (SECTION BB)
  6. CHILD SUPPORT FOR FOCAL CHILD (SECTION CC)
  7. FOCAL CHILD'S HEALTH AND HEALTH CARE (SECTION DD)
  8. PARENTING OF FOCAL CHILD (SECTION EE)
  9. SELF-ADMINISTRED QUESTIONAIRE, PARTS 1 AND 2, PARENTING (SECTION FF)
  10. SELF-ADMINISTRED QUESTIONAIRE, PART 3 DEPRESSIVE SYMPTOMS (SECTION GG)
  11. SELF-ADMINISTRED QUESTIONAIRE, PART 4, CHILD POSITIVE AND PROBLEM BEHAVIOR (SECTION: HH)
  12. INTERVIEWER ASSESSMENT
  13. SUBGROUPS
  14. COVARIATES

Note: the Child Outcomes Study sample is nested within the Client Survey and Full Impact samples. Codebooks, file layouts, and other key documentation are stored with the data files. For details, see:

As noted above, the two survey samples are nested within the full impact sample. All sample members who responded to the 2-Year Client survey and Child Outcomes Study surveys have background characteristics and administrative records data on the Full Sample Impact file. Likewise, all sample members who responded to the Child Outcomes Study Survey have additional survey data on the 2-Year Client Survey file.

Each record of each file contains a unique sample member IDNUMBER, which varies from 1 to 44569. For any sample member, the same IDNUMBER is used on each file. Researchers may build one or several analysis files, depending on their research interests, by identifying the samples (and records) they wish to study and then merging files BY IDNUMBER.

The key subsamples are identified on the Full Sample Impact file SAMPLES record. They having a value of 1 on the following variables:

  • FULLSAMP: Full impact sample
  • SRV2RESP: 2-Year Survey respondent
  • COS2RESP: 2-Year Client Outcomes Study respondent
  • POS: Completed Private Opinion Survey at baseline
  • TESTSAMP: Competed literacy and/or math test at baseline

The research design for the NEWWS Evaluation is complicated. So are the sample selection strategies for the 2-Year Client and Child Outcomes Study surveys. We strongly suggest that users of these files do the following before conducting any further analyses:

  1. Read the _README files which give a brief description of all files included in this set.
  2. Read the reports, including chapters which describe the research designs, samples, and data sources.
  3. Review the codebooks, file layouts, output, annotated tables, and memos.
  4. After reading the data into SAS or another statistical or econometric software package, replicate the sample sizes and means.

IMPORTANT:

It is also suggested that researchers estimate program impacts using the same dependent variables, independent variables, and procedures as MDRC used for the Full Impact sample (see IMP_MEMO.TXT on CD #1) and 2-Year Client Survey sample (see SIMPMEMO.TXT on CD #2) and Child Trends used for the Child Outcomes Study sample (See C2COVER.TXT on CD #3). But with the following caveat:

The annotated tables included with the file documentation are from the published two-year reports and are based on calculations from a data file created in 1997. However, the restricted access file was created in 2001, using the most recent files at MDRC. In a small number of cases, data updates (from statewide UI systems and state and county welfare and Food Stamp payment systems) were received after the two-year report impacts were calculated. These updates, which represent the most accurate data, are included in the two-year restricted access files. As a result, there are small discrepancies between the two-year impacts calculated with data from the two-year restricted access files and those published in the two-year reports.

The following table shows the differences in impacts for four key economic outcomes:

Differences in Impacts Calculated for Two-Year Impact Report
and Calculated with Restricted Access File Data

                                                           Impact
                                                 Impact    from
                                                 from      Restricted    Difference
                                                 2-Year    Access        in
                                                 Report    File          Impacts

Site and Program                              Ever employed in years 1 and 2 (%)

Atlanta Labor Force Attachment                   4.5         4.5            0.0
Atlanta Human Capital Development                2.8         2.7            0.1

Grand Rapids Labor Force Attachment              7.6         7.7           -0.1
Grand Rapids Human Capital Development           5.3         5.4           -0.1

Riverside Labor Force Attachment                15.1        15.1            0.0
  Lacked high school diploma or basic           16.6        16.5            0.1
Riverside Human Capital Development              9.3         9.4           -0.1

Columbus Integrated                              1.7         1.7            0.0
Columbus Traditional                             1.3         1.3            0.0

Detroit                                          4.1         4.1            0.0

Oklahoma City                                   -0.9        -0.7           -0.1

Portland                                        11.2        11.2            0.0

                                             Total earnings in years 1 and 2 ($)

Atlanta Labor Force Attachment                   813         805              9
Atlanta Human Capital Development                496         482             14

Grand Rapids Labor Force Attachment            1,035       1,041             -7
Grand Rapids Human Capital Development           580         589             -9

Riverside Labor Force Attachment               1,276       1,276              0
  Lacked high school diploma or basic            992       1,001             -9
Riverside Human Capital Development              317         336            -19

Columbus Integrated                              673         676             -3
Columbus Traditional                             677         675              2

Detroit                                          367         370             -2

Oklahoma City                                      5          11             -6

Portland                                       1,842        1,829            12


                                               Total months of welfare receipt

Atlanta Labor Force Attachment                 -1.15       -1.14          -0.01
Atlanta Human Capital Development              -0.57       -0.56           0.00

Grand Rapids Labor Force Attachment            -2.21       -2.21           0.00
Grand Rapids Human Capital Development         -1.22       -1.22           0.00

Riverside Labor Force Attachment               -1.46       -1.46           0.00
  Lacked high school diploma or basic          -1.37       -1.37           0.00
Riverside Human Capital Development            -0.96       -0.97           0.01

Columbus Integrated                            -1.58       -1.58           0.00
Columbus Traditional                           -1.03       -1.03           0.00

Detroit                                        -0.48       -0.48           0.00

Oklahoma City                                  -0.78       -0.78           0.00

Portland                                       -2.41       -2.48           0.06

                                                Total welfare payments ($)

Atlanta Labor Force Attachment                  -369        -367             -1
Atlanta Human Capital Development               -288        -286             -1

Grand Rapids Labor Force Attachment           -1,404      -1,404              0
Grand Rapids Human Capital Development          -835        -835              0

Riverside Labor Force Attachment              -1,308      -1,305             -3
  Lacked high school diploma or basic         -1,408      -1,407             -1
Riverside Human Capital Development           -1,049      -1,055              7

Columbus Integrated                             -694        -694              0
Columbus Traditional                            -530        -529              0

Detroit                                         -158        -158              0

Oklahoma City                                   -233        -233              0

Portland                                      -1,196      -1,224             28

 

As may be seen, the impact estimates are very close. Impacts on total earnings varied by less than $20 for all programs. In addition, impacts on total welfare payments varied by less than $10, except for Portland ($28).

It should be noted that impacts on survey outcomes estimated with data from the restricted access files will also vary slightly from impacts published in the two-year reports. That is because the regression model used to estimate program impacts includes, as covariates, measures of pre-random assignment employment and earnings and pre-random assignment welfare and Food Stamp receipt and payments that were calculated with administrative records. These covariates were updated for some sample members after the tables for the reports were completed. These updates can change the value of the impact estimates slightly, even when the outcome measure was created from survey responses and was not updated.