Families on TANF in Illinois:  Employment Assets and Liabilities

Appendix A.
Survey and Data Weighting Methods

[ Main page of Report | Contents of Report ]

Contents

• Survey Methodology
• Comparison of Survey Respondents and Nonrespondents
• Data Weighting Procedure
• Assessment of the Weighted Survey Respondents

Endnotes

This appendix describes the methodology for the telephone survey of 416 TANF clients in Illinois and the procedures that were used to develop weights for the resultant data. Mathematica Policy Research (MPR) conducted the survey from November 2001 through February 2002.

Survey Methodology

In this section we present the methods that were used to design and conduct the client survey. We discuss the design of the survey sample, the survey instrument, data collection and processing, and the survey completion rates.

Sample Design

The sampling frame for this survey consisted of single-parent TANF cases in Illinois in November 2001. More specifically, the sampling frame consisted of TANF cases that, according to administrative records of the Illinois Department of Human Services (DHS):

1. Were classified as "single-parent"
2. Were authorized to receive a cash grant during the routine benefit issuance cycle for November 2001; however, 9 percent of the cases in the sampling frame received a $0 grant⁽¹⁾
3. Included the grantee as a case member, thus excluding "child-only" cases
4. Had a TANF status of "active" or "suspended" when the sampling frame was identified, thus excluding cases whose status had changed to "cancelled" after benefits were issued but before the sampling frame was identified⁽²⁾

These criteria were satisfied by 33,495 TANF cases. They constituted the sampling frame for the survey of families on TANF in Illinois.

We implemented a simple stratified sample design. There were just two strata, defined by whether a case was located in Cook County or "downstate" (all other counties). A key objective for this study by Illinois DHS was that the survey data support the description of the statewide TANF caseload, as opposed to supporting separate descriptions for Cook County and downstate. Consistent with that objective, the probability of selection of a case from the frame into the sample was designed to be uniform across the two strata. We selected 532 cases into the sample, of which 431 were located in Cook County and 101 were located downstate. We attempted to interview every case in the sample and succeeded in interviewing 416 (78 percent) of them.

Survey Instrument and Pretest

We developed the survey instrument in consultation with the Office of the Assistant Secretary for Planning and Evaluation (ASPE) in the U.S. Department of Health and Human Services under a separate task-order agreement. The instrument was designed for either paper and pencil administration or computer assisted telephone interviewing (CATI) and was designed to take 35 minutes. Several questions were taken from the Michigan Women's Employment Survey (WES, Wave 2) and MPR's Nebraska Client Barriers Survey. Specific scales covering the topics of learning disabilities, mental health and depression, alcohol and drug abuse, and domestic violence were taken from Washington State's Learning Disabilities screener, the Composite International Diagnostic Interview (CIDI), and the Conflict Tactics Scale, respectively. In addition, we developed a series of questions to assess job readiness skills in collaboration with researchers at the University of Colorado.

We drafted the survey instrument between August and October of 2001 and pretested it in early November. The pretest interviews were conducted with the heads of ten TANF cases in Illinois who had received a cash benefit in October. Those interviews averaged 40 minutes in length. The goals of the pretest were to: (1) identify ways to improve the administration procedures, (2) measure the length of the survey, (3) test the flow and sequencing of questions, (4) clarify question wording for the interviewees, and (5) clarify instructions for the interviewers. Based on information obtained from the pretests through debriefings with the interviewers and through the monitoring of interviews by supervisory staff, we made minor modifications to the newly developed job readiness questions.

Data Collection

Survey data collection began on November 19, 2001 and continued through March 3, 2002--a field period of 16 weeks. At the outset, our target interview completion rate was 75 percent and we exceeded that goal by three percentage points. The interviews averaged 42 minutes in length, all interviews were conducted by telephone using a hard-copy survey instrument, and no in-person follow-up was employed on this study.

Immediately prior to the commencement of interviewing, we held an eight-hour interviewer training session spread over two days, November 14^th and 15^th 2001, led by the survey director. In attendance were the survey director's assistant, the telephone supervisor, the locating supervisor, the telephone interviewers, and the quality control monitors.

We contacted sample members by mail and by telephone to participate in the survey. DHS administrative records were the source of the initial addresses and telephone numbers. We mailed advance letters to all sample members prior to the first telephone contact. The letters introduced the study, identified the study sponsor and MPR, and invited the sample members to call our toll-free telephone number and participate in the survey at their earliest convenience. The letter explained that participation was voluntary and that the identities and responses of all participants would be kept confidential. It offered sample members $35 if they would call and complete the survey within two weeks of receiving the letter. Otherwise they would receive $20 for completing the survey after that.

Our next step was to call the sample members. We timed the first telephone calls to begin after sample members received the advance letter. This resulted in a number of completed interviews and also helped us to identify the sample members with either no phone number or for whom the number from DHS records was incorrect and would require additional searching efforts. In addition, the advance letters served to identify cases that required additional searching. Some of the advance letters were returned to us if the addresses that we had obtained from DHS records were out of date. Advance letters that were returned with forwarding addresses marked on the envelopes were remailed to the new addresses. Advance letters that were returned without a forwarding address required additional searching.

Our principal searching effort consisted of running identifying information (name, date of birth, last known address and phone number) for sample members through a database owned by Lexis-Nexis, a personal database search company. That generated some new addresses and phone numbers to which we then sent letters or called. We also obtained updated contact information for some sample members through a search of DHS records that we conducted approximately halfway through the field period.

Throughout the 16-week field period we continued mailing letters and postcards to sample members with whom we had not completed interviews. The format and content of the letters and postcards changed every few weeks, as well as the size and appearance of the envelope and method of mailing (first-class mail versus priority mail). This was done to spark the sample members' interest in reading the items sent. However, the most salient information remained the same in each version of the letter or postcard.

A small number of sample members initially refused to participate in the survey. For these cases, we waited approximately one month from the telephone contact in which the refusal occurred and then mailed them a specially crafted letter. The letter reiterated the importance of the study and of their participation. It again invited them to call our toll-free telephone number to participate and reminded them that we would pay them $35 if they completed the interview. We waited until we were confident that a sample member had received the letter, and then a specially trained "refusal-conversion" interviewer called to attempt to gain his or her cooperation. If the result of these steps was a second refusal, we ceased attempts to contact the sample member until the end of the field period. At that time, we sent out a final mailing to all sample members who had not completed an interview to alert them that the study was ending and to offer an increased incentive of $50 for participating.

Data Preparation

As interviews were completed, they were reviewed for completeness, consistency, and accuracy. Based on guidelines developed by MPR, interviewers called back respondents to obtain information or to clarify contradictory answers. Reviewers back-coded "other-specify" responses to prelisted choices where appropriate, or assigned new codes if responses were common enough to warrant the additions. They also assigned numeric codes to open-ended questions and to industry and occupation responses using standard coding manuals.⁽³⁾

After the completed interviews had been reviewed and coded, they were sent through the data entry process. A customized data entry program restricted entries to values that were consistent with the skip patterns in the survey instrument and were within allowable ranges. The data were entered two times by different people to verify that they had been entered correctly. After data entry was verified, frequencies for all data elements were produced and reviewed for inconsistencies and out-of-range values. Questionable data were reconciled based on review of the source data and, in some cases, on callbacks to sample members. Following this process, a final data file was produced and turned over to MPR's Research Division for further processing and analysis.

Sample Disposition and Survey Response Rate

We completed interviews with the TANF grantees in 416 of the 532 sampled cases, for an overall survey response rate of 78 percent. Of the completed interviews, 335 were with TANF clients from Cook County and 81 were with clients from downstate. Only two percent of the sample members refused to participate in the survey and only one sample member failed to complete an interview after starting. Table A.1 shows the final survey disposition of all cases in the sample by Cook County, downstate, and combined.

Two factors accounted for almost 90 percent of nonresponse to this survey. Nearly half of the nonresponse occurred because sample members could not be located (10.5 percent of all sample members). These were sample members whose addresses and phone numbers, as provided by Illinois DHS, were incorrect and we were unable to locate them by other means, such as searching through various databases for contact information and using the forwarding addresses provided by the U.S. Postal Service on letters returned to us after we had mailed them to sample members. Forty percent of nonresponse (8.6 percent of all sample members) occurred because the sample members were never available to participate in the survey. We believe that our contact information for these sample members was good, but they did not call us in response to our letters, did not answer the telephone in response to our calls, and were not available to take our calls when another household member answered the telephone.

A language barrier--lack of proficiency with English or Spanish--resulted in only one case of nonresponse to the survey (0.2 percent of all sample members). There were very few Spanish-only members of the survey sample and we chose to interview them in their native language so they could be included in the study. We conducted three interviews in Spanish, all with the same interviewer, who is a native Spanish speaker. That interviewer participated in the initial interviewer training session and completed roughly 40 interviews in English prior to conducting interviews in Spanish. She translated the instrument on her own and used the same translation for all three interviews. A native Spanish speaker monitored these interviews.

[Go To Contents]

Comparison of Survey Respondents and Nonrespondents

In a survey that achieves less than a 100 percent response rate there is a risk that respondents may be systematically different from nonrespondents. Such differences would imply that the respondents should not be regarded as a random subsample of the full survey sample. If the survey data are not adjusted to mitigate these differences, such as by weighting the survey respondents, it may be inappropriate to draw inferences about the sampling frame from statistics computed on the basis of the survey data.

To assess whether the sample members who responded to the survey are different from those who did not respond, we regressed each of 12 characteristics of the case or case head (i.e., the TANF grantee) on a 0/1 variable that indicates the sample member's response status: 1 for respondents and 0 for nonrespondents.⁽⁴⁾ The regression coefficient on the indicator variable is the difference between the two groups in a characteristic. We conducted a t-test to determine whether the estimate of the coefficient is significantly different from zero for each of the 12 characteristics. The results of that analysis are presented in Table A.2.

Table A.2 shows significant differences between survey respondents and nonrespondents in only three of the twelve selected characteristics. Cases with grantees who were 28 years old or less were more likely to participate in the survey than were cases with older grantees. Consistent with that difference, respondent cases tended to have fewer children and to have been on TANF for fewer consecutive months than nonrespondent cases.

To reduce the bias that systematic differences between survey respondents and nonrespondents might introduce to inferences based on the survey data, we developed survey weights that directly correct for the age difference. The development of the weights is described in the next section. We expected the weights to indirectly mitigate differences in age-related characteristics, such as the number of children on the case and the duration of the current TANF spell. Those expectations were realized, as discussed in the final section of this appendix.

[Go To Contents]

Data Weighting Procedure

We computed weights for the survey respondents using three factors, as summarized in Table A.3 and discussed below. The product of these three factors for a given survey respondent is the final sampling weight for that case.

Component 1: Sample Weight

The sample weight for each case in the survey sample accounts for the number of cases it represents in the sampling frame, based on the sample selection procedure. As indicated earlier, we designed a stratified sample in which the theoretical sampling rate was constant across the two strata, Cook County and downstate. Because we had to select whole cases, we actually sampled Cook County cases at a very slightly higher rate than downstate cases. The sample weight is the inverse of the actual probability of selection. For sampled cases in Cook County and downstate, respectively, its value is 62.93 and 63.09.

Component 2: Survey Response Adjustment

This component compensates for the reduction in the sample due to cases that could not be interviewed for the reasons given in Table A.1. It is the inverse of the survey response rate. Because the response rate was slightly lower in Cook County than downstate, the value of this component is slightly higher in Cook County (1.28) than downstate (1.25).

Component 3: Post-Stratification Adjustment

This component of the survey weights is based on a post-stratification of the survey respondents into five cells as shown in Table A.4. This factor causes the sum of the weighted survey respondents to equal the number of cases in the sampling frame in each cell. The five cells were defined by three variables that were extracted from the Illinois DHS administrative data system in November 2001: residence in Cook County or downstate, the grantee's age less than or equal to 28 years or greater than 28 years, and a zero or positive TANF benefit amount. While in principle, these variables could be used to define eight cells, the infrequency of zero-benefit cases led us to consolidate them in a single cell. The values of the post-stratification adjustment factor range from 0.86 to 1.25. In general, the larger values are for cells containing cases with older grantees, which had a lower survey response rate than cases with younger grantees.

Final Survey Weights

The final weights for the survey respondents are the product of the three components discussed above. There is a unique weight for each of six cells, ranging in value from 69.17 to 98.24.⁽⁵⁾

[Go To Contents]

Assessment of the Weighted Survey Respondents

We assessed the representativeness of the weighted sample of survey respondents by comparing them with the entire 33,495 cases in the sampling frame. The comparisons were based on the same 12 characteristics that we used to test for differences between survey respondents and nonrespondents. Those include demographic characteristics of grantees and cases as of November 2001, the TANF benefit amount in November 2001, and employment and earnings in 2000 and 2001. Characteristics that were expressed as percentages in the comparison of survey respondents and nonrespondents are expressed as case counts in this analysis. Thus, we are comparing the weighted count of survey respondents that possess the selected characteristic with the count of cases in the sampling frame that possess the characteristic. The results of the tests are presented in Table A.5.

Table A.5 shows a high degree of correspondence between the weighted survey respondents and all of the cases in the sampling frame. For ten of the twelve characteristics considered the weighted count of survey respondents differs from the count of all cases in the sampling frame by 2 percent or less. Most notably, the difference with respect to the grantee characteristic, "age 28 or younger," is just one percent. Recall that this is the characteristic on which survey respondents and nonrespondents differ most sharply. The survey weights virtually eliminate the effect of nonrandomness in survey response along this dimension.

The survey weights mitigate, but fall short of eliminating, the effects of nonrandomness in survey response with respect to two characteristics of TANF cases: the number of children per case and the duration of the current spell on TANF. For both of these characteristics the error rate in the sample of weighted respondents is 5 percent relative to all cases in the sampling frame. These error rates, while not trivial, are less than half of the relative difference in these characteristics between survey respondents and nonrespondents.

To summarize, the weighted survey respondents resemble very closely the cases in the sampling frame. This correspondence is a consequence of a high survey response rate, the general absence of large systematic differences between survey respondents and nonrespondents, and the survey weights. Consequently, we can be confident in drawing inferences about the sampling frame from statistics computed on the basis of the survey data.

[Go To Contents]

Endnotes

1. The routine TANF benefit issuance cycle for November 2001 extended from October 19 to November 8. The sampling frame was identified upon the completion of benefit issuance on November 8. Four factors accounted for 95 percent of the zero benefit cases: recoupment of prior overpayments, failure to cooperate with eligibility determination, participation in Illinois' Work First program, and sanctioning.

2. A case whose November TANF benefit was processed prior to November 8, 2001, may have closed by the time the sampling frame was identified following completion of routine benefit processing on November 8. We excluded cases such as this (recent TANF leavers) from the sampling frame in order to maximize the proportion of survey respondents that were current TANF recipients when interviewed.

3. For coding industry responses we used the 1987 Standard Industrial Classification Manual. For coding occupation responses we used the 2000 Standard Occupational Classification Manual.

4. The characteristics were obtained from TANF administrative records for November 2001 maintained by the Illinois DHS and from Unemployment Insurance earnings records for the years 2000 and 2001 maintained by the Illinois Department of Employment Security.

5. The six cells and their associated survey weights are: 1) zero TANF benefit, Cook County, 95.96; 2) zero TANF benefit, downstate, 93.48; 3) positive TANF benefit, Cook County, less than or equal to age 28, 69.17; 4) positive TANF benefit, Cook County, greater than age 28, 93.60; 5) positive TANF benefit, downstate, less than or equal to age 28, 70.38; 6) positive TANF benefit, downstate greater than age 28, 98.24.

Where to?

Top of Page | Contents

Main Page of Report | Contents of Report

Home Pages:
Human Services Policy (HSP)
Assistant Secretary for Planning and Evaluation (ASPE)
U.S. Department of Health and Human Services (HHS)