The purpose of this paper is to review methods used to conduct telephone surveys of low-income populations. The motivation for this review is to provide information on best practices applicable to studies currently being conducted to evaluate the Personal Responsibility and Work Opportunity Reconciliation Act of 1996 (PRWORAВ hereafter referred to as Welfare Reform). The National Academy of Sciences panel observed that many of the states are conducting telephone surveys for this purpose and that it would be useful to provide them with information on the best methods for maximizing response rates. The information provided in this paper is intended to assist these individuals, as well as others, to either conduct these studies themselves or to evaluate and monitor contractors conducting the studies.
We have divided the telephone surveys into two types. The first, primary, method is to sample welfare recipients or welfare leavers from agency lists. This can take the form of a randomized experiment, where recipients are randomly assigned to different groups at intake, with a longitudinal survey following these individuals over an extended period of time. More commonly, it takes the form of a survey of those leaving welfare during a particular period (e.g., first quarter of the year). These individuals are then followed up after X months to assess how they are coping with being off welfare.
The second type of telephone survey is one completed using a sample generated by random digit dialing methods (RDD). In this type of study, telephone numbers are generated randomly. The numbers then are called and interviews are completed with those numbers that represent residential households and that agree to participate in the interview. To effectively evaluate welfare reform, this type of survey would attempt to oversample persons who are eligible and/or who are participating in welfare programs.
The issues related to these two types of telephone surveys, one from a list of welfare clients and one using RDD, overlap to a large degree. The following discussion reviews the common issues as well as the unique aspects related to each type of survey. In the next section, we discuss methods to increase response rates on telephone surveys, placing somewhat more emphasis on issues related to conducting surveys from lists of welfare clients. We chose this emphasis because this is the predominant method being used by states to evaluate welfare reform. The third section reviews a number of welfare studies that have been implemented recently. In this section we discuss how the methods that are being used match up with the best practices and how this may relate to response rates. The fourth section provides an overview of issues that are unique to RDD surveys when conducting a survey of low-income populations. To summarize the discussion, the final section highlights practices that can be implemented for a relatively low cost but that could have relatively large impacts.
Methods to Increase Response Rates
In this section we discuss the methods needed to obtain high response rates in a telephone survey. These methods include locating, contacting, and obtaining the cooperation of survey subjects. The review applies to all types of telephone surveys, but we have highlighted those methods that seem particularly important for conducting surveys from lists of welfare clients. A later section provides issues unique to RDD.
Studies of Welfare Leavers
Table 2-1 summarizes the procedures discussed previously. It is organized around the three primary activities required to conduct a study: (1) locating the subject, (2) contacting the subject, and (3) obtaining cooperation.
In this section we discuss how these best practices have been applied in a number of surveys that have been conducted to evaluate welfare reform in different states. The purpose of this review is to provide a picture of the range of practices that have been used and how these practices relate to results.
Summary of Best Practices for Conducting Telephone Surveys of Welfare Leavers
- Accurate address and telephone number
- Contact for persons not living with subject
|Collect at intake and update regularly
||Try to collect consent to search other databases
- Use other sources to locate subject
|Use available administrative databases (e.g., food stamps, Medicaid, drivers licenses); use commercially available sources (reverse directories, credit bureaus)
||Start with the least expensive methods
- Telephone tracing; in-person tracing
- In-person tracing
|Review tracing record and follow leads
||Very expensive and requires specialized skills
- Incentives and continued contact
|Send letter prior to making contact
Repeated mailings to subjects
|Use express delivery if possible
||Spread out calls over day/night; weekdays/weekends
- Interviewer training and experience
|Provide interviewers with answers to common questions
||Try to use experienced interviewers with good records
||Minimize redundant questions
Keep length as short as possible
|Pretest questions and allow for time to revise after the pretest
||Keep initial introduction as short as possible
||Prenotify with express mail and incentives
Special Issues for RDD Surveys of Low-Income Populations
In many ways, RDD surveys pose a much different set of challenges than those for list-based samples, especially on issues related to nonresponse. For surveys of welfare clients, the target population is identified clearly and quality issues have to do with finding sample members to conduct the interview. For RDD surveys, the primary issues have to do with efficiently identifying low-income subjects and, once identified, convincing them to participate in a survey.
Highlighting Low-Cost Actions
This paper has attempted to provide information on methods to achieve high response rates on telephone surveys of low-income populations. We have concentrated much of the review on studies that start with a list of welfare recipients, but we also have provided information for persons conducting RDD interviews. The second section of this paper provided a list of best practices that should be considered when conducting telephone surveys. The third section provided examples of what is currently being practiced in recently completed welfare-leaver studies and how these practices relate to results. The fourth section provided special issues related to RDD surveys. In this section we concentrate on highlighting suggestions that seem practical and relatively low cost.
1. A related concern is whether respondents are using caller ID in a similar way.
2. This rate is surprisingly low, given that probation officers should be in regular contact with probationers.
3. Most studies had, as an eligibility criteria, that leavers had to stay off the welfare program for at least 2 months. Sampling within a month of leaving the program, therefore, eventually results in having to drop subjects because they return to the program within 2 months.
4. Telephone interviews would be conducted by having the respondent call into a central facility using a cellular telephone.
- Anglin, D.A., B. Danila, T. Ryan, and K. Mantius
- 1996 Staying in Touch: A Fieldwork Manual of Tracking Procedures for Locating Substance Abusers for Follow-Up Studies. Washington, DC: National Evaluation Data and Technical Assistance Center.
- Bogen, K.
- 1996 The effect of questionnaire length on response rates: A review of the literature. In Proceedings of the Section on Survey Research Methods. Alexandria, VA: American Statistical Association.
- Brick, J.M., M. Collins, and K. Chandler
- 1997 An Experiment in Random Digit Dial Screening. Washington, DC: U.S. Department of Education, National Center for Education Statistics.
- Brick, J.M., I. Flores-Cervantes, and D. Cantor
- 1999 1997 NSAF Response Rates and Methods Evaluation, Report No. 8. NSAF Methodology Reports. Washington, DC: Urban Institute.
- Camburn, D.P., P.J. Lavrakas, M.P. Battaglia, J.T. Massey, and R.A. Wright
- 1995 Using Advance Respondent Letters in Random-Digit-Dialing-Telephone Surveys. Unpublished paper presented at the American Association for Public Opinion Research Conference; May 18-21, 1995; Fort Lauderdale, FL, May.
- Cantor, D.
- 1995 Prevalence of Drug Use in the DC Metropolitan Area Juvenile and Adult Offender Populations: 1991. Washington, DC: National Institute on Drug Abuse.
- Cantor, D., P. Cunningham, and P. Giambo
- 1999 The use of pre-paid incentives and express delivery on a Random Digit Dial Survey. Unpublished paper presented at the International Conference on Survey Nonresponse. Portland, October.
- Cantor, D., and K. Wang
- 1998 Correlates of Measurement Error when Screening on Poverty Status for a Random Digit Dial Survey. Unpublished paper presented at the meeting of American Statistical Association, Dallas, August.
- Collins, M., W. Sykes, P. Wilson, and N. Blackshaw
- 1988 Non-response: The UK experience. Pp. 213-232 in Telephone Survey Methodology, R.M. Groves, P.P. Biermer, L.E. Lyberg, J.T., Massey, W.L. Nicholls, and J. Waksberg, eds. New York: John Wiley and Sons.
- Couper, M., R.M. Groves, and R.B. Cialdini
- 1992 Understanding the decision to participate in a survey. Public Opinion Quarterly 56:475-495.
- Dillman, D.
- 1978 Mail and Telephone Surveys: The Total Design Method. New York: John Wiley and Sons.
- Dillman, D., J.G. Gallegos, and J.H. Frey
- 1976 Reducing refusal rates for telephone interviews. Public Opinion Quarterly 40:66-78.
- Everett, S.E., and S.C. Everett
- 1989 Effects of Interviewer Affiliation and Sex Upon Telephone Survey Refusal Rates. Paper presented at the Annual Conference of the Midwest Association for Public Opinion Research, Chicago.
- Flores-Cervantes, I., J.M. Brick, T. Hankins, and K. Wang
- 1999 Evaluation of the Use of Data on Interruption in Telephone Service. Unpublished paper presented at the meeting of American Statistical Association in Baltimore, August 5-12.
- Frankel, M.R., K.P. Srinath, M.P. Battaglia, D.C. Hoaglin, R.A. Wright, and P.J. Smith
- 1999 Reducing nontelephone bias in RDD surveys. Pp. 934-939 in Proceedings of the Section on Survey Research Methods. Alexandria, VA: American Statistical Association.
- Goyder, J.
- 1987 The Silent Minority, Nonrespondents on Sample Surveys. Cambridge, Eng.: Polity Press.
- Groves, R., D. Cantor, K. McGonagle, and J. Van Hoewyk
- 1997 Research Investigations in Gaining Participation from Sample Firms in the Current Employment Statistics Program. Unpublished paper presented at the Annual Meeting of the American Statistical Association, Anaheim, CA, August 10-14.
- Groves, R.M., and M.P. Couper
- 1998 Nonresponse in household interview surveys. New York: John Wiley & Sons.
- Groves, R., and N. Fultz
- 1985 Gender effects among telephone interviewers in a survey of economic attributes. Sociological Methods and Research 14:31-52.
- Groves, R.M., C. Cannell, and M. ONeil
- 1979 Telephone interview introductions and refusal rates: Experiments in increasing respondent cooperation. Pp. 252-255 in Proceedings of the Section on Survey Research Methods. Alexandria, VA: American Statistical Association.
- Keeter, S.
- 1995 Estimating telephone noncoverage bias with a telephone survey. Public Opinion Quarterly 59:196-217.
- Massey, J., D. OConnor, K. Krotki, and K. Chandler
- 1998 An investigation of response rates in random digit dialing (RDD) telephone surveys. In Proceedings of the Section on Survey Research Methods. Alexandria, VA: American Statistical Association.
- Moeller, J.F., and N.A. Mathiowetz
- 1994 Problems of screening for poverty status. Journal of Official Statistics 10:327-337.
- Oldendick, R.W., and M.W. Link
- 1994 The answering machine generation: Who are they and what problem do they pose for survey research? Public Opinion Quarterly 58:264-273.
- Sobal, J.
- 1982 Disclosing information in interview introductions: Methodological consequences of informed consent. Sociology and Social Research 66:349-361.
- Stapulonis, R.A., M. Kovac, and T.M. Fraker
- 1999 Surveying Current and Former TANF Recipients in Iowa. Unpublished paper presented at the 21st Annual Research Conference of the Association for Public Policy Analysis and Management, Washington, DC: Mathematica Policy Research, Inc.
- 1995 Answering machines hold off voice mail challenge. STORES 77(11):38-39.
- Thornberry, O.T., and J.T. Massey
- 1988 Trends in United States telephone coverage across time and subgroups. Pp. 25-50 in Telephone Survey Methodology, R.M. Groves, P.P. Biemer, L.E. Lyberg, J.T., Massey, W.L. Nicholls, J. Waksberg, eds. New York: Wiley.
- Traugott, M., J. Lepkowski, and P. Weiss
- 1997 An Investigation of Methods for Matching RDD Respondents with Contact Information for Validation Studies. Unpublished paper presented at the Annual Meeting of the American Association for Public Opinion Research, Norfolk, VA, May 15- 17.
- Tuckel, P.S., and B.M. Feinberg
- 1991 The answering machine poses many questions for telephone survey researchers. Public Opinion Quarterly 55:200-217.
- Tuckel, P., and H. ONeil
- 1996 New technology and nonresponse bias in RDD surveys. Pp. 889-894 in Proceedings of the Section on Survey Research Methods. Alexandria, VA: American Statistical Association.
- Xu, M., B. Bates, and J.C. Schweitzer
- 1993 The impact of messages on survey participation in answering machine households. Public Opinion Quarterly 57:232-237.