High response rates are critical in surveys that are part of an impact evaluation in order to minimize the potential for nonresponse to bias the impact estimates. Nonresponse may bias the impact estimate because those who do not respond to the survey may experience different program impacts from those who do respond. If nonresponse is not correlated with experimental/control status, estimated impacts are unbiased for those who do complete the survey, but not for the overall population. If nonresponse is correlated with experimental/control status (as, for instance, when experimental group members leave assistance earlier, and are then harder to locate because contact information is more out of date) then the impact estimates will be biased even for respondents.
Thus, there are two major issues:
- Should there be a minimum standard for an acceptable response rate in a follow-up survey to be used in developing impact estimates and, if so, what should the standard be, both for initial and (if applicable) later rounds of followup?
- What knowledge is there in the evaluation community concerning survey practices that are conducive to achieving and maintaining high response rates, and how can federal and state officials promote use of these practices?
In discussing these issues, we draw heavily on our experiences as working evaluators and on the insights of the expert panel convened for this project.
a. Standards for Response Rates
The Office of Management and Budget sets the standard of a minimum 80 percent response rate in surveys funded directly by the federal government. For surveys of low-income populations, achieving high response rates is a particular challenge. Low-income families tend to be more mobile, often do not have telephones, and may be suspicious of outsiders asking them questions because of concern about losing government benefits. In some areas, large subgroups of the low-income population do not speak English. However, the authors and experts consulted for this report believe that response rates in the 75 to 80 percent range are achievable with low-income populations when quality survey methods are used. Response rates may be raised to around 85 percent with ample resources for tracking and repeated interview attempts, but they rarely exceed this level.
b. Survey Methods that Promote High Response Rates
In the evaluation community, a number of factors are known to be important in obtaining high survey response rates in follow-up surveys of low-income populations:
- Initial Contact Information. Obtaining contact information from all sample members at the time of demonstration intake, before any of them can "fall through the cracks," is critical in locating respondents later.
- Updating Contact Information. It is important to update the initial contact information at least every 18 months. If the time between the initial intake and the first followup is longer, it is good practice to contact the respondents just to update contact information.
- Sophisticated Tracking Methods. Survey research organizations experienced in surveys of low-income populations typically use computerized tracking systems that can check for addresses and phone numbers in multiple databases (such as credit bureau files and motor vehicle registration records). In an evaluation with several rounds of follow-up data collection, good tracking databases can make it worthwhile to attempt to recontact individuals who were nonrespondents in earlier waves, particularly if the response rate on the first wave was less than ideal.(3)
- Mixed-Mode Interviewing. Mixed-mode surveys, which initially contact respondents by telephone and then in person if telephone attempts are unsuccessful, achieve higher response rates than telephone surveys alone. Field interviewers are more able to contact individuals without phones and also may be able to locate individuals on the basis of information from neighbors. In-person interviews are generally more expensive than phone interviews, however.
- Respondent Payments. Paying respondents for their time in responding to the survey, even if the payment is small, generally fosters higher response rates and greater cooperation in obtaining quality data. One problem in paying respondents on AFDC, however, has been that such payments generally were counted as income in computing benefits unless a specific federal waiver was obtained.
The lack of any of these factors does not necessarily imply in itself that a survey will have poor response rates, but it may be seen as a risk factor that requires careful monitoring. An additional risk factor exists if an organization that lacks a track record in interviewing low-income populations conducts the survey. Experienced survey organizations have staff from the level of survey director to interviewer who are adept in the techniques of reaching low-income respondents, as well as resources to plan and organize survey operations to achieve high response rates within the time frames needed for timely followup.