Approaches to Evaluating Welfare Reform: Lessons from Five State Demonstrations. 1. Issues


High response rates are critical in surveys that are part of an impact evaluation in order to minimize the potential for nonresponse to bias the impact estimates. Nonresponse may bias the impact estimate because those who do not respond to the survey may experience different program impacts from those who do respond. If nonresponse is not correlated with experimental/control status, estimated impacts are unbiased for those who do complete the survey, but not for the overall population. If nonresponse is correlated with experimental/control status (as, for instance, when experimental group members leave assistance earlier, and are then harder to locate because contact information is more out of date) then the impact estimates will be biased even for respondents.

Thus, there are two major issues:

  1. Should there be a minimum standard for an acceptable response rate in a follow-up survey to be used in developing impact estimates and, if so, what should the standard be, both for initial and (if applicable) later rounds of followup?
  2. What knowledge is there in the evaluation community concerning survey practices that are conducive to achieving and maintaining high response rates, and how can federal and state officials promote use of these practices?

In discussing these issues, we draw heavily on our experiences as working evaluators and on the insights of the expert panel convened for this project.