Approaches to Evaluating Welfare Reform: Lessons from Five State Demonstrations. 3. Analysis and Recommendations


We recommend specifying a response rate standard for surveys that are to be used for impact analysis. An appropriate standard would be from 70 to 80 percent (with the lower end of the range for later rounds of followup); such response rates are achievable with the types of practices discussed in Section C.1, including contact information collected at intake and updated regularly. Welfare agencies could also encourage achievement of high response rates by exempting respondent payments from countable income. In addition, it should be standard practice to compare the characteristics of respondents and nonrespondents in the available administrative data, to assess the likely magnitude of response bias.

We also recommend that the sponsors of welfare reform evaluations monitor data collection plans carefully to ensure that survey practices needed to achieve high response rates are being used. In particular, we recommend not approving any survey plan that lacks two or more of the "best practices" described in Section C.1. If the state is not able to invest the level of resources implied by these practices, then the survey may not produce data of sufficient quality for an impact analysis. Finally, we recommend close monitoring of surveys that lack any one of these practices or that are conducted by a survey organization that is relatively inexperienced with low-income populations. If such surveys produce low response rates early on, then states should carefully consider whether to add resources to survey operations (for example, by adding field followup to a telephone survey) or whether to discontinue the survey altogether.