Both spillover and displacement are threats to a welfare reform evaluation, because they change the policies received by members of the control group and bias estimates of the impacts from welfare reform. It is usually difficult or impossible to adjust impact estimates for this bias, so states implementing welfare reform evaluations should take steps to minimize spillover and displacement.
Certain measures can help states reduce the risk of the spillover of experimental policies to the control group and the displacement of control group policies as a consequence of welfare reform. In particular, we recommend that states keep experimental and control group members well informed, both in writing and in person, of the policies that apply to them. In this way, states can counteract any spillover of attitudes and impressions from the implementation of welfare reform. We also recommend that states administer experimental and control policies using separate but equivalent staff, with minimal disruption and displacement of the services already being provided to recipients in the control group.
Evaluators can gather evidence on the extent to which spillover or displacement has occurred through a study of the implementation of a welfare reform evaluation. Such a study could include interviews with program administrators in which these individuals are asked to provide the following information:
- How cases are informed of the policies that apply to them
- Whether program staff members process experimental and control cases differently
- How program staff members distinguish experimental and control cases
- Whether program staff members ever confuse experimental and control cases, applying one group's policies to the other group's members
- The extent to which separate staff members handle experimental and control cases
- Whether welfare reform has expanded or narrowed the opportunities available to cases in the control group
- Whether welfare reform has otherwise affected the way in which policies are administered to cases in the control group
Evaluators can also gain useful insights through interviews with participants about their perceptions of the policies that apply to them and the services they received. The process evaluation of APDP/WPDP in California is one example of a study that carefully considers these issues. This information will not solve the problems of spillover or displacement, but it will provide evidence of the extent to which these problems are present in an evaluation.