Even if control group policies consistently represent the polices that would have existed in the absence of welfare reform, individuals originally belonging to the control group may be exposed to experimental policies (or vice versa) in some situations:
- A case may relocate to a site in which it receives a different set of policies
- A case may merge with a case of a different experimental/control status
- A case may split off from another case and be assigned a different status
- A case's official experimental/control status may be changed as the result of administrative error or manipulation(4)
All of these situations are examples of cases crossing over from one experimental/control status to another. We distinguish migrant crossover cases as cases that experience a change in experimental/control status because of migration; merge/split crossover cases as cases that experience a change in experimental/control status because of a case merger or split; and administrative crossover cases as cases that experience a change in experimental/control status because of administrative error or manipulation. In general, crossover from control to experimental status is more likely when most of the welfare cases in the state are subject to welfare reform policies, while crossover from experimental to control status is more likely when most of the welfare cases in the state are not subject to welfare reform policies.
Regardless of how crossover occurs, the presence of crossover cases in the research sample may result in biased impact estimates because some cases receive the other group's policies instead of the policies to which they were originally assigned. In particular, impact estimates may be too small, since a fraction of original control cases becomes subject to welfare reform policies, and/or a fraction of original experimental cases becomes subject to control group policies. Statistical methods for adjusting for crossover exist (they are discussed in Chapter VI); however, these methods have certain theoretical and practical limitations, so it is in a state's interest to minimize crossover.
1. Waiver Standards for Minimizing Crossover of Cases
The terms and conditions of Section 1115 welfare waivers specify several steps designed to reduce the incidence of cases crossing over from control group policies to experimental group policies (or vice versa):
- When a case relocates from one research site to another research site, the case's experimental/ control status is to be preserved.
- When a research case splits into multiple cases remaining in a research site, the original case's experimental/control status is to be preserved for the new cases.
- When an experimental/control case merges with another case in the same research site, the experimental/control status of the head of the new case is to be preserved.
These standards do not address crossover that occurs through administrative errors in the classification of cases' experimental or control status.
2. State Approaches
Although the reported incidence of crossover is seldom high for welfare reform waiver evaluations, the evaluations we reviewed had some difficulties in minimizing the risk of crossover. Most research sites were not next to each other, because the desire to obtain a sample representative of the state took precedence over the desire to reduce migration to nonresearch sites. In the four states with experimental evaluation designs, the lack of contiguous research counties may have increased the risk of crossover through migration.
The absence of experimental/control status information in individual records in state administrative systems (as opposed to case records) often made crossover from splits and mergers more likely by failing to identify individuals with previous membership in a research case. In California's evaluation, for example, county-specific automated systems made it difficult for caseworkers to identify crossovers from other counties in the state; the evaluator was able to achieve this identification by relying on state Medicaid records noting receipt of AFDC during the previous 12 months. In Michigan's evaluation, the state was unable to identify the previous research status of individuals reapplying for assistance (although the evaluator later obtained this information by merging case- and individual-level files).
3. Analysis and Recommendations
Crossover is a potentially serious threat to a welfare reform evaluation, because it blurs the distinction between experimental and control cases and can lead to biased estimates of the impacts from welfare reform. The terms and conditions of Section 1115 welfare waivers have devoted considerable attention to minimizing the risk of crossover, and we recommend that states seek to adhere to the waiver standards for administering experimental or control policies to cases that migrate, merge, or split.
States can reduce the incidence of crossover in welfare reform evaluations in at least three additional ways:
- A large portion of the state's welfare population could be included in the research sample. This measure, while costly, would reduce the likelihood of crossover through migration to nonresearch sites. However, it could increase the risk of spillover, if control group cases were a smaller proportion of each worker's caseload.
- Research sites could be located in contiguous counties, since migration might be more likely to nearby counties than other counties. However, this could create trade-offs with goals of external validity, since it may be difficult to choose representative sites that are also contiguous.
- Statewide administrative records could append original experimental/control status information to individual records as well as case records. This would make it easier to identify, at the time of application or redetermination, individuals who have split off from research cases or who are merging with other cases and make it less likely that individuals' experimental/control status will be changed as the result of administrative error or manipulation.