Approaches to Evaluating Welfare Reform: Lessons from Five State Demonstrations. 1. Issues


Several approaches are possible to selecting sites for an impact evaluation:

  • Sampling of Sites. This approach involves random selection of sites to be representative of the state as a whole. To ensure inclusion of large and small sites, sites often are selected with probability proportional to (caseload) size (PPS). To ensure geographic diversity or diversity in other important characteristics, stratification may be used, with random or PPS sampling within strata. Formal sampling procedures ensure representative sites. However, when the intervention is a random-assignment demonstration that places considerable administrative burdens on the sites selected, random selection of sites may not be feasible.
  • Selecting Judgmentally Representative Sites. Instead of using a formal sampling process, sites may be selected judgmentally (from among the subset of sites in which implementation is feasible) so that they are approximately representative of the state as a whole. For instance, in selecting sites, states may work to ensure that all regions are represented, that both urban and rural areas are represented, or that key program variations or types of local economic conditions are included.
  • Selecting Exemplary Sites. If the major goal of an evaluation is to assess the feasibility of implementing a new program or to study operation of that program, a state may wish to choose those sites that are most interested in operating the new program or that are otherwise best suited to testing the program. Such an approach clearly limits the generalizability of the impact results, but that may be a worthwhile trade-off. In addition, impact results in such circumstances can be seen as program impacts in the "best case"; a finding of little or no impact is thus still informative.

Regardless of how sites are selected, it is always possible later to compare the characteristics of the sites to the state as a whole and to compare the characteristics of the AFDC caseload in the demonstration sites to the state caseload. If the demonstration sites appear reasonably similar to the rest of the state, this makes generalizing net impact results to the state level more plausible. If major differences exist, the evaluator can assess the possible implications of these differences. It may also be possible to reweight the demonstration sample (especially if it is relatively large) to be more representative of the state caseload.