Approaches to Evaluating Welfare Reform: Lessons from Five State Demonstrations. 2. State Approaches


The five evaluations range from those that were not concerned with external validity in selecting sites to those that have made a serious attempt to achieve a representative sample:

  1. Wisconsin made no claims of representativeness for the two demonstration counties for the WNW evaluation--Pierce and Fond du Lac; instead, it selected these sites because they were very interested in implementing the demonstration and seemed likely to achieve success (Bloom and Butler 1995). The two counties are small, relatively prosperous, and overwhelmingly white (as is most of the state outside Milwaukee). The state's main goal was to test the feasibility of the new approach. The selection of these sites severely limits the ability to assess impacts, however, even within these two sites. In particular, sample sizes are small, and MAXIMUS was not able to select comparison sites with unemployment rates quite as low as in the two demonstration sites.
  2. California has implemented the APDP/WPDP demonstration in four counties that are judgmentally representative of the state and that contain 49 percent of the state's caseload: Los Angeles, San Bernadino, Alameda (Oakland-Berkeley), and San Joaquin. The first process report states: "Planners chose Los Angeles because of its critical importance to the state, San Bernardino because it is adjacent to Los Angeles, Alameda for its ethnic diversity, and San Joaquin to represent the Central Valley and because of its proximity to Alameda"(UC DATA 1994). Los Angeles and Alameda have large urban areas, while the other two are largely rural. They range in population: Los Angeles has a population of 9 million, San Bernardino and Alameda populations of 1.2 to 1.5 million, and San Joaquin a population of 0.5 million. San Joaquin had the highest percentage of the population on AFDC and the highest percentage of two-parent cases. In California, welfare reform outcomes also are being tracked in the rest of the state, and state staff members are working on methods for reweighting the research sample to make it more representative of the state as a whole (for example, in terms of ethnicity).
  3. Colorado chose 5 research counties from among 13 that applied to be in the demonstration, on the basis of their capacity to implement the demonstration and to represent the state's diverse geographic, economic, and demographic conditions.
  4. Michigan's sample of four offices was designed to be judgmentally representative along the dimensions of gender, race, age, earned income status, months since the case opened, and family size. Sampling rates were set so that the share of cases in Wayne County (Detroit) was the same as in the statewide caseload.
  5. Minnesota's MFIP is operating in 7 of the state's 87 counties, of which 3 are urban (including one Twin Cities county) and 4 are rural. The sample was designed to overrepresent rural cases in relation to the statewide caseload but to choose representative counties within the urban and rural groups. For the urban sample, the state wished to include the county containing St. Paul or that containing Minneapolis (Ramsey or Hennepin); it ruled out Ramsey because it was participating in another demonstration. The state also chose one of the two large suburban counties at random, but ended up including both because the second county offered to pay planning costs itself in order to be included. The rural counties (all remaining counties in the state) were originally to be part of a nonexperimental evaluation, and two clusters of counties were chosen randomly to represent rural counties statewide.(20) When the state moved to an experimental design for the rural counties as well, one of the two clusters was chosen for the demonstration.

In summary, the selection of the rural counties in Minnesota is the only example of sites being selected through a formal sampling procedure. All of the other states except Wisconsin chose sites that were, to some extent, judgmentally representative. Most evaluators also analyze site representativeness after sites have been chosen.