Approaches to Evaluating Welfare Reform: Lessons from Five State Demonstrations. 2. State Approaches


Table III.1 shows the planned applicant and recipient sample sizes in four of the five evaluations reviewed (the Minnesota evaluation plan did not provide estimated sample sizes for the two groups). The decisions each state made concerning whether to oversample applicants and the time period for sampling applicants are summarized as follows:

  • Minnesota is oversampling new applicants in its two core experimental groups to better assess impacts for this group. Wisconsin is also oversampling applicants in the demonstration counties because that is the only way it can meet overall sample size goals in such small counties. California, Colorado, and Michigan all are attempting to sample applicants and recipients at the same sampling rates. Nonetheless, because of differences across the states in projected flows and applicant sampling periods, applicants outnumber recipients in Colorado's design, but recipients outnumber applicants in Michigan's design. California originally expected the applicant sample to be slightly larger than the recipient sample but (for reasons discussed later) actually has a much smaller applicant than recipient sample.
  • Wisconsin planned to sample applicants over seven years, California over four years, Colorado over two years, and Minnesota over 18 months in urban counties and 24 months in rural counties. Michigan initially planned to sample applicants over two years; the sampling period was extended by two years, however, as new waivers were passed and applicant samples were not as large as expected after the first two years.
  • The states furthest along in sampling--California and Michigan--found that the flow of applicants was smaller than expected; in Michigan, this may reflect the move from sampling all applicants to sampling approved applicants. Lower-than-expected applicant flows also seem likely to occur in Wisconsin.

The rest of this section discusses these findings in more detail, by state.

Wisconsin. Because of the small size of the demonstration counties in Wisconsin, the state planned an applicant sample that would be three times the recipient sample. It planned to achieve this sample by sampling for seven years. This lengthy intake period seems unrealistic, given the current rapid pace of policy development; indeed, Wisconsin now is proposing a new program (Wisconsin Works) that would supersede WNW. A key problem with the Wisconsin design is that the two demonstration sites are too small to meet sample goals within a policy-relevant time frame.

As of April 1996, the evaluator for WNW had received from the state a list of cases enrolled in the demonstration in the first nine months, but no data on applicants who did not complete their applications or on cases in the comparison counties. Thus, it is difficult to assess how well the demonstration has been achieving the planned sample goals. Anecdotal evidence, however, suggests that the demonstration is having substantial entry effects, which may reduce the sample sizes well beyond those expected. One indication is that the caseloads in the two counties declined nearly 20 percent between the time the demonstration was announced and actual implementation, leading to a recipient sample of 818 cases rather than 1,000.(19)

California. California's APDP sample was designed with the goal of sampling at the same rate from the recipient pool and the applicant flow, to continue to have a representative sample of the caseload in the demonstration counties over time for cost neutrality. To support the cost neutrality calculations, California originally planned to continue sampling for four of the five years of the follow-up period. A second set of waivers, implemented 16 months after the first set, extended the duration of the demonstration; at this time, however, state officials do not plan to extend the sampling period.

The original sampling plan estimated that the applicant sample would be slightly larger than the recipient sample (see Table III.1). The actual flow of applicants into the sample has been roughly one-third of what was expected, however, resulting in an applicant sample much smaller than the recipient sample. As of April 1996, 40 months after implementation, only 5,460 applicants had been sampled (1,824 controls and 3,636 experimental cases). Three reasons for the discrepancy are:

  1. Applications have been declining because of the economic recovery and, possibly, because of the multiple cuts in the maximum benefit. Because California is a "fill-the- gap" state, which allows applicants to fill the gap between the AFDC payment and the need standard with other income without losing eligibility, the cuts in the maximum benefit would not make anyone ineligible, but they could have behavioral effects if participation in AFDC becomes less attractive.
  2. The state uses a sample frame to select the sample that does not include all approved applicants. In particular, the sample is selected from the statewide Medicaid system. Very short-term cases (which may never be entered into the Medicaid system) and cases that are entered into the Medicaid system late are not included in the sample frame. The state believes this problem reduces the size of the sample but does not bias its composition (except that very short-term recipients are excluded).
  3. The state did not accurately estimate the number of cases that would be excluded from sampling because they had received AFDC within the recent past.

Colorado. Sample intake in Colorado has proceeded pretty much according to plan. The state recently stopped intake one month early with an applicant sample of about 6,000 cases, about 600 less than the number planned, stating it had more than met the requirement in the terms and conditions for 4,000 applicant cases.

Michigan. In Michigan, the projected applicant sample after two years was 8,374, but the actual sample intake was about 6,600. The shortfall seems to reflect the omission of denied applicants (those denied for both AFDC and SFA) from the sample. Denied applicants were dropped because Michigan's data system does not retain data on them. In addition, the state has argued that the same cases would be denied all benefits under both TSMF and control group rules; the intervention merely affects whether cases are approved for AFDC or SFA.

The intake period for new applicants was extended for two years largely because of the implementation of new waivers that substantially changed the TSMF program. As a result, the applicant sample may approach the recipient sample in size. Most analyses will examine applicants in the first two years and the second two years separately, however.

Minnesota. In the design for the MFIP evaluation, sample goals are not broken down into goals for applicants and recipients in the usual way. Instead, the sample design discusses a "basic" single-parent sample, which is a proportional sample of recipients coming up for redetermination, reapplicants (defined as those on AFDC in the past three years), and new applicants (defined as those not on AFDC in the past three years) (see Table III.2). In addition, there is a plan to oversample 1,800 single-parent new applicants in urban counties, equally divided between the two larger experimental groups (E1 and C1). Again, new applicants are defined as those not on AFDC in the past three years. The report states that this implies a sampling rate in the urban counties of 13 percent for single-parent recipients, but 80 to 86 percent for single- parent applicants. In the rural counties that are part of the MFIP demonstration, all applicants and recipients are subject to random assignment to one of the two core experimental groups.

MDRC staff members report that the intake for reapplicants and new applicants has exceeded expectations; this led to shortening of the intake period for reapplicants and to assigning a greater proportion of single-parent new applicants and two-parent applicants to nonexperimental groups.