Approaches to Evaluating Welfare Reform: Lessons from Five State Demonstrations. b. Trade-Offs Between Long and Short Sampling Periods for Applicants


In designing applicant samples, states should consider the following trade-offs between long and short sampling periods:

  • If the sample of active cases at all points in time is to be representative of the current caseload, it is necessary to continue sampling applicant cases throughout the demonstration. Sometimes, sampling over long periods also is necessary to meet sample size goals. In addition, sampling over a longer period may permit assessment of the impact of a more mature program, since implementation problems may mark the early period.
  • If the planned sampling of applicant cases is stretched out over more than two years, there are substantial risks for the impact analysis: (1) the length of followup on applicant cases will be shorter, on average, than if sampling was more front-loaded, and may be insufficient to observe full impacts; (2) the intervention may change, resulting in different programs being tested at different times; (3) sampling may be cut off early (if the evaluation is cut back for financial or substantive reasons) and then the applicant case sample may be too small to be useful; and (4) the application rate may fall over time, so that the sample ends up smaller than intended.