Approaches to Evaluating Welfare Reform: Lessons from Five State Demonstrations. a. Selecting a Method of Random Assignment

10/01/1996

The four evaluations we examined differed in the method they used to perform random assignment. California's evaluation used an interval sampling approach relying on sequential case numbers with a random start. Colorado's and Minnesota's evaluations used computer-generated random numbers to perform random assignment. Michigan's evaluation used the eligibility worker's number to perform random assignment of recipients and the case head's SSN to perform random assignment of applicants.

All of these approaches appear to have been acceptable means of performing random assignment, but each has advantages and disadvantages. Use of a computer-generated random number best ensures that cases will not learn of their experimental or control status before random assignment, even when cases go through random assignment over a period of a time. The main disadvantage of this approach is that program administrators may lack the resources to generate a random number for each case at the time of random assignment.

Interval sampling has the advantage of being a simple and straightforward approach. If case numbers are assigned sequentially, interval sampling will produce experimental and control groups in which cases of different welfare tenures are guaranteed to be represented in the same proportions in both groups. The disadvantage of this approach is that steps must be taken to ensure that the interval sampling begins with a random starting point and that the subsequent assignment of recipient and applicant cases to the list used in sampling is immune from manipulation by state employees or the cases themselves. An example of such manipulation would be program staff altering the sequence in which applicants are entered on the list.

Random assignment of recipients by eligibility worker number has the advantage of preserving the relationship between caseworkers and ongoing cases. This relationship is more likely to be disrupted by random-assignment approaches that assign recipients individually, since cases may then be moved to new eligibility workers or even to new welfare offices. The main disadvantage of random assignment by eligibility worker number is that cases are assigned experimental and control status in groups rather than on an individual basis, thereby increasing the possibility of large differences in the baseline characteristics of the experimental and control group samples.

Random assignment by the last two digits of the case head's SSN has the advantage of employing a number readily available from a state's administrative records. This approach introduces a slight risk of giving advance notice to recipients and potential applicants of the rules by which experimental or control status is determined. Anticipating their experimental or control status, recipients may decide to leave welfare before random assignment, and potential applicants may decide either not to apply for assistance or to apply with another individual identified as the case head.