Approaches to Evaluating Welfare Reform: Lessons from Five State Demonstrations. b. Timing of Random Assignment for Applicants


Two of the evaluations implemented random assignment of applicants before eligibility determination, while the other two implemented random assignment after eligibility determination. Both Michigan's and Minnesota's evaluations assigned applicants to experimental and control groups before determining eligibility. In Minnesota's evaluation, random assignment was immediately preceded by baseline data collection; it was immediately followed by eligibility determination (under the rules for the appropriate group) and then separate group orientation sessions for the experimental and control groups. Minnesota's research sample included denied applicants. Denied applicants who had gone through random assignment were excluded from Michigan's sample because of data limitations. Fortunately, Michigan's welfare reform and control group policies differed little in eligibility rules, so the comparability of the experimental and control groups probably was not seriously compromised by the exclusion of denied applicants from the sample.

In California's evaluation, random assignment of applicants occurred after intake and eligibility determination (under experimental group rules, which were identical to the control group rules for applicants). Originally, random assignment of applicants was to occur at intake, but this "proved to be too expensive and disruptive to county operations.(2) Control group members were subject to experimental policies for one or two months. A control case whose initial AFDC benefits were too low would receive a retroactive supplement. A control case whose initial AFDC benefits were too high (the rarer case) would not receive a retroactive reduction. Short-term AFDC recipients in the control group may have left AFDC without being exposed to control group policies. Control cases receiving AFDC-UP who began working over 100 hours after enrollment (but before random assignment) were dropped from the research sample rather than disenrolled from AFDC-UP. To preserve comparability of the experimental and control group samples, the corresponding AFDC-UP cases in the experimental group should also have been dropped from the research sample, but this action does not appear to have been taken.

In Colorado's evaluation, random assignment of applicants occurred one month after cases were observed in the state's data system. Only applicant cases approved for AFDC under the (slightly more restrictive) control group policies were included in the research sample. Experimental cases in the sample of approved applicants were subject to control group policies for one month before the application of experimental group policies.

None of the welfare evaluations we reviewed performed dual eligibility calculations. As states upgrade the computer systems used to administer their assistance programs, this sort of random- assignment process may be more feasible in the future.