Approaches to Evaluating Welfare Reform: Lessons from Five State Demonstrations. b. Timing of Random Assignment for Applicant Cases

10/01/1996

When randomly assigning applicants to treatment or control status, the goal is to include all applicants who could be eligible for assistance under either experimental or control policies, but to exclude applicants who are twice-ineligible-ineligible under both sets of rules. The timing of random assignment before or after eligibility determination may crucially affect whether the sample meets this goal.

If welfare reform does not change eligibility rules at all, then eligibility determination should precede random assignment. Then, only approved applicants are kept in the sample.

If eligibility rules change, there is no ideal solution, unless it is feasible to determine eligibility under both sets of rules. When random assignment of applicants occurs before the determination of eligibility for benefits, then the sample of applicants will likely include twice-ineligible cases. These cases probably will not be affected by welfare reform programs, unless they later become eligible and reapply for welfare benefits. Ignoring the possibility of future changes in eligibility, it is reasonable to assume that the impact of welfare reform on twice-ineligible cases is zero. Under this assumption, the estimated impact of welfare reform on the entire sample of applicants will be smaller than the estimated impact of welfare reform on the sample of eligible applicants--those eligible under at least one set of rules. The extent to which impact estimates are diluted for applicants will depend on how large a portion of the sample of applicants is twice-ineligible.(1)

When eligibility rules change and random assignment of applicants occurs after the determination of eligibility for benefits, then the sample of applicants will be restricted to cases eligible for assistance under the rules used to determine eligibility prior to random assignment. The sample will contain all applicants eligible for assistance under either set of policies only under the following circumstances:

  • If one set of eligibility rules is strictly broader than the other set of eligibility rules, and the first set of rules is used for the initial eligibility determination, with a second set of rules applied shortly thereafter to cases subject to the narrower set of rules
  • If eligibility under both sets of rules is determined for each case at the time of random assignment

The first example is of a situation in which one eligibility calculation occurs prior to random assignment and another shortly thereafter, and the second is of a situation in which a dual eligibility calculation occurs prior to random assignment.

States generally will find it most convenient to perform a single eligibility calculation for each case, with no subsequent eligibility calculation occurring until the time of redetermination. Sequential or dual eligibility calculations usually will impose greater administrative burdens on states. For example, a state may prefer to have separate staff administer experimental and control policies to avoid confusion of case status or corruption of the random-assignment process. Performing a sequential or dual eligibility determination while maintaining separate staff could effectively double the staff time needed to determine a case's eligibility for welfare benefits.

Sequential eligibility determinations also may create awkward situations for states. For example, if welfare reform expands eligibility, and initial eligibility is determined under this broader set of rules, the state will need to recalculate eligibility and benefits under the narrower set of rules for cases subsequently assigned to the control group. This second eligibility calculation may result in benefits being lowered or eliminated entirely for cases in the control group. Specifically, to ensure that control cases receive only control group polices following random assignment, benefits would need to be reduced retroactively to account for the broader rules initially applied.

It might be feasible to perform dual eligibility calculations if a computer determined eligibility. This could be done in five steps:

  1. The applicant provides the necessary information needed to determine eligibility under either welfare reform rules or control group rules.
  2. The applicant's eligibility information is entered into a computer.
  3. The computer determines welfare eligibility and benefits under both sets of rules.
  4. If the applicant is eligible for assistance under either set of rules, the computer randomly assigns the case to either the treatment group or the control group.
  5. The computer informs the applicant of the benefits (if any) for which the applicant is eligible and directs the applicant to a caseworker trained to administer the underlying set of policies to which the applicant has been assigned.