Approaches to Evaluating Welfare Reform: Lessons from Five State Demonstrations. c. Sample Balance


Dividing the sample into equal numbers of experimental (demonstration) and control (comparison) cases (this is referred to as a "balanced" design) leads to estimates with the highest level of precision, for a given total sample size.

(1) However, substantial deviations from this balance may occur with only minor losses in precision (Bloom 1995). States may prefer an unbalanced sample because of a desire to implement the reform program as completely as possible (if the reforms are implemented statewide for all cases except control cases). By having the minimum allowed number of control cases but more experimental cases, states can increase sample precision while keeping the control group as small as possible. Thus, in many evaluations in which the intervention is implemented for everyone except the control group, the sample is designed to include two experimental group members for every control group member. Increasing the ratio of experimentals to controls beyond 2:1, for a fixed total sample size, leads to more substantial loss in precision. Increasing the total sample size by adding additional experimental group members (but keeping the control group sample the same) increases precision only slightly.