A social experiment is a field test of one or more social programs — or, to use the phraseology of the natural sciences, a test of one or more "treatments." A social experiment is a field test in the sense that families or individuals are actually enrolled in a pilot social program offering some type of special benefit or service. It is "experimental" in the sense that families or individuals are enrolled in each of the tested programs on the basis of a random assignment process, for example, the flip of a coin. To draw conclusions about the effects of the treatment, it is necessary to collect information about the people who are enrolled in each experimental program and about those who receive no special treatment (called the control group), and then to compare them on the basis of the collected information. If the composition of each of the groups is determined by a random process and the groups are composed of similar families or individuals, any differences in measured behavior between the groups can be attributed to the effects of the programs being tested.
It is the random assignment procedure that gives social experimentation its advantage as a policy research tool. Most non-experimental date sources are simply observations of behavior or situations resulting from a myriad of unspecified and indeterminate influences. With such non-experimental data, one can observe factors that appear to change simultaneously or sequentially. However, at best, only tentative conclusions can be drawn about causality. Since experimentations can deliberately inject a new element into an environment, keeping everything else the same, subsequent changes can be attributed to the influence of the new element within known statistical confidence intervals.
This cause-effect characteristic does not, of course, guarantee that social experiments will produce definitive findings; the treatment may turn out to have no effect or to have an effect that is not precisely measurable. And even if there are definitive findings, there is no guarantee that these findings will be those that were expected by the designers of the experiment or will be useful for policy purposes. The scientific success of the experiment depends on several important factors:
- How precisely the propositions to be tested can be formulated;
- How well the experiment is designed to test those propositions;
- How large the experimental and control groups are and how long is the period over which they experience the treatment;
- How much difference the treatment makes to the environment faced by the experimental group compared to the otherwise identical environment faced by the control group — in other words, how strong the treatment is; and
- The extent to which unforeseen or uncontrollable situations and events distort the observed experimental-control comparison.
It is worth expanding briefly on each of these factors.