Investigators must also decide whether the study should focus on a subgroup of the families that have left welfare (i.e., a sample) or whether the study should include all families. The key determinant for using this approach is the purpose of the study. If the primary purpose is to identify families at risk of extreme deprivation, then it is important to include all families. If the primary purpose is to measure outcomes and/or to inform ongoing policymaking efforts, a sample can be used. Sampling enables the study to collect more-detailed information without making the cost prohibitive.
Sampling can produce accurate results at a lower cost than surveying an entire population. The size of the sample determines the confidence level of the results (i.e., the probability that observed results would have been obtained by chance). For example, if the difference between two groups is significant at the 95 percent confidence level, a 95 percent chance exists that the difference between the groups is real and there is only a 5 percent chance that the difference represents chance or sampling error. Those commissioning surveys must determine the desired confidence level for each statistic studied. Generally, larger samples increase confidence that smaller observed differences are statistically significant. The findings from larger samples may also be more defensible against critics. Many studies have used a stratified design in which two or more subgroups are compared (e.g., recipients in county A compared with recipients in county B or welfare recipients compared with diverted applicants).
Generally, a sample size of 500 is sufficient for a general-purpose survey, such as a statewide survey of former recipients to determine if they are working and if they are able to support their families. Larger sample sizes would be required to focus on particular regions in the state or to examine the reasons why some recipients could not find or keep jobs. (Alternatively, the state could focus on recipients without jobs by drawing a smaller sample from the recipients whose names do not appear on unemployment insurance lists.) Larger surveys of 3,000 or more provide substantial versatility and precision in analysis, and they can support many subgroup comparisons. However, the increased cost suggests that these surveys should be reserved for critical data needs.