Approaches to Evaluating Welfare Reform: Lessons from Five State Demonstrations. b. Distinguishing Impacts Using Nonexperimental Methods


Regardless of whether the underlying evaluation design includes random assignment, evaluators can use several nonexperimental methods to attempt to assess the impacts of different welfare reform policies. First, a process study can often identify the components of a program likely to have been most (or least) effective through interviews with program staff and with clients; for instance, such interviews can identify components that were never implemented or that were misunderstood, versus components that were implemented well. Second, impacts of particular provisions of a welfare reform package may be analyzed by comparing outcomes for cases that participate in those components of the overall package (such as a JOBS program) to outcomes for cases that do not participate. Third, the staggered implementation of welfare reform policies in particular sites can help to distinguish impacts of different measures. Fourth, the evaluation design may call for certain research sites to implement only a subset of the total welfare reform package; this allows separate impacts to be estimated in a way similar to the use of a partial experimental group in an experimental design.

When the evaluation design does not incorporate random assignment, nonexperimental methods must be used. Unfortunately, these approaches are less likely than experimental methods to lead to reliable estimates of the impacts of different welfare reform provisions. The main disadvantage of nonexperimental approaches is that the groups being compared most likely differ not only by being subject to different policies, but also in other ways. For example, cases that decide to participate in a program are probably systematically different from cases that decide not to participate.(1) Similarly, when welfare reform policies are implemented in stages, applicants subject to both the first and second stage of reforms probably differ in systematic ways from cases initially subject only to the first stage of reforms. Finally, when different research sites implement different combinations of welfare reform policies, systematic differences probably exist between the sites that are confounded with the impacts of the policy combinations. Although statistical procedures such as multivariate regression can adjust for observed differences between different groups of cases, only random assignment can ensure that the unobserved characteristics of different groups of cases are, on average, the same.

Nonexperimental analysis of component impacts may be most useful as a supplement to experimental estimates of the impacts of the entire reform package. In this situation, the experimental design can be relied upon to determine whether a welfare reform package is associated with statistically significant differences in outcomes. Nonexperimental analyses (particularly process analyses) can help to establish which policy changes appear be most responsible for the observed impacts of the entire package.