At the meeting of the expert panel, Timothy Bartik introduced the idea of analyzing "natural experiments" in localities where assignment of TANF recipients to employment and training providers is essentially random, as is the case in Detroit and New York City. Depending on the amount of variation in service providers and in the practices they use, such studies could be very valuable.
One of the main attractions of natural experiments is that they could be studied quickly and inexpensively. A valuable study, relying primarily on WIASRD and UI data, could be carried out for a few hundred thousand dollars. Natural experiments also provide a way to compare the activities and outcomes and, potentially the impacts of different practices and services in an environment that has not been manipulated by researchers.
For impact estimation, natural experiments face two challenges. One is that assignment of clients rarely is genuinely random. Often assignment is partially random, as is the case when assignment to service providers located within a geographic area or within an education grouping is random. Assignment also can be chiefly random, as when most clients are randomly assigned and a few are assigned nonrandomly. These nonrandom aspects of assignment can be addressed statistically, but the corrective actions taken often raise doubts about the credibility of the resulting impact estimates.
The other challenge is that service providers bundle groups of services, making it difficult to identify the value added by particular services. Researchers typically exert little or no influence on the composition of these service bundles. However, with sufficient variation in services across several sites, the bundles can be unpacked statistically.
Because conditions cannot be controlled by researchers, natural experiments cannot be expected to be as definitive as planned experiments (discussed below) in addressing questions such as those raised earlier in this chapter. Departures from true random assignment and well-defined service packages are to be expected when there is no agreement governing such matters (such an agreement is essential to the success of a controlled experiment). However, at a minimum, natural experiments like these can provide an excellent opportunity for an exploratory study.
"report.pdf" (pdf, 212.99Kb)