Private Employers and TANF Recipients. 4.4  Rigorous Evaluation Studies

05/25/2004

Planned random assignment studies involving employers, labor market intermediaries, and/or other institutions would be very valuable. They could provide credible estimates of the net impacts of alternative work development approaches, and perhaps even of particular employer practices, on employment, earnings, TANF receipt, income, training and support provided by employers, as well as other outcomes.

The advantage of designed experiments over natural experiments may not be great with respect to random assignment, assuming that assignment practices in New York, Detroit, and other places are truly random. However, their superiority is clear in regard to controlling the mix of services received by welfare recipients in the study. In principle, planned experiments could control the services received by the randomly assigned groups. This would allow researchers to draw inferences more confidently about what does and does not work.

  • In practice, however, such experiments could be difficult to implement. Designing random assignment evaluations of intermediaries would be challenging for many reasons, including the following:
  • Many employer-led workforce development initiatives and innovative community-based organization programs involve relatively small numbers of workers. This raises serious sample size concerns.
  • Inserting random assignment into intermediary or employer processes often would be difficult, if not impossible. In other cases, it might cause disruptions or raise serious ethical issues.

Many intermediaries do not seem ready for rigorous study. It is critically important that the service be strong enough to generate measurable impacts and that the organization have the capacity to accommodate random assignment and collect necessary data.

In addition, designed experiments are subject to the criticism that they are artificial and thus less robust than natural experiments with respect to real-world conditions. This is especially true if too much control is exerted over the service mix (or other aspects of the discretionary decisions) of program operators.

However, at least three candidates for random assignment study merit consideration. First, financial and other incentives for employers represent one option. By providing specific incentives to some employers and not others, or for some employees and not others, the experiment would be designed to induce different employer practices, which would be expected to generate different outcomes for TANF recipients. However, it might not be feasible to modify an existing tax incentive (such as the WOTC), or to establish a new tax incentive, only for randomly selected employers. In addition, past experience with wage subsidies paid to employers suggests that offering tax incentives to employers for some job candidates and not others can stigmatize those individuals.(19) However, some form of time-limited payments to employers or intermediaries might be used instead of changes in tax rules.

Second, TANF/WIA One-Stop Centers are another candidate for study. They work with large numbers of welfare recipients and employers, random assignment seems feasible, and they are accustomed to collecting data. The available research suggests that One-Stop Centers vary in terms of their employer focus, TANF-WIA coordination, and performance outcomes. Therefore, perhaps, an intensive, employer-focused One-Stop treatment could be delivered at selected centers to TANF recipients randomly assigned to a treatment group. A less intensive, "standard" treatment could be provided to control group members.

Third, community intermediaries such as Project QUEST are potential candidates for rigorous study. Many are sophisticated initiatives serving a large number of TANF recipients. Were intermediaries such as Project QUEST to be evaluated using an experimental research design, either employers or employees could be randomly assigned to treatment and control groups.(20) Several intermediaries have been assessed as part of one or more qualitative studies. The Center for Employment Training (CET) was rigorously evaluated in three different random assignment evaluations, although many of the key outcomes associated with its labor market intermediary role  notably its interactions with and impacts on employers  were not considered.(21)

Without question, comprehensive surveys and random assignment experiments would be the best way to address the unanswered questions identified earlier in this chapter. However, a well-run experiment would require substantial time (at least four years) and money to implement. With a large sample (2,000 or more in both the treatment and control groups), multiple waves of survey data collection, and intensive field research, the cost of such a study could easily reach $15-20 million.

A random assignment experiment also would require cooperation  in terms of random assignment, service provision, and data collection  from the intermediaries and other organizations participating in the research. As a result, it may make sense to study natural experiments and/or conduct other research first, and pursue larger-scale planned experiments later, after more has been learned.

View full report

Preview
Download

"report.pdf" (pdf, 212.99Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®

View full report

Preview
Download

"appendix.pdf" (pdf, 433.16Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®