[ Main page of Report | Contents of Report ]
Based on the findings of this study, which were presented in the previous two chapters, this chapter explores options for potential successor projects. The research evidence that is most needed by employers and policymakers requires comprehensive knowledge of employer and intermediary attitudes, policies and practices, and explicit comparisons of activities and outcomes across employers and intermediaries. Such comparisons, however, are only meaningful if the data to be analyzed (1) are consistent for different employers and intermediaries; and (2) capture important employer practices in sufficient detail. Most existing data sources do not satisfy these two criteria.
This chapter begins with a discussion of potential analyses of two existing data sources, and then turns to the advantages and disadvantages of embarking on three kinds of new data collection and analysis.
[ Go to Contents ]
As indicated in the last chapter, the employer surveys directed by Harry Holzer in the late 1990s stand out as some of the most important available data sources. Several noteworthy research documents based on analyses of these survey data were cited in Chapter 2. However, these surveys clearly offer opportunities for additional analysis. The survey questions concentrate on employer willingness to hire welfare recipients, the extent to which they have actually hired recipients, and their experience with the recipients they have hired. Yet Holzer's survey instrument also includes a number of questions about employer practices, notably recruitment, screening, and hiring. In addition, the instrument has several questions about labor market intermediaries, including questions about whether an employer has worked with specific intermediaries in each of the four metropolitan areas.
This is the only existing dataset that could be used to address the unanswered questions highlighted at the outset of this chapter. These survey data, which were collected consistently from a large sample of urban employers, cover several employer practices in meaningful detail. It would be possible, for example, to analyze employment outcomes for individual intermediaries or groups of intermediaries in the four metropolitan areas. It would also be possible to compare outcomes for employees who were hired using different recruitment and screening practices.(17)
The advantages of such analyses are apparent: The data are readily available, the analysis would be relatively inexpensive and, at least for some employer approaches and practices, additional analysis could be very informative. The disadvantages arise principally from the limitations of these survey data: they are limited to four cities, were collected a full five years ago, in a different economic and welfare policy environment than presently exists, and do not address some of the most important issues examined in this report (e.g., the specific ways in which employers utilize labor market intermediaries).
The surveys of labor market intermediaries carried out for the Welfare to Work Partnership in the Atlanta and New Orleans metropolitan areas also could provide the basis for valuable new analyses. Surveys were completed with more than three-quarters of all intermediaries identified as working with TANF recipients in those two areas. Straightforward analyses — such as an assessment of the types of services provided to employers by different types of intermediaries — would complement other research on intermediaries and the employment of TANF recipients.
The disadvantage of other existing data sources, in terms of addressing the unanswered questions, is that they lack information on employer practices. For example, the WIA program data system — WIASRD — contains information on publicly supported training, but not on private training supplied by employers. This means it would be possible to assess the effects of alternative training practices for a subset of labor market intermediaries (those with WIA funding), as discussed in the next section. However, the practices of employers could not be examined.
WIASRD can be linked to data on employee wages — based on Unemployment Insurance (UI) reporting system used by employers — but the quarterly records offer limited insight into employer compensation practices. For example, hourly wage rates and fringe benefits cannot be determined from these data.
While WIASRD would not help with the unanswered questions, it could provide, once linked to TANF data, a more complete and up-to-date answer to ASPE's fifth question — Which segments of the TANF population present the greatest challenges? This question could be addressed with data on individual characteristics, deferrals, and exemptions. While analysis of these data would not shed new light on some issues (which are not fully captured in these administrative data), such as the effects of mental illness on employment, it would provide better estimates of the prevalence of a wide range of barriers.
Another important dataset is the one developed by the Census Bureau's LEHD program. A number of studies have already utilized the LEHD data, including two cited in the last chapter. The core of the dataset is UI wage record files, and the LEHD program currently includes states accounting for roughly half of total U.S. employment over an extended period, with that coverage expected to grow in the near future. These quarterly UI data have been integrated with employer records, basic demographic information and, for a limited subsample, survey data from the Current Population Survey (CPS), Survey of Income and Program Participation (SIPP), and decennial census.(18)
There is limited but important information about employer practices in the LEHD dataset. For example, detailed questions about employer-provided training are asked on one wave of the SIPP. This employee-reported information could potentially be combined with other SIPP and non-survey data in the LEHD dataset- such as employer size and industry, job classification and characteristics, and worker characteristics, length of employment, and earnings and fringe benefits - to compare activities and outcomes across different types of employee training. The data might also be useful in providing additional information on the types of employers that hire TANF recipients.
[ Go to Contents ]
A survey is the most practical way to obtain quantitative data to support systematic comparisons of practices used by employers and intermediaries around the country. At present, Holzer's survey of employers and Abt Associates' survey of intermediaries are the only sources of quantitative data on employer and intermediary practices regarding TANF recipients.
A new survey of employers and labor market intermediaries would be valuable for two important reasons.
As explained in the next chapter, a new survey could use several alternative sample frames and could concentrate on different substantive issues. A single survey could be administered to a sample of all employers, perhaps stratified in terms of employer size, location, or other factors. Alternatively, the survey could be supplemented by in-depth interviews with a subset of employers, such as those that regularly hire TANF recipients or those that have relationships with a particular set of labor market intermediaries. A survey could also be conducted of administrators from the labor market intermediaries working with employers that are also being interviewed.
The key advantage of a new survey is that it could be designed to collect specific types of information of interest, such as data on labor market intermediary practices or employer practices used in managing TANF workers. The key disadvantages are cost (up to several million dollars) and the length of time required before reporting findings. As discussed in the next chapter, the cost of a survey would depend on factors such as the sample size, the methods used to administer the survey, the response rate, whether or not secondary interviewing is done, and how the survey data are analyzed.
Qualitative data on the practices used by employers and intermediaries could be collected in new case studies. Existing case studies typically have addressed small groups of employers or intermediaries. Sometimes the studies have focused on a single organization. Because the studies have used different approaches and applied different criteria, it is difficult to draw wide-ranging conclusions from this qualitative research.
Given the need for systematic comparisons, it is important that such studies collect consistent information on specific practices among varied employers and intermediaries. This would require the use of consistent definitions and field research methods across a representative group of employer and intermediary settings.
The two chief advantages of such case studies, compared to other types of research, are that they could explore issues in much more depth (and receive input from a greater number of employer or intermediary informants), and respond better to issues encountered after the research begins. This greater depth of information comes at a cost — a small sample size that cannot be subjected to statistical analysis. The cost of a case study project depends on its features.
[ Go to Contents ]
At the meeting of the expert panel, Timothy Bartik introduced the idea of analyzing "natural experiments" in localities where assignment of TANF recipients to employment and training providers is essentially random, as is the case in Detroit and New York City. Depending on the amount of variation in service providers and in the practices they use, such studies could be very valuable.
One of the main attractions of natural experiments is that they could be studied quickly and inexpensively. A valuable study, relying primarily on WIASRD and UI data, could be carried out for a few hundred thousand dollars. Natural experiments also provide a way to compare the activities and outcomes — and, potentially the impacts — of different practices and services in an environment that has not been manipulated by researchers.
For impact estimation, natural experiments face two challenges. One is that assignment of clients rarely is genuinely random. Often assignment is partially random, as is the case when assignment to service providers located within a geographic area or within an education grouping is random. Assignment also can be chiefly random, as when most clients are randomly assigned and a few are assigned nonrandomly. These nonrandom aspects of assignment can be addressed statistically, but the corrective actions taken often raise doubts about the credibility of the resulting impact estimates.
The other challenge is that service providers bundle groups of services, making it difficult to identify the value added by particular services. Researchers typically exert little or no influence on the composition of these service bundles. However, with sufficient variation in services across several sites, the bundles can be unpacked statistically.
Because conditions cannot be controlled by researchers, natural experiments cannot be expected to be as definitive as planned experiments (discussed below) in addressing questions such as those raised earlier in this chapter. Departures from true random assignment and well-defined service packages are to be expected when there is no agreement governing such matters (such an agreement is essential to the success of a controlled experiment). However, at a minimum, natural experiments like these can provide an excellent opportunity for an exploratory study.
[ Go to Contents ]
Planned random assignment studies involving employers, labor market intermediaries, and/or other institutions would be very valuable. They could provide credible estimates of the net impacts of alternative work development approaches, and perhaps even of particular employer practices, on employment, earnings, TANF receipt, income, training and support provided by employers, as well as other outcomes.
The advantage of designed experiments over natural experiments may not be great with respect to random assignment, assuming that assignment practices in New York, Detroit, and other places are truly random. However, their superiority is clear in regard to controlling the mix of services received by welfare recipients in the study. In principle, planned experiments could control the services received by the randomly assigned groups. This would allow researchers to draw inferences more confidently about what does and does not work.
Many intermediaries do not seem ready for rigorous study. It is critically important that the service be strong enough to generate measurable impacts and that the organization have the capacity to accommodate random assignment and collect necessary data.
In addition, designed experiments are subject to the criticism that they are artificial and thus less robust than natural experiments with respect to real-world conditions. This is especially true if too much control is exerted over the service mix (or other aspects of the discretionary decisions) of program operators.
However, at least three candidates for random assignment study merit consideration. First, financial and other incentives for employers represent one option. By providing specific incentives to some employers and not others, or for some employees and not others, the experiment would be designed to induce different employer practices, which would be expected to generate different outcomes for TANF recipients. However, it might not be feasible to modify an existing tax incentive (such as the WOTC), or to establish a new tax incentive, only for randomly selected employers. In addition, past experience with wage subsidies paid to employers suggests that offering tax incentives to employers for some job candidates and not others can stigmatize those individuals.(19) However, some form of time-limited payments to employers or intermediaries might be used instead of changes in tax rules.
Second, TANF/WIA One-Stop Centers are another candidate for study. They work with large numbers of welfare recipients and employers, random assignment seems feasible, and they are accustomed to collecting data. The available research suggests that One-Stop Centers vary in terms of their employer focus, TANF-WIA coordination, and performance outcomes. Therefore, perhaps, an intensive, employer-focused One-Stop treatment could be delivered at selected centers to TANF recipients randomly assigned to a treatment group. A less intensive, "standard" treatment could be provided to control group members.
Third, community intermediaries such as Project QUEST are potential candidates for rigorous study. Many are sophisticated initiatives serving a large number of TANF recipients. Were intermediaries such as Project QUEST to be evaluated using an experimental research design, either employers or employees could be randomly assigned to treatment and control groups.(20) Several intermediaries have been assessed as part of one or more qualitative studies. The Center for Employment Training (CET) was rigorously evaluated in three different random assignment evaluations, although many of the key outcomes associated with its labor market intermediary role — notably its interactions with and impacts on employers — were not considered.(21)
Without question, comprehensive surveys and random assignment experiments would be the best way to address the unanswered questions identified earlier in this chapter. However, a well-run experiment would require substantial time (at least four years) and money to implement. With a large sample (2,000 or more in both the treatment and control groups), multiple waves of survey data collection, and intensive field research, the cost of such a study could easily reach $15-20 million.
A random assignment experiment also would require cooperation — in terms of random assignment, service provision, and data collection — from the intermediaries and other organizations participating in the research. As a result, it may make sense to study natural experiments and/or conduct other research first, and pursue larger-scale planned experiments later, after more has been learned.
[ Go to Contents ]
It is clear that rigorous impact research is needed to measure the relative effectiveness of these practices. However, it is equally clear that several preliminary research steps must be taken before such research can be carried out. First, it is important to document the use of various employer practices in recruiting, hiring, supporting, and managing employees. What are these practices and how often are they used by different types of employers? How often are recruitment, hiring and employment functions outsourced, and how frequently are different types of intermediaries used for this purpose? A new survey is probably the best way to take this important first step, and would be an important end in itself. However, further analysis of the Holzer employer surveys and the Welfare to Work Partnership intermediary surveys also would be beneficial.
Second, it is critical to develop testable hypotheses for alternative practices and intermediaries. A new survey would also help in doing this, particularly if it involves in-depth data collection on particular practices and intermediaries. Other research efforts, notably systematic case studies on innovative practices and specific analyses of existing quantitative datasets, would also be useful toward this end.
[ Go to Contents ]
(17) In the survey employers were asked a series of questions about the most recent TANF recipient they hired, including "How was this employee recruited?" and "Did you have the applicants take any test?" Later they were asked a series of questions about the job performance of the same individual, including questions about absenteeism, tardiness, and overall performance.
(18) The LEHD dataset is described on the Census Bureau's website at http://lehd.dsd.census.gov.
(19) See G. Burtless, "Are Targeted Wage Subsidies Harmful? Evidence from a Wage Voucher Experiment", Industrial and Labor Relations Review. vol. 39 (1985).
(20) A classic experimental research design would entail randomly assigning job applicants and/or employees from a variety of employers to treatment and control groups; the intermediaries would work only with the treatment group, and employment and other outcomes for these two groups would be compared. An alternative design would involve random assignment of employers. After identifying enterprises with an interest in partnering with the intermediaries, the enterprises could be assigned to treatment or control status and the intermediaries would provide services only to enterprises in the treatment group. The outcomes of new employees of the treatment enterprises would be compared to those of new employees in the control enterprises.
(21) Community Development Financial Institutions (CDFIs) also offer an interesting alternative approach to promoting demand for labor from welfare recipients and other low-income individuals. Among the more than 500 CDFIs in the country overseen by the U.S. Treasury Department, there are a number of community venture funds that make equity investments in local businesses that locate in and employ people from high-poverty neighborhoods.
Main Page of Report | Contents of Report
Home Pages:
Human Services Policy
(HSP)
Assistant Secretary for Planning and Evaluation
(ASPE)
U.S. Department of Health and Human Services
(HHS)