[ Main page of Report | Contents of Report ]
This chapter has two purposes. The first is to assess the value of available research studies and data sources, which the study has found to be substantial in some areas, but weak in others. The second is to identify the most important questions presently unanswered by the existing research. Generally speaking, these are questions that require nationally representative data on the practices of employers and labor market intermediaries and/or credible evaluation evidence on these practices.
[ Go to Contents ]
Three features of the research literature are especially important from the standpoint of this project. First, few studies have concentrated on the important practices of employers and labor market intermediaries in recruiting, hiring, retaining, and managing TANF recipients. Exhibit 3.1 lists some of these practices, most of which can be performed by the employers themselves or outsourced to intermediaries.
The vast majority of studies that have examined employer practices regarding TANF recipients have done so in the context of a larger set of research concerns. The focus of the research typically has been employment and public assistance outcomes for TANF recipients and the demand for low-wage and entry-level workers, of which TANF recipients are a segment. There are some exceptions, however, and these studies play a prominent role in this assessment.
Second, when studies have considered employer and intermediary practices, this attention has been almost entirely descriptive rather than evaluative. A number of studies have identified and explained employment practices, but only one study has collected consistent and detailed data for a large group of employers and no research has gathered this kind of information on intermediaries. No study has systematically assessed the effectiveness of employer and intermediary practices.
Third, for many important employer topics, the available research literature is exclusively qualitative. Similarly, very little quantitative information has been assembled for most types of labor market intermediaries. This is not surprising, because many intermediaries do not work with large numbers of TANF recipients and/or do not track recipients separately from other clients, making quantitative analysis very difficult.
| Category | In-House Employer Practices | Intermediary Practices |
|---|---|---|
| Recruitment and Hiring | Job Description Outreach Advertising Contact with Referral Sources Applicant Screening Applications/Resumes Drug/Criminal History Screening Aptitude/Skills Testing Use of Screening Criteria Applicant Evaluation Interviewing Reference Checks |
Prescreening Training General Training Job-Specific Training Applicant Screening (outsourced by employer) Applicant Evaluation (outsourced by employer) |
| Training | In-House Training Soft Skills-Workplace Soft Skills-Life Skills Training Hard Skills-Classroom Hard Skills-On-the-Job Outside Training College Technical/Vocational School Other |
Employer Training (outsourced by employer) In-House Training Soft Skills-Workplace Soft Skills-Life Skills Training Hard Skills-Classroom Hard Skills-On-the-Job Training Referrals |
| Employee Support | Mentoring Employee Assistance Job Coaching Services Counseling Transportation Assistance Assistance with Obtaining Public Supports (e.g., Food Stamps, Medicaid) Work Schedule Flextime Work Options |
Mentoring Employee Assistance (outsourced by employer) |
| Employee Management | Communication Compensation Wages Fringe Benefits Job Performance Assessment Formal Assessments Feedback Career Planning Job Advancement Policies |
Conflict Management (outsourced by employer) |
If Workforce Investment Act (WIA) One-Stop Centers are considered a type of intermediary, they constitute a noteworthy exception to this characterization. However, even for studies of One-Stop Centers the case study method is the dominant approach and data collection has generally been limited to open-ended interviews and site visits.
Although some studies attempt to relate participation and outcomes to specific program approaches, only one that we have reviewed has done so using quantitative data and statistical analysis (U.S. General Accounting Office, 2002a). Moreover, only a few of the One-Stop studies present quantitative participation and outcome data.(11)
As a result, the available research evidence on employers — the demand side of the labor market — is quite limited. In sharp contrast, there is a substantial body of research evidence, much of it based on rigorous evaluations, addressing the labor supply of welfare recipients under varying conditions. Studies have carefully compared recipients' labor supply with and without employment mandates, with and without financial incentives, and with alternative packages of employment-related services.
Comparable demand-side studies do not exist. Moreover, few studies have sought to compare outcomes — either employee outcomes (e.g., job retention) or employer outcomes (e.g., utilization of government-funded work supports) — for different employer or intermediary practices in a systematic fashion. There are several potential barriers to conducting random assignment experiments and reliable statistical studies of alternative employer and intermediary practices; notably, these studies' cost and necessary prerequisite knowledge (discussed in the next chapter). Thus, it may be difficult to conduct more systematic research. That said, the "best practices" and policy recommendations offered by available studies often rest more on assumptions and judgments than on strong evidence.
Among existing databases, one stands out in importance for addressing the questions discussed in this report. This is the telephone survey of approximately 3,000 employers, focusing on employer experiences with TANF recipients and directed by expert panel member Harry Holzer in 1998 and 1999. The survey covered employers in four metropolitan areas: Chicago, Cleveland, Los Angeles, and Milwaukee. The sample of business establishments was chosen randomly, with the probability of an employer's selection proportional to the number of its workers. Holzer also conducted the Multi-City Telephone Employer Survey (MCTES) in the early 1990s, which asked employers about their recruitment, hiring and employment of low-skill workers — including, but not limited to TANF recipients.
Holzer and his collaborators have conducted a number of analyses of each of these two surveys of employers. As discussed in the next chapter, valuable additional analyses could be done on issues covered by the surveys, but not yet addressed analytically by Holzer or others.
The importance of these surveys stems primarily from two features. First, the content of these surveys is entirely "on target." They were carefully designed to collect objective and detailed information from employers on important aspects of employers' experience with TANF recipients. For example, one series of questions addressed the specific skills sought by employers for particular jobs that have recently been filled. Respondents were asked "Does this position involve filling out forms on a daily basis?" and "Does this position involve keeping a close watch over gauges, dials, or instruments of any kind?" No other data source comes close to providing pertinent quantitative information at this level of detail.
Second, the surveys were administered to a stratified, random sample of employers in four major cities, including Chicago and Los Angeles, two of the largest cities in the U.S. The response rate for the interviews exceeded 60 percent in all four cities. The resulting employer sample is much more representative than comparable data sources (the exception is databases based on employer-reporting systems such as Unemployment Insurance and Bureau of Labor Statistics).
The surveys, however, have several noteworthy limitations. First, they were administered in only four cities, three of which are in the Midwest, and thus may not be nationally representative. As noted in the last chapter, research indicates that employers in urban centers are much more likely to hire TANF recipients than other employers. The absence of nonmetropolitan employers in these surveys may consequently skew their findings from a national perspective.
Second, the surveys were administered near the peak of the economic expansion during the 1990s. Employer demand for unskilled workers was extraordinarily high, which undoubtedly affected the survey responses.
Third, the surveys focused on recruitment and hiring practices, paying much less attention to employer practices following hiring. Indeed, the respondents for the surveys were the individuals identified as responsible for company hiring. As a result, the other practices shown in Exhibit 3.1 were largely ignored.
Fourth, the surveys paid limited attention to labor market intermediaries. Respondents were asked about organizations with which employers worked in recruiting and hiring individuals. However, they were asked little either about the intermediaries' mission or about the specific services they performed. This has implications for the interpretation of the survey findings. For example, an employer's interest in hiring TANF recipients can be expressed in different ways. One is an employer's willingness to hire individual recipients. Another is an employer's tolerance for recipients' relatively poor soft skills and relatively high support service needs. Both of these issues are addressed quite well in Holzer's surveys. However, another way employers show their interest is in working with labor market intermediaries that focus on TANF recipients, including welfare agencies. While expressions of willingness and tolerance on a survey suggest potential interest in welfare recipients as employees, established working relationships with the right intermediaries demonstrate this interest.
Two surveys of labor market intermediaries were undertaken at about the same time as Holzer's surveys of employers. The surveys, which were completed by more than three-quarters of the 214 intermediaries identified by the Welfare to Work Partnership in the Atlanta and New Orleans metropolitan areas,(12) identified the services they provided to employers in connection with TANF recipients. The surveys in Atlanta and New Orleans were used to develop guides for use by employers in those cities (Abt Associates, 1999a and 1999b), and have not been used for research purposes.
These surveys are important for two reasons. First, they collected detailed information from intermediaries on (1) the missions of the intermediaries, (2) all types of services they provided to employers, and (3) the types of companies and business sectors they served. Second, the survey sought to interview all pertinent labor market intermediaries identified in two major metropolitan areas. The survey response rate for these interviews was 72 percent in New Orleans and 81 percent in the Atlanta area. This is the only survey of intermediaries that has been administered in this way.
Yet these surveys have clear limitations. Like the Holzer surveys, they were administered in urban settings — in this case two metropolitan areas — during the late 1990s. Importantly, in large part because the intermediary universe is a fraction of that for employers, the sample of completed interviews is very small — only 164. Finally, while intermediaries were asked to identify several employers with which they worked, follow-up interviews were done with only a small sample of these employers.
[ Go to Contents ]
The existing research provides solid answers to some of ASPE's questions and less satisfactory answers to others. The questions in the first category are the ones that can be addressed with descriptive information. The available research explains reasonably well which types of employers are most likely to hire TANF recipients, the main reasons employers hire or do not hire welfare recipients, and identifies those segments of the TANF population that present the greatest challenges to employers. While there are answers to these questions, the answers could be sharpened with additional research using a combination of TANF and UI data, WIASRD, and the U.S. Census Bureau's Longitudinal Employer-Household Dynamics (LEHD) data described below.(13)
The questions that cannot be answered reliably are the ones that require comprehensive descriptive or credible evaluation evidence. The available research studies cannot tell us what works because they have not compared, in any systematic way, the performance of different types of employer and intermediary practices. Ultimately, their performance should be compared in terms of multiple outcomes. "Success," from the standpoint of employers, must take account of outcomes such as job retention; from the perspective of TANF recipients, TANF administrators, and intermediary organizations, comparisons should also consider outcomes such as job advancement, improvement in job skills, and measures of progress toward economic self-sufficiency.(14)
Thus, ASPE's second question — What do employers relying on low-skill, entry-level workers do to employ TANF recipients successfully? — currently can only be answered based on descriptive information, which is far from comprehensive, in terms of industry and geography, and the range of important practices. Some claim to have identified practices for employing TANF recipients "successfully." However, if "successfully" has a comparative meaning — for example, if it means with better than average employment outcomes — these claims are on shaky ground. The practices have not been systematically compared.
ASPE's last question — What would enable employers to increase and improve their efforts? — at present must be answered based on limited information. We simply do not know what types of employer or intermediary practices work best or what government actions can be taken to improve employer practices. The research literature is filled with case study evidence that particular practices hold promise. However, in only a few areas, notably temporary employment agencies, does the evidence involve consistent data collected across multiple practitioners.
Fundamentally, what is needed is research evidence that compares inputs and outcomes — and the interactions between the two — for alternative employer practices used to achieve specific objectives in recruiting, hiring, and employing TANF recipients. Ideally, such evidence should be assembled for different types of practices in recruitment and outreach, screening, hiring, training, performance assessment, employee support, and compensation and advancement. Ideally, too, the information on inputs and outputs should be sufficiently detailed to support analysis of the reasons particular practices are effective or ineffective. For example, if specific types of employer training result in higher job retention and employee earnings, it is important to identify the features of the training — such as its curriculum, teaching methods, and setting — that lead to these improvements. This would be easier if consistent data were available on inputs, such as types of computerized instruction, and intermediate outcomes, such as skill test scores for individual employees.
While systematic comparisons of this kind are needed for the full range of employer practices, several practices deserve priority attention given the findings of the research summarized in Chapter 2 and other research. For example, given the research evidence on the effectiveness of mentoring in other settings,(15) a comparison of employers with and without mentoring programs would be instructive. The presence or absence of a meaningful soft skills training program could be examined in a similar fashion.
Comparisons could be based on a sample of employers and their TANF recipient employees. For example, assessing the experience of employers using different recruitment strategies would entail identifying the recruitment methods and resources used by the employers, measuring short- and longer-term outcomes for job candidates and employees, and analyzing differences associated with individual practices.
Alternatively, such comparisons could focus on labor market intermediaries. One critical area in which employer practices differ is in the use of intermediaries, and the practices of these labor market organizations vary as well. Recent research attention has focused on community organizations such as the Center for Employment and Training (CET) in San Jose, California, and Project QUEST in San Antonio, Texas (McGahey, 2003), and on temporary help firms such as Manpower, Inc. (Autor and Houseman, 2002). However, other institutions, such as community colleges and WIA One-Stop Centers, also play an intermediary role. If we take this broad view of intermediaries, a substantial fraction of the firms that employ TANF recipients may be said to utilize them.
In some cases intermediaries exert a strong influence on employer practices and secure explicit commitments to hire welfare recipients and other low-wage workers. For example, Project QUEST has sometimes operated as "an extension of the firm's human resources department" (Osterman and Lautsch, 1996). In other cases, good working relationships have been established, but the employers' hiring behavior has not been noticeably altered by the intermediaries.
Much of the research reviewed in the last chapter focused on employer behavior, ignoring the fact that employers use intermediaries. This omission, though less important in many other circumstances, is problematic in the context of TANF recipients. As noted earlier, intermediaries represent one of the main vehicles used by employers to recruit, hire, and support workers from the welfare population.
Exhibit 3.2 depicts a simple comparison of activities and short- and longer-term outcomes for employers using two types of intermediaries and employers relying on their own human resources (HR) procedures without the involvement of intermediaries. The first row of boxes in the figure is the one that involves employers working with intermediaries focusing on welfare recipients. The second row shows employers partnering with intermediaries that serve a broader group of workers, and the third shows employers relying on their own HR offices.
Each column of boxes in the figure involves potential comparisons. Comparing the boxes in the first column — that is, the first box in each row — raises a number of questions. For example, what are the reasons some employers work with TANF-focused intermediaries while others do not? Which types of intermediaries do employers most often use — private firms (such as Manpower, Inc.), public agencies (such as the TANF/WIA One-Stop Centers), or community-based organizations (such as Project Match in Chicago)? For what reasons do employers choose not to work with such organizations?
The boxes in the second column highlight another related set of questions. For example, what services do employers want from intermediaries? How do these services affect employer procedures? What are the services intermediaries arrange for and provide, particularly for TANF recipients? How do recruitment practices differ for employers using TANF-focused intermediaries, other intermediaries, and their own HR departments? How does involvement with an intermediary change employer practices such as screening? How does applicant screening differ across these three sets of circumstances? How do hiring practices differ? How do employee support, performance assessment, and other practices differ?
Exhibit 3.2
Activities and Outcomes with and without Labor Market Intermediaries
For the third and fourth columns in this figure, how do key short- and long-term employment outcomes — in terms of hiring and job retention and advancement — differ across these three situations? In particular, are the employment outcomes significantly different? If the outcomes are significantly better for employers using intermediaries than for employers that do not — and if there are no differences between the employers and employees other than in the use of intermediaries — then intermediaries are producing a net impact on employment. While less than ideal,(16) such comparisons would provide more credible evidence of the intermediaries' effectiveness than currently exists. If the outcomes are indeed different, we would want to know why they are different. What specific activities or practices account for the value added by intermediaries? Gaining answers to such questions would be challenging for researchers, but also critically important for employers and policymakers.
Several studies have assessed the activities and outcomes of individual intermediaries such as Project QUEST (e.g., Rademacher et al., 2001). Clearly, however, the value of such information would be enhanced if it could be compared to the activities and outcomes of other intermediaries and/or employers that handle employment matters without the help of intermediaries. The same can be said for several other potentially important comparisons of employer practices. For example, one crucially important group of employer practices for the TANF population involves the management of workers once they are hired. Practices such as the assessment of employee performance are likely to be critical to job retention and advancement for welfare recipients.
In this important and broad area of employee management, it is clear that employer behavior differs. However, the pertinent research evidence is limited. As discussed earlier in this chapter, more attention has been paid to recruitment, screening and hiring methods, as well as to compensation matters, than to other practices. However, several of the post-hiring practices listed in Exhibit 3.1 — such as soft skills training, the use of different formal and informal job performance methods, the use of mentoring and other types of worker support, and the availability of flexible scheduling options for employees — may be extremely important. Some of these practices, such as scheduling, only involve employers. Other practices, such as soft skill training, may involve intermediaries, either by referral or through outsourcing arrangements.
[ Go to Contents ]
(11) Quantitative data on service utilization at One-Stop Centers is maintained in the WIASRD (Workforce Investment Act Standardized Record Data) data system, which is described in Chapter 4. However, as explained in Werner et al. (2002), the vast majority of available studies on the Centers have utilized case study information collected from individual programs rather than aggregate data from WIASRD.
(12) The Welfare to Work Partnership identified 237 potential intermediaries in the two metropolitan areas, 23 of which said they did not work with TANF recipients or did not provide pertinent labor market services. Of the remaining 214 organizations, 164 (77 percent) completed the full survey.
(13) These data sources provide extensive individual-level information on TANF recipients, but limited information on their employers.
(14) According to one definition, full economic self-sufficiency involves independence from poverty as well as from TANF and other forms of public assistance. Researchers have developed different measures of full and partial self-sufficiency. For a discussion of self-sufficiency outcomes based on this definition, see D. A. Long, "From Support to Self-Sufficiency: How Successful Are Programs in Advancing the Financial Independence and Well-Being of Welfare Recipients?" Evaluation and Program Planning, vol. 24 (2001), pp. 389-408.
(15) See, for example, J. P. Tierney and J. B. Grossman, Making a Difference: An Impact Study of Big Brothers/Big Sisters (Philadelphia, PA: Public/Private Ventures, 2000); and C. L. Sipe, Mentoring: A Synthesis of P/PV's Research (Philadelphia, Pa: Public/Private Ventures, 1996).
(16) Unless employers are randomly assigned to relationships with intermediaries, the results of these comparisons would be subject to selection bias. Such an assignment process is probably not feasible. However, in comparing outcomes for employers, an effort could be made to statistically control for differences between employers that use intermediaries and those that do not. Alternatively, job applicants or employees could be randomly assigned to intermediaries that provide recruitment, screening, and other services to employers (see Chapter 4 for further discussion).
Main Page of Report | Contents of Report
Home Pages:
Human Services Policy
(HSP)
Assistant Secretary for Planning and Evaluation
(ASPE)
U.S. Department of Health and Human Services
(HHS)