This technical appendix describes the two-phased approach to comparing the impact of nine work promotion strategies used in health and welfare programs in the United States.
Literature Review
We used a variety of methods to conduct a search of rigorous research on strategies to promote work in health and welfare programs. First, a preliminary search was conducted using the Pathways to Work Evidence Clearinghouse to find recent, rigorous evidence on interventions that provide services such as case management, childcare, apprenticeships, and sanctions. We included relevant syntheses and studies from these sources. Next, we conducted Google Scholar searches separately by project question. Third, we conducted an advanced search in Academic Search Premier. Finally, we included relevant studies found in snowball searches. To be reviewed, studies met the following criteria:
- Written in English
- Examine a program or policy relevant to work penalties or work incentives
- Examine employment outcomes or mechanisms to increase employment
- Not an opinion piece
- Implemented in the United States
To decide which studies to include, we followed the Pathways Clearinghouse evidence standards for randomized controlled trials and quasi-experimental designs. We used the Clearinghouse for Labor Evaluation and Research (CLEAR) and What Works Clearinghouse (WWC) standards for difference in differences and regression discontinuity designs. We then sorted high- and moderate-quality studies from the Pathways Clearinghouse as well as newly assessed rigorous studies into the following nine strategies. Each study could include more than one strategy.
- Work requirements
- Work First approach
- Programs that match optimal credentials with job requirements
- Cash transfers conditional on working
- Programs that offer work supports such as child care or transportation
- Programs that offer coaching or case management
- Incentives for employers to hire apprentices or other temporary workers
- Programs that combine work and training
- Programs that collaborate with labor market partners.
Estimating Impact of Strategies
For all strategies and policies, we assessed the strongest evidence we found, prioritizing studies that took place in the United States, that were randomized controlled trials, and that were published since 2015. We specified whether evidence comes from a randomized controlled trial, quasi-experimental design, or descriptive study, and ranked the strength of the evidence for strategies that promote work into strong, moderate, or additional evidence.
We assessed changes in employment rate and earnings between intervention and comparison groups as reported by study authors. We categorized findings on short-term and long-term earnings and employment following the Pathways protocol, not just positive or statistically significant findings. We defined short-term as 18 or fewer months and long term as between 18 months and 5 years. We included the most recent available finding (for example, findings after 18 months rather than after 12 months for the short term and findings after year 4 rather than findings after year 2 for the long term).
To standardize across different studies and populations to produce an average impact for each strategy, we adjusted these author-reported impacts by standard deviations and report out impacts in 2018 percentages and dollars. We collected effect sizes for each study, averaged them, and converted back to 2018 measures. The effect size is the strength of the effect measured in standard deviations, based on the distributions of outcomes for adults with low earnings potential in the 2019 Current Population Survey Annual Social and Economic Supplement.
The table accompanying this page includes the 132 studies included in our analysis.