Evaluation Design Options for the Long-Term Care Registered Apprenticeship Program. 5.2. Evaluation Designs to Determine Effects on Apprentices

09/01/2011

Exhibit 3 presents a range of illustrative evaluation designs for the evaluation of the LTC RAP’s effects on apprentices, ranked from strongest to weakest according to scientific rigor. The strength of the design is an assessment of how well the findings can be defended as measuring the true effect of the intervention as opposed to resulting from a methodological weakness in the design. The strongest designs have the highest costs; conversely, the lowest cost designs have major limitations. Although this exhibit ranks these potential evaluation designs in terms of their scientific rigor, it does not necessarily rank them in terms of their feasibility and desirability for the evaluation of the LTC RAP.

EXHIBIT 3. Possible Research Designs for Evaluation of Effects of LTC RAP on Apprentices, Ranked from Strongest to Weakest in Scientific Rigor
Design Pros Cons
Randomly assign eligible applicants for long-term care positions to a LTC RAP or to a standard long-term care training program (randomized controlled trial) with data collection at assignment, when apprenticeship ends, and one year afterwards. Apprentices who do not complete the program would be followed
  • Strongest possible design, with recognized ability to attribute effects to the intervention
  • Able to address a wide range of outcome variables
  • LTC RAPs and control group employers not likely to accept a randomized design because they lose control of an important component of their business
  • LTC RAP is already an ongoing program, not a demonstration
  • LTC RAP has too few employers for randomization
  • Requires at least two rounds of expensive data collection
Compare the changes over time in outcomes of apprentices entering the LTC RAP with entrants to standard long-term care jobs in other facilities/ agencies (quasi-experimental design with a comparison group). Apprentices who do not complete the program would be followed
  • Relatively strong design, commonly used in social science evaluations
  • Comparison groups could be made more similar through matching, propensity scoring, or multivariate analyses
  • Able to address a wide range of outcome variables
  • Results may be from unmeasured differences in workers or providers and not the apprenticeship program
  • No compelling reason for comparison group employers and workers to participate, reducing response rate
  • Requires at least two rounds of expensive data collection
Rigorous non-experimental methods, including natural experiments
  • Offers methods for obtaining rigorous impact estimates without requiring employers to use random assignment
  • Yields estimates based on actual program operations not based on a change in approach necessitated by the evaluation
  • May go together with the administrative record option listed below
  • Natural experiments rely on events not under the control of the evaluator
  • No known natural experiments are currently available for a LTC RAP experiment.
  • Likely to be limited to a few sites, rather than a national evaluation
Use administrative records, such as Unemployment Insurance data, to examine how earnings and retention of long-term care workers who participated in LTC RAP compare with long-term care workers in other agencies/ facilities who have not participated in LTC RAP, controlling for work history, previous earnings, and other available variables. (Quasi-experimental design with comparison group). All persons who ever participated in the LTC RAP program would be included, regardless of whether they completed the program.
  • Low data collection costs if researchers can gain access to Social Security numbers and Employer Identification Numbers, which should be possible with proper protections
  • No new reporting burden on employers or workers
  • Offers direct measures of turnover and earnings
  • Earnings and turnover information will be relevant to any cost-benefit assessment
  • Limited matching or control variables available in administrative datasets
  • Cannot control for unmeasured differences between the two groups
  • From case studies, wage rates are unlikely to increase much. Turnover for apprentice population ambiguous measure because it may signify that workers left for more formal education required to advance in field
  • Outcome variables limited to what is in administrative dataset
  • No data on views of workers or employers
Compare apprentices when they begin their training after 1-2 years (pre/post design). Only new apprentices would be included, but they would be followed regardless of whether they completed the program
  • Does not require recruitment of comparison group
  • Lower cost than gathering information for separate treatment and comparison group
  • Able to address a wide range of outcome variables
  • Changes cannot be definitively attributed to LTC RAP because design does not control for secular trends, such as recessions, inflation, changes in demand for services, and increasing experience of workers
  • Requires long data collection period as new apprentices enroll in LTC RAP on a flow basis, driving up costs
Compare apprentices at a point in time with workers who did not participate in LTC RAP in other agencies/ facilities. Persons who had ever been in the apprentice program and were still working for the same employer would be included
  • Relatively low-cost because only one round of data collection
  • Relatively weak design because there is no comparison of change in outcomes over time
  • Since cross-sectional analysis, cannot say that differences were “caused” by LTC RAP; can only say that there is an “association” between variables
  • Cannot fully control for potential selection bias
Collect single wave of data from apprentices with focus on comparing subgroups, such as Whites versus ethnic/ racial minorities. Persons who had ever been in the apprentice program and were still working for the same employer would be included
  • Lower cost because only single wave of data collection
  • Obtain information on views of apprentices on LTC RAP, which may be useful for program improvement
  • Cannot answer question of program effectiveness because no comparison to people who did not participate in program
  • Does not help build “business case” for apprenticeship programs
Conduct focus groups of workers who are apprentices and of workers who are not apprentices. Persons who had ever been in the apprentice program and were still working for the same employer would be included
  • Low-cost
  • Provides detailed views of workers
  • Can provide detailed recommendations for improving LTC RAP
  • Allows for some comparison with workers not in apprenticeship programs
  • Qualitative data cannot be used to determine effectiveness of intervention
  • Representativeness of views expressed cannot be directly assessed
  • Comments provided cannot be easily summarized or quantified
Conduct focus groups only with apprentices. Persons who had ever been in the apprentice program and were still working for the same employer would be included
  • Lowest cost option
  • Provides information on views of apprentices
  • Can provide detailed recommendations for improving LTC RAP
  • Qualitative data cannot be used to determine effectiveness of intervention
  • Representativeness of views expressed cannot be directly assessed
  • Views by apprentices provided cannot be easily summarized or quantified
  • Comparisons cannot be made to workers who did not participate in LTC RAP

View full report

Preview
Download

"LTCRAPedo.pdf" (pdf, 870.86Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®