Evaluation Design Options for the Long-Term Care Registered Apprenticeship Program. 5.3. Evaluation Designs to Determine Effects on Employers

09/01/2011

Exhibit 4 presents a range of research designs for the evaluation of the program’s effects on employers and pros and cons for each design. These designs are not necessarily mutually exclusive and could be combined in various ways. The relatively small number (119 across four occupational categories) of programs limits the evaluation design options for employers, but new LTC RAPs may expand options. In addition to gathering information on the effect of the apprenticeship programs on employers, data on employers would also provide control variables for the analyses of the effect of LTC RAP on direct care workers.

EXHIBIT 4. Research Designs for Evaluation of Effects of LTC RAP on Employers, Ranked from Strongest to Weakest in Scientific Rigor
Design Pros Cons
Randomly assign long-term care employers into two groups: a treatment group that will heavily use LTC RAP and a control group that will not use LTC RAP
  • Strongest possible design, with recognized ability to attribute effects to the intervention
  • Obtaining a large enough sample of employers to do quantitative analysis will be difficult and expensive
  • Providers recruited because they are interested in improving their training programs may not be satisfied with being in control group and may adopt other training programs
  • Some providers in the intervention group may not implement LTC RAPs
  • Requires a long time period for employers to adopt the program and master its use
Compare outcomes for multi-site employers who use LTC RAP in some sites but not in other sites; also compare changes in outcomes by site
  • Holds constant many employer-specific factors not linked to the type of training
  • Might attract employer participation and interest
  • Captures worker and employer impacts, including potential organizational effects
  • Offers a direct way of estimating costs and benefits to the employer
  • Number of LTC RAP employers which have multiple sites is small
  • Might involve selection bias because sites that employers choosing to implement LTC RAP may be systematically different from sites not using long-term care
  • Outcomes might depend non-LTC RAP site-specific factors
  • If number of apprentices per employer is small, program is not likely to have an impact on organizational performance
Compare outcomes for LTC RAP employers with matched employers not offering LTC RAPs
  • Relatively strong design, commonly used in evaluations
  • Captures worker and employer impacts, including potential organizational effects
  • Offers a direct way of estimating costs and benefits to the employer
  • Results may be result of selection bias, if employers who are more (or less) effective in other ways disproportionately adopt LTC RAP
  • Cross-sectional design limits ability to interpret differences as due to the intervention
  • Comparison group may include facilities with some other training initiative
  • If number of apprentices per employer is small, program is not likely to have an impact on organizational performance
Cost-benefit analysis to ascertain costs for local LTC RAP design and implementation and benefits realized as improvements in worker productivity and quality
  • Low-cost
    Commonly used in social science evaluations
    Requires only a modest number of employers
  • Does not employ statistical controls found in regression approaches listed above for non-LTC RAP effects and selection bias
    Less generalizable than regression-based approaches -- no comparison group
    Requires significant cooperation of employers to provide data
Conduct focus groups of employers who operate LTC RAPs and of employers who do not offer LTC RAPs. Participants will be primarily administrators attending national conferences
  • Low-cost
  • Provides detailed views of employers
  • Can provide detailed recommendations for improving LTC RAP
  • Allows for some comparison with employers not offering LTC RAPs
  • Qualitative data cannot be used to yield quantitative impact estimates of the effectiveness of intervention
  • Representativeness of views expressed cannot be directly assessed
  • Comments provided cannot be easily quantified and counted
Conduct focus groups only with employers operating LTC RAPs
  • Low-cost option
  • Provides information on views of employers
  • Can provide detailed recommendations for improving the LTC RAP
  • Focus groups could be organized at national provider conventions
  • Qualitative data cannot be used to determine effectiveness of intervention
  • Representativeness of views expressed cannot be directly assessed
  • Comments provided cannot be easily quantified and counted
  • Comparisons cannot be made to employers who do not participate in LTC RAP
Case studies of LTC RAP employers compared to employers without LTC RAPs
  • Low-cost
  • Can obtain “rich” description of programs and employer views
  • Offers comparison to organizations without programs
  • Qualitative data cannot be used to determine effectiveness of LTC RAP
  • Results may be result of selection bias, if employers who are more (or less) effective in other ways disproportionately adopt LTC RAP
  • Case studies recently have been conducted of major LTC RAPs. Relatively little to be gained by more case studies at this time
Case studies of LTC RAP employers with no comparison group
  • Lowest cost
  • Can obtain rich description of programs and employer views
  • Qualitative data cannot be used to determine effectiveness of LTC RAP
  • No comparison group
  • Case studies recently have been conducted of major LTC RAPs. Relatively little to be gained by more case studies at this time

View full report

Preview
Download

"LTCRAPedo.pdf" (pdf, 870.86Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®