The essence of evaluation research is to answer the question: How do the outcomes of the participants in the intervention compare with the outcomes they would have experienced had they not participated? For this evaluation, the focus is on how the LTC RAP training affects workers and employers compared to the standard training that direct care workers normally receive.
A key issue, then, is how to identify a comparison group that is similar to apprentices in the LTC RAP. In the case of random assignment, the control group is very likely to yield reliable estimates of what would have happened in the absence of the intervention (also known as the counterfactual) because the assignments to treatment and control groups are random. However, for other types of evaluations, a potential threat to the validity of the findings is that the comparison group and the LTC RAP group differ in terms of characteristics that affect the outcomes of interest but are not related to the operation of the intervention. This could well be the case if apprentices are more motivated and ambitious than people who are not in the LTC RAP. Similarly, employers that voluntarily participate in the LTC RAP are likely to have an organizational culture that places a higher priority on the importance of training direct care workers than employers that do not participate in the program.
An additional complication in choosing comparison groups is that the effect of the LTC RAP is likely to vary over time; that is, the effect of the program on an apprentice is likely to be greater towards the end of the training program than it is at the very beginning. Thus, ideally, the comparison group would start with people at the beginning of their employment at the provider and follow them over time. However, collecting data on a flow basis is more difficult and expensive than data collected from people all starting at the same time because data need to be collected whenever a new person starts work, which could be over a long period of time.
Some possible comparison groups are:
Randomized treatment and control groups. The classic solution to the problem of selection bias is randomization between the treatment and control groups. However, randomization is more difficult to implement in ongoing programs because employers may be reluctant to randomly assign workers to an apprenticeship if they do not think they are capable of benefiting from it. Similarly, they may be unwilling not to assign workers to the treatment group if they think the workers could benefit the organization by receiving the apprenticeship training. In addition, employers may not be willing to be part of a control group if their motivation in participating in the study is to improve their training programs.
Matched sample of employers and matched sample of workers working for these employers. While matching employers based on simple characteristics, such as number of nursing home beds, would be simple; gathering information about other variables, such as organizational culture, would be difficult and time consuming. Employers who do not have an apprenticeship program would have limited incentives to participate in the survey and to provide confidential contact information on their employees. Two levels of sampling -- facility/agency and workers -- could result in relatively low final response rates.
Apprentices with matched workers within the same employer. It would be easier to identify the sample and compare people without having to control for the employer. However, if LTC RAP participation causes changes for the organization as a whole (or if employers choose to participate because of certain employer characteristics), then differences between apprentices and non apprentices may be minimal. Moreover, findings from the site visits suggest that employers consciously use apprentices as peer-mentors for persons not in the apprenticeship program, thus contaminating workers who are not formally in the apprenticeship program. This approach would also leave the evaluation without an employer-level comparison.