We used an exploratory case-study methodology, examining five carefully selected CER studies of the recent past to identify some of the potential root causes for the failure of CER to change clinical practice. We anticipated that our results would inform the development of subsequent qualitative or quantitative studies using more-rigorous designs. Given the need to inform the current portfolio of federally funded CER, our priorities were to distill the key lessons from our case studies, using expert opinion from a broad range of stakeholders in a relatively short time frame. Because of these factors, a number of limitations regarding our study’s scope, methodology, and generalizability should be kept in mind when interpreting our results.
We struck a specific balance between depth of content and breadth of inclusion of case studies, because our primary objective was to identify and synthesize themes across cases. Since we had a limited sample of expert discussions and stakeholder perspectives for each case study, we emphasized the early phases of the CER translation process, limiting discussions about the local implementation phase, on which there is a large existing literature. As a result, we had only a limited number of discussions with primary-care providers, particularly those who practice in nonacademic settings. Community-based providers might have perspectives that differ considerably from those of the opinion leaders with whom we tended to speak. In particular, we might not have identified the full range of implementation barriers that individual physicians confront when making treatment decisions on the basis of CER evidence. A larger sample of end users from a diversity of practice settings could identify additional barriers and enablers. In cases where we could undertake only a small number of discussions per stakeholder group, the information provided by each stakeholder might not have been representative of the larger group. In addition, we did not have time for discussions with some stakeholder groups, including popular media and individual patients.
Despite considerable effort and multiple invitations, the device industry and, to a lesser extent, the pharmaceutical industry were underrepresented in our sample of discussants. While we were able to identify potential discussants, most of them declined to participate, citing concerns about the sensitivity of the information they might be asked to share. The perspectives of the device industry would have been useful for the COMPANION case study and, to a lesser extent, for SPORT and COURAGE. Our retrospective study design also proved challenging in this regard, as general industry turnover meant that many potential discussants were not employed by the industry in a suitable capacity at the time the results of studies such as CATIE were released and so were unable to provide insight into the strategies and dissemination activities surrounding the CER translation process. This challenge will confront any retrospective examination of CER dissemination and will be true of many potential discussants.
While we covered a broad range of topics, we did not include a case study on diagnostic imaging, which might have added value given the role of diagnostic imaging in determining subsequent utilization of medical care. We also might have considered CER studies that compared behavioral interventions (e.g., dietary therapy for patients with diabetes); however, we believed that behavioral interventions would involve too many unique factors that would limit our ability to draw conclusions over multiple case studies. Finally, we might have expanded our scope to specifically include one or more “success stories,” to better draw contrasts between factors that impeded and facilitated translation. We ultimately chose not to use this criterion to select topics because of the difficulty of identifying a sufficiently large sample of unambiguous success stories. Instead, we prioritized topics in a way that ensured diversity of types of treatment comparisons.
Because of the formative goals of our study, we elected not to use a formal qualitative research methodology that included the coding of themes with the use of specialized analytic software. Biased interpretations of the data by the research team were mitigated by requiring a minimum of three investigators to be present for each discussion. In addition, we held a debriefing immediately following each discussion to identify key themes, and we drafted case-study-specific summaries to refine the presentation of themes and to arrive at a consensus interpretation. Because we did not record each discussion, our analysis was based on examination of the transcripts generated by the dedicated note takers. Although we were not always able to derive exact quotes from discussants, we used email follow-up for areas that needed clarification. While we were unable to verify the veracity of statements or double-check the accuracy of quantitative estimates, our root-cause analysis drew primarily on themes that were mentioned repeatedly by stakeholders. Future studies might use more-structured interviews to capture similar types of information from a greater number of discussants.
Because our case studies dated from as early as 1998, stakeholders’ recollections of specific events may have been subject to recall bias. We believe this bias, if present, was likely to be mitigated by the fact that most stakeholders were intimately involved with the relevant studies. Moreover, the crosscutting discussions we held with certain stakeholders, including payers and representatives of integrated delivery systems, did not require detailed knowledge of each case study. We might have used greater numbers of group discussions to guard against recall bias introduced by any individual stakeholder or to ensure that each discussion more accurately reflected the perspective of the larger stakeholder group. It was logistically challenging to schedule group discussions, so we prioritized individual discussions with senior managers or key opinion leaders who we assumed would be most informed about a broad array of issues.
The environmental scans we conducted for each case-study topic were helpful in structuring discussions. However, we were unable to integrate findings from the grey literature, which contains highly redundant information (e.g., similar press releases were picked up by dozens of trade publications) and has few identifiable authoritative sources. Because we were thus unable to conduct an efficient search of the grey literature, we may have missed an opportunity to gain greater insight into additional perspectives—particularly those of insurers and device and pharmaceutical manufacturers.
Because of the limited scope of topics we could address and the limited number of discussions we were able to hold, some of our findings regarding the root causes of failed CER translation or the facilitators of practice change may not be generalizable to other topics or to a broader range of practice settings. For example, small health plans may face a different set of barriers to the use of CER to guide policymaking than those faced by payers that operate nationally. Community physicians may have different interpretations of the meaning of these studies and may also face a different set of implementation challenges. Because of the extensive heterogeneity within each stakeholder group, our findings may be significantly more nuanced than we have presented them to be.
The CPOE case study, in particular, may not provide results that generalize to the broader set of delivery-system interventions. CPOE is a heterogeneous intervention that can be simple or complex. It may be integrated into EHRs or stand alone, and its successful adoption is much more highly context-dependent than are the subjects of our other case studies. Our findings from the CPOE case study may be more generalizable to informatics topics than to the broader set of delivery-system interventions, which have substantial heterogeneity. As identified by the IOM, this is a high priority for future CER, and we emphasize that while our findings concerning adoption of CPOE were relatively negative, CER may have considerable potential to bring about changes in practice for the class more broadly.