Dissemination and Adoption of Comparative Effectiveness Research Findings When Findings Challenge Current Practices. Key Barriers to Clinical-Practice Change


Issues with the Interpretation and Dissemination of the SPORT Results

SPORT is an unusual example of CER in that two distinct interpretations based on the intention-to-treat analysis and the observational-cohort study could have different effects on clinical practice, depending on which is most prominent in the practice community and among patients. The SPORT results could be expected to either confirm current practice (yielding no change in the use of surgery), suggest that surgery is used too frequently given its limited benefit (yielding a reduction in the use of surgery), or suggest that surgery is not used frequently enough given its potential for benefit (yielding an increase in the use of surgery). The wide acceptance among key stakeholders of the observational-cohort study results—which tended to favor surgical intervention—suggests that observational CER study designs can provide useful guidance to practice and policy if a randomized clinical trial is not a feasible alternative (or is flawed because of crossover, as in this case).

Financial Incentives

Among orthopedic surgical procedures, spinal surgery is relatively highly reimbursed. The financial incentives of physicians, hospitals, and device manufacturers are aligned in a way that may promote the use of more-complex surgery. SPORT did not specifically address the relative benefits of decompression surgery and complex spinal surgery (i.e., fusion). In particular, differences in professional fees for the two types of surgery are quite large. Most of the experts we spoke with agreed that the trial results probably have not contributed to the growth in complex surgery for spinal stenosis—a trend that began nearly a decade ago. Others believe that some surgeons have used the SPORT results as justification for recommending more complex procedures, because it included patients who underwent fusion surgery.

Two other types of financial incentives may be contributing to the growth of surgery, as well as complex surgery. First, some medical device distributors pay surgeons “dividends” based on the number of devices they use. The Office of the Inspector General of HHS and the Centers for Medicare and Medicaid Services (CMS) have both warned that these arrangements may violate federal anti-kickback statutes and laws governing patient referrals (Carreyrou and McGinty, 2011). The prevalence of these arrangements is unclear but likely to be small. Second, patients who are covered by workers’ compensation policies may also be more likely to undergo complex spinal surgery, as insurers in this area are less likely to impose restrictions on coverage than those in the commercial health insurance market. Experts noted that repeat surgery rates are higher in patients covered by workers’ compensation policies.

The Importance of Referring Physicians in the Treatment Decision

Nearly all experts expressed the view that consultation with a spine surgeon greatly increases the likelihood that a surgical procedure will be recommended. This suggests that dissemination of the CER results to referring physicians may be as important as dissemination to spine surgeons. The appropriateness and timing of the referral decision could be an important area of focus for decision support tools that use evidence from SPORT.

Challenges of Conducting CER Using Randomized Surgical Trials

Evidence from past trials suggests that RCTs in surgery are fraught with methodological challenges. First, the “blinding” of treatment assignment in surgical trials represents an ethical challenge, even though studies have shown that certain spinal procedures have little benefit. According to our experts, surgical interventions are associated with strong placebo effects, so randomization is critical to balance treatment groups with regard to surgical preferences and expectations of outcomes. Randomization is even more critical when outcomes are subjective, such as the assessment of pain, which is a common outcome for spinal surgery trials. Controlled trials involving sham surgery may have a role in some contexts where ethical concerns are mitigated by the low risks of minor incisions, such as in knee arthroplasty; however, sham spinal surgery would be difficult, if not impossible, to implement without ethical concerns.

The other main challenge in CER trials involving surgical interventions is crossover, which threatens a trial’s internal validity. The high rate of crossover in both directions (from the nonsurgical arm to the surgical arm and vice versa) significantly weakened SPORT’s intention-to-treat analysis. Some experts indicate that crossover is not inevitable but, rather, can be carefully controlled by trial investigators. They cite trials conducted in European countries that tend to have much lower rates of crossover (Deyo, Nachemson, et al., 2004) as evidence that these problems can be overcome. SPORT was conducted at a small number of academic institutions specializing in spine surgery, where, according to one expert, surgeons and referring physicians may have lacked equipoise in their willingness to recommend surgery. Surgeons in other settings might be better able to control crossover in these types of trials. Our discussants suggested that non-adherence in SPORT may also not have been detected sufficiently early.

Trial Results Provided Insufficient Detail to Enable Clinicians to Tailor Treatment

The primary comparison of two or more treatments among cohorts analyzed by treatment may not provide detailed data on subgroups of patients that physicians could identify as particularly likely to benefit from a particular treatment. Thus, physicians may ignore the group-level data. This might be especially important for SPORT, where patients may have experienced a wide range of anatomical abnormalities and differences in pain symptoms.

Reluctance of Clinicians to Use All Available Evidence

Most of the strongest evidence on the limited efficacy of spinal surgery comes from randomized trials conducted in Europe. European trials have tended to use specific nonsurgical treatments (unlike SPORT) and in this context have shown that patients can improve dramatically without surgery. According to one expert, American spinal surgeons are either unaware of the results of European studies or believe that the results are not generalizable to the U.S. context. This suggests that available evidence about the benefits and harms of surgery for lumbar spinal stenosis is not being used and therefore does not influence decisionmaking. However, there may be key differences that do limit the generalizability of non-U.S. trials. For example, some pain-management options, such as cognitive behavioral therapy, may not be widely available in the United States, may not be reimbursed, or both. Also, in the United States, the financial incentives associated with fee-for-service reimbursement may lead to patients proceeding more quickly to surgery, so the profile of patients enrolled in European trials may be clinically different from that of those evaluated for surgery in the United States.

Conflicting Guidelines from Multiple Specialty Societies

Physicians who treat patients with spinal conditions may draw on any of several practice guidelines, and these guidelines are often in conflict with one another, possibly because of limitations of the evidence regarding the procedures. One discussant mentioned that the American College of Physicians and the American Pain Society have produced more-conservative practice guidelines, while the North American Spine Society (NASS) has been far more likely to recommend surgical procedures. With strong backing from the device industry, the International Society for the Advancement of Spine Surgery was recently formed, because many thought that the leadership of NASS and its journal were not favorable enough to the industry.

The Absence of Registries

RCTs may exclude high-risk patients and are often of inadequate size or duration to identify harms, particularly rare events (Chou and Helfand, 2005). At the same time, administrative data, such as claims, do not contain information on patients’ symptom severity, extent of anatomic abnormalities, functional impairments, or specific implants used during surgery (Deyo, Mirza, et al., 2010). The dearth of available data with which to estimate the benefits and risks of alternative treatment strategies for patients with lumbar spinal stenosis suggests a potential role for registries. Given the findings from earlier research that more-invasive surgical procedures are associated with greater complication rates (Deyo, Mirza, et al., 2010), experts have suggested that registries might be most helpful in providing additional data about the risks of surgery. They may fill gaps in evidence and also serve as the basis for development of performance measures and appropriateness criteria for procedures. Currently, there are few registries for spinal surgery. The American Academy of Orthopedic Surgeons attempted to develop a registry nearly a decade ago, but the effort was severely underfunded and the registry never materialized. NASS is seeking to develop a registry, and this effort is being viewed as a key vehicle for future CER relating to spinal surgery. This research could help support the development of appropriateness criteria, which, according to one surgeon, is “inevitable.”

Stakeholders identified at least three factors that may have contributed to the lack of registries for spinal surgery. First, surgeons have few incentives to develop a registry and may face the risk of bringing to light performance problems and other safety risks. As one expert suggested, “Not everyone can practice at the level of the SPORT surgeons.” He noted that the majority of spine surgeries are performed in low-volume practices, where outcomes may be worse than in high-volume practices, and that professional societies have historically challenged studies that assess volume-outcome relationships in orthopedic surgery for this very reason. Second, patient-reported outcomes, the primary measure of benefit in spinal surgery, display remarkable variability and may limit the power-of-outcomes analyses. For this reason, some experts believe that registries may be most useful for quantifying the harms associated with spinal surgery. Third, the lack of standardized definitions for both diagnosing conditions and measuring outcomes poses a significant barrier. Diagnosing orthopedic conditions (including spinal stenosis) is challenging, because patient-reported symptoms and imaging results are not always correlated, so diagnostic criteria have not been developed. According to one expert, the FDA is the entity best positioned to enforce standardized definitions for measuring outcomes of surgery and adverse events, but it has not done so to date. Claims data, while difficult to use for measuring benefits, might be well suited for measuring harms, according to this expert, but these data have not been used effectively. Registries in other countries have identified device problems much earlier than those in the United States. One expert suggested that neither hospitals nor physicians have any interest in drawing attention to the risks of surgery; in contrast, patients have an interest, but they have few advocates.

Lack of Performance Measures and Appropriateness Criteria

Development of quality measures for orthopedic surgery, including measures of readmissions and mortality for select procedures, is advancing. However, there are few measures for spinal surgery specifically, and no appropriateness criteria currently exist. According to one expert, “It is in everyone’s interest to not have clear definitions of appropriateness.”

Medical Innovation and Marketing

New surgical devices and their marketing increased significantly over the past decade and may be key drivers of the use of complex procedures in spinal surgery. According to experts, the rapidity of technological development and its marketing pose a barrier to CER, which often takes several years to complete. Even in the absence of evidence about the benefit of new technologies, arguments in favor of new devices are often compelling and may speed their adoption. This may be an important factor in regional variation in the use of complex surgery, as some surgeons have a desire to be local innovators (Deyo, Mirza, et al., 2010), However, experts noted that hardware was not involved in a large share of spinal stenosis surgery procedures.

The device industry played an active role in the dissemination of findings from SPORT, although it initially attempted to discredit the study even before the results were released. After the results were found to be favorable to surgery, the industry began a campaign to promote awareness of the benefits of surgery. The messaging was vague and neglected to focus on the procedures used in the trial, only 7 percent of which involved spinal hardware.

Few Payers Have Restricted Insurance Coverage for Lumbar Surgical Procedures

By nearly all accounts, payers and purchasers have few policies in place that limit access to surgery for patients with lumbar spinal stenosis. This appears to be the case for both Medicare and private health plans. Recently, BCBS of North Carolina discontinued coverage of fusion procedures for spinal stenosis following a comprehensive review of the evidence. A broad coalition of professional societies representing spine surgeons submitted comments and were successful in modifying parts of the coverage policy. According to one expert, the action taken by BCBS represents a “shot across the bow,” and similar scrutiny may be placed on spinal fusion surgery in the future.

View full report


"rpt_RANDFinal.pdf" (pdf, 2.01Mb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®