Rapid Evaluation Approaches for Complex Initiatives. IV. Process Change—quality Improvement


This section provides an example of a process change evaluation used in the implementation of the Safe Surgery Checklist.  Table 2 features the unique factors of process change evaluations. In Table 2, the ten evaluation factors described in the previous section are listed (in the left-hand column), the factors are applied to process change projects (in the middle column), and the factors are illustrated through one surgery checklist example (in the right-hand column). The rapid evaluation practice of process improvement is illustrated in Figure 1.

Figure 1. Continuous Quality Improvement

Figure 1. Continuous Quality Improvement

Quality improvement methods are appropriate rapid evaluation techniques for simple, discrete process improvements in the daily functions of organizational units). Dr. Atul Gawande demonstrated the value of this technique in the World Health Organization’s Safe Surgery Checklist Project, when his team used quality improvement methods to test early prototypes of a surgical checklist (Haynes et al. 2009, Gawande 2010). The checklist was designed to reduce preventable surgical errors.

Through numerous PDSA cycles of prototype testing, the team improved the checklist, reducing the number of items, limiting the checklist to one page, and streamlining its application to two minutes. At that point, the team implemented a pilot study, in which surgical teams in eight hospitals adapted and implemented the checklist in their operating rooms. Time series analyses from the pilot study showed that surgical complications and deaths were reduced significantly (36 and 47 percent respectively) after the checklists were introduced (Gawande 2010).

Although many hospitals have implemented the checklist, it is not yet being used to its full capacity. By the end of 2009, approximately 10 percent of American hospitals had adopted the checklist or had started taking steps to implement it. Globally, more than 2,000 hospitals had started using the checklist. The uptake of the checklist has been slower than expected, however. Gawande recognized that on its own, a single process change such as the checklist cannot institute a complex, system-wide culture shift among hospitals and physicians to test and improve their surgical practices:

Just ticking boxes is not the ultimate goal here. Embracing a culture of teamwork and discipline is. And if we recognize the opportunity, the two-minute World Health Organization checklist is just a start. It is a single, broad-brush device, intended to catch a few problems common to all operations, and we surgeons could build on it to do even more (Gawande 2010).

Through iterative testing of adaptations of the Safe Surgery Checklist, other hospitals have improved its practice. Some researchers have found that the introduction of the hospital checklist initially lowered the risk of mistakes, but then the error rate gradually returned back to near its former level. To address this problem, one surgeon, Marc Parnes, tested an adaptation of the checklist by having a “personal check-in conversation” with the patient while rolling the patient into the procedure room. The conversation with the patient and entire operating team allowed each person to see the situation through the eyes of the others, including the patient. This adaptation reduced the hospital’s surgical error rate more sustainably than did the original checklist (Scharmer and Kaufer 2013).

Table 2. Rapid Evaluation: Simple Process Change

Evaluation Factor Process Change World Health Organization’s Safe Surgery Checklist
1.  Situational dynamics Simple The team’s dynamics are simple; the surgeon leads the surgical team. In surgery, the patient’s unstable health adds some complexity to the situation.
2.  Intervention complexity Simple projects Simple project: the surgical team completes a two-minute, 19-step checklist designed to prepare teams better for surgery and to respond better to unexpected problems that occur during surgery.
3.  Governance structure Organizational unit Hospital operating room teams implement the checklist.
4.  Scale of outcomes Single, discrete process changes The checklist changes the quality of the procedures used to prepare the patient and the surgical equipment before surgery and to prepare the team for potential problems during the operation.
5.  Timeline of expected results Immediate change expected within weeks The checklist was introduced to operating rooms over a period of one week to one month. Data collection started during the first week of checklist use.
6.  Theory of change Implementing an evidence-based practice The same procedure was used in all situations, with minor adaptations in language, terminology, and the order of the checklist items for different hospitals.
7.  Execution strategy Fidelity to a set of documented procedures Hospital surgical teams received the checklist with a set of how-to PowerPoint slides and YouTube videos.
8.  Purpose Implementation, efficacy, and outcome questions The eight-hospital pilot study tested the checklist’s efficacy; major complications were reduced by 36 percent and deaths were reduced by 46 percent after the introduction of the checklist.
9.  Reporting and use of findings Unit operations managers and staff receive and use evaluation results To create the prototype of the checklist, Gawande’s surgical team tested multiple versions of the original checklist using operation process and outcome data, and team feedback, to track improvement of the checklist.
10.  Rapid evaluation methods Quality improvement—PDSA cycle The team developing the checklist used the results of each PDSA cycle to improve the checklist, so that it was ready to be tested in operating rooms in other hospitals.


View full report


"rs_EvalApproach.pdf" (pdf, 598.02Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®