Rapid Evaluation Approaches for Complex Initiatives. V. Organizational Change—rapid Cycle Evaluation

03/31/2014

This section provides an example of an organizational change evaluation used in the implementation of the Partnerships for Patients Program (PfP). Table 3 features the unique factors of organizational change evaluations. In Table 3, the ten evaluation factors are listed (in the left-hand column), the factors are applied to organizational change projects (in the middle column), and the factors are illustrated through the PfP example (in the right-hand column). The practice of rapid cycle evaluation is illustrated in Figure 2.

Figure 2. Rapid Cycle Evaluation

Figure 2. Rapid Cycle Evaluation

Rapid cycle evaluations are appropriate for testing models of organizational change. The Center for Medicare and Medicaid Innovation (CMMI) developed the rapid cycle evaluation approach to test innovative health care payment and service delivery models that preserve or improve the quality of care while reducing costs (HHS 2011, Shrank 2013). The Centers for Medicare & Medicaid Services (CMS) is using this approach to evaluate several national initiatives, including PfP. PfP aims to prevent hospital-acquired infections and hospital complications, increasing patient safety and cutting related hospital readmissions. More than 3,700 hospitals are participating in the initiative (CMS 2013a). To reduce preventable inpatient harms by 40 percent and readmissions by 20 percent by the end of 2013, the PfP hospitals need to invest in and redesign their organization infrastructures to support improved care. This requires “substantial learning and adaptation” on the part of health care providers as “there are no simple turnkey solutions” (Shrank 2013).

To help hospitals identify and implement effective solutions, CMS awarded $218 million in 2011 to 26 state, regional, national, or hospital system organizations to become Hospital Engagement Networks (HENs).1 The HENs support the initiative by identifying hospitals’ current solutions to reducing hospital-acquired conditions and disseminating them to other hospitals and health care providers. The HENs are using a range of strategies to help hospitals, including providing financial incentives, developing learning collaboratives, conducting intensive training programs, providing technical assistance, implementing data systems to monitor hospitals’ progress, and identifying high-performing hospitals (CMS 2013b). The HENs are required to develop, collect, and report PfP process and outcome measures that monitor the early progress of the initiative. The timing, content, and quality of the HEN data have varied considerably (Felt-Lisk 2013).

Rapid cycle evaluation methods are being used in PfP’s formative implementation and summative impact evaluations that are being conducted by the team of Health Services Advisory Group, Inc. (HSAG) and Mathematica. The evaluations’ goals are to provide real-time information monitoring the hospitals’ progress, and ongoing feedback to support improvement of PfP activities and outcomes. Specifically, the formative evaluation is (1) documenting the organizational context of the PfP hospitals, including the hospitals’ infrastructure (staffing and operational systems) and level of commitment to the initiative’s 11 clinical areas of focus; (2) monitoring hospitals’ site-specific process measures that are reported monthly by their HENs; and (3) documenting hospitals’ activities and challenges in monthly HEN reports to CMS. Whether PfP’s goals are met will be determined by time series analyses of pre-post patient data derived from medical chart reviews conducted by the Agency for Healthcare Research and Quality, using a sample representative of the entire nation. In addition, an impact evaluation conducted by the HSAG/Mathematica team will determine the extent to which changes over time can be attributed to PfP and will identify if there are certain types of interventions associated with greater harm reduction (Felt-Lisk 2013).

Key internal audiences for the evaluation findings are CMS staff, support contractors, and the leadership and staff of the HENs, all of whom have a voracious appetite for information (Felt-Lisk 2013). Monthly Formative Feedback Reports include a one-page visual summary of the initiative’s progress, key news, appendices for evaluation methods and supplemental tables organized by clinical area (200+ pages) and by HEN (200+ pages). CMS has also made special requests for graphs of individual site-level (hospital) progress and for “success stories” of downward trends in preventable harms, accompanied by a description of associated interventions.

There are additional costs associated with being able to “provide formative information to feed program needs at any given point in concert with the flow of the program” beyond what is done in traditional evaluation. New reporting methods are also required for the impact evaluation, with frequently updated deliverables that are timed as soon as data are available, using PowerPoint presentations and simple memos with tables, rather than formal reports. As a result, the program’s formative evaluator noted that while rapid cycle evaluation’s methods might be somewhat more costly than traditional evaluation, the approach appeared to be more useful to the evaluation’s primary audiences than other approaches (Felt-Lisk, personal communication, Sept. 2013).

Table 3. Rapid Evaluation: Complicated Organizational Change

Evaluation Factor Organizational Change CMS Partnership For Patients Campaign
1.  Situational dynamics Complicated Complicated campaign, with some complexity of learning across HENs in an “all teach and all learn” environment.
2.  Intervention complexity Complicated programs National initiative to reduce preventable patient harms in 3700 hospitals across 27 HENs.
3.  Governance structure Federal funder of multiple grants CMS is the single federal funder of the PfP.
4.  Scale of outcomes Short list of individual-level outcomes PfP has two overarching outcomes (reduced hospital-acquired patient harms, reported in 11 clinical areas, and hospital readmissions.
5.  Timeline of expected results Incremental change expected in months Results are expected in months; the focus is on speeding up the pace of change to achieve the goals in three years.
6.  Theory of change Testing a specific program model HENs facilitate sharing of best practices among their aligned hospitals and offer hospitals training, technical assistance, learning collaboratives, and reporting systems to help them achieve the PfP goals.
7.  Execution strategy Fidelity to work plans outlining program goals, objectives, and strategies Detailed hospital-specific work plans and measures of success are developed and implemented by the hospitals.
8.  Purpose Implementation and efficacy questions In what contexts are the PfP hospitals working to achieve the PfP goals? What progress are the hospitals making? What are early results? How do PfP implementation and early outcomes vary by hospital, HEN, and condition?
9.  Reporting and use of findings Program management separates reporting and learning functions Monthly feedback report to CMS and HENs, and ad hoc reports in response to special requests. No external evaluation linkage to CMS’s internal Learning Team to maintain objectivity.
10.  Rapid evaluation methods Rapid cycle formative and summative evaluation methods Rapid cycle formative and summative evaluation methods; time-series analyses of changes in hospital-specific processes and outcomes, with documentation of hospitals’ contexts, culture, and PfP activities.

CMS = Centers for Medicare & Medicaid Services; HEN = Hospital Engagement Network; PfP = Partnership for Patients


1 Since then, one more HEN has been awarded, increasing the number of HENs to 27.

View full report

Preview
Download

"rs_EvalApproach.pdf" (pdf, 598.02Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®