Rapid Evaluation Approaches for Complex Initiatives. Overview

03/31/2014

This paper addresses the challenges of conducting rapid evaluations in widely varying circumstances, from small-scale process improvement projects to complex, system transformation initiatives. Rapid approaches designed to evaluate projects at lower levels of complexity do not take into account the inter-organizational aspects of more complex initiatives, especially those designed to build capacity and integrate activities across organizations, sectors, and levels. Providing a framework that recognizes key differences in the scope and complexity of interventions helps to advance implementation science beyond a program-centric focus on process and organizational improvements to encompass a whole systems approach.

This paper presents a comparative framework of rapid evaluation methods for projects of three levels of complexity: quality improvement methods for simple process improvement projects; rapid cycle evaluations for complicated organizational change programs, and systems-based rapid feedback methods for large-scale systemic or population change initiatives. The paper also provides an example of each type of rapid evaluation and ends with a discussion of rapid evaluation principles appropriate for any level of complexity. The comparative framework is designed as a heuristic tool rather than as a prescriptive how-to manual for assigning rapid evaluation methods to different projects. No one best rapid evaluation method works in all circumstances; the right approach addresses the goals of the evaluation and captures the complexities of the intervention and its environment.

Different rapid evaluation methods are appropriate for different circumstances. Quality improvement and performance measurement methods are appropriate for process improvement projects.  Rapid program evaluation methods are appropriate for organizational change programs. Developmental evaluation and other systemic change evaluation methods are best for larger-scale systems change initiatives. However, these methods are not mutually exclusive. They may be more effective when nested, just as simple checklists are used to reduce preventable surgical errors within larger hospital safety campaigns that are funded through national payment reforms that reward system-wide shifts in health care costs and quality. Evaluating an intervention from process, organization and systemic perspectives allows us to implement change more effectively from multiple vantage points.

Regardless of the level of complexity, rapid evaluation methods should maintain a balance between short-term results and long-term outcomes so that there is an alignment of task control, management control, and strategic control. They should also be part of an interactive and adaptive management process in which internal operational results and external environmental feedback are used together in an iterative process to test and improve the initiative’s overall strategy.

View full report

Preview
Download

"rs_EvalApproach.pdf" (pdf, 598.02Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®