Rapid Evaluation Approaches for Complex Initiatives. B. The Intervention


Factor 2: Intervention complexity. Is the intervention a simple, direct process change, a test of a program model, or a larger initiative addressing multisector, multilevel population or systems change?

Factor 3: Governance structure. Who is funding and overseeing the initiative—a single organization, a federal funder of a cohort of grantees, or a consortium of funders?

How complicated is the program, including its structure? An intervention’s governance may include its funding, management, organizational structure, and implementation. A simple intervention is typically implemented by a single organizational unit. More-complicated efforts involve teams of experts or program units within an organization. Complex, networked interventions often involve the collaboration of multiple entities or organizations working across sectors and levels. Examples include comprehensive public health initiatives addressing childhood asthma and obesity; community capacity building initiatives to prevent child abuse and neglect; integrated urban development initiatives alleviating poverty; and networks of federal, state, and local agencies working together on homeland security (Kamarck 2007; Goldsmith and Kettl 2009).

Factor 4: Scale of outcomes. What is the expected change—a process improvement, changes in a small, specified set of individual-level outcomes, or broader system-wide change?

Factor 5: Timeline of expected results. Will early results be seen immediately or in weeks, months, or years?

What is the scope of the program’s desired results? Simple, linear interventions are designed to produce specific, narrowly focused, and measurable outcomes, such as increasing student reading skills through a literacy program, or increasing the accuracy of the results of lab tests through changes in testing procedures. Interventions with more-complicated dynamics may target multiple, potentially conflicting outcomes, such as improving personal safety while maintaining the quality of life and level of independence for seniors in community-based, long-term care programs (Brown et al. 2008). In complex system interventions, stakeholders may share a common vision, such as reduced poverty at a regional level, improved quality of life for people with developmental disabilities, or rapid advances in biomedical science, but they may not be able to predict in advance which specific outcomes will emerge from the complex interactions of the initiative’s entities or organizations.

Factor 6: Theory of change. Is the program developing, adapting, or implementing a promising or best practice, testing a program model, or applying general principles to a complex systems change process?

How complex are the program’s dynamics? In simple, straightforward interventions, linear logic models can be used to trace a single stream of program inputs, activities, and outputs that lead to a limited set of outcomes. In more-complicated interventions, multiple coordinated pathways of activities may lead to a broader set of complementary outcomes. Systems change interventions may target specific drivers of systemic change at strategic leverage points. Potential drivers might include the development of a shared vision, collaborative partnerships, pooled resources, and shifts in community norms. The activities used to facilitate those changes might include convening meetings of potential partners or changing policies to allow the pooling of resources.

Factor 7: Execution strategy. How prescribed is the intervention’s implementation strategy? Is the implementation work plan a clearly specified set of procedures, a program manual, set of program guidelines, or contract requirements outlining the specific functions, timing, and sequence of program components, or is the execution strategy developed collectively and adapted over time with input from the initiative’s partners and stakeholders?

View full report


"rs_EvalApproach.pdf" (pdf, 598.02Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®