Rapid Evaluation Approaches for Complex Initiatives : Table 3


: Table 3

Evaluation FactorSystems ChangeCMS Medicaid and CHIP Learning Collaborative
1.     Situational dynamicsComplexThere are complex dynamics between CMS and states and among CMS, states, and facilitators; the project is also operating within the volatile political dynamics of federal health care reform.
2.     Intervention complexityComplex initiativesThree organizations contracted by CMS to operate six learning collaboratives addressing ACA and non-ACA topics.
3.     Governance structureAlliance of multiple funders and stakeholdersCMS funds LC facilitation teams and provides in-kind CMS expertise; 50 states voluntarily participate in one or more LCs by invitation.
4.     Scale of outcomesLarge-scale population or system-wide changeBalance of short-term outcomes (LC-created policies, tools, and practices) and long-term outcomes, such as successful implementation of the ACA.
5.     Timeline of expected resultsTransformative change expected in months or yearsThe coordination of state and federal learning around Medicaid policy is ongoing. Several initiatives, such as the implementation of the ACA, will require several more years of implementation.
6.     Theory of changeApplying change principles to strategic leverage pointsThe LCs provide a new forum for federal-state dialogue, creating new communication channels that increase federal-state collaboration on critical issues.
7.     Execution strategyChange strategies developed and revised as initiative evolvesThe content, format, frequency, and facilitation of the learning sessions were modified over time in response to participant feedback and changes in the federal policy landscape.
8.     PurposeImplementation and efficacy questionsDevelopmental and formative questions: What are the LCs doing to develop and implement learning sessions that meet the needs of CMS and state representatives? How can the structure and process of the LCs be improved?
9.     Reporting and use of findingsStrategic leadership incorporates findings into adaptive management cycleMonthly project direct debriefings, quarterly project management presentations, annual performance reports to CMS—no external publication of results.
10.  Rapid evaluation methodsDevelopmental evaluation, systems change evaluation, action research methodsDirect observation of sessions, immediate post-session web surveys of state participants, monitoring of session attendance metrics, in-depth interviews with facilitation members and CMS staff, and ongoing review of project documentation.