This section provides an example of a system change evaluation used in the implementation of the Medicaid and CHIP Learning Collaboratives (MAC LC) project. Table 4 features the unique factors of system change evaluations. In Table 4, the ten evaluation factors are listed (in the left-hand column), the factors are applied to systemic change projects (in the middle column), and the factors are illustrated through the MAC LC example (in the right-hand column). The rapid evaluation practice of systems change evaluation is illustrated in Figure 3.
Figure 3. Nested, Systems-based Evaluation
There are evaluation methods designed for initiatives that engage many different entities in collaborative efforts to shift larger systems. The system-shifting strategies in those initiatives are not predetermined best practices, or fully specified program models (although they may incorporate both). Their strategies are developed over time as key actors work across organizations, sectors, and levels to achieve common goals. Development-oriented, systems-based evaluation methods with rapid feedback mechanisms were created to address these systems change initiatives. These evaluation methods differ from quality improvement and rapid cycle evaluation methods in four ways. First, they use nested logic models and theories of change that show how the interactions of unit (micro)-, organization (meso)-, and policy or community (macro)-level activities impact individual-level outcomes. Second, they recognize, document, and incorporate the changing dynamics of the initiative’s environment into the evaluation. Third, they include systems concepts in the evaluation’s conceptual framework to measure changes in collective capacity, networked relationships, and shared perspectives of the entities and organizations involved in the initiative. Fourth, they embed the evaluation function into the initiative as an integral set of feedback loops to support decision making at tactical unit, program management, and strategic initiative levels.
This development-oriented, systems-based approach is currently being used by CMS in the MAC LC project, established in 2011 to achieve high-performing state health coverage programs, a goal that requires “a robust working relationship among federal and state partners.” The original two-year MAC LC project brought federal and state Medicaid agencies together “to address common challenges and pursue innovations in Medicaid program design and operations as well as broader state health coverage efforts” (CMS 2013c). The project created six collaborative work groups, each consisting of 6 to 10 states plus federal partners and national experts, which addressed a range of topics “critical for establishing a solid health insurance infrastructure,” including policies related to the implementation of the ACA. The original six learning collaboratives (LCs) were the (1) Exchange Innovators in Information Technology LC, (2) Expanding Coverage LC, (3) Federally Facilitated Marketplace Eligibility and Enrollment LC, (4) Data Analytics LC, (5) Promoting Efficient and Effective IT Practices LC, and (6) Value-Based Purchasing LC.2
The MAC Collaboratives activities are coordinated by Mathematica, the Center for Health Care Strategies (CHCS), and Manatt Health Solutions, with additional assistance from external experts and in close association with CMS. Over a period of two years, LC meetings, called learning sessions, were conducted, mostly by webinar or conference call, on a monthly or biweekly basis, and were moderated by Mathematica, CHCS, and Manatt facilitation teams. In the learning sessions, state representatives, technical experts, and CMS staff discussed policy issues, reviewed draft rules and other federal guidance, and created technical assistance tools, background materials, and other state resources (CMS 2013d).
The MAC LC project included an internal assessment function operating at three levels (learning session, LC, and project), using a systems-based, multilevel conceptual framework. First, the assessment team observed and rated the quality of the content, logistics, and facilitation of individual learning sessions, for quality improvement purposes. Second, the assessment team tracked LC session attendance rates, conducted participant feedback surveys, and reviewed LC documents to evaluate the performance of each LC, for formative purposes. Third, the assessment team also interviewed the CMS staff and LC facilitation teams to obtain project-level information about the overall functioning and effectiveness of the project. Assessment feedback is provided on different cycles, through monthly debriefings with the project director on the LC sessions, group feedback to the LC facilitation teams in quarterly project management meetings, and annual reports to CMS on the performance of the project as a whole. This feedback is intended primarily for an internal audience.
In 2013, the project was renewed for two more years; some aspects of the project were modified, informed by internal and external feedback. Over the next two years, the assessment team will continue to provide an internal monitoring and rapid feedback function on the project. There are costs associated with this function; the internal assessment is a separate project task.
Table 4. Rapid Evaluation: Complex Systems Change
|Evaluation Factor||Systems Change||CMS Medicaid and CHIP Learning Collaborative|
|1. Situational dynamics||Complex||There are complex dynamics between CMS and states and among CMS, states, and facilitators; the project is also operating within the volatile political dynamics of federal health care reform.|
|2. Intervention complexity||Complex initiatives||Three organizations contracted by CMS to operate six learning collaboratives addressing ACA and non-ACA topics.|
|3. Governance structure||Alliance of multiple funders and stakeholders||CMS funds LC facilitation teams and provides in-kind CMS expertise; 50 states voluntarily participate in one or more LCs by invitation.|
|4. Scale of outcomes||Large-scale population or system-wide change||Balance of short-term outcomes (LC-created policies, tools, and practices) and long-term outcomes, such as successful implementation of the ACA.|
|5. Timeline of expected results||Transformative change expected in months or years||The coordination of state and federal learning around Medicaid policy is ongoing. Several initiatives, such as the implementation of the ACA, will require several more years of implementation.|
|6. Theory of change||Applying change principles to strategic leverage points||The LCs provide a new forum for federal-state dialogue, creating new communication channels that increase federal-state collaboration on critical issues.|
|7. Execution strategy||Change strategies developed and revised as initiative evolves||The content, format, frequency, and facilitation of the learning sessions were modified over time in response to participant feedback and changes in the federal policy landscape.|
|8. Purpose||Implementation and efficacy questions||Developmental and formative questions: What are the LCs doing to develop and implement learning sessions that meet the needs of CMS and state representatives? How can the structure and process of the LCs be improved?|
|9. Reporting and use of findings||Strategic leadership incorporates findings into adaptive management cycle||Monthly project direct debriefings, quarterly project management presentations, annual performance reports to CMS—no external publication of results.|
|10. Rapid evaluation methods||Developmental evaluation, systems change evaluation, action research methods||Direct observation of sessions, immediate post-session web surveys of state participants, monitoring of session attendance metrics, in-depth interviews with facilitation members and CMS staff, and ongoing review of project documentation.|
CMS = Centers for Medicare & Medicaid Services; CHIP = Children’s Health Insurance Program; ACA = Patient Protection and Affordable Care Act; LC = learning collaborative
2 In 2013, the Promoting Efficient and Effective IT Practices LC ended and was replaced by the Basic Health Plan LC.