Using Evidence-Based Constructs to Assess Extent of Implementation of Evidence-Based Interventions

About This Issue Brief

This issue brief was written by Gene Hall, Ph.D., of the University of Nevada Las Vegas; Allison Dymnicki, Ph.D., of the American Institutes for Research; Jennifer Coffey, Ph.D., of the Office of Special Education Programs, and Melissa Brodowski, Ph.D., of the Children’s Bureau.

In 2012, ASPE awarded the American Institutes for Research to manage the Investing in What Works (IWW) project, to continue ASPE’s efforts to keep building the knowledge and supports that evidence-based programs and initiatives can use to improve the quality and outcomes of interventions funded through federal investments.

Executive Summary

Implementing interventions in communities and organizations and with individuals requires an understanding of the complexities involved with making any major change. A good idea and evidence that an intervention has been effective elsewhere do not automatically lead to the widespread adoption and use of an intervention. Studying the implementation of interventions has increased our understanding of how to monitor more effectively what happens during implementation to promote success.

In this brief, we introduce several constructs that can be used to describe, monitor, and facilitate implementation. Then we describe how to apply these constructs in several ways, including data collection to monitor progress and developing an implementation monitoring plan. We also address how much time each phase of implementation is expected to take. The brief concludes with implications for policy and for supporting practice.

Key Take Away-Messages

• Scholars who study change have identified and categorizedimplementation milestones common to most interventions.These can be used to improve the likelihood of achievingfull implementation of interventions.
• Policymakers could require plans for identifyingimplementation milestones, monitoring progress, and usingdata to inform continuous quality improvement duringimplementation.
• Practitioners should assess implementation progress bycollecting a variety of indicators from differentperspectives (e.g., frontline staff, leadership) to pinpointbarriers to implementation and ways to overcome them.
• Policymakers evaluating intervention implementation should be realistic about what can be done within certain time frames, but they should also encourage grantees to move thoughtfully and quickly through the first phase, which is often overlooked.

What Is the Change Process?

One important aspect of understanding change is that it is a process, not an event. An intervention¹ involves three major phases of change:

1. Creating includes the research and development activities that provide the knowledge base for developing interventions, programs, and innovations. This phase also includes establishing strategies for making interventions available for use, assessing readiness, and scaling up.
2. Implementing is when individuals, agencies, or communities apply the new intervention, program, technology, or method.
3. Sustaining is when the entire intervention, or select components of it, becomes the established way of doing things; use of the intervention has been “institutionalized.”

Figure 1 illustrates how these three phases unfold. It also illustrates several ideas important to understanding the change process in general and implementation in particular. One especially important idea in figure 1 is the Giant Leap. Sometimes, there is an implicit expectation that accomplishing a change will be simple and quick, that practitioners who are implementing an intervention need only to jump across the gap and start using the new way. This perception does not accurately represent the complexity of how change occurs in organizations, in communities, or with individuals. Also, most interventions require implementing a set of practices and structures simultaneously. A common misunderstanding is the expectation that all practitioners in an organization or large system make the Giant Leap at the same time.

Figure 1. Three Phases of the Change Process and the Implementation Bridge

Source: Hall & Hord (in press).

Figure 1 represents another idea about change: the size of the gap and the distance that must be covered in one leap. Most expectations for implementing new interventions require leaps too long for many to achieve. Change initiatives will be more successful if there is an Implementation Bridge. As a metaphor, the Implementation Bridge can help us examine the activities, challenges, and barriers involved in achieving successful implementation. For example, each practitioner who implements an intervention has to make it across the bridge. As practitioners move across the bridge, they gain knowledge, skills, and confidence in using the new intervention. Individuals and organizations also need ways to measure, monitor, and assess how far across the Implementation Bridge an intervention has progressed. These steps and the information gathered during these steps are important to inform further progress, identify where additional support is needed, gauge the effectiveness of an intervention to produce desired outcomes, and identify unintended consequences. The effort and level of support required during the implementation phase will vary dramatically based on characteristics of the change and the context in which it is introduced.

¹ In other related work, we refer to the terms “innovation” versus interventions. In this brief, we define the term “intervention” as: (a) a procedure, or set of procedures (b) designed for use in a specific context (or set of contexts) (c) by a specific set of users (d) to achieve defined outcomes (e) for (a) defined population(s) (c.f. Cook, Tankersley, & Landrum, 2009; Dunst, Trivette, & Cutspec, 2002; Flay et al., 2005; Horner, Sugai & Anderson, 2010). Interventions are what we do to achieve desired outcomes.

How To Measure Implementation Milestones

Several constructs can be used to monitor progress in implementing interventions:

Fidelity of Implementation—Today’s interventions are complex and have many parts, which can lead to varying aspects of the intervention being implemented at different times.

Competence in Use—The expertise and competence of practitioners carrying out an intervention increase during the implementation process.

Feeling and Perceptions—The personal side to the change process is reflected in practitioners’ increasing confidence in or resistance to implementing an intervention.

Context of the Organization and Community—Implementation takes place within a broader context that can support or impede implementation efforts.

Supporting Implementation—Addressing factors that promote the effective implementation of interventions can advance its progress.

In this section, we describe these constructs and demonstrate how each can be used to monitor implementation.

Fidelity of Implementation: To what extent are the core content, pedagogical, and implementation components of an intervention being implemented?

One set of implementation milestones involves Fidelity of Implementation. “Fidelity is the faithfulness with which an intervention is implemented; that is, how well the intervention is implemented without compromising its core content, pedagogical, and implementation components which are essential for the intervention’s effectiveness” (National Resource Center for HIV/AIDS Prevention Among Adolescents, n.d.). Implementing an intervention with fidelity improves the likelihood of replicating positive intervention effects (see the Administration for Children and Families, n.d., for more information).

One way to think about and measure fidelity is to identify the major components of a change. In a recent research brief developed by the Office of the Assistant Secretary of Planning and Evaluation titled Core Intervention Components: Identifying and Operationalizing What Makes Programs Work, Blase & Fixsen (2013) described core components as “the essential functions or principles, and associated elements and intervention activities (e.g., active ingredients, behavioral kernels) that are judged necessary to produce desired outcomes” (Embry, 2004, p. 2). Century, Cassata, Rudnick, and Freeman (2012) have proposed a framework for measuring the “enactment of innovations,” which includes identifying the characteristics of fidelity. In these works there is an expectation that practitioners develop fidelity gradually with practice instead of suddenly halting what they are doing and fully implementing a new approach.

An important first step when implementing an intervention is to establish what achieving fidelity will look like for that intervention and then establish what indicators will suggest progress in achieving fidelity. This involves identifying the critical components of an intervention, and then identifying steps during the implementation process that practitioners are likely to take as they move toward implementing intervention practices.

We recognize that at any point in time there will be variation in how well each component of the innovation is being used. For example, in a counseling program the assessment component might be implemented very well, but the feedback sessions leave key elements left out, suggesting partial fidelity of the whole program. This can be described in terms of different “Innovation Configurations” (Hall & Loucks, 1981; Hall & Hord, 2015), where each configuration represents a different combination of the components. As fidelity increases, more and more core components are included and are used as intended. In the final, "ideal" configuration, all critical components are in use and full Fidelity of Implementation has been achieved.

Important questions about fidelity must be addressed for each change initiative. How will the key components of the change be identified? Who will monitor the extent to which core components have been established and are being performed consistently? What data collection method will be used? How much information is needed? How frequently should fidelity milestones be measured? What standards will be used to interpret what is found? These are challenging decisions that have to be answered on a case-by-case basis; answers depend on the particular change being introduced. For example, Chinman et al. (2005) addressed some of these questions in their work about building community capacity for effective preventive interventions, including how to identify core components.

Developing Competence: How do practitioners progress from nonusers to expert users?

Another set of implementation milestones describes practitioners’ increasing competence in using an intervention. Studies conducted from the practitioners’ perspective suggest that developing the knowledge and skill to implement an intervention effectively requires time and experience (Bellg et al., 2004; Durlak & DuPre, 2008). As practitioners approach and move across the bridge, different levels of expertise can be described:

Nonuser—Is not engaged in using the change and is using the “old” way. (In other words, is not on the bridge.)

Novice—Is applying the change but struggling to use all pieces in effective ways. (Is on the bridge.)

Expert—Has mastered the complexity and understands everything entailed in full use. (Is across the bridge.)

Change process researchers have identified eight unique behavioral profiles, called “Levels of Use” (LoUs), which include three behavioral profiles of nonusers and five profiles of users (Hall & Hord, 2015). Each of these profiles describes a different level of understanding and degree of skill in using (or not using) an intervention. The LoUs range from the nonuser, who knows nothing and is taking no action to learn about an intervention (LoU 0, Nonuse), to the novice, whose use is inefficient and disjointed (LoU III, Mechanical Use), to the skilled expert who makes deliberate efforts to coordinate with others in using the intervention (LoU V, Integration).

LoUs can be used to identify several important implementation milestones and to ensure that progress continues toward full implementation and sustainability. For example, one very important milestone is LoU III Mechanical Use, in which practitioners have begun to implement certain practices but their use is disjointed, inefficient, and incomplete. During this phase, practitioners might plan lessons 1 day in advance, follow the user’s manual very closely, and possibly be uncomfortable with the technology involved in the program.

These stages have important implications for training and technical assistance (TTA) providers, administrators, and evaluators. In the LoU III Mechanical Use stage, the TTA should focus on “how to do it,” not on philosophy or theory. Administrators should plan problem-solving meetings with staff to understand the challenges being encountered. Evaluators should be documenting the extent and duration of this stage and what facilitates progress.

Developing Confidence: How do feelings, emotion, and motivation develop as interventions are being implemented?

Charting feelings and perceptions represents a third way to establish implementation milestones. Change science scholars have identified and described the different concerns that people can have when engaged with a change process; these can be used to chart movement across the Implementation Bridge. Four major categories of concern have been identified (Cheung, Hattie, & Ng, 2001; George, Hall, & Stiegelbauer, 2008; Hall & Hord, 2015; Yan & Huang, 2008):

Unconcerned—At the beginning of a change process, practitioners will not be concerned about an intervention; they will be more concerned about other things. “I think I heard something about it, but I'm too busy right now with other priorities to be concerned about it.”

Self—As practitioners get close to implementation, concerns about what the change could mean for them personally become intense. “I don’t know whether I can do this.”

Task—After implementation has begun, concerns about time, organizing tasks, and fitting everything in become intense. “I have to spend a lot of time preparing to use this tomorrow.”

Impact—Ultimately, as practitioners become more familiar with the intervention, they have more intense concerns about how well the intervention is working for clients and what can be done to produce even better outcomes. “I just heard about this app that might be useful with several clients who don’t seem to be making good progress.”

Different TTA and coaching are needed for each area of concern. For example, consider the resistance often experienced during change processes that can occur at any time. The source of the resistance will vary by concern. For example, Self-Concerns (e.g., “Well, I don’t see how this is better than what I have been doing) can lead to disagreement with engaging the change at all. When overwhelmed by complexities (Task Concerns, e.g., “It takes too much time to get ready and nothing works out as it is supposed to”), the resistance will have a different theme. Table 1 presents examples of different types of concern-based resistance that might occur and some possible resolutions. Understanding the sources of resistance helps inform the development of more targeted solutions.

Table 1. Examples of Resistance and Solutions for Difference Stages of Concern

Organization and Community Context: What system factors are key to implementation success?

Another important consideration is the broader context in which an intervention is being implemented. Different aspects of the context can support or inhibit successful implementation. One aspect of context is the “unit of change.” The primary implementer can be an individual (e.g., a case manager in a foster care agency), an organization (all staff in a community-based organization), or a whole community (city or state). It is useful to monitor progress in implementing interventions when considering whole-organization and systemic change. The same three constructs (fidelity, competence, and concerns) can be applied to the whole system as well as the various subgroups. The same considerations regarding TTA, coaching, and assessing movement during implementation apply, but now it is to groups and whole communities.

Figure 2: Understanding Practitioners in Their Broader Context

Source: Goodman, 2011.

Implementation will be more successful when there is clear and consistent support for using an intervention across the system. The plan for monitoring implementation should include measuring how well the system is supporting its use. Figure 2 illustrates an education system in one state, but the concept behind the graphic can be applied to any state system because most large-scale change efforts should be supported at the organizational, local, and state levels, which can be considered units of change. Implementation progress needs to be monitored within each unit, and overall. When a change initiative is being led from the “top,” there is a “cascading” relationship in which each unit of change supports the next. In addition, needs and barriers are expressed to the next unit (e.g., the building team expresses its needs and barriers to the district team) so the right supports will be provided. Regular measures of fidelity, use, and concerns within each unit are important to maintain an informed, strategic view of progress and needs.

Supporting Implementation: What supports can help advance progress in implementing interventions?

Another use of implementation milestones is as diagnostics for determining which supports will advance the implementation of interventions. The idea of “implementation drivers” proposed by Fixsen, Naoom, Blase, Friedman, and Wallace (2005) represents one way to think about actions and processes to support implementation at different levels (organization, local, and state). Their model of support is based on a set of implementation drivers, which have three main areas of focus: (a) drivers that build competence (selection, training, coaching); (b) drivers that support that competence through organizational structures (systems interventions, administrative facilitation, decision support data systems); and (c) leadership drivers (technical drivers such as time, and adaptive drivers like motivation; see http://nirn.fpg.unc.edu/learn-implementation/implementation-drivers).

These drivers can be applied to each level of the system. For example, if state mental health providers are providing multisystemic therapy, they might rely on regional TTA providers, who in turn rely on state agency personnel to offer them the training, guidance, and facilitative policies that help them do their work. Staff at the state, regional, and local levels need appropriate training and coaching and the use of a data system (that can provide an assessment of fidelity, use, and concerns) to help them make decisions about how to change or continue to do their work.

Collecting information about implementation milestones and context supports the diagnostic assessments that identify the implementation supports needed for each level, system, subunit, and individual. In combination, these constructs and perspectives provide a complex yet coherent understanding of what is involved in moving implementation forward.

Implementation Milestones: Applying These Constructs

In this section, we explore the following questions:

4. What data should be collected to determine progress across the Implementation Bridge?
5. What would an implementation monitoring plan look like?
6. How much time does it take?

What data should be collected to determine progress across the Implementation Bridge?

Several evidence-based constructs have been introduced for thinking about and developing implementation milestones. Applying these milestones to an intervention involves (1) identifying core components (i.e., What will the practitioners be doing?); (2) projecting a timeline of the likely rate of movement across the LoUs; (3) anticipating the Stages of Concern and what might cause resistance; and (4) identifying TTA that will promote the use of critical components, address concerns as they arise, and support movement toward optimal implementation. All of these actions and steps can be aligned along a timeline. We present several suggestions regarding different types of data to collect and ways to use these data during the implementation process, including data about:

• Practitioners’ use of different intervention practices to inform conversations about intervention practices that are and are not being implemented consistently, and why
• Practitioners’ skills and knowledge of intervention practices, to understand their competence in different intervention practices and where additional TTA is needed
• Practitioners’ personal concerns that arise during the implementation process, to pinpoint sources of resistance and possible solutions
• The organization and community where the intervention is being implemented, to understand how aspects of an organization or community support or inhibit the implementation of an intervention

What would an implementation monitoring plan look like?

 Creating an implementation monitoring plan could involve tracking specific activities anticipated during each stage of implementation. To illustrate, table 2 presents sample implementation monitoring activities during a trial that implemented multidimensional treatment foster care (MTFC) at 3 sites in 51 counties (Chamberlain, Brown, & Saldana, 2011). Researchers involved in this work also conceptualized three phases of the change process similar to the three phases listed on page 3 of this brief. They collected information about a number of implementation activities associated with each phase, the amount of time associated with completing each implementation stage, and the percentage of activities completed. Table 2 provides readers with activities that could be included in an implementation monitoring plan for an evidence-based intervention in a human services setting.²

² We acknowledge that the dynamic nature of change is not reflected in this table (e.g., there is not necessarily a linear relationship among phases, fidelity, and levels) and that all people in an organization or community might not be at the same level or stage.

How much time does it take to move across the bridge?

The amount of time it takes to progress across the Implementation Bridge can vary greatly. Data from MTFC suggest that the creating and implementing phase could take 2 to 3 years, and other work suggests that the creating phase could take between 6 months and a year (Blase & Fixsen, 2013). Implementing new programs, curricula, and reforms in schools takes 3 to 5 years (Hall & Hord, 2015).

The time it takes to achieve implementation success is based on a number of factors, including (a) the initial readiness of an organization or setting (Dymnicki et al., 2014), (b) intervention complexity, (c) how new or different the intervention is from existing practices, (d) the resistance expressed by different stakeholders and sources of that resistance along the way, and (e) how valuable or beneficial people consider the intervention will be once it is being implemented.

Information collected during the MTFC trial provides an example of the time needed to complete each stage (Chamberlain et al., 2011). Progression through the stages differed markedly by site. For example, counties in Site 1 spent almost 2 years in the creating stage but then moved quickly through the implementing stage in about a year before applying for and achieving certification. Counties in Site 2 moved more quickly through the creating stage (in about 8 months) but took about 4 years to implement and achieve certification. Counties in Site 3 discontinued implementation activities during the creating stage and skipped 7 of 13 recommended activities in that stage. Followup analyses found that sites were more likely to be successful in implementing MTFC if they progressed quickly and thoroughly completed activities in the creating stage (Saldana, Chamberlain, Wang, & Brown, 2012).

Implications for Policy

Policy can be related to identifying and monitoring implementation milestones in several ways: (a) Include expectations about the likely time and supports necessary to meet implementation milestones in funding announcements and interim project deliverables; (b) align performance management and continuous quality improvement (CQI) with implementation milestones; (c) Encourage funding for proactive and ongoing technical assistance; and (d) invest in support for intervention infrastructure.

Include Expectations To Meet Implementation Milestones in Funding Announcements and Interim Project Deliverables

Policymakers and funders can play an important role in fostering the successful implementation of interventions by supporting the use of implementation milestones as a key monitoring strategy across the life cycle of a program. At the front end, funding announcements can outline (a) expectations around the exploration work that applicants must complete before submitting their applications and (b) specific activities and implementation milestones expected for each year of the grant. Some federal agencies have incorporated an initial planning year for grant funding, which culminates in the development of a detailed implementation plan to guide subsequent years’ activities. In other federal projects, after 2 years of funding, a panel of external reviewers hears evidence of a project’s outputs and preliminary outcomes and provides recommendations about activities or modifications to activities that will help the project move forward with implementation. A recommendation to terminate or continue funding is based on the project’s ability to incorporate these recommendations and make additional progress in implementation. In this way, continued funding depends on an approved implementation plan and the achievement of major milestones such as staff recruitment and training, identifying referral sources, establishing a Memorandum of Understanding with partner agencies, and identifying TTA resources.

Align Performance Management and CQI Efforts With Implementation Milestones

Many funded interventions have required—and in some cases, legislatively mandated—that participant-level outcomes, benchmarks, and indicators be tracked. However, these rarely include implementation milestones. Adding implementation milestones to grant or contract requirements provides an opportunity to embed these measures into a comprehensive framework for assessing program performance. As different sectors develop data systems for performance management, policymakers and funders are in a position to require that such indicators be included and tracked by programs. Once data are available, it will be important for policymakers to provide incentives for programs to use their data and engage key stakeholders in ongoing conversations about the data. Information sharing with funders and others who support the program can be facilitated through practice-to-policy feedback loops. These data can also advance CQI efforts throughout the project period. CQI is the complete process of identifying, describing, and analyzing strengths and problems and then testing, implementing, learning from, and revising solutions (Children’s Bureau, 2012). Such data can also help distinguish between programs that do not work and sites that have implemented a potentially effective program inadequately.

Encourage Funding for Proactive and Ongoing Technical Assistance

Policymakers need to be prepared to provide funding for TTA proactively and in response to identified implementation needs. Developing effective and usable implementation monitoring systems takes time. These systems need to carefully consider the specific context and larger system in which a program is being implemented. Policymakers understand that programs often experience barriers and challenges to achieving implementation milestones at various points during a program’s life cycle. This is part of the implementation process and should not be considered failure. Technical assistance can help programs reach critical milestones and also provide targeted consultation to identify the reasons that certain programs cannot achieve these milestones within a specific time frame. Some funders require remedial action or program improvement plans for projects that may need more time to reach the next milestone. For example, the Maternal, Infant, and Early Childhood Home Visiting Program legislation requires TTA for states that cannot measurably improve in required benchmark areas within 3 years.

Invest in Systems Support for Interventions

Policymakers’ awareness that investments in evidence-based interventions must include support for implementing interventions will help ensure success. Differences in context, capacity, and readiness should be considered. Identifying and tracking implementation milestones in these settings will involve different timelines and resources. Settings that are less ready will need different starting points and development structures. Infrastructure activities might include (1) creating activities such as strategic planning and relationship building with partners; (2) implementation and delivery services and workforce supports; and (3) sustaining activities such as financing, communication, evaluation, and monitoring (Hargreaves, Cole, Coffee-Borden, Paulsell, & Boller, 2013). These costs are not insubstantial and should be factored into intervention budgets as part of initial startup and ongoing implementation costs.

Implications for Supporting Practice

Several implications for supporting practice are related to identifying and monitoring implementation milestones: (a) the importance of engaging in planning activities early in implementation, (b) the importance of providing support for implementing an intervention, and (c) the importance of measuring infrastructure for implementation support.

Importance of Engaging in Planning Activities Early in Implementation

Sometimes, activities such as including stakeholders in planning and assessing the fit of an intervention, identifying the knowledge and skills needed, anticipating likely concerns, exploring the availability of resources to support the intervention, and building the infrastructure to support the intervention might seem like a waste of time and causes of delay. However, if such activities are absent, it is less likely that key milestones will be met later. Initial implementation is typically an awkward learning period for practitioners. Novice implementers need TTA support to resolve challenges that arise during implementation; they must also have the necessary time, energy, and support to incorporate intervention practices into their daily behaviors. Matching the right implementation supports to these stages and milestones can be encouraging and motivating.

Importance of Providing Support for Implementing an Intervention

Implementation drivers can support implementation and provide information about the training, coaching, administrative facilitation, systems intervention, and performance assessment needed to remove barriers to progress. These drivers build staff competence through selection, training, and coaching; they assess performance; and they provide organizational support through data-driven decision making, facilitative administration, and systems interventions. For example, if offsite training is needed, it may be necessary for alternate staff members to substitute for those who are attending training. An administrator would need to facilitate this type of systems intervention. Leaders can support implementation by demonstrating how an intervention connects to the mission and vision of an organization and convincing staff of the organization’s long-term support of and commitment to the intervention.

Importance of Measuring Infrastructure To Support Implementation

Measuring improvements in the infrastructure to support implementation can produce feedback that helps decision making and encourages those building the infrastructure to keep moving toward the next milestone (Blase & Fixsen, 2013). Each part of the system that supports the practitioner implementing an intervention should have the appropriate infrastructure to support the larger initiative. Information should be collected at multiple levels (e.g., organization, local, and state) of the support system. This enables one level of the system to prevent barriers that are impeding successful implementation from occurring at another level.

Conclusion

Studying the implementation of interventions can help us understand the challenges and opportunities that arise during the implementation process. Several research-verified constructs, including Fidelity of Implementation, Levels of Use, and Stages of Concern, can be used to measure the progress of systems, communities, organizations, and practitioners as they implement interventions. Each offers a way to understand an important element of the change process. Determining milestones at regular intervals can help to identify appropriate supports and necessary resources. Milestones can also help practitioners understand where they are now and what is required to take the next steps. Charting milestones for different stages of implementation and measuring success in reaching those milestones can provide motivation to continue the work and promote more successful implementation of interventions.

References

http://www.acf.hhs.gov/sites/default/files/private/fysb/prep-fidelity-monitoring...

Bellg, A. J., Borrelli, B., Resnick, B., Hecht, J., Minicucci, D. S., Ory, M., et al. (2004). Enhancing treatment fidelity in health behavior change studies: Best practices and recommendations from the NIH Behavior Change Consortium. Health Psychology, 23(5), 443.

Hargreaves, M., Cole, R., Coffee-Borden, B., Paulsell, D., & Boller, K. (2013). Evaluating infrastructure development in complex home visiting systems. American Journal of Evaluation, 34, 147–169.