Using Evidence-Based Constructs to Assess Extent of Implementation of Evidence-Based Interventions. Implications for Policy

09/01/2014

Policy can be related to identifying and monitoring implementation milestones in several ways: (a) Include expectations about the likely time and supports necessary to meet implementation milestones in funding announcements and interim project deliverables; (b) align performance management and continuous quality improvement (CQI) with implementation milestones; (c) Encourage funding for proactive and ongoing technical assistance; and (d) invest in support for intervention infrastructure.

Include Expectations To Meet Implementation Milestones in Funding Announcements and Interim Project Deliverables

Policymakers and funders can play an important role in fostering the successful implementation of interventions by supporting the use of implementation milestones as a key monitoring strategy across the life cycle of a program. At the front end, funding announcements can outline (a) expectations around the exploration work that applicants must complete before submitting their applications and (b) specific activities and implementation milestones expected for each year of the grant. Some federal agencies have incorporated an initial planning year for grant funding, which culminates in the development of a detailed implementation plan to guide subsequent years’ activities. In other federal projects, after 2 years of funding, a panel of external reviewers hears evidence of a project’s outputs and preliminary outcomes and provides recommendations about activities or modifications to activities that will help the project move forward with implementation. A recommendation to terminate or continue funding is based on the project’s ability to incorporate these recommendations and make additional progress in implementation. In this way, continued funding depends on an approved implementation plan and the achievement of major milestones such as staff recruitment and training, identifying referral sources, establishing a Memorandum of Understanding with partner agencies, and identifying TTA resources.

Align Performance Management and CQI Efforts With Implementation Milestones

Many funded interventions have required—and in some cases, legislatively mandated—that participant-level outcomes, benchmarks, and indicators be tracked. However, these rarely include implementation milestones. Adding implementation milestones to grant or contract requirements provides an opportunity to embed these measures into a comprehensive framework for assessing program performance. As different sectors develop data systems for performance management, policymakers and funders are in a position to require that such indicators be included and tracked by programs. Once data are available, it will be important for policymakers to provide incentives for programs to use their data and engage key stakeholders in ongoing conversations about the data. Information sharing with funders and others who support the program can be facilitated through practice-to-policy feedback loops. These data can also advance CQI efforts throughout the project period. CQI is the complete process of identifying, describing, and analyzing strengths and problems and then testing, implementing, learning from, and revising solutions (Children’s Bureau, 2012). Such data can also help distinguish between programs that do not work and sites that have implemented a potentially effective program inadequately.

Encourage Funding for Proactive and Ongoing Technical Assistance

Policymakers need to be prepared to provide funding for TTA proactively and in response to identified implementation needs. Developing effective and usable implementation monitoring systems takes time. These systems need to carefully consider the specific context and larger system in which a program is being implemented. Policymakers understand that programs often experience barriers and challenges to achieving implementation milestones at various points during a program’s life cycle. This is part of the implementation process and should not be considered failure. Technical assistance can help programs reach critical milestones and also provide targeted consultation to identify the reasons that certain programs cannot achieve these milestones within a specific time frame. Some funders require remedial action or program improvement plans for projects that may need more time to reach the next milestone. For example, the Maternal, Infant, and Early Childhood Home Visiting Program legislation requires TTA for states that cannot measurably improve in required benchmark areas within 3 years.

Invest in Systems Support for Interventions

Policymakers’ awareness that investments in evidence-based interventions must include support for implementing interventions will help ensure success. Differences in context, capacity, and readiness should be considered. Identifying and tracking implementation milestones in these settings will involve different timelines and resources. Settings that are less ready will need different starting points and development structures. Infrastructure activities might include (1) creating activities such as strategic planning and relationship building with partners; (2) implementation and delivery services and workforce supports; and (3) sustaining activities such as financing, communication, evaluation, and monitoring (Hargreaves, Cole, Coffee-Borden, Paulsell, & Boller, 2013). These costs are not insubstantial and should be factored into intervention budgets as part of initial startup and ongoing implementation costs.

View full report

Preview
Download

"ib_Milestones.pdf" (pdf, 421.66Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®