Building and Sustaining Community Partnerships for Teen Pregnancy Prevention: A Working Paper. 3. Monitoring & Evaluation

06/01/1998

Most studies noted the importance of monitoring and evaluation. These activities should begin early in the partnership.(209) The impartiality of independent, external evaluation is the standard for scientific study.(210) There is value as well in evaluation efforts that are integrated into the system and collaboratively developed with partnership members.(211) Members of the partnership should be involved in setting evaluation goals, identifying data needed, and collecting and interpreting data.(212) The monitoring system should be dynamic, changing with developments in the coalition.(213) Whether internally or externally conducted, evaluation results should be reported at regular intervals, more frequently in early stages of development, and should be openly communicated to coalition members, the community and funders.(214) These recommendations reflect the view that monitoring and evaluation should not be one-time, after-the-fact judgments of partnership effectiveness but rather on-going feedback that can be used for improving the association.(215) In other words, evaluation should encourage creativity and innovation, not penalize failure.(216) Several types of evaluation are important in community partnerships.

Process or formative evaluation measures whether the community has been mobilized to address the problem by assessing the number of members and volunteers recruited; by-laws, goals and objectives, mission statements, committees, and reports created; financial resources generated; member satisfaction; and other measures internal to the partnership. In a variety of partnerships, these measures appear as the group takes form, usually within the first three to six months. They continue to be produced over the life of the partnership and are produced in greater numbers than other types of measures.(217) In the Plain Talk initiative, those partnerships that successfully moved from planning into the implementation phase completed a number of process measures defined by the funder and identified as "milestones" in the planning process.(218) In the CSAP Community Partnership programs evaluated on the FORECAST model, "markers" were used to verify that the program was implemented according to plan. If markers were not achieved, the plan might need review or the program might need "mid-course correction."(219) Many noted the importance of frequent feedback on these measures to assess fidelity to plans, to detect and communicate early successes to coalition members, the community, and current and potential funders, and to assess and consider the partnership course and needed adjustments.(220) Where resources are not available to conduct extensive evaluator-administered surveys and the like, self-administered checklists have been found to be useful.(221)

Partnerships also conduct outcome evaluation to assess whether partnership actions resulted in changes in the community. Outcome evaluation measures immediate results, such as services provided, actions taken in the community, and changes in community programs, policies and practices. These measures, which appear regularly after eight to 12 months of partnership existence and continue at fairly high and steady rates in successful partnerships, help sustain momentum, focus efforts, and justify activities to members, the community, and funders.(222) Regular feedback on immediate outcomes prompts discussion among members of partnership activities, focuses attention on the mission and objectives, communicates progress, and indicates where adjustments are needed in partnership functioning.(223) Feedback also leads to an increased number of changes in the community.(224)

Services, community actions, and community changes tend to increase at about the same rate in partnerships that are functioning well.(225) Community changes appear after community actions, and increases in community actions tend to produce community changes.(226)

Finally, it is necessary to document the connection between various partnership activities and the achievement of the partnership's mission, in other words, whether the partnership has had community-level impact. As one review noted "a well formed and maintained coalition is not necessarily effective in accomplishing its mission, even if it is effective in generating programs and activities or member satisfaction and commitment."(227) Impact or summative evaluation measures the effect of the partnership on intermediate and ultimate outcomes. Intermediate outcomes are behavioral objectives, such as increased abstinence and use of contraception, connected by research and theory to the ultimate health concern; and ultimate outcomes are the partnership's mission, for example, a reduction in the rate of teen pregnancy.

Measuring long term effects and system change is difficult; thus, impact evaluation demands skill, time, and resources, often requiring technical assistance to implement successfully.(228) For these reasons, some suggest limiting impact evaluation to the most promising strategies.(229) Others hypothesize that the greater the number of important changes in the community, the greater the likelihood of achieving a positive impact. They propose tracking process and activity outcomes with relevant risk factors and health indicators to establish links.(230) In the evaluation of a partnership to prevent alcohol and other substance abuse among youth, preliminary results indicate such a link.(231)

The long delay between the implementation of partnerships and measurable impacts creates problems for groups that need to justify their expenses, if not their existence, to the community, to policy makers, and to funding organizations. Large scale initiatives have been abruptly ended because they failed to show impact.(232) This situation is not easy to remedy; however, educating others about these delays, building short term goals into program design, and measuring process and intermediate outcomes can help indicate progress on the way to impact.(233)