Literature Review: Developing a Conceptual Framework to Assess the Sustainability of Community Coalitions Post-Federal Funding. A. Challenges of and Approaches for Assessing the Impacts of Community Coalitions


When several organizations and resources come together in the form of a community coalition, the assumption in the literature and among practitioners is that the pooling of intellectual and material resources will result in improved program effectiveness with the sum total of the coalition’s results being greater than the results for organizations working independently. A search of the literature reveals several community coalition studies which failed to find evidence of this effect for health outcomes (Community Intervention Trial for Smoking Cessation, 1995; Feinberg, Bontempo, & Greenberg, 2008; Hallfors, Cho, Hyunsan, Livert, & Kadushin, 2002), while other studies, as listed in Berkowitz and Wolff (2000), demonstrated significant improvements in health outcomes.

One reason for the inconsistent evidence on the ability of community coalitions to affect change and influence health outcomes is methodological. According to the literature, the traditional set of methodological tools is not well equipped to handle the realities of the complex world in which community coalitions operate (Berkowitz, 2001). For example, community coalitions often operate multiple interventions and program activities aimed at multiple levels and populations in multiple venues (Koepsell et al., 1992). This structure poses several challenges for traditional health outcomes evaluation designs such as an undefined universe for sampling, contamination of control groups, and the inability to identify and control key variables (Berkowitz, 2001).

In response to the challenges associated with demonstrating the impacts of community coalitions through evaluation, researchers are beginning to develop evaluation models of community coalitions that capture both their impacts at the individual level (e.g., health outcomes) and at the community level (e.g., capacity and environment) (Backer, 2003; Taylor-Powell, Rossing & Geran, 1998). Three evaluation models that are frequently used to capture the impacts of coalitions are the Targeting Outcomes of Programs (TOP) model, the Community Toolbox Evaluation model, and the theory of Empowerment Evaluation.

The TOP model is a framework for identifying and assessing outcomes by integrating program development, process evaluation, and impact evaluation elements (Bennett & Rockwell, 1995). Within the model, there are seven levels of evidence that can be used to assess whether the coalition is making an impact, from measuring resources to measuring social, economic, and environmental impacts. The evaluation model helps the evaluator track the coalition’s individual, community, and systems-level health outcomes, as well as its long-term impacts. 

The Community Toolbox Evaluation model is another evaluation model that provides a logical framework for assessing change throughout the stages of the coalition process, from assessing the success of problem identification to disseminating best practices identified throughout the evaluation (Fawcett et al., 2001). Given the dynamic nature of coalition activities, this model moves beyond the notion of evaluating a unidirectional causal relationship to evaluating a series of impacts.

Finally, an alternative approach to assessing the impacts of coalitions is the theory of Empowerment Evaluation (American Diabetes Association, 2009; Cramer, Mueller, & Harrop, 2003; Fetterman & Wandersman, 2005). Empowerment Evaluation theory is a frequently employed and thoroughly studied approach that intends to increase the probability of attaining program success through two key components. The first component is to provide stakeholders with the tools and resources they need to assess planning, implementation, and self-evaluation of their activities. The second component is to include evaluation as an integrated part of the planning and management of program activities. Empowerment Evaluation theory can also be used alongside traditional, external evaluation methods (Fetterman & Wandersman, 2005; Fetterman & Wandersman, 2007). 

Although the theoretical foundations of each model differ (including both traditional and participatory evaluation approaches), a shared goal of all three models is to provide a tool that is specific enough to measure and assess a particular coalition, yet general enough to allow for valid comparisons between coalitions. Even using these models, however, it is difficult to demonstrate that a community coalition has had direct and positive impacts over time.

Researchers have argued that the primary impacts of community coalitions may stem from their ability to alter their environment and/or increase their community’s capacity to continually identify and address health problems (Cheadle, Wagner, Koepsell, Kristal, & Patrick, 1992; Kegler, Twiss, & Look, 2000; Mittelmark, Hunt, Health, & Schmid, 1993). Therefore, evaluators have tried to demonstrate the ability of community coalitions to increase a community’s capacity and social capital, which is then leveraged to attain a positive impact over time (Hawe, King, Noort, Jordens, & Lloyd, 2000; Laverack & Wallerstein, 2001; Stephens & Studdiford, 2008). These outcomes can then be linked to the existing science showing a connection between the physical and social environment and long-term health outcomes. However, policymakers and funders are often left unconvinced by this solution, seeking instead direct evidence of health outcomes to justify a continued reliance on the community coalition as a mechanism for improving health.

View full report


"report.pdf" (pdf, 574.58Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®