Skip to main content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

HHS Logo Eagle IconU.S. Department of Health and Human Services

An Assessment of the Sustainability and Impact of Community Coalitions once Federal Funding has Expired

Publication Date

By:
NORC at the University of Chicago

Abstract

This report examines the long-term sustainability and impact of community coalitions that were funded by the Community Access Program (CAP) and its successor, the Healthy Communities Access Program (HCAP). CAP/HCAP was one of the largest federal investments to strengthen local safety nets through community coalitions providing $525 million in grants from 2000 to 2006 to 260 coalitions across 45 states, the District of Columbia, and the U.S. Virgin Islands. This study utilized the experience of the HCAP grantees as a lens to explore the sustainability and impact of community coalitions once their initial federal funding has expired.

Disclaimer

This study was conducted by the Urban Institute under contract number [CONTRACT_NUMBER], task order number [TASK_ORDER_NUMBER], with the HHS's Office of Assistant Secretary for Planning and Evaluation. The authors take full responsibility for the accuracy of material presented herein. The views expressed are those of the authors and should not be attributed to ASPE or HHS.

"

Executive Summary

The Office of the Assistant Secretary for Planning and Evaluation (ASPE), within HHS, contracted with NORC at the University of Chicago (NORC) to examine the long-term sustainability and impact of community coalitions that were funded by the Community Access Program (CAP) and its successor, the Healthy Communities Access Program (HCAP). CAP/HCAP was one of the largest federal investments to strengthen local safety nets through community coalitions providing $525 million in grants from 2000 to 2006 to 260 coalitions across 45 states, the District of Columbia, and the U.S. Virgin Islands. This study utilized the experience of the HCAP grantees as a lens to explore the sustainability and impact of community coalitions once their initial federal funding has expired.

Background

CAP was originally funded by Congress and implemented by the Health Resources and Services Administration (HRSA) in September 2000. In 2002, Congress passed authorizing legislation creating HCAP, which began in fiscal year 2003. Congressional funding for CAP began with a $25 million appropriation in the Fiscal Year (FY) 2000 budget, used to make grants to 23 coalitions of community organizations and safety net providers. Additional funding provided to HRSA for the program comprised approximately $500 million from FY 2001 through FY 2005. The program was unfunded for FY 2006. Between 2000 and 2006, HCAP provided grants to local communities to strengthen the health care safety net that served the uninsured and underinsured. In total, HRSA awarded 260 grants in 45 states and the District of Columbia and the Virgin Islands. Most grantees received an initial grant of $750,000 to $1 million in the first year, with additional, but reduced, funding for two additional years. Many grantees within the initial cohorts of the HCAP program had a track record of building partnerships, having been recipients of earlier national foundation funding from the W.K. Kellogg and Robert Wood Johnson Foundations, or having received support from other contributors such as health systems, corporations, or non-profit organizations.

Although federal and foundation grants have long supported safety net providers’ efforts to increase access to care and the quality of care for underserved populations, the HCAP program distinguished itself on three fronts: first, by requiring collaboration—grants were given only to consortia of local providers, not to individual institutions; second, the funds were to be directed to infrastructure development, rather than direct service provision; and, third, the program afforded grantees wide latitude to formulate programs based on their communities’ specific needs (e.g. care coordination, patient education, disease prevention, service integration). Through these design features, the program sought to overcome the fragmented nature of safety net care by bringing the major players together and providing funds to address problems that cannot adequately be addressed by individual providers or organizations.

Methods

This project sought to use the experience of HCAP to learn about the long-term sustainability of federally-funded coalitions and answer three key research questions:

  1. How many community coalitions funded under HCAP are still in existence?
  2. What coalition characteristics are associated with sustainability and what factors promote or hinder community coalition sustainability?
  3. What are the impacts of HCAP coalitions post-federal funding and what are the types of outcomes achieved? To what extent have the coalitions evolved to address the needs in the community?

To examine these important issues, NORC conducted a multi-method assessment using qualitative and quantitative research methods, including a review of literature on coalition sustainability; a survey of the 260 former HCAP grantees; key informant interviews with sustained and not sustained HCAP coalitions; and case studies involving site visits with sustained coalitions.

The first component of the study consisted of a literature review and the development of the conceptual framework. The literature review focused on community coalition sustainability. Specifically, the literature review explored the ways in which researchers, policy makers, and practitioners have defined and measured sustainability for community coalitions. The literature review also explored the facilitators of and barriers to sustainability in community coalitions. Findings from the literature review were synthesized to develop operational definitions for key concepts, such as what it means for a coalition to be "sustained" versus "not sustained." With a working definition of sustainability in the context of community coalitions and their activities, NORC developed a conceptual framework for assessing the sustainability of community coalitions after their initial federal funding has expired. In the second component of the project, the literature review and conceptual framework supported the development of a survey to assess the HCAP coalitions based on the sustainability definition and the factors driving sustainability included in the conceptual framework. A mailed, self-administered questionnaire was sent to all 260 HCAP grantees between March 10 and May 31, 2011 and achieved a 63% response rate. The questionnaire included a screening question to determine the coalition’s current operational status; other questions focused on the coalition’s structure, mission and goals, funding sources, activities, evaluation methods, sustainability plans, and overall impact.

From the pool of grantees that completed the survey (113 (68%) sustained and 52 (32%) not sustained), a total of 25 (15%) coalitions were invited to participate in telephone interviews. Telephone interviews with key informants of these coalitions were conducted in August and September 2011. The purpose of the interviews was to gather more detailed information and confirm survey responses regarding the coalitions’ experience and strategies for trying to sustain the coalition after initial federal funding ended, coalition outcomes, and future plans. In the final phase of the project, NORC conducted site visits with six high performing sustained coalitions. Two members from the NORC team traveled to each location to conduct site visits. All site visits occurred during October 2011. The site visits involved in-depth interviews with the coalition directors and lead staff and representatives from the coalition’s partner organizations. NORC staff facilitated the interviews using a semi-structured interview protocol that focused on facilitators and obstacles of sustainability, the coalition’s structure and dynamics, and lessons learned.

Results from the surveys, key-informant interviews, and site visits were analyzed to identify overarching themes and differences among sustained coalitions and not sustained coalitions. Survey data analyses were conducted in STATA 10, and focused on the calculation of descriptive statistics, such as frequencies, central tendencies, and distributions for all questions. Standard difference of means and difference of proportions tests were used for making statistical comparisons between sustained and not sustained coalitions. Logistic regression models were employed to assess specific hypotheses derived from the conceptual framework. Qualitative data analysis of the 25 phone interviews was conducted using QSR NVivo 9 (NVivo). NVivo facilitated the identification of common themes across community coalitions in addition to major differences between sustained and not sustained coalitions. NORC also analyzed the site visit data by validating and confirming key information and themes; reviewing notes from each site visit; identifying significant quotes and comments; and conducting debrief conversations with each grantee director either at the end of the site visit or shortly thereafter by telephone to confirm the interpretation of key findings and ensure that important information was not omitted.

Findings

The study identified a number of findings related to the sustainability of community coalitions post initial federal funding:

More than two-thirds of the HCAP coalitions were sustained, with many pursuing a range of activities in their communities.Even using a fairly rigorous definition of sustainability which required three or more organizations to be collaborating for purposes of one or more of the original HCAP goals, a total of 113 (68%) of the 165 responding coalitions were sustained. The fact that more than two-thirds of the coalitions were sustained is encouraging, particularly given that the HCAP project did not have an explicit emphasis on sustainability. These coalitions are still actively pursuing a range of activities in their communities (continuing existing activities and taking on new ones), and have achieved both individual-level impacts (e.g. increased access to primary and specialty care, increased enrollment into health insurance plans) and other policy and system impacts (e.g. streamlined eligibility systems, new processes for care coordination across community providers). Moreover, even among the 52 (32%) of coalitions classified as not sustained, the majority (80% of not sustained coalitions) report that at least one of the activities they conducted under the HCAP grant was still being conducted at the time of the survey. However, all of the 52 coalitions reported that they were no longer a coalition of three or more organizations, and as such these coalitions were considered to be not sustained. Of the not sustained coalitions, 35 (67%) of the coalitions dissolved due to issues such as funding or organizational problems and 17 (33%) disbanded because the coalition achieved all of its goals or was no longer needed in the community.

When asked to define sustainability in the context of their coalition, many HCAP coalitions described sustainability in terms of the institutionalization of key activities in the community and maintenance of their impacts. Sustainability can be defined in terms of whether organizations are still working together, whether specific activities continue, or some combination of the two. This study required continued collaboration among organizations in order to be considered sustained, though the survey instrument also sought the perspectives of coalition staff on how they defined sustainability. The vast majority of sustained and not sustained coalitions (84%) included some aspect of institutionalized activities or lasting impacts in their definition of sustainability. Only 16 percent of all coalitions defined sustainability in terms of the coalition membership and structures only. Additionally, a greater proportion of sustained coalitions (40%) defined sustainability in terms of the coalition and its activities compared to the not sustained coalitions (19%). The majority of not sustained coalitions (75%) defined sustainability in terms of either its activities or impacts continuing to benefit the community even if the coalition was no longer operating.

Coalition characteristics associated with sustainability include coalition leadership, membership diversity, coalition structure, vision guiding action, and resources.

  • Leadership. The results of this study are consistent with others indicating that strong leadership has a direct and positive influence on the sustainability of coalitions. Survey results showed that leadership experience, measured in terms of coalition management, years of experience in the field, and experience working or living in the community, increases the likelihood of sustainability even when controlling for other key predictors of coalition strength and sustainability, such as membership, funding, and coalition structure variables. The findings from the key informant interviews and site visits also underscored that coalitions with strong leaders were able to overcome many of the challenges facing coalitions as they transitioned out of the core HCAP funding and into a self-sustaining mode. Beyond leadership from individuals, the survey results revealed the importance of having governing bodies comprised of high level leaders from the membership organizations or external organizations. Sustained coalitions were more likely than not sustained coalitions to have a Board of Directors (60% compared to 26%, p<.01) or Executive Committee (59% compared to 43%, p<.10).
  • Coalition Membership. Sustained coalitions are comprised of 40 member organizations, on average, with 79 percent of members considered active. Not sustained coalitions were comprised of 22 member organizations, on average, with 74 percent of members considered active. Although survey results showed no significant differences in the size of sustained and not sustained coalitions, the proportion of active members was a significant predictor of sustainability controlling for other coalition characteristics. Additionally, membership turnover was not a significant, predictor of sustainability. Comparing sustained and not sustained coalitions, the findings suggest that it may be beneficial to have some churning of the membership. The turnover among sustained coalitions tended to be driven by issues or changes at the member organization level and not due to constraints at the coalition level. During the site visits, coalition leaders noted that the loss of a member organization that may have been distracting to the coalition’s efforts may ultimately strengthen the commitment of the remaining organizations.
  • Structure. Results of the influence of coalition structure on sustainability were a departure from expectations. Sustained and not sustained coalitions had equal rates of formal membership structures and the vast majority with formal structures (92% and 89%, respectively) had Memoranda of Agreement (MOA), Memoranda of Understanding (MOU), or Interagency Agreements (IAAs) in place with members. There was no association between having formal structure with members (e.g., MOAs or IAAs) and sustainability. Survey results also indicated that not sustained coalitions were significantly more likely than sustained coalitions to hold more frequent meetings compared to sustained coalitions. Qualitative findings support that sentiment with several sustained coalitions discussing the need to reduce the frequency of meetings and increase the quality of meetings. Not sustained coalitions mentioned that the number of meetings and operational requirements for members may have been too burdensome to sustain, and could have played a role in the loss of membership organizations in the coalition.
  • Vision Guiding Action. The ability to stay focused on the overall goals of the coalition while maintaining action steps that are manageable in the day to day was positively associated with sustainability. Survey results showed a high level of agreement among sustained coalitions about the importance of the coalition’s vision and using the vision to focus activities and services. From the perspective of interviewed and site visited coalitions, these factors are key contributors to sustainability.
  • Resources. Sustained coalitions were significantly more likely to have funding from more diverse sources, including state level agencies and departments (other than the health department), local health departments, foundations, and community-based organizations. Fifty percent of sustained coalitions said they had more diverse funding at the time of the survey than during HCAP. Additionally, on the survey as well as in the telephone interviews and site visits, many sustained coalitions indicated that member organizations contribute funds for coalition operations and programs or services. Also, when controlling for other coalition characteristics, a higher degree of flexibility to use funding for operational or programmatic purposes was a positive predictor of sustainability.

Evaluation was considered an important aspect of coalition work for both sustained and not sustained coalitions; however, sustained coalitions were more likely to use quantitative evaluation methods and to conduct outcomes evaluations than not sustained coalitions. Coalitions noted that evaluation activities were important for fine tuning the coalition’s work and for demonstrating results to funders and key stakeholders. The ability to make a business case that shows how resources are leveraged by the coalition and how coalition activities save the lead organization and member organizations money is also important. While some sustained coalitions had sophisticated means for collecting, processing, and analyzing these data (e.g., dummy claims processing through Blue Cross Blue Shield for all enrolled patients), other simple analyses were effective in making the business case for the program as well (e.g., adding the value of all donated pharmaceuticals, lab, and imagery services and dividing by the coalition’s total funding in order to demonstrate that every dollar of funding yields so many more dollars in patient products and services). While sustained coalitions tended to have more sophisticated evaluation methods and processes, they explained that the anecdotes were also important to demonstrate the health and social outcomes of their programs and identify new funding opportunities.

Implementation of sustainability actions mattered more than the type and degree of sustainability planning. Sustained coalitions differed in terms of the sustainability actions they undertook and the findings revealed important predictors of coalitions sustainability including reassessing the coalition’s goals, activities, or priorities and developing an infrastructure in the community to support systems-level activities. The qualitative interviews suggested that sustained coalitions were always thinking about what needed to happen for sustainability, such as obtaining additional types of funding, developing a focus on showing results, getting institutional support for coalition activities, evolving with the community needs, incorporating new partners, as well as strategizing to handle external factors like the Patient Protection and Affordable Care Act of 2010 (Affordable Care Act). While sustainability plans were an important first step, they had no impact on sustainability status. Additionally, establishing a committee for sustainability decreased the likelihood of sustaining the coalition, holding all other variables constant. Findings suggested that the coalitions’ ability to carry out sustainability actions differentiated the sustained from the not sustained coalitions.

Even though many of the HCAP coalitions could be considered "successful" in terms of their continued existence, some coalitions remained apprehensive about the long-term stability of their coalition. Two-thirds of the sustained coalitions said they have sufficient funding for the next year, but that proportion drops to only 38 percent when asked to forecast for two years out. Nearly all of the sustained coalitions (68% of HCAP grantees) are confident they will exist in two years, but less than three-quarters of those believe they will exist in 10 years. Coalition work is a challenging endeavor and even the most successful coalitions face uncertainty which may encourage greater attention to long-term planning and emphasis on action steps.

Sustained and not sustained coalitions reported success in achieving individual, systems-level, and policy impacts. The coalitions were asked how successful they have been in achieving individual health and behavior outcomes, systems-level outcomes, and policy change outcomes since the end of the HCAP grant period. Overall, the coalitions reported fairly widespread success for all types of impacts. For individual and behavior outcomes, 75 percent of sustained coalitions and 70 percent of not sustained coailtions reported that they were successful or very successful in achieving individual health and behavior outcomes. For systems-level outcomes, 70 percent of sustained coalitions and 53 percent of not sustained coalitions reported that they were successful or very successful in achieving systems-level outcomes. Thirty seven percent of sustained coalitions and 28 percent of not sustained coalitions reported that they were successful or very successful in changing policies. Sustained and not sustained coalitions had similar perceptions of how many of the coalition’s individual health and behavior outcomes and systems-level outcomes would continue to benefit the community without coalition support. Forty six percent of not sustained coalitions estimated that none or only a few of their policy outcomes would be sustained compared to 29 percent of sustained coalitions (p<.10).

Sustained and not sustained coalitions reported what they would do differently if they could start their coalition over again. First, coalitions reported they would diversify their membership (e.g. expanding to business, specialty care providers, private primary care providers and medical groups, and engaging all health systems in the community including competitors, consumer groups, and faith-based groups). The overall sentiment was that a more diverse membership would have better represented all perspectives of the community and the target population, as well as provided different types of knowledge, skills, and resources for the coalition. Second, coalitions would establish more formal structures. The most frequently cited example was establishing formal contracts, MOAs, etc. with the member organizations that explicitly state and describe the expectations and responsibilities for the member and the coalition so that both parties share a common understanding of the relationship from the very beginning. Several coalitions also expressed a desire to increase the leadership structure through the executive committee or board of directors, and several regreted that the coalition did not establish as an independent not-for-profit. The third frequently cited lesson learned was to focus the coalition’s mission and activities to concentrate on what the coalition did well and where it could make the greatest impact. Several coalitions stressed the importance of assessing activities regularly and being willing to let go of those that do not work well.

Implications

Findings from this study have implications for community coalitions at all stages of development, as well as evaluators and funders of coalitions.

Coalitions should invest time and resources into developing characteristics and capacities that facilitate sustainability. Such actions might include identifying leaders with experience working in the community and committed staff that are highly effective managers; incorporating diverse, multi-sectorial partner organizations with a shared sense of mission and a willingness to invest in the coalition’s success; pursuing diverse funding sources that will finance both the coalition’s programs and its operations; and investing in robust outcome evaluations that demonstrate the coalition’s performance and impact. Taking discrete action steps towards sustainability—from securing funding to institutionalizing some program activities in other organizations—was more important than only making plans for sustainability.

In setting expectations of coalition sustainability, both funders and evaluators must carefully identify their definition of sustainability. The conceptual framework developed as part of this study illustrates the importance of defining what is meant by sustainability. Is the funder or evaluator interested in the sustainability of the coalition or in the sustainability of its activities and impacts? The answer to this question will impact the funder or evaluators’ research questions, hypotheses and activities. Additionally, when providing initial program funding for community coalitions, funders must consider whether they expect the coalitions to continue post-grant. If so, funders and coalitions can collaborate during the funding period to lay the necessary groundwork for sustainability. Findings from this study also suggest that, when assessing coalition sustainability and outcomes, evaluators should pay close attention to the presence of specific characteristics and capacities that facilitate sustainability. This study presented numerous comparisons of the characteristics and capacities of sustained and not sustained coalitions, which may provide a useful starting point for framing future evaluation studies. For example, sustained coalitions were significantly more likely than not sustained coalitions to have diversity in the types of members involved, both within the health sector and across sectors. Future studies may further explore the sustainability of multi-sectorial coalitions, specifically.

Community coalitions are a promising approach for implementing key aspects of the Affordable Care Act.The Affordable Care Act established a Prevention and Public Health Fund for public health activities, which states and communities are using to support community and clinical prevention activities. Coalitions are a natural fit for implementing key prevention activities through the Prevention Fund given their focus on applying evidence-based, population-based, and culturally-appropriate interventions. Additionally, policymakers have an opportunity to leverage diverse, multi-sectorial networks that have the ability to effectively enroll individuals into the health insurance exchanges. Community coalitions—and the organizations that participate in them such as community and faith-based organizations—are particularly well-suited to enroll individuals into these exchanges and provide guidance during the enrollment process. Further, community coalitions that serve the most vulnerable populations will be particularly important in this process, given that these populations traditionally trust the organizations that participate in the coalitions.

Key facilitators of community coalition sustainability and coalition building should be considered when developing future programs. Policy makers may consider investing in local leaders and building leadership capacity in communities across the country through technical assistance and training opportunities. The most effective and highly sustainable coalitions were led by leaders with prior experience working within their community, suggesting that leadership training in community-based participatory approaches may be valuable. Policy makers should also consider investing in other mechanisms for supporting sustainability, such as funding rigorous data collection and evaluation to encourage results-oriented planning and assist coalitions in appealing to additional funders. Finally, while capacity building activities may wane over time, policy makers should structure funding to account for an ongoing investment in these activities throughout the life of the coalition.

Future Research

Researchers are investigating the formation and development of community coalitions—from their initial mobilization and development of key structures and processes, to their implementation and sustainability. These findings shed light on the sustainability outcomes of community coalitions and the factors that impact sustainability. However, there is a dearth of empirical information on what happens to coalitions once they have formed, developed, and reached a sustained state of activity. Future research should explore whether there are different stages of coalition sustainability, how coalitions mature and evolve in each of these stages, and whether their characteristics and capacities change over time. Another area for future research is identifying funding models for coalition sustainability and the optimal length of funding. While sustainability is a process that occurs over time, funders and policy makers would benefit from more information about the optimal length of time to fund a coalition or program to achieve its goals. Finally, no single study can investigate every coalition characteristic or capacity that may affect sustainability. Future research should continue to explore different factors and conditions that foster community coalition sustainability.

1. Introduction

Community coalitions have increasingly been used as a vehicle to foster improvements in community health. When they function well, they offer a powerful means of mobilizing individuals, raising the visibility of issues that are of concern to the community, minimizing duplication of effort and services, making efficient use of new resources, and convening diverse organizations that have a common goal or concern in order to develop comprehensive solutions. Funders, including the U.S. Department of Health and Human Services (HHS), have often provided grants to community coalitions to promote health, and have encouraged partners to work together at the community level to achieve common aims. Further, many initiatives have required grantees to form coalitions because they offer a means of coordinating multi-faceted approaches to address complex problems that are rooted in the social and physical environment of a community.

While research has shown that coalitions can successfully address health concerns and support improved health behavior, less is known about the sustainability of coalitions and their outcomes once their initial funding ends. Further, there is significant ambiguity around the meaning and assessment of coalition sustainability. There are no standard guidelines for evaluating sustainability, and no common definition of sustainability currently exists. Some studies have operationalized the concept of sustainability as the continuation of all or part of the coalition and its structure after initial external funding ends. Others have focused less on the idea that a coalition must maintain the membership and structure to be considered sustainable, and more on the idea that it continues to meet its initial goals and objectives to provide benefits to the community. Furthermore, many HHS initiatives provide seed money to community coalitions to promote their sustainability, but very little research has been done to examine the impact of community coalitions that have continued to exist after the initial funding has ceased.

To fill this important research gap, and to learn more about what happens to the dynamics of collaborations that have been built around grant funding once that funding is no longer present, the Office of the Assistant Secretary for Planning and Evaluation (ASPE), within HHS, contracted with NORC at the University of Chicago (NORC) to examine the long-term sustainability and impact of community coalitions that were funded by the Community Access Program (CAP) and its successor, the Healthy Communities Access Program (HCAP).1 HCAP was one of the largest federal investments to strengthen local safety nets through community coalitions providing $525 million in grants from 2000 to 2006 to 260 coalitions across 45 states, the District of Columbia, and the U.S. Virgin Islands. Many grantees within the initial cohorts of the HCAP program had a track record of building partnerships, having been recipients of earlier national foundation funding from the W.K. Kellogg and Robert Wood Johnson Foundations, or having received support from other contributors such as health systems, corporations, or non-profit organizations.

The conclusion of the HCAP funding posed a number of problems for grantees that lacked other funding sources, and the sustainability of several programs was threatened at the end of the funding stream. As such, exploring what has happened to the HCAP coalitions during the intervening years since they stopped receiving initial federal funding can provide important information—not only on the effectiveness of the HCAP coalitions, but also on the post funding experiences of coalitions that successfully competed for grant funding from HHS. This project sought to use the experience of HCAP to learn about the long-term sustainability of federally-funded coalitions and had several research questions:

  1. How many community coalitions funded under HCAP are still in existence?
  2. What coalition characteristics are associated with sustainability and what factors promote or hinder community coalition sustainability?
  3. What are the impacts of HCAP coalitions post-federal funding and what are the types of outcomes achieved? To what extent have the coalitions evolved to address the needs in the community?

To examine these important issues, NORC conducted a multi-method assessment using qualitative and quantitative research methods, including a review of literature on coalition sustainability; a survey of the 260 former HCAP grantees; key informant interviews with sustained and not sustained HCAP coalitions; and case studies involving site visits with sustained coalitions.

This report examines the long-term sustainability and impact of community coalitions that were funded by HCAP. It synthesizes the findings from our study of the sustainability, evolution, and impact of the HCAP coalitions since the expiration of HCAP funding.

2. Background on the Healthy Communities Access Program

CAP was originally funded by Congress and implemented by the Health Resources and Services Administration (HRSA) in September 2000. In 2002, Congress passed authorizing legislation creating HCAP, which began in fiscal year 2003 (Health Care Safety Net, 2002). From 2000 through 2006, HCAP provided grants to local communities to strengthen the health care safety net that served the uninsured and underinsured. Congressional funding for CAP began with a $25 million appropriation in the Fiscal Year (FY) 2000 budget, used to make grants to 23 coalitions of community organizations and safety net providers. Additional funding provided to HRSA for the program comprised approximately $500 million from FY 2001 through FY 2005. The program was unfunded for FY 2006.

In total, HRSA awarded 260 grants in 45 states and the District of Columbia and the Virgin Islands. Most grantees received an initial grant of $750,000 to $1 million in the first year, with additional, but reduced, funding for two additional years. Although federal and foundation grants have long supported safety net providers’ efforts to increase access to care and the quality of care for underserved populations, the HCAP program distinguished itself on three fronts: first, by requiring collaboration—grants were given only to consortia of local providers, not to individual institutions; second, the funds were to be directed to infrastructure development, rather than direct service provision; and, third, the program afforded grantees wide latitude to formulate programs based on their communities’ specific needs. Through these design features, the program sought to overcome the fragmented nature of safety net care by bringing the major players together and providing funds to address problems that cannot adequately be addressed by individual providers or organizations.

HCAP coalitions focused on a variety of activities, including service integration, expansion of the delivery system, cultural competency, provider education, community and patient education, disease detection and prevention, service integration, and new health insurance plans for the uninsured, among others. The outcomes of their grants were documented in an evaluation of the HCAP program conducted by NORC in 2006, including:

  • A total of over 640,000 people were enrolled in Medicaid and SCHIP;
  • About 156,000 were enrolled in private insurance or a new program for the uninsured;
  • Over 560,000 individuals were assigned to a medical home;
  • Over 1.2 million individuals were assigned to a primary care provider;
  • 483,000 and 438,000 patients were referred to primary and specialty care, respectively;
  • A total of 650,000 patients were reached through care coordination or navigation systems; and
  • Over 500,000 patients were reached through programs targeting appropriate use of the emergency department.

The 2006 evaluation also identified several lessons learned related to collaboration and sustainability:

  • A broad consortium of diverse members was critical to developing a successful program. In many of the more successful HCAP coalitions, a history of collaboration and shared experiences provided a solid foundation on which to undertake additional endeavors.
  • The HCAP community collaboration model supported increased integration of services and coordination of care. The development of shared infrastructure contributed to increased communication and collaboration, administrative efficiencies, and improvements in coordination of care and health care access for the under- and uninsured.
  • Planning for sustainability at the outset and documenting program results were key to securing ongoing funding and creating lasting improvements to the safety net. The most successful HCAP grantees worked towards securing funds from the beginning of their programs, pursuing a variety of funding sources, including foundations, other community organizations, or the use of internal funds. Being able to demonstrate results (in terms of grantees’ access or health outcomes) and, in particular, cost savings was frequently instrumental in gaining financial support.

Using the HCAP grantees as a lens to explore sustainability of community coalitions is particularly important today given new federal investments in clinical and community-based strategies. Signed into law in February 2009, the American Recovery and Reinvestment Act (ARRA) provided funding for community-based health strategies through a Prevention and Wellness fund. A year later, in March 2010, the Patient Protection and Affordable Care Act (Affordable Care Act) was signed into law, expanding coverage and access to health services for Americans. HHS has implemented several programs that build on ARRA and the Affordable Care Act—including the Communities Putting Prevention to Work (CPPW) Initiative and the Community Transformation Grants (CTGs). These programs are creating policy, systems and environmental changes, and demonstrate the federal government’s investment in innovative community-based strategies to improve health outcomes. Understanding the facilitators and barriers to community coalition sustainability is an issue of paramount importance both for the multi-sectorial community coalitions that are implementing this work as well as their funders.

3. Conceptual Framework for Assessing Community Coalition Sustainability

The section of this report summarizes key findings from an extensive literature review on community coalition sustainability that identified how researchers, policy makers, and practitioners have defined and measured sustainability for community coalitions (NORC, 2010). The findings from the literature review were synthesized to develop operational definitions for the key concepts and a conceptual framework to assess the sustainability of community coalitions. The definitions and framework presented in this section were then applied to the HCAP population and guided the study’s approach to additional primary data collection, as described in Section 4.

Summary of Approaches to Studying Coalitions, Impacts, and Sustainability

The federal government and foundations are increasingly investing in community coalition-driven programs to create capacities within and across organizations, convene different community segments, conduct innovative activities, and extend health and social benefits to underserved populations. Prior research supports this approach with evidence showing that community coalitions can fulfill three main functions: create collaborative capacity; build community capacity; and foster change at the local level. Coalitions mobilize community resources, creating collaborative capacity among coalition members, within member relationships, and through organizational structure and programs (Foster-Fishman, Berkowitz, Lounsbury, Jacobson, & Allen, 2001). In addition to increasing collaborative capacity, community coalitions strengthen community capacity by building social capital that can be applied to other health and social issues (Fawcett et al, 1995). Finally, because community coalitions are more focused on the implementation of services at the local level, they are positioned to bring about social change and improve the health of communities by creating new programs or services, developing new or more coordinated systems or infrastructure, advocating for stronger policies, influencing individual health or behavior, and disseminating products or materials, among other activities (Butterfoss, 2007; and Wolff, 2001).

Since community coalitions have emerged as a popular vehicle for addressing community health issues, significant resources have been invested in assessing best practices for the development and implementation of community coalitions. However, few systematic studies have been conducted to examine trends in community coalition outcomes and impacts (Berkowitz, 2001; Cramer, Mueller, & Harrop, 2003; Payne, 1999). Researchers developed and continue to refine evaluation models of community coalitions that capture both their impacts at the individual level (e.g., health outcomes) and at the community level (e.g., capacity and environment) (Backer, 2003; Taylor-Powell, Rossing & Geran, 1998). These models share a goal of providing an evaluation strategy that is specific enough to measure and assess a particular coalition, yet general enough to allow for valid comparisons between coalitions. However, even with these frameworks, a full accounting of the issues, populations, or intervention methods addressed by community coalitions is lacking.

Factors Identified in the Literature that Affect Coalition Functioning and Sustainability
Key factors that affect coalition functioning and sustainability are leadership, membership, structure, vision guiding action, funding diversity, sustainability planning, and contextual factors.

Given the research and programmatic investments in community coalitions to date, funders often expect that community coalitions and their activities will be sustained post-funding. While the concept of sustainability is germane to research on both community-based programs and community coalitions, a consensus definition of sustainability has not emerged in either body of research. The primary divergence among definitions in both bodies of literature relates to the unit of analysis—what is being sustained. Some definitions focus on sustaining the coalition (Rog et al., 2004; Butterfoss, 2007; Edwards et al., 2007), while others focus on sustaining the activities and impacts of the program or coalition (Scheirer, 2005; Alexander et al., 2003).

Researchers have developed conceptual frameworks and frameworks to define sustainability in the context of community coalitions (Alexander et al., 2003; Edwards et al., 2007; Mancini & Marek, 2004; Beery et al., 2005; Shediac-Rizkallah & Bone, 1998; Rog et al., 2004). Collectively, the models demonstrate the importance of measuring both the sustainability of the coalition and the coalition’s activities separately. The models also highlight different coalition-specific and contextual factors that affect sustainability. In reviewing these models and other literature, NORC identified several factors as key for coalition functioning and sustainability: leadership, membership, structure, vision guiding action, funding diversity, sustainability planning, and contextual factors.

  • Leadership. The CCAT, Empowerment Theory, and other studies have identified effective leadership as a facilitator of coalition action and sustainability (Butterfoss, Goodman, Wandersman, Valois, & Chinman, 1996a; Butterfoss, Goodman, & Wandersman, 1996b; Goodman et al., 1998). Leadership can consist of one or both of the following: the member organizations of a coalition, and the individual leaders within a coalition (Bailey & McNally Koney, 1995). Research suggests that the convening or "lead" agency must have organizational capacity, commitment, and vision, among other characteristics to build an effective coalition (Butterfoss, 2007). In addition, leadership from individual staff members in the member organizations is also critical. Coalitions and partnerships with action-oriented leadership (Bazzoli et al., 2003; Hasnain-Wynia, 2003) and competent, committed leaders are most effective (Conrad et al., 2003). Hasnain-Wynia et al. (2003) found that partnerships with effective or ethical leadership were more likely to be perceived by their memberships as effective in achieving their goals. Wagenaar and Wolfson (1993) found that coalition leaders from diverse cultural groups, especially those that reflect the community, are more successful in obtaining community buy-in for coalition activities.

  • Membership. Coalition membership includes a variety of organizations in the community that provide time or other resources to the coalition. Butterfoss (2007) noted that coalitions with a diverse membership of community gatekeepers, and professional and other grassroots organizations are most successful. A diverse membership brings a variety of perspectives from different sectors, backgrounds, and constituencies. Hays, Hays, Deville and Mulhall (2000) found that representation of a large number of community sectors was associated with achieving coalition outcomes. Diverse membership may create challenges for the coalition in the short-run (e.g., difficulty in obtaining consensus, divergent perspectives), but facilitates the achievement of community improvements in the long run (Easterling, 2003).

    Other membership factors that are associated with coalition effectiveness are the number of partners in the membership and the amount of time that member organizations can contribute to the coalition’s activities. There is an inverse relationship between the number of partners and the successful completion of activities: the more partners in the membership, the fewer activities successfully completed by the coalition (Hasnain-Wynia et al., 2003). Additionally, coalitions with a dedicated staff (those who are wholly committed to working on the activities of the coalition) demonstrate more results than coalitions without their own staff (Wolff, 2001b).

    The expertise of the membership can also affect the success of the coalition. Coalitions benefit from having staff members with experience in community planning and organization, as they understand what is required to engage the community and conduct activities that meet the community’s needs (Butterfoss, 2007; Wolff, 2001b). Finally, the commitment of the membership to the coalition and its activities facilitates coalition effectiveness (Butterfoss, 2007). Research shows member satisfaction is associated with coalition effectiveness, as satisfied members are more invested in the coalition and its activities (Kumpher, Turner, Hopkins, & Librett, 1993).

  • Structure. Structural characteristics refer to the administrative rules in place that facilitate the management of the community coalition. The CCAT illustrates that coalitions with structures are more likely to achieve collaborative synergy in the coalition. Researchers have also identified the importance of coalition structures in predicting coalition progress. Butterfoss (2007) notes that the development of structure, rules, and responsibilities early in a coalition’s development enables community coalitions to operate effectively. Butterfoss (2007) indicates that structures can take the form of written policies and laws (e.g., memoranda of understanding, bylaws, and policy and procedure manuals). Bryson (1988) adds that coalitions benefit from developing clear mission statements. Another important structure is a steering committee or executive board that provides guidance and governance to coalition activities (Butterfoss, 2007). The steering committee or executive board, comprised of representatives of the member organizations, convenes regularly to assess the goals and activities of the coalition. Such structures facilitate collaboration, as they help members to more fully understand the purpose of the coalition and their individual roles and responsibilities.

  • Vision Guiding Action. Vision guiding action refers to the extent to which the membership agrees on the long-term goals of the coalition (vision) and is committed to pursuing activities (action) that will move the coalition toward this shared vision. Action based on a shared vision affects sustainability in the long-term because it reflects the commitment of the membership to achieve the goals of the coalition. Each coalition has a number of goals—from allocating resources and providing services to suggesting new policies. Vision helps coalition members to understand the future direction of the coalition, and to recognize the benefits of their participation (Foster-Fishman et al., 2001). A clear vision helps the coalition to raise awareness of its activities within the community, identify partners and resources, and reduce conflict within its membership.

  • Funding Diversity. Funding diversity is a key predictor of sustainability in community coalitions (Butterfoss, 2007), and in organizations more generally (Leviton et al., 2006; Rog et al., 2004). Programs are more likely to survive when they have political, financial, and institutional resources (Feinberg et al., 2008). Resources include money, people, goods, and services. Funds can be obtained from membership dues, the lead agency, community donations, financial partners, and in-kind contributions, grants, and contracts (Butterfoss, 2007). Diverse funding reduces the imbalance in power that occurs when a single funder is controlling the coalition’s budget. With multiple funding sources, the coalition can ensure that it is fulfilling its own goals, as well as funders’ requirements. Two studies in particular demonstrate that resource diversity contributes to sustainability in community coalitions. Leviton et al. (2006) found that the size of the budget (at least $25,000 per year) and funding diversity (resources from at least three different community organizations) were associated with program survival. In another study, Rog et al. (2004) found that the community coalitions with diverse funding portfolios were expanding into new areas or becoming institutionalized in the community. Rog et al. found that funding should be flexible enough to support the core activities of the collaborative—rather than earmarked for specific programmatic activities that are carried out by the collaborative (e.g., service delivery).

  • Sustainability Planning. Moving sustainability from a goal to a reality requires creating goals and objectives, developing and implementing sustainability strategies, and continuously evaluating those strategies (Shediac-Rizkallah & Bone, 1998). Developing a comprehensive sustainability plan at the outset is critical to a coalition’s success (Friedman & Wicklund, 2006). Program results, strategic funding, and staff involvement and integration are also related to planning early for sustainability (Mancini & Marek, 2004). Models of the sustainability of community-based programs and community health initiatives also identify sustainability planning as an important step to achieving program sustainability (Mancini and Marek, 2004; Beery et al., 2005).

  • Contextual Factors. Contextual factors may also affect the formation and effectiveness of community coalitions. Contextual factors are external conditions that either exist or are lacking in the environment, and thus can enhance or inhibit the coalition’s activities. Both the CCAT and Empowerment Theory highlight the impact of contextual factors, suggesting their importance in predicting coalition effectiveness. Specifically, Butterfoss, Lachance, and Orians (2006) found that contextual factors such as politics, the history of collaboration among member organizations in the coalition, geography, and community readiness can impact coalition formation. Population demographics, the cultural climate in the community, overall community attitude toward a particular issue, and precipitating events in the community are other contextual factors that may impact coalition effectiveness (Butterfoss, 2007).

These findings, along with the background information gathered around the functions, characteristics, and impacts of community coalitions, led to the development of a conceptual framework of sustainability in community coalitions. The purpose of this framework was to guide the specific design of this study using the HCAP coalitions, as well as provide a conceptual approach to the study of sustainability issues for community coalitions more broadly.

Defining Community Coalitions and Sustainability

The literature review demonstrated that coalitions have different memberships, patterns of formation, functions, goals, activities, and organizational structures. As such, it was important to clearly define the term "community coalition." A commonly used definition of community coalitions, developed by Feighery and Rogers (as cited in Butterfoss, 2007, p. 31), defines a community coalition as "a group of individuals representing diverse organizations, factions, or constituencies within the community who agree to work together to achieve a common goal." To add specificity to the conceptual framework, the Feighery and Rogers definition is expanded to define the number of organizations. Thus, for this study, a community coalition is defined as an alliance of three or more organizations who agree to come together to achieve a common goal. Adding the requirement of at least three organizations to the definition excludes direct partnerships between two entities from qualifying as a coalition, while ensuring the inclusion of coalitions of all sizes. This definition serves two purposes. First, by building on the well-accepted Feighery and Rogers definition, findings about the sustainability of community coalitions generated with this definition can be compared to other findings in the literature. Second, this definition is broad and will therefore be inclusive of community coalitions even if their form or function changed over time. Furthermore, it is necessary to define what is meant by the sustainability of a community coalition.

Post initial federal funding, some community coalitions continue to function exactly as they did previously—with the same membership, goals, activities, managerial structures, intensity of collaboration, community buy-in, and vision. Others are sustained with a different composition of members, although the coalition still continues to address its original goals. Some community coalitions have the same composition of members, but have scaled back their work by addressing only one (rather than all) of their original goals. Other community coalitions continue to evolve since they were initially federally funded, addressing their original goals and expanding to work toward new goals. Some coalitions adopt entirely new goals as a result of a shift in the economic or political environment or in response to a change in the community’s needs. In addition, some coalitions dissolve because of internal problems, or actively disband because they have found new homes for their activities within the community (e.g., institutionalization of the benefits within the community) or because they have achieved their original goals. A definition of sustainability in the context of community coalitions must recognize these different scenarios.

In the conceptual framework, a sustained community coalition is defined as an alliance of three or more organizations that is addressing one or more of the original goals of the coalition. There is an important distinction between the community coalition’s "goals" and its "activities." For example, the original goals of the HCAP community coalitions were connected to the vision of the coalition. Within HCAP, common goals were to increase insurance coverage and access to services for the uninsured and underserved; increase coordination and integration of services in the community; improve the quality of health care for the uninsured and underserved; and reduce the cost of care for the uninsured and underserved. The activities are the ways in which each coalition works toward its goals. Activities are unique to each coalition and may be refined over time to reflect the economy, funding priorities, population demographics, evaluation results, or other factors.

Definition of a Sustained Community Coalition
A sustained community coalition is an alliance of three or more organizations that is addressing one or more of the original goals of the coalition.

This model assumes that there will be membership turnover in the community coalition, and therefore, the alliance of three or more organizations does not need to be the same one that was part of the community coalition when it was initially federally funded. The decision to define a sustained coalition in terms of the presence of an alliance of three or more organizations that is addressing one or more of the original goals—rather than the continuation of the coalition’s activities post-initial federal funding—was based on the literature, and made in collaboration with ASPE.

Furthermore, of the sustained community coalitions (i.e., those that have satisfied both conditions), some may have been partially sustained. A coalition is considered partially sustained if it satisfies both conditions of sustainability but is not addressing all of its original goals. Similarly, some community coalitions may be continuing to work toward all of their original goals while also addressing a new goal or goals. This community coalition would be considered expanded. Community coalitions may also be partially sustained and expanded. These coalitions are partially sustained because they are addressing at least one of their original goals. However, they have also expanded because they have taken on at least one new goal. The new goal may or may not be synergistic to the original goals of the coalition. Rather, the new goal is reflective of the evolving needs of the community. Partially sustained and expanded coalitions have an important adaptive capacity, given that they have responded to community conditions over time.

Post initial federal funding, some community coalitions will not be sustained. The coalitions that do not have an alliance of three or more organizations, may have either dissolved because of a lack of resources, conflicts, or other reasons, or actively disbanded because they have achieved their original goal(s), and/or were no longer needed in the community. Additionally, in some cases, the coalition may have an alliance of three or more organizations that is no longer addressing at least one of the coalition’s original goals. This coalition is addressing a new goal, perhaps as a result of a shift in the economic or political environment or in response to a change in the community’s needs. Additionally, a coalition may address a new goal to meet the requirements of a new funder. Regardless of whether the coalition dissolved, actively disbanded, or is addressing a new goal to meet the needs of the community, the coalition is considered not sustained. Thus, even coalitions that have an active membership and/or were successful in institutionalizing the activities in the community may not necessarily be considered sustained.

In order to provide a foundation for the definition of a sustained community coalition, Exhibit 1 displays a sustainability decision tree that was used to guide our study.

Exhibit 1: Sustainability of the Community Coalition

Exhibit 1: Sustainability of the Community Coalition

Notes:

  1. The alliance of two or more organizations does notneed to be the same alliance that was part of the community coalition when it was initially federally funded.
  2. The original goals are those goals that the coalition was addressing when the coalition was initially federally funded. The coalition’s original goals should not be confused with its activities. The coalition’s activities are the ways in which the coalition works towards its goals. These activities may change over time to reflect the economy, funding priorities, population demographics, evaluation results, or other factors.

Coalitions that are "sustained" in Exhibit 1 are composed of an alliance of three or more organizations that are working toward one or more of the coalition’s original goals. However, these coalitions are not necessarily pursuing their original activities, i.e., the same activities that they did when they were initially federally funded. Activities are the ways in which each coalition addresses its goals, and may include programs or services, systems, policies, health behavior interventions, dissemination of products, and community capacity building. Therefore, upon determining whether the coalition itself has been sustained, it is necessary to explore whether the coalition has been able to sustain all, some, or none of its original activities. Given that coalitions evolve over time, it is possible that activities will also change to reflect the needs of the community or the requirements of a funder.

Exhibit 2 demonstrates that all, some, or none of the original activities of the coalition may have been sustained, regardless of whether the coalition itself has been sustained. Additionally, the coalition may take on new activities to reflect the economy, funding priorities, population demographics, evaluation results, or other factors. Below are three hypothetical cases of community coalitions whose activities have been sustained after their initial federal funding has ended. In the first case, all of a sustained coalition’s original activities have been sustained. In the second case, some of a sustained coalition’s original activities have been sustained. In the third case, none of a sustained coalition’s original activities have been sustained.

  • All of a sustained community coalition’s original activities are sustained. For example, suppose a community coalition’s original goal was the improvement of coordination and integration of services. The coalition decided to implement an electronic medical record (EMR) application at 15 different clinics in its service area. The coalition’s funds were used to conduct several activities: to integrate clinic messaging standards, train providers in the county clinics about how to use the EMR, and purchase some of the needed hardware for the rollout. After the initial federal funding ended, the coalition was able to continue all of these activities because it found a suitable benefactor to continue the project.

  • Some of a community coalition’s original activities are sustained. For example, suppose a community coalition’s original goal was the improvement of access to health care services for the uninsured. The coalition conducted a variety of activities to meet this goal when it was initially federally funded. First, the coalition expanded the network of providers in the community that would serve the uninsured at a reduced cost. Second, the coalition implemented patient navigation services to expand access to rural members of the community. Finally, the coalition disseminated health education materials throughout the community. After initial federal funding expired, the coalition has been sustained because it has an alliance of three or more organizations that continues to meet the original goal of improving access for the uninsured. However, after the initial funding ended, the coalition also had to discontinue several original activities because of budgetary constraints, and it now only focuses on expanding the network of providers that serve the uninsured.

  • None of a sustained community coalition’s original activities are sustained. For example, suppose a community coalition’s original goal was to increase access to primary care and prevention services. The coalition’s original activities were related to health education and community outreach. Post initial federal funding, the coalition was sustained because it received a large grant from a foundation. As part of this grant, the coalition conducted a needs assessment that found that transportation was the largest barrier to accessing services in the coalition’s catchment area. In response to this new information, the coalition discontinued its original activities, and conducted new activities that address transportation needs in rural areas.

In addition to these examples, there may be many other variations (e.g., a partially sustained coalition that has sustained some/all of its activities, an expanded coalition that sustained none of its original activities but conducts several new activities, a coalition that was not sustained even though some or all of its activities live on in the community, etc.).

Exhibit 2: Sustainability of the Community Coalition’s Activities

Exhibit 2: Sustainability of the Community Coalition’s Activities

Notes:

  1. The alliance of three or more organizations does notneed to be the same alliance that was part of the community coalition when it was initially federally funded.
  2. The original goals are those goals that the coalition was addressing when the coalition was initially federally funded. The coalition’s original goals should not be confused with its activities.
  3. The coalition’s original activities are the ways in which the coalition works towards its original goals. Coalitions may sustain some, all, or none of their original activities. Additionally, the coalition may take on new activities to reflect the economy, funding priorities, population demographics, evaluation results, or other factors.

A Conceptual Framework for Assessing Coalition Sustainability

With a working definition of sustainability in the context of community coalitions and their activities, it was possible to develop a conceptual framework for assessing the sustainability of community coalitions after their initial federal funding has expired. The framework in Exhibit 3 depicts the relationships between sustainability enabling characteristics, actions, intermediate outcomes, and long-term outcomes. The components of the conceptual framework are discussed in detail in the remainder of the section.

Exhibit 3: A Conceptual Framework for the Assessment of Community Coalition Sustainability

Exhibit 3: A Conceptual Framework for the Assessment of Community Coalition Sustainability

Components of this model were adapted from Beery et al., 2005.

Notes:

  1. Structure refers to the administrative rules in place that facilitate the management of the coalition
  2. For example, data sharing or integrated data systems, electronic medical records, financial management information systems, standardized registry systems, screening and eligibility systems
  3. Goals that the community coalition was addressing during the initial federal funding period
  4. Activities that the community coalition was conducting during the initial federal funding period
  5. For example, community newsletters, community bulletin boards or web sites, community hotlines

1. Enabling Characteristics of the Community Coalitions

The enabling characteristics of the community coalitions are those defining features that affect whether they will be sustained over time. While there are a number of characteristics that may affect sustainability, this conceptual framework includes strength of leadership, diversity of membership, structure, vision guiding action, strategic planning, resource stability and diversity, and evaluation. These characteristics were selected because they were identified in the literature as important facilitators of coalition effectiveness and/or sustainability. Additionally, the framework includes an "other" category in order to represent the array of additional characteristics that may affect sustainability.

2. Community Coalition Activities

The community coalition’s activities operationalize the original goals of the coalition and may include but are not limited to the following:

  • Programs/services (e.g., enrollment assistance in Medicaid/SCHIP, pharmacy assistance, coordination with social services, language/translation services, patient navigation)

  • Systems change activities (e.g., data sharing or integrated data systems, electronic medical records, financial management information systems, standardized registry systems)

  • Policy advocacy (e.g., collaborations with local or state government)

  • Health behavior change (e.g., prevention, healthy diet, screenings, health education)

  • Dissemination of products (e.g., community newsletters, community bulletin boards or web sites, community hotlines)

  • Community capacity building activities (e.g., providing technical assistance and training to other organizations, establishing networks of organizations, implementing community leader development programs, developing community resource guides)

As noted earlier, these activities may have changed over time, although the coalition’s goals remained the same. Activities may be refined over time to reflect the economy, funding priorities, population demographics, evaluation results, or other factors.

3. Sustainability Actions

Sustainability actions are those activities that the coalition undertakes to plan for the future of the coalition and its activities. Sustainability actions may begin during the coalition development phase, and continue throughout the life of the coalition, to ensure a seamless transition to new funding mechanisms. For example, coalitions attempt to identify and implement certain activities to sustain themselves, such as locating new homes for programs/services; identifying new funding streams from foundations or other organizations; and securing additional resources from their memberships. Coalitions may also choose to restructure their membership or modify processes for collaboration (e.g., meet less often or in a smaller core group). Sustainability actions can include working to identify ways to maintain the impacts of coalition activities—ranging from changes in individual behavior, community-level systems, and policies. A coalition’s sustainability actions may not necessarily involve developing a formal sustainability plan, but can include determining key priorities, and identifying the most effective activities to be sustained and strategies for sustaining them.

4. Sustainability of the Community Coalition

Based on the success of their sustainability actions, an intermediate outcome is the sustainability of the community coalition. The community coalition may or may not have been sustained after initial federal funding ended. The sustainability of the community coalition means that there is an alliance of three or more organizations that is addressing one or more of the original goals of the coalition. The alliance does not need to include the same membership organizations that were part of the coalition when it was initially federally funded. Of the coalitions that have been sustained, some may only be partially sustained. The coalition is partially sustained if it is not addressing all of its original goals. The community coalition is not sustained if the coalition has dissolved (i.e., lack of an alliance of three or more organizations) or disbanded (i.e., alliance of three or more organizations disbanded because the coalition is no longer needed in the community). Additionally, coalitions are "not sustained" if they have an alliance of three or more organizations, but are not addressing the original goals of the coalition.

5. Sustainability of the Community Coalition’s Original Activities

The sustainability of the community coalition’s original activities is another intermediate outcome. The coalition may sustain all, some, or none of the original activities that it conducted. A community coalition does not need to be sustained for activities to continue in the community.

6. Expansion of the Community Coalition

In addition to coalition sustainability, another intermediate outcome is the expansion of the community coalition. In this case, the community coalition is sustained and is addressing new goals, in addition to continuing to address one or more of its original goals. A coalition can also be partially sustained and expanded, meaning that it continues to address at least one of its original goals and has also taken on a new goal.

7. Long-Term Outcomes

Outcomes are the conditions to be achieved by the community coalition. The coalition may have long-term outcomes regardless whether the coalition itself has been sustained. The impacts are the cumulative effects of these outcomes at multiple levels in the community. For the purposes of this assessment, there are three types of impacts:

  • Individual impacts are changes in health or behavior at the person level. Some community coalitions may have focused on changing individual behavior—tobacco use, utilizing screening tests, healthy weight, and disease prevention.

  • Systems impacts are changes in infrastructure or capacity at the community level. This may include the implementation of new systems (e.g., data sharing or integrated data systems, electronic medical records, financial management information systems, standardized registry systems, screening and eligibility systems), clinics, and other infrastructure.

  • Policy impacts are changes in policy at the local, state, and federal levels.

8. Contextual Factors

Contextual factors are the external factors or conditions that exist within the environment that can enhance or inhibit sustainability. These factors permeate the conceptual framework at all levels. They can impact the coalition’s enabling characteristics, activities, sustainability actions, sustainability, expansion, and outcomes. Examples of contextual factors include: a seminal or precipitating event impacts the responsiveness of the community to the coalition’s activities (e.g., community rallies around the coalition’s goal to improve access to dental services because a local child died from complications associated with a tooth abscess); policies, laws, and regulations (e.g., reduced Medicaid reimbursement rates affect a provider’s ability to provide free services to the uninsured); the implementation of a new organization in the community (e.g., a sustained coalition expands its goals because it included a new community partner that brings a new perspective); and/or the priorities of a funder (e.g., a new funder indicates that resources can only be used for delivering services rather than for core coalition operations).

4. Methodology

This section presents the study design based on the conceptual framework presented in Section 3 and data collection techniques, analysis activities, and study limitations. All aspects of the study were reviewed and approved by NORC’s Institutional Review Board (FWA00000142) and received clearance from the Office of Management and Budget (OMB) under the Paperwork Reduction Act (OMB No. 0990-0368).

Study Design and Data Collection Techniques

The study design included four components that were subsequently carried out. The first phase consisted of the literature review and development of the conceptual framework presented in Section 3 (see also NORC, 2010). The next step involved a self-administered questionnaire sent to all HCAP grantees. Next, a subset of grantees that responded to the survey was chosen for interviews with key informants from the coalition. The final phase included site visits with a smaller subset of high performing, sustained coalitions and discussions with lead and partner organizations.

Self-administered questionnaire. The literature review and conceptual framework supported the development of a survey to assess the HCAP coalitions based on the sustainability definition and the factors driving sustainability included in the conceptual framework. The survey was administered to all 260 HCAP grantees between March 10 and May 31, 2011. Contact information for each coalition director was obtained using grantee records from HRSA and confirmed using web searches, the White Pages, and when necessary, confirmation emails or phone calls. A mailed, self-administered questionnaire was sent to the grantees, along with a cover letter describing the study, providing contact information, and explaining that their participation was voluntary. The cover letter also provided an email address and toll-free number if the designated respondent did not feel qualified to provide content. When a more knowledgeable contact was provided for a coalition, a new survey packet was mailed.

The questionnaire included a screening question to determine the coalition’s sustainability status, 66 questions for sustained coalitions, and 52 questions for not sustained coalitions. The survey included questions regarding the coalition’s structure, mission and goals, funding sources, activities, evaluation methods, sustainability plans, and overall impact. Non-responders received follow-up prompting via postcard, phone, and emails at two week intervals throughout the field period that included a link to a web-based version of the questionnaire. Exhibit 4 provides a breakdown of the responses received and the mode of completion. During the fielding period, three coalitions were identified as duplicate cases. These coalitions received a second grant, a continuation grant, or had merged with another HCAP coalition and responded as a single case. NORC achieved a 63 percent response rate. All analyses are conducted on complete cases only (n=165).

Exhibit 4: Coalition Survey Response by Mode of Administration
Sustainability of Community Coalitions Number Percent
Paper SAQ 65 25
Web SAQ 99 38
Phone Administration 1 0
Partial Complete 18 7
No Respondent with Knowledge of HCAP Grant 5 2
Coalition Disbanded and No Respondent with Knowledge of HCAP Grant 2 1
Nonresponders 67 26
Duplicate Cases 3 1
Total 260 100

The responding HCAP coalitions showed variability on a number of key characteristics. There was considerable variation in the size of the population served by the HCAP coalitions. The average size of the population served by the sustained coalitions was 5.7 million (median=380,800), with a range of [5,000, 300,000,000]. The average size of the population served by the not sustained coalitions was 7.4 million (median=307,896), with a range of [1,000, 50,000,000]. There was no statistically significant difference in the size of community served by sustained versus not sustained coalitions.

As shown in Exhibit 5, both sustained and not sustained coalitions served diverse populations in term of racial and ethnic composition, urbanicity, insurance status, and income. Additionally, sustained coalitions were significantly more likely than not sustained coalitions to serve urban (p<.01), White (p<.01), Hispanic/Latino (p<.10), Asian American/Pacific Islander (p<.01), Native American (p<.01), and mixed race respondents (p<.01). Not sustained coalitions were more likely to report serving underinsured/underserved populations (p<.05).

Exhibit 5: Characteristics of Community Served by Coalition
Demographics of Population Served by Coalition % Sustained Coalitions (n=112) % Not Sustained Coalitions (n=52)
*p<.10; **p<.05; ***p<.01, two-tailed difference of proportions test
Urban*** 81 52
Rural 66 63
Suburban 44 33
Uninsured 95 94
Underinsured/underserved** 89 98
White*** 92 79
African American 85 83
Hispanic/Latino* 89 79
Asian American/Pacific Islander*** 63 33
Native American*** 48 25
Mixed race or other racial or ethnic group (others included Latino mixed race, Hmong, immigrants, refugees, and Haitians)*** 55 27
Low-income 100 100
Middle-income 38 42
High-income 12 6
Other characteristics (others include undocumented workers, seniors, frontier communities, specific conditions like asthma) 15 12

Interviews with sustained and not sustained coalitions. From the pool of grantees that completed the survey (113 (68%) sustained and 52 (32%) not sustained), a subset of 25 (15%) coalitions including 16 sustained and 9 not sustained was selected to participate in telephone interviews. Telephone interviews with key informants of these coalitions were conducted in August and September 2011. The purpose of the interviews was to gather more detailed information and confirm survey responses regarding the coalitions’ experience and strategies for trying to sustain the coalition after federal funding ended, coalition outcomes, and future plans. The key informant interviews allowed for an evenly distributed sampling of cohort, region, urban versus rural, and sustained versus not sustained. HCAP cohorts I-VII were all represented and each cohort had grantees from different regions except those in cohort V where all coalitions were from the Northeast region. All cohorts also had a mix of sustained and not sustained grantees that were interviewed.

Site visits with sustained coalitions. From the group of key informant interviews, we invited six diverse coalitions (sustained and expanded) to participate in a site visit. These coalitions represented high-performing grantees from across the nation. Two grantees from each of the Western, Middle, and Eastern Regions of the United States were chosen. Two members from the NORC team traveled to each location to conduct site visits. All site visits occurred during October 2011. The site visits involved in-depth interviews with the coalition directors and lead staff and representatives from the coalition’s partner organizations. NORC staff facilitated the interviews using a semi-structured interview protocol that focused on facilitators and obstacles of sustainability, the coalition’s structure and dynamics, and lessons learned. Case study reports on the site visits are provided in Appendix A.

While valuable, findings from the key informant interviews and case studies should be considered from a qualitative frame of reference. Results are not generalizable to the entire population of HCAP coalitions. The key informant and case study data are integrated with the quantitative findings throughout the report. The qualitative data are most often utilized to provide additional in-depth explanation, context, and nuance to support the survey findings.

Data Analysis Activities

Survey data analyses focused on the calculation of descriptive statistics, such as frequencies, central tendencies, and distributions for all questions. Standard difference of means and difference of proportions tests were used for making statistical comparisons between sustained and not sustained coalitions. In cases where multiple items were used to assess a concept, the factor structure of the data was analyzed and an additive index or factor variable was created. Confirmatory factor analysis was used to provide evidence that the multiple items load together and are related to the latent concept. When the confirmatory factor analysis supported the latent concept, principal components analysis was used to extract an empirical indicator, using the Bartlett method. These indicators were then utilized as predictor variables in subsequent analyses (e.g., the influence of coalition leadership strength on sustainability). This procedure was used for the validated scales included in the questionnaire from past research as question ordering effects and the application of the scale to a slightly different population can jeopardize the statistical integrity of the scale.

Logistic regression models were employed to assess specific hypotheses derived from the conceptual framework (see Exhibit 3). Use of multivariate models allowed the explanatory power and fit of theories from the literature to be assessed for the HCAP population. Significance was determined through two-tailed tests and statistical significance is noted at the p<.10, .05, and .01 level. All references to significant differences refer to statistical significance. Analyses were conducted in STATA 10. Appendix B contains the results from the survey.

Qualitative data analysis of the 25 phone interviews was conducted using QSR NVivo 9 (NVivo). NVivo facilitated the identification of common themes across community coalitions in addition to major differences between sustained and disbanded coalitions. All notes from the interviews were uploaded into NVivo and sorted into folders based on the coalition statuses of Not Sustained, Sustained and Expanded, and Sustained and Not Expanded (see Exhibit 3). Based upon the topics addressed in the interview protocol, nodes (containers for categories and coding that represent concepts, processes, people, abstract ideas, or places, etc.) were developed to capture data from the transcribed interviews and subsequently coded (linking text to nodes). From this analysis, a list of key themes was developed for each topic area that emerged as particular to sustained coalitions and not sustained coalitions. Sub-nodes were made according to these themes to capture additional data specific to these themes.

Site visit data analysis began with the site visit teams engaging in a process of respondent validation to confirm the key information and themes emerging from the visit and the framing of these key themes in the site visit report. First, NORC reviewed the notes from each site visit, identified significant quotes and comments, and drafted the reports. The site visit teams conducted debrief conversations with each grantee director either at the end of the site visit or shortly thereafter by telephone to confirm the interpretation of key findings and ensure that important information was not omitted. Reports highlighted important facilitators for the coalition’s sustainability and lessons learned. Once internal reviews were completed, the draft reports were shared with the lead organization at each site for verification.

Results from the surveys, key-informant interviews, and site visits were analyzed to identify overarching themes and differences among sustained coalitions and not sustained coalitions and are integrated with the key survey findings throughout the report.

Study Limitations

The purpose of this study was to understand the impacts, outcomes, and strategies for sustainability post-federal funding. Although a complete census of HCAP coalitions was not achieved, the 63 percent response rate on the survey and high levels of cooperation for interviews and site visits produced a valid representation of the perspectives of the study respondents. However, it is unknown whether the respondents that completed the survey are a representative sample of the full population of HCAP grantees. Therefore, it is possible that non-response is related to the coalition-specific characteristics and may bias the analysis. Causes for non-response are unlikely to affect the findings related to the key research questions as the coalitions that did not complete the questionnaire were distributed across all grantee cohorts.2 Additionally, given that the coalitions were funded up to ten years ago through HCAP, it is possible that the staff who completed the survey did not work on the program originally. No efforts were made to validate the survey data through administrative records or public data. However, out of the 25 key informant interviews, only one coalition’s sustainability status needed to be corrected from their survey response of sustained to not sustained based on the interview discussion. If one of every 25 surveys were misclassified, roughly 6 coalitions would be misclassified. This is not expected to affect the analyses as characteristics and themes shared across sustained and shared across not sustained are being considered rather than individual coalition characteristics.

5. Findings

The following sections build on the literature review and conceptual framework of coalition sustainability by describing the most salient findings from the survey, key informant interviews, and site visits with the former HCAP community coalitions. We begin with a discussion of our findings related to the primary research question: How many coalitions funded under HCAP are still in existence? Next, we discuss the coalition characteristics associated with sustainability, including coalition membership, structure, vision, and funding. Finally, we present results examining how coalitions have evolved and coalition impacts and lessons learned by coalition staff over time. Case study reports from the site visits and topline survey results are presented as Appendices.

Coalition Sustainability Status: Sustained or Not Sustained?

Based on the survey results, NORC was able to classify the HCAP grantees according to their experiences sustaining their coalitions and activities. Exhibit 6 provides a breakdown of the sustainability of the community coalitions. The coalitions responded to questions about their goals; their responses were classified into categories according to the sustainability decision tree (Exhibit 1) that was first introduced in the conceptual framework.

Overall, 113 (68%) of the 163 responding coalitions were sustained. Nearly half (49%) of the sustained coalitions reported addressing all of their original goals from the HCAP grant. Of those 55, 33 (60%) expanded to add new goals while 22 (40%) did not add any additional goals. More than half (51%) of the sustained coalitions reported addressing at least one of their original goals from the HCAP grant. Of those 58, 43 (74%) expanded to add new goals and 15 (26%) did not add any new goals. None of the HCAP coalitions reported a coalition of three or more organizations that was not addressing at least one of its original goals.

In total, 76 (67%) of the 113 sustained coalitions are categorized as sustained and expanded, while 37 (33%) are categorized as sustained but not expanded (Exhibit 7, Boxes 9 and 11). Respondents provided reasons why they expanded to add one or more new goals. Of the 76 sustained coalitions that expanded, 13 percent reported expanding to attract new coalition members, 29 percent reported expanding to qualify for new funding sources, and 90 percent reported expanding to address new or additional needs of the community. Thirteen percent cited other reasons for expansion including expanding the coalition’s mission, aligning goals with the Affordable Care Act legislation, and leveraging and building upon the existing coalition to tackle new problems.

Fifty-two (32%) of the coalitions reported that there was no longer a coalition of three or more organizations, and as such these coalitions were considered to be not sustained. Of these, 35 (67%) of the coalitions dissolved due to issues such as funding or organizational problems and 17 (33%) disbanded because the coalition achieved all of its goals or was no longer needed in the community. The majority of the not sustained coalitions (61%) dissolved or disbanded between 2005 and 2007. Eighteen percent dissolved or disbanded between 2002 and 2005. Twenty one percent dissolved or disbanded between 2008 and 2010.

Exhibit 6: Sustainability of Community Coalitions
Sustainability of Community Coalitions Number Percent
All Responding Coalitions (Exhibit 7, Box 1) 165 100
Coalition of 3 or More Organizations (Box 2) 113 68
Overall, Not Sustained (Box 15) 52 32
Not Sustained and Dissolved (Box 12) 35 67
Not Sustained and Disbanded (Box 13) 17 33
Is coalition addressing 1 or more original goal? (Exhibit 7, Box 3) 113 100
Yes, sustained (Box 4) 113 100
No, but Addressing New Goals (Box 14) 0 0
Is coalition addressing all original goals? (Exhibit 7, Box 5) 113 100
Yes, fully Sustained (Box 6) 55 49
No, partially Sustained (Box 7) 58 51
Is fully sustained coalition addressing new goal(s)? (Exhibit 7, Box 8) 55 100
Yes, expanded 33 60
No, not expanded 22 40
Is partially sustained coalition addressing new goal(s)? (Exhibit 7, Box 10) 58 100
Yes, expanded 43 74
No, not expanded 15 26
Sustained (Exhibit 7, Box 4) 113 100
Overall, Sustained and Expanded (Box 9) 76 67
Overall, Sustained but Not Expanded (Box 11) 37 33

Exhibit 7: Sustainability Decision Tree

Exhibit 7: Sustainability Decision Tree

Coalition Characteristics

As noted in the literature review, there are a number of coalition characteristics associated with coalition effectiveness. This section explores whether coalition characteristics identified in the literature review as potentially supporting coalition effectiveness — membership, structure, vision guiding action, and funding— may also be associated with coalition sustainability.

Coalition Membership

When responding to the survey, the former HCAP community coalitions reported on several items related to the size and composition of the coalition membership, including membership turnover and membership diversity. Sustained coalitions provided responses regarding the current membership characteristics while not sustained coalitions responded with information about the coalition at the time the coalition disbanded. No statistically significant differences emerged between sustained and not sustained coalitions in terms of the number of member organizations and the proportion of those members considered active in the coalition. Sustained coalitions are comprised of 40 member organizations, on average, with 79 percent of members considered active. Not sustained coalitions were comprised of 22 member organizations, on average, with 74 percent of members considered active. Membership size ranges from 3 to 500 members for sustained coalitions and was 2 to 125 for not sustained coalitions at the time the coalition disbanded.

Tracking Different Levels of Partnership in One Large Sustained Coalition
A sustained coalition in the Midwest has around seventy-five member organizations—sixty of which are active members. In order to manage its large size, the coalition developed a rubric to measure the varying levels of involvement from partner organizations. There were three levels of partnership under the rubric. A Level 1 partnership generally referred to a loose partnership consisting of a one-time, in-kind exchange, e.g. hosting a conference together. A Level 2 partnership involved a one-time exchange of resources that were not just in-kind, e.g. a one-time exchange of money. A Level 3 partnership suggested an ongoing exchange of resources, both financial and in-kind. When the coalition was formed, about sixty percent of members were Level 1 or Level 2 partners, and forty percent were Level 3 partners. Today, most of the coalition’s members are operating as Level 3 partners.

In discussing coalition membership, key informants emphasized the importance of coalition quality and cohesion—having the right people participating in the coalition is more important than having the most members. A key informant from a coalition that was not sustained reflected that:

If we had a smaller group to define what we want to do and work on…[we] could have brought the pertinent people right away, and worked towards [our] goal more quickly...You don’t need a lot of people. It’s better to have five than thirty if you can actually accomplish something.

For larger coalitions, members played a number of different roles. Some members were more active than others, and contributed by providing in-kind resources, financial resources, and frequently, some combination of both.

Membership turnover. Exhibit 8 shows the data on the frequency of and reasons for membership turnover in terms of organizations leaving the coalition or new organizations joining the coalition after the formation of the coalition, for both sustained and not sustained coalitions. As illustrated in Exhibit 8, there was a low turnover of coalition members. The modal category for both sustained and not sustained coalitions was less than once every two years. Less than 10 percent of sustained and not sustained coalitions reported the addition of new members on a quarterly basis or more often. There were no statistically significant differences between sustained and not sustained coalitions on this measure.

Panel B shows the frequency of members leaving the coalition for sustained and not sustained coalitions. Overall, the loss of members was infrequent with over 80 percent of coalitions reporting the loss of a member no more than once every two years. On this measure of turnover, however, there are significant differences between sustained and not sustained coalitions (p<.01). A majority of not sustained coalitions, 57 percent, reported never losing members compared to 35 percent of sustained coalitions. Sustained coalitions were more likely to report very minimal turnover with 49 percent reporting the loss of a member less than every two years compared to 24 percent of not sustained coalitions. One sustained coalition explained the importance of early buy-in for maintaining members. Within this coalition, members were engaged from the initial development of the program, which gave them a sense of ownership. Furthermore, member organizations, rather than specific staff members, "owned" a seat on the board. This meant the organization maintained membership in the coalition, even if there was staff turnover within the organization. A key informant from this coalition noted: "As we developed the bylaws…we defined [on the] board of directors, the organization owns the seat...They sit as an organization because you can have turnover but you have that organization committed."

Additionally, as shown in Panel C, the reasons for coalition turnover varied significantly by sustainability status (p<.05) with sustained coalitions more likely to say turnover was related to issues with the member organization rather than the coalition. Thirty four percent of sustained coalitions cited changes within the member organization as the most frequent reason for turnover compared to 20 percent of not sustained coalitions. Coalitions that were not sustained cited coalition funding and a reassessment of membership needs and priorities slightly more often than sustained coalitions. Furthermore, as noted in Panels A and B, not sustained coalitions were more likely to say there was no turnover (33% compared to 25% of sustained coalitions).

Exhibit 8: Coalition Member Turnover>

Panel A: Frequency Members Join Coalition

Panel A: Frequency Members Join Coalition

Fisher’s Exact p<.01

Exhibit 8: Coalition Member Turnover: Panel A: Frequency Members Join Coalition
Frequency of New Members Joining % Sustained Coalitions (n=113) % Not Sustained Coalitions (n=52)
Never 8 21
Less than once every two years 41 31
About once a year 28 25
About once every six months 12 12
About once every three months 4 6
More than once every three months 6 6

Panel B: Frequency Members Leave Coalition

Panel B: Frequency Members Leave Coalition

Fisher’s Exact p<.01

Exhibit 8: Coalition Member Turnover: Panel B: Frequency Members Leave Coalition
Frequency of Losing Members % Sustained Coalitions (n=113) % Not Sustained Coalitions (n=51)
Never 35 57
Less than once every two years 49 24
About once a year 12 10
About once every six months 4 8
About once every three months 1 0
More than once every three months 0  

Panel C: Reason for Turnover

Panel C: Reason for Turnover

Fisher’s Exact p<.05

Exhibit 8: Coalition Member Turnover: Panel C: Research for Turnover
Reason for Member Turnover % Sustained Coalitions (n=113) % Not Sustained Coalitions (n=51)
Changes in coalition funding 8 12
Shifts in projects and activities 18 18
Reassessment membership needs and priorities 8 12
Changes within the member organization 34 20
Other 8 6
Not applicable, no turnover 25 33

Membership diversity. Coalitions were asked to note the types of organizations and sectors represented in the coalition membership. The options included a list of 26 different types of health organizations, as well as seven other sectors (government, social services, education, faith, business, foundations, and other). Overall, there was a great deal of diversity among both sustained and not sustained coalitions, as shown in Exhibit 9. Sustained coalitions showed signficantly more diversity within the health sector than not sustained coalitions (p<.05). The average number of health organization types was 9.9 for sustained coalitions with a range [1, 21] compared to an average of 7.9 for not sustained coalitions with a range [1, 24]. Sustained coalitions were signicantly more likely to have members representing several types of health organizations including public or private practices and providers; medical or dental associations; specialty care providers; school based health centers; migrant health centers; managed care organizations; Medicaid programs; laboratories; long-term care providers; and community based organizations. Key informants emphasized the need to engage diverse health providers from the community. A representative from one coalition said, "we definitely have to stay connected with the administration or leader of the local health facility (hospital or FQHC), health department, etc." Another said, "The hospitals need to be [involved], the county needs to be [involved],..[we] need to get funders and providers involved."

Exhibit 9:Types of Organizations and Sectors Represented in HCAP Coalitions
Types of Sectors and Organizations Members Represent Number Sustained Coalitions % Sustained Coalitions Number Not Sustained Coalitions % Not Sustained Coalitions
*p<.10; **p<.05; ***p<.01, two-tailed difference of proportions test
Health Sector Members Represent:
Federally Qualified Health Centers 100 90 45 88
Hospitals with a low-income utilization rate greater than 25% 82 74 39 76
Community-based organizations* 79 71 29 57
Free clinics/other community health centers 76 70 31 61
Public or private health care providers/ practices** 70 64 24 47
Mental health programs/providers 64 58 23 45
Other hospitals* 62 56 21 41
Academic medical centers 56 51 25 49
Specialty care providers*** 49 45 9 18
Medicaid programs* 44 41 14 27
Oral health providers 41 38 15 29
Substance abuse programs 38 35 13 25
Medical/dental societies** 35 32 9 18
Pharmacies 35 32 12 24
School-based health centers** 34 31 8 16
Managed care organizations* 33 30 9 18
Area health education centers 31 29 17 33
Primary care associations 27 25 18 35
Rural health clinics 27 25 11 22
Home health providers 26 24 7 14
Laboratories* 23 21 5 10
Other health care coverage programs 21 19 8 16
Migrant health centers* 20 19 4 8
Long-term care providers* 16 15 3 9
Private insurance providers 16 15 6 12
Public housing primary care programs 12 11 5 10
Non-Health Sector Members Represent:
Government (e.g., local health department or elected officials) 86 77 40 78
Social services (e.g., juvenile justice programs or temporary housing assistance)* 55 50 18 35
Education (e.g., elementary schools or university public health programs) 48 44 18 35
Foundations (e.g., philanthropic organizations) 44 40 15 29
Faith (e.g., churches or faith-based organizations) 42 39 17 33
Business (e.g., chambers of commerce or local nonprofits) 30 28 11 22
Other (including ambulance/ems, public housing, agriculture, individual community members, and interest/advocacy groups) 16 15 4 8

Looking across health and the seven other sectors in Exhibit 9, the average number of sectors represented in the membership of sustained coalitions was 3.9 with a range of 1 to 8. Similarly, the average number of sectors represented in the membership of not sustained coalitions was 3.4 with a range of 1 to 8 . The government sector was the only non-health sector represented in a majority of the coalition memberships by 77 percent of sustained coalitions and 78 percent of not sustained coalitions. Sustained coalitions (50%) were more likely to have members representing the social services sector than not sustained coalitions (35%, p<.10).

Diversity of Membership as a Facilitator of Sustainability
One sustained coalition that participated in a site visit explained that a major facilitator of their success and sustainability is the diversity of their membership. The two major hospitals, their respective clinics, and a local federally qualified health center were the major players, but the state department of mental health and addiction, the local homeless shelter, other specialists, laboratory and diagnostic service providers, and the local social services agency provided needed wrap-around services for the under/uninsured population served by the coalition.

Several other coalitions noted the importance of involving partners that are not historically connected to the cause. For example, a representative from a sustained coalition that participated in a site visit noted: "We definitely have to stay connected…we have typically partnered with community development organizations, which not only have manpower and resources that can be used, but [can also] connect you beyond those people who have a personal invesment in health care." Similarly, a key informant from a not sustained coalition said: "[You] can’t sell this program to mission folk, social service, mental health folk. They get it, what you have to do is convince the finance and provider folks. If you got them on board, you’re good to go."

While membership diversity helped some coalitions to expand services available to the target population, other coalitions experienced conflicts resulting from this diversity. A representative from one coalition that was not sustained commented on this fragmentation:

Members were some businesses, insurance-oriented people, and there was representation from faith-based, public health, health care….but I think each of us perceived our roles in the coalition with our organizations’ hats on. We did not perceive the activity as…’how we could make the community better?’ We all perceived our role as…’how can we make [our] organization better?’

Other coalitions did not experience this fragmentation, and found that their member organizations were able to come together, and put their own agendas aside, to discuss issues that affect the community. For example, one sustained coalition commented: "[The coalition] is recongized as an organization that is totally neutral—all of the people on our board are really competitors, but they do not sit in a competitive mode when they are in the [coalition] mindset."

Coalition Structure

In addition to discussing their membership, the coalitions answered several questions in the survey about their structure, including the formality of the leadership structure and relationships with members. This section discusses findings related to coalition lead organizations, leadership committees, membership structures, coalition leader experience, and coalition strength.

Lead organization. Seventy three percent of sustained coalitions reported having a lead organization, which is significantly less than the 92 percent of not sustained coalitions. This difference is not surprising as some of the sustained coalitions have likely moved away from the lead organization requirement under HCAP or may not think of the coalition structure in those terms anymore. As shown in Exhibit 10, not sustained coalitions were more likely to have a Federally Qualified Health Center (FQHC) as the lead organization, while sustained coalitions were more likely to be led by a community based organization, government agency, or other type of health sector organization including public hospitals, private hospitals, and other coverage programs. Only 12 (15%) of sustained coalitions with a lead organization have changed lead organizations since the end of the HCAP grant.

Exhibit 10: Type of Lead Organization

Exhibit 10: Type of Lead Organization

Exhibit 10: Type of Lead Organization
Type of Lead Organization % Sustained with a Lead Organization (n=82) % Not Sustained with a Lead Organization (n=46)
Academic medical centers 6 7
Community-based organizations 13 9
Federally Qualified Health Centers 10 30
Government (e.g., local health department) 17 13
Foundations 2 4
Free clinics/other community health centers 2 0
Hospitals (low-income utilization > 25%) 5 9
Medical Associations (Primary Care, Dental, Other) 5 2
Non-Health Sector (Business, Education, Faith) 4 2
Other Health 35 24

Leadership committees. Coalitions were asked to provide information on the types of leadership committees established as part of their coalition operations. The coalitions could select any or all of the following: (1) Steering Committee (a committee made up of representatives from member organizations that works with the consortia leadership); (2) Board of Directors (a group of individuals external to the consortia that provides input and/or oversight to the consortia leadership); (3) Executive Committee (a small group of consortia leadership responsible for consortia operations); and (4) Community Advisory Board (a group of lay-persons from the community that provides input on consortia activities and direction). Sustained coalitions were more likely than not sustained coalitions to have a board of directors (60% compared to 26%, p<.01) or Executive Committee (59% compared to 43%, p<.10). Conversely, not sustained coalitions were significantly more likely than sustained coalitions to have a Steering Committee (65% compared to 41%, p<.01). One-third of sustained coalitions (33%) and less than one-third of not sustained coalitions (30%) reported having a Community Advisory Board.

The findings from the key informant interviews also validated the survey responses. Sustained coalitions discussed the importance of having a board of directors. One coalition said we "have doctors, universities, hospitals, etc. on the Board [of Directors]; that has been key to our sustainability." This coalition created an online system that reports hospital utilization information across the state for health planning councils, providers and policy makers. The providers on their Board recognize the importance of the information and therefore have been active in support the sustainability of this activity.

Another sustained coalition says that the coalition needs a "strong Board to help provide vision and qualified and competent staff; [and] establish key partnerships and involve key leadership in order to be successful." A sustained coalition with a Community Advisory Board referred to the Board [of Directors] as its "touchstone," and indicated that it "do[es] not start something new without getting permission from the Board;" it believes that the Board has had a real role in "grounding" the coalition.

Membership organization and structures. There was no significant difference in the proportion of sustained coalitions (57%) and not sustained coalitions (48%) reporting a formal membership structure (e.g., legally incorporated or Memoranda of Understanding (MOU)). Of those sustained coalitions with a formal membership structure (n=71), 92 percent had MOUs or inter-agency agreements (IAA) with at least some of their members. Similarly, of those not sustained coalitions with a formal membership structure (n=27), 89 percent had MOUs or IAAs with at least some of their members.

In addition to the formality of member relationships with the coalition, a common type of organizational structure was discovered during the site visits. Four of the six site visited coalitions were organized in a "hub and spoke model." In this model, the core coalition staff serve as the hub coordinating all operational activities for the coalition as opposed to a more traditional model. As the hub, they disseminate information out to the members, as well as receive information, and coordinate and manage the day-to-day operational, programmatic, and service delivery activities and decisions. These four coalitions had varying levels of direct, member-to-member activity, though at the minimum all four had mechanisms in place for periodic inter-member discussions and any major opportunities and challenges were discussed and resolved as a group. One coalition with this "hub and spoke model" considered the structure a key to their sustainability because it contributes to efficient, effective, and sustainable operations. Program staff noted that the coalition’s hub and spoke structure was a very purposeful design intended to keep the operations centralized and efficient. Furthermore, the members who are primarily partner and referral organizations clearly valued the centralized nature of the operations and appreciated the efficiency and effectiveness of direct communication with the core coalition staff. During a discussion of how members connect their patients and clients to coalition services, one representative from a member organization noted, "From my end, it’s pretty seamless; I just make a phone call."

Sustained coalitions were significantly more likely to have formal processes in place for managing conflict between members and coalition leadership than not sustained coalitions (42% compared to 23%, p<.05). One not sustained coalition subsequently instituted a more formal structure during the post-HCAP period because of the lessons learned from the HCAP grant: "The alliance is comprised of lots of the same partners from HCAP and many others, [but] the alliance is more formal, it has by-laws." However, 75 percent of sustained coalitions and 75 percent of not sustained coalitions reported the use of formal decision making processes for the coalition. As shown in Exhibit 11, not sustained coalitions were slightly more likely to hold more frequent member meetings (p<.10).

Exhibit 11: Frequency of Formal Coalition Meetings
Frequency of Formal Member Meetings Every Month or More Every 1-3 Months Every 4-6 Months Once a year or less Never
% Sustained (n=112) 38 34 11 13 4
% Not Sustained (n=51) 55 35 6 2 2

In key informant interviews, both sustained and not sustained coalitions acknowledged that making coalition meetings worthwhile and valuable—not the quantity of meetings—contributes to sustainability. For example, a key informant from a sustained coalition shared the importance of using members’ time wisely in preventing member attrition:

I…needed to make sure at every Board meeting I had the data that the city and the county wanted, the anecdotal information that the other organizations [wanted]...if I left that out they were less likely to come back.

A representative from a member organization that participates in a sustained coalition attributed members’ continued commitment to the high quality of the coalition meetings. The key informant from the member organization explained that the coalition meetings are "very well-run," "organized," and that the "presentations are informational," which engages participants. The opposite is true as well, as a key informant from a not sustained coalition indicated that their meetings were cumbersome, and that "reducing meeting time so it’s not too labor intensive" would have helped the coalition to sustain itself.

Coalition leader experience. Sustained and not sustained coalitions provided information on the years of experience of the coalition leader in directing/managing the coalition, working in the field of health care delivery or health care administration, and living or working in the community served by the coalition. Sustained coalitions were asked, "How long has the coalition leader or director…" and not sustained coalitions were asked, "How long had the coalition leader or director…" Therefore, by definition, sustained coalitions, on average, report leaders with greater years’ experience and the data should not be compared without controlling for the years of coalition operation from the baseline of the receiving the HCAP grant. As can be seen in Exhibit 12, sustained coalitions’ leaders had more experience than not sustained coalitions’ leaders. For example, 54% of sustained coalitions reported their leader having 5 or more years of experience directing or managing the coalition; whereas only 14% of not sustained coalitions reported that length of experience.

Exhibit 12: Coalition Leadership Experience
  % Sustained Coalitions % Not Sustained Coalitions
How long has the coalition leader or director: 1 year or less Between 1 year and 3 years 3-5 years 5-10 years More than 10 years No Opinion/Not Applicable 1 year or less Between 1 year and 3 years 3-5 years 5-10 years More than 10 years No Opinion/
directed or managed the coalition? 8 16 16 39 15 5 10 45 29 10 4 2
worked in the field of health care delivery or health care administration? 1 2 5 15 68 9 6 12 8 24 47 4
lived or worked in the community served by the coalition? 3 0 4 14 71 8 6 14 12 14 53 2

Coalition strength. The coalitions responded to a series of agree or disagree statements about the strength of the coalition structure using a four point scale ranging from (1) strongly disagree to (4) strongly agree, with a fifth no opinion/not applicable option. An exploratory factor analysis revealed three underlying concepts: the strength of the coalition structure, the level of member organization involvement, and the strength of the coalition’s leadership. No statistically significant differences exist between sustained and not sustained coalitions for these three concepts using either an additive scale or the mean for each concept.

In addition to these survey questions, key informants from the interviews and site visits discussed the importance of coalition strength for being able to absorb the influence of contextual factors beyond the control of the coalition. The sustained coalitions provided information on the key contextual factors that affected their coalitions in both positive and negative ways during the post-HCAP period. With the exception of a ubiquitous impact of the economic downturn, which has increased demand for their services but reduced resources like funding donated supplies and services, the contextual factors mentioned were idiosyncratic across sites (e.g., particular changes to Medicaid eligibility in the state, or, mergers between two local hospitals). One theme that emerged across these particular contextual factors is that sustained coalitions are resilient and are able to absorb these exogenous events. To the extent possible, sustained coalitions planned for these events. For example, the two major hospitals, both coalition members, in one coalition’s community were going through a merger and the coalition took a very proactive position in advocating for the coalition to ensure a continued home for the coalition after restructuring. Additionally, sustained coalitions did their best to capitalize on positive contextual factors. For example, one coalition was positioning itself within their state’s discussion of Primary Care Medical Home models and programs established through ACA. The coalition was using its experiences with medical homes to contribute to the discussion, as well as looking for new opportunities for the coalition to engage in future programs.

Vision Guiding Action

Sustained coalitions were asked about their overall agreement with several statements to assess vision guiding action, shown in Exhibit 13. Over 85 percent of the sustained coalitions agreed or strongly agreed with all statements, except for the statement that "vision, mission, and goals are understood by residents and institutions in your community" to which 72 percent of the sustained coalitions agreed with the" Additionally, seven percent responded "not applicable" to that item compared to one to three percent for all other statements. The high level of agreement with the survey questions was supported by several statements from coalition leaders and partners during the key informant interviews and site visits.

Vision Guiding Action in a Sustained Coalition
A sustained coalition that exemplifies the concept of vision guiding action reported that their partners are highly motivated to accomplish the coalition’s mission of improving health care for underserved populations. The coalition was comprised of a range of social service agencies, community organizations, and hospitals systems that traditionally did not collaborate. As a result of their commitment to improving health care in their community, the partners developed a sophisticated network of charity care for uninsured patients. Prior to the HCAP grant, a culture of donated care existed in the community; many hospitals and providers were providing free care to patients. The HCAP grant was an opportunity to develop a coalition that focused on the issue of improving care for the underserved and uninsured in their community, and formalize the existing network of donated services. Through its case management and access activities, the coalition has created a coordinated system of care for uninsured patients, and as a result, decreased duplicative and fragmented care. An employee of a hospital partner in the coalition summarized the power of their mission-driven focus: "[There is] no fighting, because the bottom line is we’re all here for the same purpose. The patient is first and we try to do what we can."

Sustained coalitions frequently discussed the importance of having a vision, mission, and goals that are shared by member organizations. During a site visit with a sustained coalition, the coalition leader explained that a key to their sustainability has been that members share a common purpose "to serve the patients and [create] a continuum of care." A member organization of a sustained coalition also pointed out that they continue to participate in the coalition because they are committed to pursuing activities to achieve the coalition’s vision, mission, and goals: "It is [a] commitment the Board and CEO have made to the community."

Another sustained coalition emphasized that their mission has been an important facilitator of their outcomes as well as their sustainability. A key informant explained that "outcomes in health is the bottom line, but you have to go through the process of keeping the partnership together…to [get to] the outcomes." This coalition is located in a county that is geographically isolated, increasing the need for collaboration within the community. One key informant said, "I don’t think that the federal government or the state is going to come and solve our problems…We have been the ones to solve our own problems."

Additionally, several coalitions noted that the coalition must have a "mission driven reputation" that resonates with the community and target population. For example, a key informant from a sustained coalition said that "we have….a reputation now for being a go-to place for people who have no other means of accessing healthcare." Another sustained coalition shared that it has had such an impact that when a discussion of cutting the coalition’s program arose, the community rallied to "support the fact that having kids insured is saving [the] county a lot of money."

Exhibit 13: Sustained Coalitions Vision Guiding Action
Your coalition’s: % Strongly Disagree % Disagree % Agree % Strongly Agree % No Opinion/Not Applicable
vision, mission, and goals are documented. (n=112) 2 4 28 65 1
vision, mission, and goals take into account what is happening in the community. (n=113) 3 2 24 71 1
member organizations agree on the coalition’s vision, mission, and goals. (n=113) 2 2 31 64 2
member organizations are committed to pursuing activities to achieve the coalition’s vision, mission, and goals. (n=112) 2 4 41 52 2
vision, mission, and goals are understood by residents and institutions in your community. (n=113) 4 17 54 18 7
vision, mission, and goals are periodically re-evaluated or updated. (n=113) 5 8 40 45 2
activities are evaluated in relation to its vision, mission, and goals. (n=113) 4 8 38 48 3

Funding and Resources

Funding and resources are critical facilitators of sustainability in community coalitions. This section discusses several significant differences between sustained and not sustained coalitions in terms of their funding and resource structures. Next, we present findings on the coalitions’ financial resources, the stability and flexibility of coalition funding, diversity of funding and in-kind resources, and trends related to full-time equivalent coalition staff.

Financial resources. In terms of sources of financial resources, as shown in Exhibit 14, local health departments and foundations were the most commonly cited financial sources for both sustained and not sustained coalitions. Faith based organizations were the least common sources, cited by less than 20 percent of sustained and not sustained coalitions. Sustained coalitions were significantly more likely to receive resources from state level agencies and departments (other than the health department), local health departments, foundations, and community-based organizations. In some cases, sustained coalitions named their state senators as being critical during the end of HCAP funding in getting earmarks for the coalition. Other sources of funding mentioned by coalitions included direct support from member organizations, local donations, and support from local hospitals. The mean number of financial sources for sustained coalitions was 3.9, which is significantly higher (p<.05) than the mean for not sustained coalitions of 3.0.

Exhibit 14: Financial Sources

src=

*p<.10; **p<.05, two-tailed difference of proportions test

Exhibit 14: Financial Sources
Financial Source % Sustained % Not Sustained
*Indicates p<0.10, two-tailed difference of proportions test
**Indicates p<0.05, two tailed difference of proportions test
State Health Dept. 30 18
Other State Agency/Dept* 30 16
Local Health Department* 49 32
Other Local Agency/Dept 31 29
United Way 25 16
Foundations** 57 37
Health Insurers/ Managed Care Organizations 25 19
Businesses 28 24
Univerisities/ Academic Institutions 26 22
Community Based Organizations* 34 20
Faith Based Organizations 16 18
Other 32 27

Twenty-four percent of sustained coalitions were still receiving financial resources from HHS.3 For example, one coalition that participated in a site visit was able to secure funds from two different HHS agencies post-HCAP, allowing them to continue all of their HCAP programming. This coalition received funding from the Office on Women’s Health to continue the behavioral health aspect of their program, and HRSA to develop a comprehensive electronic health network (EHR) that builds on their HCAP work. Several of these sustained coalitions noted that their federal funding has helped them to continue activities related to eligibility or enrollment into entitlement programs.

Stability of funding. The sustained and not sustained coalitions varied significantly (p<.05) in terms of the stability of their funding. Forty-four percent of sustained coalitions and 54 percent of not sustained coalitions said that in general their funding was stable from year to year. However, 32 percent of sustained coalitions report funding that fluctuated year to year compared to 8 percent of not sustained coalitions. Thirty five percent of not sustained coalitions reported funding that decreased over time compared to 14 percent of sustained coalitions. Only 10 percent of sustained coalitions and 2 percent of not sustained coalitions reported funding that generally increased over time.

Flexibility of funding. Sustained and not sustained coalitions reported similar levels of flexibility for how they can allocate financial resources. On average, sustained coalitions had 50 percent of their funds dedicated to programmatic activities, with 19 percent dedicated to funding coalition operations, and 31 percent of funds that could be used for both. Similarly, on average, 48 percent of not sustained coalitions’ funds were dedicated to programmatic activities, 23 percent to operations, and 30 percent that could be used for both.

Diversity in funding sources. Sustained coalitions reported on the current characteristics of their funding situation compared to the HCAP period, as shown in Exhibit 15. For almost all categories, sustained coalitions reported having less than during the HCAP grant. The one exception being diversity of funding, in which 50% of sustained coalitions reported having more diverse funding now than they did during the HCAP grant.

Exhibit 15: Sustained Coalitions' Current Funding Characteristics Compared to HCAP Period
  Less than during HCAP grant Same as during HCAP grant More than during HCAP grant
Diversity of funding sources (n=111) 38% 12% 50%
Funding earmarked for programs, services, or activities (n=110) 43% 20% 37%
Funding for coalition operations (n=109) 58% 25% 17%
Flexibility to allocate funds wherever they are needed or can be most effective (n=109) 40% 28% 32%
Certainty about sources of future funding (n=110) 62% 20% 18%

Because of spending cuts, coalitions utilize creative solutions to securing funding. Rather than applying for large-scale grants, sustained coalitions aim to secure smaller grants or "funding opportunities that others may not be looking for." A leader of one sustained coalition that is focusing on building infrastructure in the community noted that their strategy has been to secure smaller grants: "We don’t go for the large grants very often because they are very competitive and a lot of funders don’t see us as being very important because we don’t provide services directly."

In-kind resources. There were few differences in terms of the types of in-kind support received by sustained and not sustained coalitions. As shown in Exhibit 16, facilities such as office space and equipment, and supplies such as computers or health educational materials were the most common type of in-kind support received by over two-thirds of sustained and not sustained coalitions. The least common types of in-kind support, received by less than 40 percent of sustained and not sustained coalitions, were evaluation support and fundraising and development support. Other types of in-kind services included donated pharmaceuticals, donated clinical services such as radiology, and data management and analysis support. Sustained coalitions received significantly more (p<.10) types of in-kind support with a mean of five types compared to a mean of four types for not sustained coalitions.

Exhibit 16: In-Kind Support

Exhibit 16: In-Kind Support

Exhibit 16: In-Kind Support
In-Kind Support % Sustained % Not Sustained
Facilities 77 82
Equipment and Supplies 71 67
Salary Sharing/Time Coverage 49 53
Volunteers 63 50
Administrative Staff or Services 60 57
Provider Staff or Services 55 48
Grant Writing 59 53
Fundraising/ Development 35 28
Evaluation Support 40 35
Other 19 10

Full-time equivalent staff. Additionally, in terms of full-time equivalent (FTE) staff, sustained and not sustained coalitions differed very little. The median number of FTEs for sustained coalitions was three with a range [0, 125]. Similarly, not sustained coalitions had a median of four FTEs, but a smaller range [0, 12]. Sustained coalitions were also asked if the size of the FTE staff changed since the end of the HCAP grant. Thirty one percent (n=34) reported "more staff now than during the HCAP grant;" fifty percent (n=55) had fewer staff; and 20 percent (n=22) report "the same number of FTE staff now as during the HCAP grant."

Modeling Sustainability as a Function of Coalition Membership, Structure, and Resources

In light of the many significant bivariate relationships between sustained and not sustained coalitions in terms of their membership, structure, and resources, a logistic regression model was utilized to understand the multivariate relationships between these variables and coalition sustainability. The results of this model are summarized in Exhibit 19 and the full table of regression coefficients can be found in Appendix C. The dependent variable is whether the coalition is sustained or not. Additionally, the predicted probabilities discussed below are included in Exhibit 17.

There are several factors that were significant predicators of sustainability even controlling for the other variables. The greater the experience of the coalition leader in the field and working in the community (controlling for the coalition’s age), the more likely a coalition was to be sustained. For example, holding all other variables at their mean or modal value, the predicted probability of sustainability for a coalition whose leader fell in the minimum experience response category is 48 percent. This probability increased significantly to 84 percent for a coalition whose leader fell in the maximum experience response category.

Exhibit 17: Modeling Sustainability as a Function of Coalition Membership, Structure and Resources
Factors Predicting Sustainability Type of Association
Leader Experience Positive Association
Proportion Active Members Positive Association
High Membership Turnover No Association
Number of Sectors in Membership No Association
Steering Committee Indicator Negative Association
Board of Directors Indicator Positive Association
Executive Committee Indicator No Association
Community Advisory Board Indicator No Association
Have Formal Structure with Members Indicator No Association
Number Funding Sources No Association
Number In-kind support Positive Association
Proportion Funding for Programs and Operations Positive Association

In terms of membership variables, neither a high membership turnover rate nor the diversity of the membership impacted sustainability. However, the proportion of the membership that was considered active did significantly increase the likelihood of sustainability.

For example, holding all other variables at their mean or modal value, the predicted probability of sustainability for a coalition with the minimum proportion of active members, 13 percent, was 48 percent. This probability increased significantly to 87 percent for a coalition with the maximum proportion of active members, 100 percent. There is no association between having formal structure with members (e.g., MOAs or IAAs) and sustainability. Lastly, the joint, or combined, influence of the membership variables (including proportion of active members, diversity of membership, and membership turnover) significantly improves the fit of the model over a model of sustainability that only includes leadership experience (likelihood ratio test with 3 d.f., p<.05).

Two measures of coalition structure had a significant effect on the likelihood of coalition sustainability in the model. Coalitions with a board of directors were more likely to be sustained, although coalitions with a steering committee were less likely to be sustained. As shown in Exhibit 18, the predicted probability of sustainability increased from 61percent for coalitions without a board of directors to 84 percent for coalitions with a board of directors. The predicted probability of sustainability decreased from 84 percent for coalitions with a steering committee to 65 percent for coalitions without a steering committee. There was no association between having an executive committee or community advisory board and sustainability status. Overall, the joint, or combined, influence of the structure variables significantly improved the fit of the model over a model of sustainability that only included leadership experience (likelihood ratio test with 5 d.f., p<.01).

Exhibit 18: Predicted Probabilities of Sustainability

Exhibit 18: Predicted Probabilities of Sustainability

Exhibit 18: Predicted Probabilities of Sustainability
Type Predicted Probability at Minimum Predicted Probability at Maximum
Leadership Experience 48 84
Proportion Active Members 48 87
Steering Committee (0/1) 84 65
Board of Directors (0/1) 61 84
Number of In-Kind Services 71 92
Proportion of Flexible Funding 84 93

Two measures of coalition funding and resources exerted a significant and positive influence on coalition sustainability. First, the more kinds of in-kind support a coalition received, the more likely it was to be sustained. For example, holding all other variables at their mean or modal value, the predicted probability of sustainability for a coalition with the minimum number of in-kind resources was 71 percent. This probability increased significantly to 92 percent for a coalition with the maximum number of in-kind resources. Second, coalitions with higher proportions of their funding that were flexible for either operations or programs had a greater likelihood of being sustained. Holding all other variables at their mean or modal value, the predicted probability of sustainability for a coalition with the minimum proportion of flexible funding was 84 percent. This probability increased significantly to 93 percent for a coalition with the maximum proportion of flexible funding. Overall, the joint, or combined, influence of the funding and resource variables did not significantly improve the fit of the model over a model of sustainability that only included leadership experience.

Evaluation Activities

Both sustained and not sustained coalitions reported a focus on several key evaluation activities, as shown in Exhibit 19. At least 70 percent of the sustained and not sustained coalitions agreed or strongly agreed that they developed evaluation plans for programmatic activities and carried out evaluations of their activities, programs, and services on a regular basis. Additionally, evaluations were used to demonstrate project effectiveness. More sustained (34%) and not sustained (24%) coalitions disagreed or strongly disagreed with a statement that evaluations of the coalition’s core operations were conducted regularly.

Similarly, as shown in Exhibit 20, there was a heavy emphasis for both sustained and not sustained coalitions on conducting evaluation activities in order to demonstrate results to funders and to compete for future resources, as well as for long-term planning and modifying programs and services. Fewer, though still over 70 percent, sustained and not sustained coalitions reported that evaluation activities were important for modifying coalition operations. In key informant interviews, numerous sustained coalitions shared that data collected through evaluation activities demonstrated that programs launched as part of HCAP were effective in helping to reduce hospital admissions and hospital inpatient visits, and saved hospitals money as a result.

Exhibit 19: Focus on Evaluation
Strongly Disagree Disagree Agree Strongly Agree No Opinion/Not Applicable
Your coalition develops evaluation plans prior to implementing programs, services, and activities.
Sustained Coalitions (n=113) 0% 18% 50% 27% 4%
Not Sustained Coalitions (n=51) 2% 16% 41% 37% 4%
Evaluations of your coalition’s core operations (e.g., community communications) are conducted on a regular basis.
Sustained Coalitions (n=113) 5% 29% 45% 12% 8%
Not Sustained Coalitions (n=50) 0% 24% 38% 34% 4%
Evaluations of your coalition’s programs, services, and activities are conducted on a regular basis.
Sustained Coalitions (n=113) 2% 9% 56% 30% 4%
Not Sustained Coalitions (n=50) 0% 8% 52% 38% 2%
Project effectiveness is demonstrated through evaluation.
Sustained Coalitions (n=113) 1% 15% 46% 31% 7%
Not Sustained Coalitions (n=51) 0% 14% 45% 39% 2%
Exhibit 20: Evaluation Purposes
  Very Unimportant Somewhat Unimportant Somewhat Important Very Important No Opinion/Not Applicable
Demonstrating results to your community
Sustained (n=113) 3% 7% 19% 67% 4%
Not Sustained (n=52) 4% 13% 23% 60% 0%
Demonstrating results to your funders
Sustained (n=113) 4% 1% 9% 81% 5%
Not Sustained (n=51) 4% 4% 16% 76% 0%
Competing for funding more successfully
Sustained (n=113) 5% 3% 17% 70% 5%
Not Sustained (n=51) 4% 4% 18% 71% 4%
Modifying coalition operations (e.g., staff, convening membership meetings)
Sustained (n=113) 5% 13% 33% 40% 8%
Not Sustained (n=51) 2% 18% 37% 41% 2%
Modifying coalition programs, services, and activities
Sustained (n=113) 4% 7% 29% 54% 6%
Not Sustained (n=51) 4% 10% 39% 45% 2%
Long-term/sustainability planning
Sustained (n=113) 3% 2% 14% 75% 6%
Not Sustained (n=50) 2% 10% 38% 48% 2%

The key informant interviews illustrated that coalitions collected data through evaluation to demonstrate program effectiveness to funders and stakeholders. One sustained coalition presented evaluation data to a hospital where the program was being implemented in order to show the hospital the benefits their patients are reaping for a minimal investment from the hospital. A key informant noted: "Even when they donate care, [the hospital gets] more out of it than they give, and we can prove that to everyone now." Another key informant from a sustained coalition said that:

Our sustainability planning is very data-driven. Right now, our strategy is to show hospitals how much money they are collecting directly as a result of the [coalition’s] services. The long-term strategy for sustainability is that hospitals have to support it, whether or not additional grant funding comes in or not.

One key informant from a sustained coalition noted that evaluation has facilitated the coalition’s expansion within the community: "We have been able to show, through data collected, that access to health care has a direct correlation to reducing visits to the hospital and inpatient visits. We have also been able to establish other relationships with other providers in the community." Internally, having data on coalition impact helped coalition members and staff to celebrate their successes. A key informant from a sustained coalition said: "We have to often pump each other up by looking at our anecdotal and data-driven success. The value of the work that we do is manifested in the people that we serve." When asked about the use of specific evaluation methodologies, a few significant differences emerged between sustained and not sustained coalitions. Sustained coalitions utilized outcome evaluations and quantitative methods at significantly higher rates than not sustained coalitions, as shown in Exhibit 21. Less than 10 percent of sustained and not sustained coalitions were utilizing newer methods including evaluation capacity building with member organizations and empowerment evaluation, or using experimental or quasi-experimental methods.

Exhibit 21: Evaluation Methods

Exhibit 21: Evaluation Methods

*p<.10; ***p<.01, two-tailed difference of proportions test

Coalition Activities

espondents provided information on the types of activities that the coalition has conducted since receiving the HCAP grant. Activities were defined as the projects, programs, products, and services the coalition worked on to serve the community.

Types of activities. Overall, a majority of sustained and not sustained coalitions conducted each type of activity. As shown in Exhibit 22, "Programs and Services" was the most commonly cited activity by both sustained (92%) and not sustained coalitions (86%). Sustained coalitions cited health behavior change activities least often (63%) while not sustained coalitions cited policy advocacy and change least often (51%). Sustained coalitions were significantly more likely to conduct several different types of activities compared to not sustained coalitions including systems change, policy and advocacy, and information dissemination.

Exhibit 22: Types of Coalition Activities
Activities Conducted Since HCAP Grant Received Sustained Coalitions (n=112) Not Sustained Coalitions (n=49)
*p<.10; **p<.05; ***p<.01, two-tailed difference of proportions test
Programs and services (e.g., enrollment assistance for entitlement programs or patient navigation) 92% 86%
Systems change (e.g., integrating data systems or pro bono provider systems)** 82% 65%
Health behavior change (e.g., wellness programs or training peer educators) 63% 53%
Capacity building (e.g., providing technical assistance to other organizations or community leader development programs) 71% 59%
Policy advocacy and change (e.g., informing local leaders and elected officials or collaborating with local institutions like school systems)*** 72% 51%
Dissemination of information and products (e.g., health fairs or community newsletters)* 79% 67%

Additionally, coalitions were asked to provide the three types of activities that made up the largest portion of their work during the HCAP grant, with sustained coalitions also being asked to provide the three types of activities that made up the largest portion of their work after the HCAP grant ended. When asked to narrow their activities down to the top three during the HCAP period, the responses of the sustained and not sustained coalitions were statistically equivalent, as shown by the blue and green bars in Exhibit 23. Among sustained coalitions, the coalitions were less likely to cite capacity building as a main activity after the HCAP grant (41% down from 50%) and more likely to cite policy and advocacy change activities (32%, up from 27%), as shown by the blue and red bars in Exhibit 23.

Exhibit 23: Activities that Made Up the Largest Portion of Coalitions’ HCAP Projects

Exhibit 23: Activities that Made Up the Largest Portion of Coalitions’ HCAP Projects

Exhibit 23: Activities that Made Up the Largest Portion of Coalitions' HCAP Projects
Main Activities Sustained Coalitions during HCAP Period (n=111) Sutained Coalitions after HCAP (n=110) Not Sustained Coalitions during HCAP Period (n=49)
Capacity Building 50 41 45
Dissemination of Information and Products 32 31 33
Health Behavior Change 32 36 33
Policy Advocacy and Change 27 32 35
Programs and Services 90 88 82
Systems Change 59 58 59

Of the 113 sustained coalitions who provided responses to the survey’s coalition activities sections, four (4%) reported that the coalition was not conducting any of the same activities it did during the HCAP grant. Fifty-nine of the coalitions (52%) reported conducting at least one of the original activities; 21 (19%) reported conducting all of the same activities; and 29 (26%) reported conducting all of the same activities plus new ones. All of the 92 sustained coalitions that have dropped or added activities since the HCAP grant period reported that the activities they conducted at the time of the survey were consistent with at least one of the coalition’s HCAP goals.

Changes in coalition activities. Sustained coalitions also reported on why they changed or added new activities. The three most common reasons for changing or adding new activities were qualifying for (or as a result of) new funding; reacting to changes in the demographics or needs of the target population; and expanding reach to a new population. An example of changing or adding new activities due to one of these reasons was seen when a sustained coalition realized that the mentally ill were not receiving adequate services, the coalition increased outreach and services provided. This activity expanded to helping the mentally ill find permanent housing when the coalition received funding through a grant specifically related to housing. Another coalition began targeting in-patients, realizing that enrolling them in a program for the under insured or uninsured was more effective. Additional reasons provided for adding or changing activities included funders’ priorities changing, evaluation results suggesting changes were needed, exogenous influences (e.g., Hurricane Ike), structural influences (e.g., all volunteer organization, so need to conduct activities people are willing to do), and more nuanced resource issues (e.g., decreases in funding resulted in scale-backs to all activities rather than eliminating a few activities). Some sustained coalitions described experiences of learning how to keep processes efficient and effective, thus changing some of the activites conducted under HCAP. For example, one coalition realized that a nursing component originally designed for the program was not valuable because those activites were already being conducted by the hospitals and clinics. As such, they made the programmatic decision to eliminate that activity, while continuing to meet the needs of their population in other ways.

Of the not sustained coalitions responding to the activities questions (51), ten (20%) reported that none of the coalition’s original activities were being conducted by the coalition/organization or another coalition/organization at the time of the survey. Twenty-six (51%) reported that at least one activity was still being conducted and 15 (29%) reported that all of the original activities were still being conducted. In some cases, some services that not sustained coalitions were providing under HCAP were transferred to another entity within the community. For example, a free care program launched by one not sustained coalition was absorbed by the hospital where it was being implemented. A key informant from another not sustained coalition said that the dollars it received through HCAP had a domino effect and "continue to bear fruit."

Coalition Sustainability Planning

Coalitions were asked several questions to understand how they defined sustainability, whether and how they planned for it, and who was responsible for sustainability.

Sustainability definition. In the survey, coalitions were asked to complete the following sentence about sustainability—"According to your coalition’s definition, sustainability of your coalition means"—with one of the following options:

  • Our coalition has the resources it needs to continue operating with our membership and structures in-tact for the long-term.
  • Our coalition’s programs, services, and activities will continue in the long-term even if our coalition is no longer in operation.
  • Both our coalition and its activities will continue in the long-term.
  • Our coalition has made a lasting impact on our community that will continue regardless of whether our coalition or its activities continue operating.
Institutionalizing the Coalition and Its Work

A sustained coalition that participated in a site visit discussed the importance of institutionalizing the coalition and its work in the community:

"If a partnership or a coalition is only as successful as the founding leader, then it has failed. I don’t think it’s as simple as succession planning, although that is an important piece of it. You have to institutionalize the idea of the coalition and the work of the coalition. You have to make it an integral part of the work that people are doing."

Overall, coalitions were more likely to include some aspect of sustaining activities or impacts in the definition of sustainability. Across both sustained and not sustained coalitions, only 16 percent selected the definition that framed sustainability only in terms of the membership and structure of the coalition. Sustained and not sustained coalitions differed significantly (p<.01) in terms of how they define sustainability, as shown in Exhibit 24. Sustained coalitions were more likely to define sustainability in terms of the long-term operation of the coalition itself, while not sustained coalitions tended to define sustainability in terms of the coalition’s activities and impacts. Further, twenty percent of sustained coalitions reported that sustainability meant that the coalition had the resources it needed to continue operating with the membership and structures in-tact for the long-term, compared to only six percent of not sustained coalitions.

To provide some context for this finding, some of the key informants from not sustained coalitions commented that they perceived sustainability as the continuation of their work in the community—even if these activities continue through another entity. For example, one key informant from a not sustained coalition noted, "We planned it so that we would make sure that the services would be provided even if we no longer existed—it was by design that we focused on the programs that we were implementing, instead of focusing on sustaining the coalition." In contrast, other respondents from sustained coalitions emphasized that in order to continue to provide services, their coalitions had to sustain. A key informant from a sustained coalition shared, "It is important to sustain the coalition, because the coalition allows us to help our patients to utilize the healthcare system as it should be utilized." Another key informant from a sustained coalition echoed this point, explaining that "someone needs to be here to coordinate services."

Forty percent of sustained coalitions reported that sustainability meant the continued operation of the coalition and activities, compared to 19 percent of not sustained coalitions. Conversely, forty percent of not sustained coalitions reported that sustainability meant that the coalition’s programs, services, and activities would continue in the long-term even if the coalition was no longer in operation, compared to 19 percent of sustained coalitions. Thirty five percent of not sustained coalitions reported that sustainability meant the coalition made a lasting impact on the community that would continue regardless of whether the coalition or its activities continued, compared to 22 percent of sustained coalitions.

Exhibit 24: Definition of Sustainability

Exhibit 24: Definition of Sustainability

Exhibit 24: Definition of Sustainability
  Sustained Coalitions (n=111) Not Sustained Coalitions (n=48)
Our coalition has the resources it needs to continue operating with our membership and structures in-tact for the long term. 20 6
Our coalition’s programs, services, and activities will continue in the long-term even if our coalition is no longer in operation. 19 40
Both our coalition and its activities will continue in the long-term. 40 19
Our coalition made a lasting impact that will continue regardless of whether our coalition or its activities continue operating. 22 35

Sustainability plan. The literature suggests the importance of developing sustainability goals and objectives in a comprehensive sustainability plan at the outset. The survey explored whether coalitions developed a sustainability plan and when. Only twenty two percent of all responding coalitions had a sustainability plan in place prior to receiving the HCAP grant. Another 34 percent developed a sustainability plan prior to the end of the HCAP grant. Nineteen percent developed a plan after the HCAP grant ended. Fifteen percent of all coalitions did not have a plan at all, but they had plans to develop one. The last 11 percent did not have a plan or intentions to develop one. As shown in Exhibit 25, there were no statistically significant differences between sustained coalitions and not sustained coalitions in terms of developing a sustainability plan.

One sustained coalition implemented a specific plan for sustainability at the outset, but used less formalized sustainability strategies as the coalition evolved. A key informant elaborated on the sustainability plan that the coalition implemented initially:

We wanted to make sure that we had a certain number of grant applications [that] went out the door per month, per year, for at least a 40 percent hit rate, and we got it. We saw it as a good exercise in bringing in resources. That was a clear sustainability plan for us in the beginning. Also, part of the sustainability plan was the processes of keeping elected officials informed, making sure everyone knew what we were doing— newsletters, electronic communication…

However, this coalition has not implemented a formal sustainability plan in the past five years, even though it continues to sustain itself and expand upon its activities. Another key informant from the same coalition explained that, in the beginning, "knowing that you have to sustain [yourself] hangs over your head," but as relationships among member organizations solidify and the coalition becomes more institutionalized within the community, the need for formalized strategies decreases.

Exhibit 25: Development of a Sustainability Plan
  % Sustained Coalitions (n=111) % Not Sustained Coalitions (n=50)
Yes, prior to receiving our HCAP grant. 21 24
Yes, within the first year of our HCAP grant. 10 16
Yes, after the first year but still prior to the end of our HCAP grant 23 20
Yes, after our HCAP grant ended. 23 10
No, but we have/had plans to develop a sustainability plan. 12 22
No, and we don’t have any plans to develop a sustainability plan. 12 8

A key lesson learned that emerged from key informant interviews with not sustained coalitions was the need to plan for sustainability earlier. Several key informants shared: "As soon as the grant was awarded, we should have developed a sustainability plan;" "It is a good idea to start planning for sustainability when you are writing your grant, when you start the project;" "Plan, and develop sustainability on the front-end;" and "Focus on sustainability early and look at the big picture early."

Sustained coalitions also echoed the need to plan for sustainability sooner. A key informant from a sustained coalition that is still conducting many of the same activities that it did during the HCAP program ended said: "We could have kept more people involved if we had planned in advance."

Although there were no differences between sustained and not sustained coalitions in terms of sustainability planning, there were significant differences in terms of the types of actions coalitions took to prepare for sustainability. These findings demonstrate that real actions—not just plans—contribute to coalition sustainability. As shown in Exhibit 26, sustained coalitions were significantly more likely to reassess the coalition’s goals, activities, or priorities; identify the most effective goals and activities to continue; develop a strategic plan for attaining resources; reorganize the coalition’s membership; hire an external consultant to assist with sustainability; and develop an infrastructure in the community to support systems-level activities. Among the sustained coalitions, the most frequent actions reported relate to identifying the most effective goals and activities (77%) and reassessing goals, activities, and priorities (75%). Additionally, a majority of sustained coalitions developed strategic plans for attaining resources (55%) and developed community infrastructure to support systems-level activities (51%).

Exhibit 26: Sustainability Actions
  % Sustained Coalitions (n=112) % Not Sustained Coalitions (n=49)
*p<.10; **p<.05; ***p<.01, two-tailed difference of proportions test
Reassessed the coalition’s goals, activities, or priorities*** 75% 39%
Identified the most effective goals and activities to continue*** 77% 55%
Developed a strategic plan for attaining resources** 55% 35%
Reduced the membership* 4% 12%
Reorganized the membership* 22% 10%
Restructured coalition operations/processes (e.g., fewer meetings, smaller leadership team) 37% 29%
Established a committee to strategically address sustainability issues 20% 18%
Hired an external consultant to advise the coalition on issues of sustainability** 21% 8%
Located partners and institutions to take over programs and services developed by the coalition 36% 37%
Developed an infrastructure in the community to support systems-level activities*** 51% 29%
Ensured appropriate mechanisms for implementation and enforcement of policy activities in the community 18% 8%
Other 8% 16%
None 4% 8%

Not sustained coalitions were significantly more likely than sustained coalitions to reduce the membership to prepare for sustainability, however only 12 percent of not sustained and four percent of sustained coalitions took this action. The only sustainability action adopted by a majority of not sustained coalitions (55%) was identifying the most effective goals and activities to continue. Only four percent of sustained coalitions and eight percent of not sustained coalitions took no sustainability actions at all.

During the key informant interviews and site visits, coalition leaders talked about their sustainability actions. Consistent with the survey findings, representatives from sustained coalitions commented on the importance of reassessing goals, activities, and priorities to meet the changing needs of their community. For example, a key informant from a sustained coalition commented on the need for behavioral health services in their rural community. During the HCAP program, this coalition focused on expanding access to behavioral health services in the community. However, due to a lack of funding, this activity was not sustained post-HCAP. Recognizing the need for this service, the coalition has worked with key entities in the community to identify new funding. Today, behavioral health services are available in the local hospital.

Another key informant for a sustained coalition talked about their sustainability actions at the beginning of the HCAP grant. While the coalition did not develop a formal sustainability plan, a reserve fund was created to support critical operations and activities. The key informant noted that the reserve fund helped the coalition to sustain its activities as the level of HCAP funding decreased each year (a feature of the program). To maximize their resources, the coalition also hired the minimum number of staff necessary conduct the program activities, with the intention of hiring additional staff over time, as needed.

Stakeholders involved in sustainability planning. Coalitions were asked to report on the different types of groups or individuals who were involved in coalition planning. As shown in Exhibit 27, the only significant differences between sustained and not sustained coalitions were for the different types of leadership groups.

As noted in the findings on coalition structure, sustained coalitions were more likely to have a board of directors or executive committee compared to a steering committee. Similarly, sustained coalitions were more likely to have the board of directors or executive committee involved in coalition planning.

Key informants from sustained and not sustained coalitions emphasized the role of stakeholders from individual member organizations in planning for sustainability. In many cases, champions from institutions such as health departments, community development organizations and local government had resources and networks that were important for sustainability. A key informant from a sustained coalition said:

"It is important to get local government involved—local department of health, county executive’s office. There is a real role for sustainability in those entities. If there is a way to have an organization to adopt it, nurture the collaboration—it goes a long way."

Exhibit 27: Groups Involved in Sustainability Planning

Exhibit 27: Groups Involved in Sustainability Planning

Exhibit 27: Groups Involved in Sustainability Planning
Groups % Sustained Coalitions (n=112) % Not Sustained Coalitions (n=50)
*** indicates p<.01, two-tailed difference of proportions test
Steering Committee 29 42
Board of Directors*** 45 18
Executive Committee*** 37 14
Community Advisory Board 12 18
Individual member organizations 40 30
Coalition staff 53 54
No one at the coalition 5 10
Other 7 6

Timing of sustainability planning. There were no significant differences between sustained and not sustained coalitions in terms of when sustainability planning took place, as shown in Exhibit 28. Across sustained and not sustained coalitions, 62 percent reported that sustainability issues were addressed in the course of regular meetings and planning activities, 24 percent reported dealing with sustainability issues toward the end of grant or funding cycles, 10 percent addressed sustainaiblity issues as they became a problem, and only 4 percent addressed them rarely or never.

Exhibit 28: Groups Involved in Sustainability Planning

Exhibit 28: Timing of Sustainability Planning

Exhibit 28: Timing of Sustainability Planning
Time % Sustained Coalitions (n=113) % Not Sustained Coalitions (n=50)
In the course of regular meetings and planning activities 65 54
As we neared the end of major grants and other funding 19 36
Only when sustainability became a problem 11 8
Rarely or never 5 2

Sustainability as a function of planning and action. These key sustainability planning and action variables are included in a logistic regression model summarized in Exhibit 29. The full regression results are presented in Appendix C. The results indicate that planning for sustainability does not increase the likelihood of sustainability, but the number of actual sustainable actions undertaken does increase the likelihood of sustaining the coalition. Controlling for the other variables in the model, there is no effect of developing a sustainability plan. Additionally, addressing sustainability issues with coalition members during regular meetings or near the end of funding cycles (compared to addressing sustainability issues through targeted attention), significantly decreases the likelihood of sustaining the coalition. However, the more sustainability actions the coalitions takes, the greater the likelihood of sustainability. Holding the other variables at their mean or modal category, the model predicted that a coalition that takes no sustainability actions had a 35 percent change of being sustained, compared to a coalition that undertook nine sustainability actions and had a 93 percent chance of being sustained.

Exhibit 29: Sustainability as a Function of Sustainability Planning and Action
Factors Predicting Sustainability Type of Association
Coalition has a sustainability plan No Association
Coalition addresses sustainability issues in regular business Negative Association
Number of sustainability actions taken Positive Association

In a second model, the individual sustainability actions were broken out to examine their individual effects on coalition sustainability. This model is summarized in Exhibit 30 and the full model is included in Appendix C. Again, the model showed no association of having a sustainability plan and a negative association of the coalition addressing sustainability issues during regular meetings or near the end of funding cycles, compared to more targeted attention on sustainability issues. In terms of the specific sustainability actions, establishing a committee for sustainability and reducing the coalition membership decreased the likelihood of sustaining the coalition, holding all other variables constant. The two sustainability actions with positive effects for sustaining the coalition were reassessing the coalition’s goals, activities, or priorities and developing an infrastructure in the community to support systems-level activities. During key informant interviews, key informants from sustained coalitions framed these sustainability actions in terms of the coalition’s natural evolution—coalitions generally moved forward with and expanded the parts of their programs that were most effective, most in-line with current funders’ priorities or most in-demand within the community.

Exhibit 30: Sustainability as a Function of Sustainability Planning and Specific Actions
Factors Predicting Sustainability Type of Association
Sustainability Plan Indicator No Association
Regular Attention to Sustainability Issues Indicator Negative Association
Reassessed Goals & Priorities Positive Association
Identified Effective Goals & Activities No Association
Developed Strategic Plan to Attain Resources No Association
Reduced Membership Negative Association
Reorganized Membership No Association
Restructured Coalition Operations No Association
Established Sustainability Committee Negative Association
Hired External Consultant for Sustainability No Association
Located Partners to Take Over Programs No Association
Developed Infrastructure in Community Positive Association
Ensured Mechanism for Policy Implementation & Enforcement No Association

Current perceptions of sustainability. Sustained coalitions were asked to report agreement with a series of statements about the coalition’s sustainability situation at the time of the survey, as shown in Exhibit 31. Eighty six percent of sustained coaltions agreed or strongly agreed that the coalitions identified alternative strategies for project survival and 78 percent agreed or strongly agreed that they have leaders who are continually planning for sustainability. Sixty two percent of sustained coaltions agreed or strongly agreed that the coaltion has sufficient funding for current operations and activities; sixty six percent agreed or strongly agreed that there is sufficient funding for the next year; and only 38 percent say the same for long-term funding. A slim majority, 53 percent, agreed or strongly agreed that they have sufficient resources to hire and retain quality staff.

Exhibit 31: Sustained Coalitions’ Current Perceptions of Sustainability
  Strongly Disagree Disagree Agree Strongly Agree No Opinion/Not Applicable
Currently, your coalition has leaders who are continually planning for sustainability. (n=109) 2% 18% 43% 35% 2%
Currently, your coalition identifies alternative strategies for project survival. (n=112) 3% 9% 53% 33% 3%
Currently, your coalition has sufficient funding for current project activities and operations. (n=113) 15% 20% 47% 15% 3%
Currently, your coalition has sufficient funding for the next year. (n=113) 11% 20% 49% 17% 3%
Currently, your coalition has sufficient funding for the long-term (2 or more years). (n=112) 16% 40% 28% 10% 6%
Currently, your coalition has sufficient funding for hiring and retaining quality staff. (n=112) 16% 28% 43% 10% 4%

Perceptions of long-term sustainability. Sustained coalitions were also asked to estimate the likelihood that the coalition will exist at several time points in the future, as shown in Exhibit 32. Only 6 percent thought the coalition was somewhat or very unlikely to exist 2 years from the time of the survey. That number doubles to 13 percent for five year survival and again to 26 percent for ten year survival. Similarly, even among those coalitions who believed it was likely they would survive over the long-term, the level of certainty decreased considerably with 75 percent saying very likely for two years, down to 40 percent for five years, and only 21 percent for ten years. Still, nearly three quarters (74%) of sustained coalitions believed they would still exist ten years out. When key informants were asked about the coalition’s perspective of the health reform changes that will be implemented as a result of the Affordable Care Act (ACA), some expected that while the nature of their activities may shift slightly, an underserved population would continue to exist. One key informant from a sustained coalition mentioned that the coalition was planning to adjust its programs to fill in gaps that would remain despite the ACA: "How do we continue to fill the gaps no matter what they are, so that we can help the community to adjust?" Another sustained coalition plans to explore opportunities for coordinating care, in the event that their safety net program for the uninsured is no longer necessary.

Exhibit 32: Coalition Perceptions of Long-term Sustainability
In your estimation, how likely is it that your coalition will continue to exist 2 years, 5 years, and 10 years from now? Very Likely Somewhat Likely Somewhat Unlikely Very Unlikely
2 years from now (n=109) 75% 19% 2% 4%
5 years from now (n=107) 40% 48% 6% 7%
10 years from now (n=108) 21% 53% 17% 9%

Coalition Impacts

Systems Impact: Coordination among Hospitals, Clinics and Providers in the Community
A sustained coalition created a streamlined eligibility intake system that provides a one-stop shop to determine the client’s eligibility for federal, state, and local coverage as well as social service and pharmacy programs. The coalition also focused on helping patients to find coverage for their care through enrollment into entitlement programs. The eligibility intake system not only helped the physicians and hospitals, but provided access to coordinated care for patients. The coalition’s activities have resulted in three different kinds of impacts.

First, the enrollment system enabled the hospitals and providers to bill for services that would otherwise be written off as charity. Second, the program provided important wrap-around services to patients that needed them. Third, the eligibility system ensured that providers have access to patients’ previous encounter information in order to provide coordinated care.

Through the literature review, broad searches for community coalition activities in public health reveal that a range of issues are being addressed (from safe streets initiatives to access for the uninsured and underinsured to diabetes management), using a range of intervention methods designed to affect change at multiple levels (e.g., individual, systems, and policy). To capture the array of accomplishments of the former HCAP coalitions, this section discusses impacts in three areas: individual impacts, systems impacts, and policy impacts. Individual impacts are changes in health or behavior at the person level (e.g. increasing screening rates in a particular population). Systems impacts are changes in infrastructure or capacity in the community (e.g. creating a county-wide case management system used by all social services agencies). Community coalitions have also achieved policy impacts at the local, state and federal levels (e.g. changing the county’s policies surrounding alcohol and tobacco billboards and storefront advertising).

Types of impacts. The coalitions were asked to provide information about the types of impacts they have realized since the end of the HCAP grant period. First, as shown in Exhibit 33, the coalitions were asked how successful they have been in achieving individual health and behavior outcomes, systems-level outcomes, and policy change outcomes since the end of the HCAP grant period. Overall, the coalitions reported fairly widespread success for all types of impacts. Sustained coalitions did differ significantly (p<.05) from not sustained coalitions as they were more likely to say individual health and behavior outcomes were not applicable (16% compared to 6%) and less likely to report unsuccessful or very unsuccessful outcomes (10% compared to 24%). Forty percent of sustained coalitions and 47 percent of not sustained coalitions reported that policy outcomes were not applicable for their coalition. One sustained coalition said that "policy work is not easy to track…we talk about tracking that, but don’t know for sure," possibly explaining the high percent of coalitions thinking that policy outcomes were not applicable to their coalition.

Exhibit 33: Impacts since Conclusion of the HCAP Grant
  Very Unsuccessful Unsuccessful Successful Very Successful Not Applicable
**p<.05, Fisher’s Exact Test
Individual Health and Behavior Outcomes**
% Sustained (n=112) 7 3 54 21 16
% Not Sustained (n=50) 10 14 48 22 6
Systems-Level Outcomes
% Sustained (n=112) 7 5 54 16 18
% Not Sustained (n=49) 10 14 43 10 22
Changing Policies
% Sustained (n=112) 4 18 29 8 40
% Not Sustained (n=49) 6 18 18 10 47

Although the survey did not provide an opportunity for the coalitions to discuss their impacts in detail, the key informant interviews and site visits allowed for more in-depth discussions. A key informant from one of the sustained coalitions reported that their program resulted in an array of impacts:

"I want people to understand that it’s the small groups that make the change in the communities. People need to redefine what sustains health care. The HCAP grant has had a huge impact. It had an impact on the residents that might not have had care prior to this, [and] had an impact in the community [which] saw this as a very good program."

Several of the sustained coalitions that participated in the site visits provided program data and evaluation reports. The coalitions frequently noted the importance of being able to demonstrate impacts, in terms of health outcomes and return on investment, as key to sustainability. Specific examples of the coalitions’ impacts discussed in the interviews and site visits are provided in Exhibit 34, and ranged from improving access and integration of health care services to new programs and initiatives.

Exhibit 34: Examples of Coalition Impacts since End of HCAP Grant
Example of Coalition Impact
Individual Impacts
Access to primary care From school-based clinics to full-service primary care, one coalition collectively provided more than 275,000 patient visits to low-income residents in 2010.
Access to primary and specialty care A sustained coalition has provided a full range of medical services, including specialty care, to 45,000 patients post-HCAP, and as of June 2011, had 7,000 enrollees in their insurance program.
Access to insurance and other programs As of 2010, a sustained coalition was able to evaluate 1,714 new patients, enroll 883 into entitlement programs, and 101 into Pharmacy Assistance Programs. In initial and follow-up surveys with their clients conducted in Fall 2010 and Spring 2011, respectively, the percentage of clients having a routine checkup in the past two years, going to a doctor’s office or clinic for medical care, experiencing better health, and having health insurance all increased.
Access to screening and care management services As of October 2011, a sustained coalition had screened over 25,000 people, enrolled roughly 25% of those people in a coverage product, and currently provides care management services for over 1,400 people. This coalition has worked with hospitals and community providers to provide over $49 million of donated medical care to their enrollees. The program has received 98% excellent on satisfaction surveys.
System impacts
Eligibility determination A sustained coalition developed a streamlined eligibility intake system that determines patient eligibility for federal, state, and local entitlement programs, social services, and pharmacy programs.
Continuum of care A sustained coalition created an improved continuum of care for patients by developing and maintaining a system which connects providers at hospitals, clinics, nursing homes and emergency management technicians.
Capacity for collaboration A sustained coalition reported that post-HCAP, they continued to build capacity for community partners to collaborate, and have been successful in identifying new opportunities and expanding needed programs and services.
Health navigation A sustained coalition is participating in the Institute for Healthcare Improvement Triple Aim Protyping Initiative. Through this project, the coalition is developing a health navigator pilot to assist uninsured patients with the transition from the hospital to primary care. The health navigator pilot has been successful in helping low-income, uninsured, recently hospitalized patients by coordinating their medical care, assisting with enrollment and eligibility, assisting with medications, facilitating primary care linkages, and accessing social services.
Care coordination An FQHC of a not sustained coalition reported that third-party payer insurance companies have found that the clinic is saving hospitals money because of its integrated and coordinated primary care and behavioral health model, pioneered through the HCAP grant.
Care coordination One sustained coalition is leading a pilot project for an ACA demonstration program that intends to follow the coalition’s program model in trying to coordinate and connect patients to additional healthcare and community resources.
Care coordination A not sustained coalition reported that their experiences in the HCAP program have positioned their FQHC for meaningful use dollars and helped them to receive patient-centered medical home recognition under the National Committee for Quality Assurance.
Information exchange One sustained coalition is one of the original fifteen Health Information Exchange Organizations serving on the Nationwide Health Information Network Trial Implementation Project supported through the Office of the National Coordinator, and today in one of only a handful of organizations across the country which is an active exchange participant.
Policy impacts
Law on kinship care A subset of one sustained coalition’s member organizations have been instrumental in increasing knowledge and awareness of caregiver families and developing more responsive public policy to address issues for these families. Their efforts contributed to the passage of a bill by the Ohio House of Representatives that became effective July 20, 2004. The law provides a grandparent, with whom a child resides, authority over their care. Other bills related to this topic have been introduced in 2005 and 2007.

Sustainability of program impacts in the absence of a coalition. The coalitions were also asked4 about how many of the coalition impacts would continue to benefit the community without additional coalition support or after the coalition disbanded. As shown in Exhibit 35, sustained and not sustained coalitions were similar in reporting how many of the individual health and behavior outcomes, and systems-level outcomes would continue to benefit the community without coalition support. However, sustained and not sustained coalitions differed significantly (p<.10) in their estimation of how many of the coalitions’ policy impacts would continue to benefit the community. Forty six percent of not sustained coalitions estimated that none or only a few of their policy outcomes would be sustained, compared to 29 percent of sustained coalitions.

Exhibit 35: Continued Impacts without Coalition Support
  All Most Some A Few None
*p<.10, Fisher’s Exact Test
Individual Health and Behavior Outcomes
% Sustained (nExhibit 35: Continued Impacts without Coalition Support=92) 16 28 30 20 5
% Not Sustained (n=44) 16 32 20 18 14
Systems-Level Outcomes
% Sustained (n=91) 21 30 24 15 10
% Not Sustained (n=37) 14 38 19 22 8
Changing Policies*
% Sustained (n=62) 27 21 23 16 13
% Not Sustained (n=26) 8 31 15 15 31

During interviews, several key informants from not sustained coalitions described the impact that their coalitions continue to have despite disbanding. In order for the coalition’s impact to continue, activities once implemented by the coalition were absorbed by other community entities. One key informant from a not sustained coalition described their coalition’s experience. In this case, the coalition’s program staff and activities were absorbed by the hospital, which was the lead organization:

"When the grant ended it didn’t make sense to continue all of the activities and it was difficult to find resources to continue them at that level. We knew the hospital wouldn’t institutionalize all our staff. We had young educated staff and they didn’t wait for the end to come, they saw the writing on the wall. By that point we were bare bones, we got down to the point where the hospital could absorb us. We whittled ourselves down to where they could."

Despite shrinking and disbanding, this coalition’s impact "continued to provide tens of thousands of dollars in free care through systems we built." Another key informant described that a pharmacy integration program launched under HCAP "has developed a life of its own" after being absorbed by the county hospital and health department. In another instance, smaller local coalitions now implement an access to care program that was launched by an HCAP coalition.

Timing of coalition impacts. As shown in Exhibit 36, the coalitions were asked about the timing of their greatest impacts. Sustained and not sustained coalitions differed significantly in their responses (p<.01). The modal category, at 52 percent, for sustained coalitions was impact occurring at a steady pace throughout the history of the coalition. The modal category for not sustained coalitions, at 53 percent, was having the greatest impact during the HCAP funding period.

Exhibit 36: Timing of Greatest Impacts

src=

Lessons Learned

Sustained and not sustained coalitions responded to a final and important question: "If you could start your coalition over again, what would you do differently?" Overall, there were three categories of responses that were repeated frequently. First, coalitions would diversify their membership. Specific examples included expanding to business, specialty care providers, private primary care providers and medical groups, and engaging all health systems in the community including competitors, consumer groups, and faith-based groups. The overall sentiment was that a more diverse membership would have better represented all perspectives of the community and the target population, as well as provided different types of knowledge, skills, and resources for the coalition. As discussed above, both sustained and not sustained coalitions emphasized the important role that institutional champions played in sustainability because of the resources and networks they offered.

Second, coalitions would establish more formal structures. The most frequently cited example was establishing formal contracts, MOUs, etc. with the member organizations that include a description of expectations and responsibilities for the member and the coalition. This desire for more formal structures was repeated in some interviews with coalitions wanting structure with formal agreeements such as a sustained coalition that reported that establishing financial roles at the beginning would help sustainability. Several coalitions also expressed a desire to increase the leadership structure through the executive committee or board of directors. Finally, several coalitions reported regret that the coalition did not establish as an independent not-for-profit.

The third frequently cited lesson learned was to focus the coalition’s mission and activities to concentrate on what the coalition did well and where it could make the greatest impact. Several coalitions stressed the importance of assessing activities regularly and being willing to let go of those that do not work well.

A common theme that emerged from not sustained coalitions during key informant interviews was the need to focus coalition activities on a common goal. Not sustained coalitions shared that main lessons learned included: "Have a clear plan about what you want to accomplish;" "have a clear goal;" "have tangible, ongoing activities to check off"; and "keep your eyes on the prize."

Additionally, coalitions frequently mentioned the need to look early and look broadly for funding sources to supplement large grants. Several coalitions also stressed the need to look for sources with longer funding cycles. A few coalitions mentioned the value of having a dedicated leader with connections throughout the community. Finally, a few coalitions emphasized the need for better data and more data support in order to improve the coalition’s capacity to measure, report, and disseminate its impacts for the community. In the interviews, a few coalitions also spoke about the need to use data to conduct assessments and to build a business case for the coalition.

Another reoccuring lesson mentioned by the coalitions was the need to build strong relationships. One coalition’s experience led them to say "the coalition needs to be based on solid relationships that have been built, or a commitment to build them, because that is what will last once the grant is over." The relationships also allow the coalition to widen the circle of their donor base because of the support from the other members.

6. Discussion and Implications

Discussion

The results of this study answer important questions about whether community coalitions formerly funded through large federal grants continue to exist after the initial federal funding, and if there are identifiable differences in the characteristics of coalitions that are sustained compared to those that are not sustained. The findings greatly improve understanding of the experience of coalitions once initial funding ends and provide important information on coalitions’ perceptions of the impact of their efforts over time. A discussion of the key findings according to the study’s research questions is presented below and followed by implications for community coalitions, planners, funders, and policy makers.

1. How many community coalitions funded under HCAP are still in existence?

2. What coalition characteristics are associated with sustainability and what factors promote or hinder community coalition sustainability?

3. What are the impacts of HCAP coalitions post-federal funding and what are the types of outcomes achieved? To what extent have the coalitions evolved to address the needs in the community?

Research Question 1: Sustainability Status

This study provided an opportunity to answer a fundamental research question: How many coalitions funded under HCAP are still in existence today? Even using a fairly rigorous definition of sustainability which required three or more organizations to be collaborating for purposes of one or more of the original HCAP goals, 68 percent of coalitions were sustained. The fact that more than two-thirds of the coalitions were sustained is encouraging, particularly given that the HCAP project did not have an explicit emphasis on sustainability. Moreover, even among the 32 percent of coalitions classified as not sustained, the majority (80% of not sustained coalitions) report that at least one of the activities they conducted under the HCAP grant was still being conducted at the time of the survey. The study results confirm that HCAP had a lasting influence on community coalitions and the communities they serve.

Research Question 2: Characteristics Associated with Sustainability

Moving beyond the basic identification of sustained coalitions, this study added to the knowledge base by identifying coalition characteristics associated with sustainability and the factors that promote or hinder community coalition sustainability. Several factors explored in the survey, key informant interviews, and case studies appear to be related to sustainability, including coalition leadership, membership diversity, coalition structure, vision guiding action, and resources.

Leadership. Good organizational leadership is often correlated with good management and successful organizational outcomes. Based on our review of the literature many aspects of leadership were explored in the survey, and several factors were found to be independently correlated with sustainability. Perhaps not surprisingly, the results of this study are consistent with others indicating that strong leadership has a direct and positive influence on the sustainability of coalitions. The regression model demonstrates that leadership experience, measured in terms of coalition management, years of experience in the field, and experience working or living in the community, increases the likelihood of sustainability even when controlling for other key predictors of coalition strength and sustainability, such as membership, funding, and coalition structure variables.

The findings from the key informant interviews and site visits also underscored the importance of good leadership on the potential for continued organizational growth and success. Coalitions with strong leaders were able to overcome many of the challenges facing coalitions as they transitioned out of the core HCAP funding and into a self-sustaining mode. For example, experienced and high quality leaders were able to communicate effectively and manage conflict at multiple levels (key executives, partner organizations, and coalition staff), which enabled the coalitions to survive challenging periods. Strong leaders were also able to help partner organizations and staff stay committed and motivated during more difficult periods.

In addition to the important role of the coalition director or manager as a leader and advocate for the coalition, coalition staff frequently discussed the value of having a champion at senior levels of the lead organization and/or partner organizations. In these cases, the coalition champions had the ear of key executives responsible for priority setting and budgetary decisions and helped ensure that continued support for the coalition, its staff, and its activities were integrated into those discussions.

Beyond leadership from individuals, the survey results revealed the importance of having governing bodies comprised of high level leaders from the membership organizations or external organizations. Coalitions with Boards of Directors or Executive Committees were more likely to be sustained. The Board of Directors and Executive Committees are defined in terms of having high level leaders from the membership organizations and/or external organizations. This type of high level or external oversight was positively associated with sustainability in the HCAP coalition population.

Coalition Membership. As with the leadership aspects, both survey and qualitative data collected in this study confirm the literature that various aspects of coalition membership are associated with coalition sustainability. Although survey results showed no significant differences in the size of sustained and not sustained coalitions, the proportion of active members was a significant predictor of sustainability controlling for other coalition characteristics. This point was echoed in the qualitative findings where a key theme emerged that it was the quality of the members’ participation that mattered far more than absolute quantity of members.

Perhaps surprisingly, high membership turnover was not a significant, negative predictor of sustainability in the model. Comparing sustained and not sustained coalitions, the findings suggest that it may be beneficial to have some churning of the membership. Additionally, the turnover among sustained coalitions tended to be driven by issues or changes at the member organization level and not due to constraints at the coalition level. The loss of a member organization that may have been distracted or distracting to the coalition’s efforts may ultimately strengthen the commitment of the remaining organizations.

The survey results also showed that sustained coalitions were significantly more likely to have diversity in the types of members involved, both within the health sector and across sectors (e.g., social service representation). The qualitative results reveal that different types of members bring different perspectives and types of resources, financial and in-kind, to the coalition. Engaging partners across diverse sectors was also noted as a source of new ideas and new perspectives in addressing problems.

Structure. Results of the influence of coalition structure on sustainability were a departure from expectations. Findings from the 2006 national HCAP evaluation suggested that the coalitions with more formal mechanisms in place to maintain a coalition structure would be a significant predictor of sustainability. However, sustained and not sustained coalitions had equal rates of formal membership structures and the vast majority with formal structures (92% and 89%, respectively) had MOAs, MOUs, or IAAs in place with members. Survey results indicated that not sustained coalitions were significantly more likely to hold more frequent meetings compared to sustained coalitions. Qualitative findings support that sentiment with several sustained coalitions discussing the need to reduce the frequency of meetings and increase the quality of meetings. Not sustained coalitions mentioned that the number of meetings and operational requirements for members may have been too burdensome to sustain.

Although not a facet of structure included in the survey component, we did observe that four of the six coalition case study sites, which represented sustained and high performing coalitions, had a hub and spoke organizational model. In these models, the core coalition staff served as the hub, coordinating all operational activities for the coalition. These coalitions had varying levels of inter-member activity, though at the minimum all have mechanisms in place for periodic inter-member discussions and any major decisions were discussed and resolved as a group. Interviews with member organizations of these hub and spoke coalitions suggest that the model provides an efficient and effective organizational structure that contributes to the coalition’s success and sustainability.

No matter the membership and structural model, coalitions ultimately do require collaboration across organizations. These findings suggest that coalitions must be mindful of a careful balance between establishing and maintaining formal structures that facilitate membership and activity and formal structures that yield too much bureaucracy and hinder member organization participation. Additionally, the type of formal member structure should be tailored to individual coalition needs. As noted in additional detail below, coalitions need resources and personnel to manage these structural and operational issues.

Vision Guiding Action. The ability to stay focused on the overall goals of the coalition while maintaining action steps that are manageable in the day to day was positively associated with sustainability. Survey results showed a high level of agreement among sustained coalitions about the importance of the coalition’s vision and using the vision to focus activities and services. These results are consistent with recent literature and research attention on collective impact organizations like the Strive Partnership, which helps communities build their civic infrastructure to support educational support for children from "cradle to career" based on a shared mission, mutually reinforcing activities, continuous communication, common metrics, and the presence of a core organization dedicated staff and resources (Hanleybrown, Kania, and Kramer, 2012; Strive Network, 2011).

Qualitative data strongly supported two particular aspects of the concept of vision guiding action: shared sense of mission and mission-driven organizations. From the perspective of interviewed and case study coalitions, these factors are key contributors to sustainability. A shared mission among member organizations was responsible for the initiation of several of the case study coalitions and that shared mission was essential in overcoming inter-member obstacles. For example, two competing hospitals were working together on a coalition because they shared the same mission and saw the value of servicing the mission as paramount. Additionally, being mission driven provided some member organizations with the justification they needed to continue providing in-kind support or donated services to the coalition. This concept is described further in the discussion of evaluation below.

Resources. Findings on the role of resources for sustainability are consistent with the literature on coalition effectiveness. Sustained coalitions were significantly more likely to have funding from more diverse sources, including state level agencies and departments (other than the health department), local health departments, foundations, and community-based organizations. Fifty percent of sustained coalitions said they had more diverse funding at the time of the survey than during HCAP. Additionally, on the survey as well as in the telephone interviews and site visits, many sustained coalitions indicated that member organizations contribute funds for coalition operations and programs or services. Some coalitions had this formalized though coalition membership dues and others determined the need and availability of funds from members on an annual or budget cycle basis. Member contributions in some coalitions were relatively large with organizations contributing twenty thousand dollars or more per year.

Perhaps not surprisingly, controlling for other coalition characteristics, the regression model showed that a higher degree of flexibility to use funding for operational or programmatic purposes was a positive predictor of sustainability. Even though greater flexibility was beneficial, 40 percent of sustained coalitions reported less flexibility now than during the HCAP period, suggesting that the sustained coalitions continued to adapt their operational approach to the constraints.

Survey results showed that sustained coalitions received more types of in-kind support and this was a significant predictor in the model even controlling for other characteristics. Additionally, the qualitative data suggested that sustained coalitions have done an excellent job of acquiring in-kind support for specific functions that require a skill-set or specific type of employee the coalition could not support on its own. These services included processing claims data, financial analysis, grant writing, and database and software development.

Although sustained and not sustained coalitions did not vary significantly in terms of the number of full time equivalent staff reported on the survey, sustained coalitions participating in the site visits clearly articulated the importance of having a dedicated position for coalition operations in order to nurture the coalition itself, and maintain and grow it. Serving as an advocate for the coalition among leadership, members, and the community was a large part of these individuals’ responsibilities. Sustained coalitions have struggled to maintain this level of support over time with 58 percent of sustained coalitions reporting that they had less funding for coalition operations at the time of the survey than they did during the HCAP period.

While these specific aspects of funding and resources impacted sustainability, survey and qualitative results suggested that the sustained coalitions have gotten by under resource constrained conditions. Even with a history of sustainability now in place, 62 percent of sustained coalitions reported feeling less certain about future funding at the time of the survey than they did during the HCAP period.

Evaluation. Evaluation was considered an important aspect of coalition work for both sustained and not sustained coalitions. Coalitions noted that evaluation activities were important for fine tuning the coalition’s work and for demonstrating results to funders and key stakeholders. Survey results revealed that sustained coalitions were more likely to use quantitative evaluation methods and to conduct outcomes evaluations than not sustained coalitions. Qualitative data supported this finding with several coalition members from the interviewed and site visited coalitions discussing the importance of having concrete figures to show key stakeholders the impact of the coalition’s work. Further, respondents mentioned the benefits of evaluation in the context of sustainability because they credited the ability to demonstrate health and social outcomes as a powerful tool in seeking additional resources and support both from formal funding opportunities as well as organizational member support and in-kind donations.

The ability to make a business case that shows how resources are leveraged by the coalition and how coalition activities save the lead organization and member organizations money is also important. While some sustained coalitions had sophisticated means for collecting, processing, and analyzing these data (e.g., dummy claims processing through Blue Cross Blue Shield for all enrolled patients), other simple analyses were effective too (e.g., adding the value of all donated pharmaceuticals, lab, and imagery services and dividing by the coalition’s total funding in order to demonstrate that every dollar of funding yields so many more dollars in patient products and services).

While sustained coalitions tended to have more sophisticated evaluation methods and processes, they explained that the anecdotes were important too. Quotes from participating providers and patient success stories provided an important publicity and communication tool for stakeholders focused on the mission but who might skim over tables of outcomes and financial data. Additionally, the anecdotal information combined with evaluation results proved an effective tool for supporting the morale of coalition staff and further strengthening the bonds between individuals working in separate organizations towards a common goal.

Contextual Factors. In qualitative interviews and case studydiscussions, the coalitions provided information on the key contextual factors that affected their coalitions in both positive and negative ways. With the exception of a ubiquitous impact of the economic downturn, which has increased demand for their services, but tightened resources like funding and donated services and products, the contextual factors were unique to the communities being served. However, one theme that emerged across these particular contextual factors is that sustained coalitions possessed a resiliency to handle the external shocks and were able to absorb these exogenous events.

To the extent possible, these coalitions planned for such events. Since planning for and reacting to these contextual factors happened primarily with the coalition director and senior leadership, some of the success in handling these situations was undoubtedly tied to strong leadership. Additionally, the ability of these sustained coalitions to be resilient to these shocks was facilitated by the fact that they have the support for these core personnel, who could focus on the issues rather than being completely dedicated to programmatic and service activities.

Additionally, sustained coalitions did their best to capitalize on positive contextual factors, looking for opportunities to enhance the coalition’s value to the community. For example, one coalition was positioning itself within their state’s discussion of patient centered medical home models and programs under the Affordable Care Act. The coalition was using its experiences with the medical home model for the uninsured to contribute to the discussion, as well as looking for new opportunities for the coalition to expand coalition activities such as participating in the design and implementation of the state’s community care teams.

Sustainability Perceptions, Planning and Actions. As noted above, sustainability can be defined in terms of whether organizations are still working together, whether specific activities continue, or some combination. This study required continued collaboration among organizations in order to be considered sustained, though the survey instrument also sought the perspectives of coalition staff on how they defined sustainability. The vast majority (84%) of responses tended to include some aspect of institutionalized activities or lasting impacts in their definition of sustainability. Only 16 percent of all coalitions defined sustainability only in terms of the coalition membership and structures. Additionally, a greater proportion of sustained coalitions defined sustainability in terms of the coalition and its activities compared to the not sustained coalitions in which the majority defined sustainability in terms of activities or impacts continuing to benefit the community. In qualitative interviews, coalitions discussed both aspects when they talked about their long-term sustainability. While, as one sustained coalition noted, the activities of the coalition are vital, "someone needs to be here to coordinate services."

Interest in sustainability issues is understandably quite high and there is limited data on the practical concerns of maintaining a coalition once initial funding has ceased. Surprisingly, these results indicated that having a sustainability plan had no impact on sustainability status and establishing a sustainability committee actually decreased the likelihood of being sustained. Rather, sustained coalitions differed in terms of the sustainability actions they undertook such as reassessing goals and priorities, developing a strategic plan for resources, and reorganizing coalition membership. The qualitative interviews suggested that sustained coalitions were always thinking about what needed to happen for sustainability, such as obtaining additional types of funding, developing a focus on showing results, getting institutional support for coalition activities, evolving with the community needs, incorporating new partners, as well as strategizing to handle external factors like the Affordable Care Act. While sustainability plans were an important first step, it was coalitions’ ability to carry out sustainability actions that differentiated the sustained from not sustained.

This project examined coalitions that have successfully sustained themselves over several years following the HCAP funding and the case studies explored the experiences of several high functioning coalitions. Even though these coalitions could be considered "successful" in terms of their continued existence, some coalitions remained apprehensive about the long-term stability of their coalition. Two-thirds of the sustained coalitions said they have sufficient funding for the next year, but that proportion drops to only 38 percent when asked to forecast for two years out. Nearly all of the sustained HCAP coalitions (68%) are confident they will exist in two years, but less than three-quarters of the sustained HCAP coalitions believe they will exist in 10 years. Coalition work is a challenging endeavor and even the most successful coalitions face uncertainty which may encourage greater attention to long-term planning and emphasis on action steps.

Research Question 3: Coalition Evolution and Impacts Post-Federal Funding

Finally, results from this study provide an understanding of the extent to which former HCAP coalitions have evolved to address the needs of their communities, and the impacts and outcomes they have achieved post-federal funding. The discussions below explore the changes in coalitions’ activities, and the individual-, system-, and policy-level impacts that they achieved.

Coalition Activities. HCAP coalitions were brought together initially to undertake a particular set of activities, and activities are still a core part of what coalitions are doing. The most common types of activities conducted by the coalitions initially included programs and services (e.g., eligibility and enrollment or patient navigation) and systems change (e.g., integrated data systems or coordinating pro bono care). Capacity building activities (e.g., providing TA to member organizations) were also prevalent among sustained and not sustained during the HCAP period.

Sustained coalitions did adapt their activities over time. Half of the sustained coalitions conducted capacity building activities during the HCAP grant, but significantly fewer (41%) did so at the time of the survey. This suggests that the need for capacity building may lesson as the coalition matures, but does not diminish completely. This may be related to mature coalitions’ continued adaptation to new issues or activities and the incorporation of new partners over time as the coalition evolves to address the changing needs of the community. As the proportion of effort spent on capacity building activities decreased, the percentage of sustained coalitions conducting policy activities (e.g., working with local officials) increased significantly. The increase may be a function of coalition evolution or may in fact be related to the increasing trend of greater emphasis on policy activities and broader attempts to make more "upstream" impacts.

A final point about the HCAP coalitions’ activities relates to the not sustained coalitions. Even among the 32 percent of coalitions that were not sustained, the majority reported that at least one of the activities they undertook as part of HCAP was still being conducted at the time of the survey. If the activity initiated under HCAP fulfilled an existing community need this significantly contributes to the positive impacts of the HCAP program.

Coalition Impacts. Overall, coalitions reported that they have continued to make individual, systems, and policy level impacts since the end of the HCAP grant. This was especially true for individual and systems level outcomes which were perceived as being part of their core functions. The coalitions were far more likely to say that impacting the policy-level was not applicable to them compared to individual and systems level impacts. This perspective may reflect the influence of the health care and human service provider sectors on the coalition. While HCAP coalitions recognized the increasing emphasis on promoting policy change as a way to promote health, most do not yet understand how to make a direct impact in the policy sphere or how to measure and track those impacts.

Sustained coalitions participating in interviews and site visits provided more specific examples and evidence of these impacts through program reports and internal statistics. In the post-HCAP period, the coalitions have achieved many different types of outcomes yielding significant improvements for the uninsured and underinsured populations and safety net system in their communities. Additionally, the majority of sustained coalitions believed that they have made impacts at a steady pace throughout the coalition’s lifespan (from pre-HCAP to the time of survey). Less than a quarter of sustained coalitions thought that their greatest impact occurred during the HCAP period.

Coalitions generally felt that they needed to sustain the coalition itself in order for their work to continue improving their communities. Fewer than half of sustained coalitions believed that all or most of their individual health and behavior outcomes would continue to benefit the community if the coalition had to disband or dissolve. Just over half said the same about systems-level outcomes and just under half about policy level outcomes. Coalitions serve to bring organizations together, to leverage and coordinate disparate resources in the community, and to fill a gap in the local health and social service system. Without the actual coalition structure in place, it is not surprising that people anticipate significant reductions in perceived benefits.

Implications for Community Coalitions, Planners, Evaluators, and Funders

Findings from this study have implications for community coalitions at all stages of development, as well as evaluators and funders of coalitions. Findings suggest that coalitions should invest time and resources into developing characteristics and capacities that facilitate sustainability. Such actions might include identifying leaders with experience working in the community and committed staff that are highly effective managers; incorporating diverse, multi-sectorial partner organizations with a shared sense of mission and a willingness to invest in the coalition’s success; pursuing diverse funding sources that will finance both the coalition’s programs and its operations; and investing in robust outcome evaluations that demonstrate the coalition’s performance and impact.

For federal and community planners, this study demonstrates the value of using community coalitions to achieve community health goals. Post-initial federal funding, 113 (68%) of the HCAP community coalitions were sustained, and nearly half of the sustained coalitions are addressing all of their original goals from the HCAP grant. These coalitions are still actively pursuing a range of activities in their communities, and have achieved both individual-level impacts (e.g. increasing access to primary and specialty care, increasing enrollment into health insurance plans) and other policy and system impacts (e.g. streamlined eligibility systems, new processes for care coordination across community providers). Federal and community planners should consider using community coalitions when developing community-level interventions. Given their ability to sustain themselves, and their resiliency, the HCAP coalitions have demonstrated the power of community coalitions as a vehicle for creating lasting community change.

Findings also speak to the risks associated with over-engineering sustainability plans, when it is action that matters. Sustainability actions such as developing plans, holding meetings, and convening special sustainability committees are not enough to sustain a coalition. Rather, this study found that a coalition’s time is best spent taking discrete action steps towards sustainability—whether that is securing funding or institutionalizing some program activities in other organizations.

This study also supports the importance of evaluation and using performance data to demonstrate outcomes. The federal government and other funders regularly require community coalitions and other programs to develop SMART goals and evaluation measures to track their progress and outcomes over time. To the extent that coalitions can invest in outcome evaluations, they will be better able to respond to funders’ evaluation requirements and also demonstrate the return on investment of their programing to coalition partners and other community stakeholders.

Finally, coalitions should discuss sustainability expectations with their funders, and define their sustainability goals at the outset. This study defined sustainability in terms of the continuation of the coalition and its goals. However, funders may choose to define sustainability in terms of the continuation or institutionalization of specific coalition activities. The conceptual framework developed as part of this study can help community coalitions have this conversation with both their funders and members, and define expectations—which will, in turn, help to direct their sustainability actions.

This study also has implications for researchers and practitioners who evaluate the sustainability of community coalitions. In crafting program evaluations of coalition sustainability, evaluators must carefully identify their definition of sustainability and use this definition to shape their study. Is the evaluator interested in the sustainability of the coalition or in the sustainability of its activities and impacts? The answer to this question will impact the evaluator’s research questions, hypotheses and activities. Additionally, findings from this study suggest that, when assessing coalition sustainability and outcomes, evaluators should pay close attention to the presence of specific characteristics and capacities that facilitate sustainability. This study presented numerous comparisons of the characteristics and capacities of sustained and not sustained coalitions, which may provide a useful starting point for framing future evaluation studies.

The study also has implications for funders of community coalitions. The federal government and foundations continue to invest in coalitions to improve health and other outcomes at the local level. When providing initial program funding for community coalitions, funders must consider whether they expect the coalitions to continue post-grant. If so, funders and coalitions can collaborate during the grant period to lay the necessary groundwork for sustainability.

Policy Implications

Community coalitions are a promising approach for implementing key aspects of the Affordable Care Act (Patient Protection and Affordable Care Act, 2010). The Affordable Care Act established a Prevention and Public Health Fund for public health activities, which states and communities are using to support community and clinical prevention activities. Given the Prevention Fund’s focus on empowering communities to apply evidence-based, population-based, culturally-appropriate interventions, policy makers should consider funding community coalitions to lead the charge. The former HCAP community coalitions are already working towards the goals of the Prevention Fund engaging in activities such as controlling the obesity epidemic through targeted interventions that focus on physical activity and nutrition; delivering preventive health screenings; conducting health education; and improving access to health care and other prevention services.

Another priority of the Affordable Care Act is to enroll individuals into health insurance exchanges. The Affordable Care Act created state-based exchanges and small business exchanges for individuals and small businesses where they can purchase qualified coverage. Policy makers have an opportunity to leverage diverse, multi-sectorial networks that have the ability to effectively identify and enroll individuals into these exchanges. Community coalitions—and the organizations that participate in them such as community and faith-based organizations—are particularly well-suited to enroll individuals into these exchanges and provide guidance during the enrollment process. Some of the former HCAP coalitions have established highly effective eligibility and enrollment systems and processes, and are familiar with the intricacies of matching the needs of an individual with the optimal health coverage products. Further, community coalitions that serve the most vulnerable populations will be particularly important in this process, given that these organizations are already established in the community and have developed and maintained the trust of the health care and social service communities and the target populations.

Implications for Future Programs

This study identified a few key facilitators of community coalition sustainability and coalition building. First, findings suggest that leadership is one of the key facilitators of community coalition sustainability. Policy makers may consider investing in local leaders and building leadership capacity in communities across the country through technical assistance and training opportunities. The most effective and highly sustainable coalitions were led by leaders with prior experience working within their community, suggesting that leadership training in community-based participatory approaches may be valuable. Policy makers should also consider investing in other mechanisms for supporting sustainability, such as funding rigorous data collection and evaluation to encourage results-oriented planning and assist coalitions in appealing to additional funders.

Findings from this study demonstrate that even ten years after HCAP began, many sustained and not sustained coalitions are still investing in capacity building and developing their networks and partnerships. Even sustained coalitions that have been operating at a particularly high level and are achieving important systems and policy level changes, still find that they need to invest resources into maintaining and nurturing their networks and partnerships. While capacity building activities may wane over time, policy makers should structure funding to account for an ongoing investment in these activities throughout the life of the coalition.

Additionally, this study provides guidance to policy makers as they invest in new demonstration programs in support of the Affordable Care Act. First, policy makers may use the findings from this study to develop standards for sustainability planning. Currently, many federal programs require community coalition grantees to develop a sustainability plan post-award. However, other programs do not require grantees to document their sustainability plans at the outset. Given that this study suggests the importance of preparing for sustainability through concrete action steps (rather than just plans), policy makers may be interested in developing tools that provide guidance about the specific actions coalitions can take to sustain themselves. This may be particularly important given that this study found that developing a sustainability plan at the outset is not enough.

Additionally, in developing future programs, policy makers should consider supporting core coalition operations to increase the likelihood of long-term sustainability. Many of the former HCAP coalitions have been able to continue to deliver programming over time because they had funding for core operations as well as programming purposes. Further, the coalitions that thrived post-funding were able to continuously support at least a few FTEs to manage and convene the coalition; these coalitions asserted that their sustainability would not have been possible in the absence of operations funding. In a down economy, federal funding for core coalition operations may enable coalitions to sustain themselves until it is possible to secure new funds.

Finally, policy makers may provide differing amounts of funding for differing amounts of time depending upon the coalition’s particular situation—for example, by making smaller, shorter duration grants available to highly effective and sustainable coalitions that have a history of programming. Such funding opportunities will help coalitions to build resiliency in difficult times (e.g., to weather an economic downturn or the loss of a key funding partner) without detracting from their momentum in the community.

Areas for Future Research

Researchers are investigating the formation and development of community coalitions—from their initial mobilization and development of key structures and processes, to their implementation and sustainability. Our findings shed light on the sustainability outcomes of community coalitions and the factors that impact sustainability. However, there is a dearth of empirical information on what happens to coalitions once they have formed, developed, and reached a sustained state of activity. One opportunity for future research is to study coalitions that have reached a sustained state—one, five, and ten years post-initial funding. Future research should explore whether there are different stages of coalition sustainability, how coalitions mature and evolve in each of these stages, and whether their characteristics and capacities change over time.

Another area for future research is identifying funding models for coalition sustainability and the optimal length of funding. For example, researchers should explore the sustainability of coalitions that received different types of funding through different funding models: coalitions that received decreasing annual resources (e.g., 100 percent year one, 50 percent year two, 10 percent year three to five); matching funds only; short-term capacity building or operations funding; and long-term infrastructure funding. While sustainability is a process that occurs over time, funders and policy makers would benefit from more information about the optimal length of time to fund a coalition or program to achieve its goals.

Finally, no single study can investigate every coalition characteristic or capacity that may affect sustainability. Future research should continue to explore different factors, their interactions, and conditions that foster community coalition sustainability.

7. Community Coalition Sustainability Bibliography

Alexander, J.A., Weiner, B.J., Metzger, M.E., Shortell, S.M., Bazzoli, G.J., Hasnain-Wynia, R., Sofaer, S., & Conrad, D.A. (2003). Sustainability of collaborative capacity in community health partnerships. Medical Care Research and Review, 60, 130S–160S.

American Diabetes Association. (2009). Taking on diabetes: Evaluation of performance measurement/outcomes. Retrieved December 14, 2009, from http://www.takingondiabetes.org/measurement/performance.htm

Backer, T. (2003). Evaluating community collaborations (1st ed.). New York: Springer Publishing Company.

Bailey, D., & McNally Koney, K. (1995). An integrative framework for the evaluation of community-based consortia. Evaluation and Program Planning, 18(3): 245–252.

Bazzoli, G.J., Casey, E., Alexander, J.A., Conrad, D.A., Shortell, S.M., Sofaer, S., Hasnain-Wynia, R., & Zukoski, A.P. (2003). Collaborative initiatives: Where the rubber meets the road in community partnerships. Medical Care Research and Review, 60(4 Suppl.), 63S–94S.

Beckham, B., & King, J. (2005). Communication in coalitions. Ohio State University Fact Sheet CDFS-6. Retrieved December 2, 2009, from http://ohioline.osu.edu/bc-fact/0006.html

Beery, W.L., Senter, S., Cheadle, A., Greenwald, H.P., Pearson, D., Brousseau, R., & Nelson, G.D. (2005). Evaluating the legacy of community health initiatives: A conceptual framework and example from the California Wellness Foundation’s Health Improvement Initiative. American Journal of Evaluation, 26(2), 150–165.

Bennett, C., & Rockwell, K. (1995). Targeting outcomes of programs (TOP): An integrated approach to planning and evaluation. Unpublished manuscript. Lincoln, NE: University of Nebraska.

Berkowitz, B. (2001). Studying the outcomes of community-based coalitions. American Journal of Community Psychology, 29(2), 213–227.

Berkowitz, B., & Wolff, T. (1999). The spirit of the coalition (1st ed.). Washington, DC: American Public Health Association.

Braithwaite, R.L., Murphy, F., Lythcott, N., & Blumenthal, D.S. (1989). Community organization and development for health promotion within an urban black community: A conceptual framework. Health Education, 20(5), 56–60.

Bryson, J.M. (1988). A strategic planning process for public and non-profit organizations. Long Range Planning, 21(1),73–81.

Butterfoss, F.D. (2007). Coalitions and partnerships in community health. San Francisco: Jossey-Bass.

Butterfoss, F.D., & Francisco, V.T. (2004). Evaluating community partnerships and coalitions with practitioners in mind. Health Promotion Practice, 5(2), 108–114.

Butterfoss, F.D., Goodman, R.M., Wandersman, A., Valois, R.F., & Chinman, M.J. (1996a). The plan quality index: An empowerment evaluation tool for measuring and improving the quality of plans. In D.M. Fetterman, S.J. Kaftarian, & A. Wandersman, Empowerment evaluation: Knowledge and tools for self-assessment and accountability (pp. 304–331). Thousand Oaks, CA: Sage.

Butterfoss, F.D., Goodman, R.M., & Wandersman, A. (1996b). Community coalitions for prevention and health promotion: Factors predicting satisfaction, participation, and planning. Health Education Quarterly, 23(1), 65–79.

Butterfoss, F., Goodman, R., & Wandersman, A. (1993). Community coalitions for prevention and health promotion. Health Education Research, 8(3), 315–330.

Butterfoss, F.D., & Kegler, M.C. (2002). Toward a comprehensive understanding of community coalitions: Moving from practice to theory. In R.J. DiClemente, R.A. Crosby, & M.C. Kegler (Eds.), Emerging theories in health promotion practice research: Strategies for improving public health (pp. 157–193). San Francisco: Jossey-Bass.

Butterfoss, F., Lachance, L., & Orians, C. (2006). Building allies coalitions: Why formation matters. Health Promotion Practice, 7(2 Suppl.), 23S–33S.

Brown, C. The art of coalition building. (1984). New York: American Jewish Committee.

California Healthy Cities and Communities Program. (2008). 20 year anniversary series report. Retrieved December 14, 2009, from http://www.cjaonline.net/TechnicalResources.htm.

Cheadle, A., Wagner, E., Koepsell, T., Kristal, A., & Patrick, D. (1992). Environmental indicators: A tool for evaluating community-based health-promotion programs. American Journal of Preventive Medicine, 8(6), 345–350.

Community Intervention Trial for Smoking Cessation (COMMIT): Cohort results from a four-year community intervention. (1995). American Journal of Public Health. Retrieved December 12, 2009, from http://ajph.aphapublications.org/cgi/content/abstract/85/2/183.

Cramer, M.E., Mueller, K.J., & Harrop, D. (2003). Comprehensive evaluation of a community coalition: A case study of environmental tobacco smoke reduction. Public Health Nursing, 20(6), 464–477. doi: 10.1046/j.1525–1446.2003.20607.x.

Conrad, D.A., Cave, S.H., Lucas, M., Harville, J., Shortell, S.M., Bazzoli, G.J., Hansain-Wynia, et al. (2003). Community care networks: Linking vision to outcomes for community health improvement. Medical Care Research and Review, 60(4 Suppl.), 95S–129S.

Cutler, I. (2002). End games: The challenge of sustainability. Baltimore, MD: The Annie E. Casey Foundation.

Easterling, D. (2003). What have we learned about community partnerships? Medical Care Research and Review, 60(4 Suppl.), 161S–165S.

Edwards, J.C., Feldman, P.H., Sangl, J., Polakoff, D., Stern, G., & Casey, D. (2007). Sustainability of partnerships projects: A conceptual framework and checklist. The Joint Commission Journal on Quality and Patient Safety, 33(12), 37–47.

el Ansari, W.E., & Phillips, C.J. (2001). Interprofessional collaboration: A stakeholder approach to evaluation of voluntary participation in community partnerships. Journal of Interprofessional Care, 15(4), 351–368.

Evaluation of AHRQ's Partnerships for Quality Program. Program evaluation. (2006). AHRQ Publication No. 08-M010-EF, December 20. Agency for Healthcare Research and Quality, Rockville, MD. Retrieved December 14, 2009, from http://www.ahrq.gov/about/evaluations/partnerships/

Fawcett, S.B., Paine-Andrews, A., Francisco, V.T., Schultz, J., Richter, K.P., Berkley-Patton, J. , Fisher, J.L., Lewis, R.K., Lopez, C.M., Russos, S., Williams, E., Harris, K.J., & Evensen, P. (2001). Evaluating community initiatives for health and development. WHO Regional Publications. European Series, (92), 241–270.

Fawcett, S.B., Paine-Andrews, A., Francisco, V.T., Schultz, J.A., Richter, K.P., Lewis, R.K., Williams, E.L., Harris, K.J., Berkley J.Y., Fisher, J.L., Lopez, C.M. (1995). Using empowerment theory in collaborative partnerships for community health and development. American Journal of Community Psychology, 23(5), 677–697.

Feighery, E., & Rogers T. (1990). How-to guides on community health promotion: Guide 12. Building and maintaining effective coalitions. Palo Alto, CA: Stanford Health Promotion Resource Center.

Feinberg, M.E., Bontempo, D.E., & Greenberg, M.T. (2008). Predictors and levels of sustainability of community coalitions. American Journal of Preventive Medicine, 34(6), 495–501.

Fetterman, D., & Wandersman, A. (2007). Empowerment evaluation: Yesterday, today, and tomorrow. American Journal of Evaluation, 28(2), 179–198. doi: 10.1177/1098214007301350.

Fetterman, D.M., & Wandersman, A. (2005). Empowerment evaluation principles in practice (1st ed.). New York: Guilford Press.

Foster-Fisherman, P.G., Berkowitz, S.L., Lounsbury, D.W., Jacobson, S., & Allen, N.A. (2001). Building collaborative capacity in community coalitions: A review of the Integrative Framework. American Journal of Community Psychology, 29(2),241–261.

Francisco, V.T., Paine, A.L., & Fawcett, S.B. (1993). A methodology for monitoring and evaluating community coalitions. Health Education Research, 8(3), 403–416.

Friedman, A.R., & Wicklund, K. (2006). Allies against asthma: A midstream comment on sustainability. Health Promotion Practice, 7(2 Suppl.), 140S–148S.

Goodman, R.M., Speers, M.A., McLeroy, K., Fawcett, S., Kegler, M., Parker, E., Smith, S.R., Sterling, T.D., & Wallerstein, N. (1998). Identifying and defining the dimensions of community capacity to provide a basis for measurement. Health Education and Behavior, 25(3), 258–278.

Granner, M.L. & Sharpe, P.A. (2004). Evaluating community coalition characteristics and functioning: A summary of measurement tools. Health Education Research, 19(5), 514–532.

Habana-Hafner, S., Reed, H.B., & Associates. (1989). Parternships for community development: Resources for practitioners and trainers. Amherst, MA: University of Massachusetts Centers for Organizational and Community Development.

Hallfors, D., Cho, H., Livert, D., & Kadushin, C. (2002). Fighting back against substance abuse: Are community coalitions winning? American Journal of Preventive Medicine, 23(4), 237–245.

Hanleybrown, F, Kania, J, & Kramer, M. (2012). Channeling change: Making collective impact work. Stanford Social Innovation Review, Winter 2012.

Hasnain-Wynia, R. (2003). Overview of the community care network demonstration program and its evaluation. Medical Care Research and Review, 60(4 Suppl.), 5S–16S.

Hawe, P., King, L., Noort, M., Jordens, C., & Lloyd, B. (2000). Indicators to help with capacity building in health promotion. NSW Health. NSW Government. Retrieved February 18, 2010, from http://www.health.nsw.gov.au/pubs/2000/pdf/capbuild.pdf

Hays, C.E., Hays, S.P., DeVille, J.O., & Mulhall, P.F. (2000). Capacity for effectivensss: The relationship between coalition structure and community impact, Evaluation and Program Planning, 23(3), 373–379.

Health Care Safety Net Amendments of 2002. (2002). Pub. L. no. 107-251, 116 Stat 1621.

Katz, D., & Kahn, R.L. (1978). The social psychology of organizations (2nd ed.). New York: Wiley.

Kegler, M., Steckler, A., McLeroy, K., & Malek, S. (1998). Factors that contribute to effective community health promotion coalitions: A study of ten project ASSIST coalitions in North Carolina. Health Education and Behavior, 25(3), 338–353.

Kegler, M.C., Twiss, J.M., & Look, V. (2000). Assessing community change at multiple levels: The genesis of an evaluation framework for the California Healthy Cities Project. Health Education & Behavior, 27(6), 760–779.

Kluhsman, B.C., Bencivenga, M., Ward, A.J., Lehman, E., & Lengerich, E.J. (2006). Initiatives of 11 rural Appalachian cancer coalitions in Pennsylvania and New York. Preventing Chronic Disease, 3(4), A122.

Koepsell, T.D., Wagner, E.H., Cheadle, A.C., Patrick, D.L., Martin, D.C., Diehr, P.H., Perrin, E.B., Kristal, A.R., Allan-Andrilla, C.H., & Dey, L.J. (1992). Selected methodological issues in evaluating community-based health promotion and disease prevention programs. Annual Review of Public Health, 13, 31–57. doi: 10.1146/annurev.pu.13.050192.000335.

Kumpher, K., Turner, C., Hopkins, R., & Librett, J. (1993). Leadership and team effectiveness in community coalitions for the prevention of alcohol and other drug abuse. Health Education Research, 8(3), 359-374.

Lasker, R.D., & Weiss E.S. (2003). Broadening participation in community problem solving: A multidisciplinary model to support collaborative practice and research. Journal of Urban Health, 80(1),14–47.

Laverack, G., & Wallerstein, N. (2001). Measuring community empowerment: A fresh look at organizational domains. Health Promotion International, 16(2), 179–185.

Leviton, L.C., Herrera, C., Pepper, S.K., Fishman, N., & Racine, D.P. (2006). Faith in action: Capacity and sustainability of volunteer organizations. Evaluation and Program Planning, 29(2), 201–207.

Lodl, K., & Stevens, G. (2002). Coalition sustainability: Long-term successes and lessons learned. Extension Journal, 40(1).

Mancini, J.A., & Marek, L.L. (2004). Sustaining community-based programs for families: Conceptualization and measurement. Family Relations, 53(4), 339–347.

Meek, J. (1992). How to build coalitions: Turf issues. Ames, IA: Iowa State University Extension. Retrieved February 18, 2010, from http://www.extension.iastate.edu/Publications/PM1486B.pdf

Minkler, M., & Wallertstein, N. (2005). Improving health through community organization and community building: A health education perspective. In M. Mikler (Ed.), Community organizing and community building for health (2nd ed., pp. 26). New Brunswick, NJ: Rutgers University Press.

Mittelmark, M.B., Hunt, M.K., Heath, G.W., & Schmid, T.L. (1993). Realistic outcomes: Lessons from community-based research and demonstration programs for the prevention of cardiovascular diseases. Journal of Public Health Policy, 14(4), 437–462.

National Opinion Research Center [NORC]. (2007). The National Evaluation of the Healthy Communities Access Program. Program evaluation. Prepared for the Health Resources and Services Administration. Unpublished.

NORC. (2010). Developing a Conceptual Framework to Assess the Sustainability of Community Coalitions Post-Federal Funding, Report to the Office of the Assistant Secretary for Planning and Evaluation, U.S. Department of Health and Human Services. January 7, 2010. Available: http://aspe.hhs.gov/health/reports/2010/sustainlit/report.shtml.

Patient Protection and Affordable Care Act of 2010, S 4002, 111 Cong., 1st Sess.

Payne, C.A. (1999). The challenges of employing performance monitoring in public health community-based efforts: A case study. Journal of Community Health, 24(2), 159–170.

Pluye, P., Potvin, L. Denis, J.L., & Pelletier, J. (2004). Program sustainability: Focus on organizational routines. Health Promotion International, 19(4), 489–500.

Rog, D., Boback, N., Barton-Villagrana, H., Marronne-Bennett, P., Cardwell, J., Hawdon, J., Diaz, J., Jenkins, P., Kridler, J., & Reischl, T. (2004). Sustaining collaboratives: A cross-site analysis of the national funding collaborative on violence prevention. Evaluation and Program Planning, 27(3), 249-261.

Scheirer, M.A. (2005). Is sustainability possible? A review and commentary on empirical studies of program sustainability. American Journal of Evaluation, 26, 320-347.

Shediac-Rizkallah, M.C., & Bone, L.R. (1998). Planning for the sustainability of community-based health programs: Conceptual frameworks and future directions for research, practice and policy. Health Education Research, 13(1), 104.

Stevens, B., & Peikes, D. (2006). When the funding stops: Do grantees of the Local Initiative Funding Partners Program sustain themselves? Evaluation and Program Planning, 29(2), 153–161.

Stephens, S., & Studdiford, D. (2008). Community use of capacity building supports as of mid-2007. Center for Assessment and Policy Development. Retrieved December 14, 2009, from http://www.capd.org/pubfiles/pub-2008-04-01.pdf

Strive Partnership. (2011). Strive Network. Retrieved March 21, 2012, from http://strivenetwork.org/strive-network.

Swerissen, H., & Crisp, B.R. (2004). The sustainability of health interventions from different levels of social organization. Health Promotion International, 19(1), 123-130.

Taylor-Powell, E., Rossing, B., & Geran, J. (1998). Evaluating collaboratives: Reaching the potential. Madison,WI: University of Wisconsin-Extension. Retrieved December 14, 2009, from http://www.uwex.edu/ces/pdande/evaluation/evaldocs.html

Valente, T.W., Coronges, K.A., Stevens, G.D., & Cousineau, M.R. (2008). Collaboration and competition in a children’s health initiative coalition: A network analysis. Evaluation and Program Planning, 31(4), 392– 402

Vicary, J.R., Doebler, M.K., Bridger, J.C., Gurgevich, E.A., & Deike, R.C. (1996). A community systems approach to substance abuse prevention in a rural setting. Journal of Primary Prevention, 16(3), 303–318.

Wagenaar, A.C., & Wolfson, M. (1993). Tradeoffs between science and practice in the design of a randomized community trial. In T.K. Greenfield, & R. Zimmerman, Experiences with community action projects: New research in the prevention of alcohol and other drug problems (pp. 119–129). Rockville, MD: Center for Substance Abuse Prevention CSAP Prevention Monograph-14. HHS Pub. No. (ADM) 93-1976.

Webber, H.S. & Karlström, M. (2009). Why community investment is good for nonprofit anchor institutions: Understanding costs, benefits, and the range of strategic options. Chicago, IL: Chapin Hall at the University of Chicago.

Weiss, H., Coffman, J., & Bohan-Baker, M. (2002). Evaluation’s role in supporting initiative sustainability. Urban Series on Children’s Health and Safety. Cambridge, MA: Harvard Family Research Project. Retrieved December 10, 2009, from http://www.hfrp.org/publications-resources/browse-our-publications/evaluation-s-role-in-supporting-initiative-sustainability

Wolff, T. (2001). Community coalition building—Contemporary practice and research: Introduction. American Journal of Community Psychology, 29(2),165–172.

Wolff, T. (2001b). A practitioner’s guide to successful coalitions. American Journal of Community Psychology, 29(2), 173–191.

West, K., Libero, F., & Shelton, A. (2005). HCAP report: Analysis of one year of impacts. Olympia, WA: Communities Joined in Action. Retrieved December 14, 2009, from http://www.cjaonline.net/TechnicalResources.htm

Footnotes

1 For the purposes of this report, subsequent references to HCAP grantees include both CAP and HCAP-funded grantees.

2 Of the 95 coalitions that did not complete the survey, 8% were cohort I, 18% cohort II, 23% cohort III, 8% cohort IV, 15% cohort V, 19% cohort VI, and 9% cohort VII, which matches the distribution of all grantees by cohort.

3 Eighty three percent of not sustained coalitions answered "yes" to the question, "Did your coalition receive financial resources from any of the following sources?: U.S. Department of Health and
Human Services" This reflects HCAP and other sources of DHHS funding at the time the coalition disbanded or dissolved.

4 Coalitions that replied Not Applicable for the question on the success of different impacts did not receive this follow-up question.

Files
rpt.pdf (pdf, 942.2 KB)
Topics
Home- & Community-Based Services (HCBS)