An Assessment of the Sustainability and Impact of Community Coalitions once Federal Funding has Expired. Evaluation Activities

06/01/2012

Both sustained and not sustained coalitions reported a focus on several key evaluation activities, as shown in Exhibit 19. At least 70 percent of the sustained and not sustained coalitions agreed or strongly agreed that they developed evaluation plans for programmatic activities and carried out evaluations of their activities, programs, and services on a regular basis. Additionally, evaluations were used to demonstrate project effectiveness. More sustained (34%) and not sustained (24%) coalitions disagreed or strongly disagreed with a statement that evaluations of the coalition’s core operations were conducted regularly.

Similarly, as shown in Exhibit 20, there was a heavy emphasis for both sustained and not sustained coalitions on conducting evaluation activities in order to demonstrate results to funders and to compete for future resources, as well as for long-term planning and modifying programs and services. Fewer, though still over 70 percent, sustained and not sustained coalitions reported that evaluation activities were important for modifying coalition operations. In key informant interviews, numerous sustained coalitions shared that data collected through evaluation activities demonstrated that programs launched as part of HCAP were effective in helping to reduce hospital admissions and hospital inpatient visits, and saved hospitals money as a result.

Exhibit 19: Focus on Evaluation
Strongly Disagree Disagree Agree Strongly Agree No Opinion/Not Applicable
Your coalition develops evaluation plans prior to implementing programs, services, and activities.
Sustained Coalitions (n=113) 0% 18% 50% 27% 4%
Not Sustained Coalitions (n=51) 2% 16% 41% 37% 4%
Evaluations of your coalition’s core operations (e.g., community communications) are conducted on a regular basis.
Sustained Coalitions (n=113) 5% 29% 45% 12% 8%
Not Sustained Coalitions (n=50) 0% 24% 38% 34% 4%
Evaluations of your coalition’s programs, services, and activities are conducted on a regular basis.
Sustained Coalitions (n=113) 2% 9% 56% 30% 4%
Not Sustained Coalitions (n=50) 0% 8% 52% 38% 2%
Project effectiveness is demonstrated through evaluation.
Sustained Coalitions (n=113) 1% 15% 46% 31% 7%
Not Sustained Coalitions (n=51) 0% 14% 45% 39% 2%

 

Exhibit 20: Evaluation Purposes
  Very Unimportant Somewhat Unimportant Somewhat Important Very Important No Opinion/Not Applicable
Demonstrating results to your community
Sustained (n=113) 3% 7% 19% 67% 4%
Not Sustained (n=52) 4% 13% 23% 60% 0%
Demonstrating results to your funders
Sustained (n=113) 4% 1% 9% 81% 5%
Not Sustained (n=51) 4% 4% 16% 76% 0%
Competing for funding more successfully
Sustained (n=113) 5% 3% 17% 70% 5%
Not Sustained (n=51) 4% 4% 18% 71% 4%
Modifying coalition operations (e.g., staff, convening membership meetings)
Sustained (n=113) 5% 13% 33% 40% 8%
Not Sustained (n=51) 2% 18% 37% 41% 2%
Modifying coalition programs, services, and activities
Sustained (n=113) 4% 7% 29% 54% 6%
Not Sustained (n=51) 4% 10% 39% 45% 2%
Long-term/sustainability planning
Sustained (n=113) 3% 2% 14% 75% 6%
Not Sustained (n=50) 2% 10% 38% 48% 2%

The key informant interviews illustrated that coalitions collected data through evaluation to demonstrate program effectiveness to funders and stakeholders. One sustained coalition presented evaluation data to a hospital where the program was being implemented in order to show the hospital the benefits their patients are reaping for a minimal investment from the hospital. A key informant noted: "Even when they donate care, [the hospital gets] more out of it than they give, and we can prove that to everyone now." Another key informant from a sustained coalition said that:

Our sustainability planning is very data-driven. Right now, our strategy is to show hospitals how much money they are collecting directly as a result of the [coalition’s] services. The long-term strategy for sustainability is that hospitals have to support it, whether or not additional grant funding comes in or not.

One key informant from a sustained coalition noted that evaluation has facilitated the coalition’s expansion within the community: "We have been able to show, through data collected, that access to health care has a direct correlation to reducing visits to the hospital and inpatient visits. We have also been able to establish other relationships with other providers in the community." Internally, having data on coalition impact helped coalition members and staff to celebrate their successes. A key informant from a sustained coalition said: "We have to often pump each other up by looking at our anecdotal and data-driven success. The value of the work that we do is manifested in the people that we serve." When asked about the use of specific evaluation methodologies, a few significant differences emerged between sustained and not sustained coalitions. Sustained coalitions utilized outcome evaluations and quantitative methods at significantly higher rates than not sustained coalitions, as shown in Exhibit 21. Less than 10 percent of sustained and not sustained coalitions were utilizing newer methods including evaluation capacity building with member organizations and empowerment evaluation, or using experimental or quasi-experimental methods.


Exhibit 21: Evaluation Methods

Exhibit 21: Evaluation Methods

*p<.10; ***p<.01, two-tailed difference of proportions test

View full report

Preview
Download

"rpt.pdf" (pdf, 942.2Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®