An Assessment of the Sustainability and Impact of Community Coalitions once Federal Funding has Expired. Study Design and Data Collection Techniques


The study design included four components that were subsequently carried out. The first phase consisted of the literature review and development of the conceptual framework presented in Section 3 (see also NORC, 2010). The next step involved a self-administered questionnaire sent to all HCAP grantees. Next, a subset of grantees that responded to the survey was chosen for interviews with key informants from the coalition. The final phase included site visits with a smaller subset of high performing, sustained coalitions and discussions with lead and partner organizations.

Self-administered questionnaire. The literature review and conceptual framework supported the development of a survey to assess the HCAP coalitions based on the sustainability definition and the factors driving sustainability included in the conceptual framework. The survey was administered to all 260 HCAP grantees between March 10 and May 31, 2011. Contact information for each coalition director was obtained using grantee records from HRSA and confirmed using web searches, the White Pages, and when necessary, confirmation emails or phone calls. A mailed, self-administered questionnaire was sent to the grantees, along with a cover letter describing the study, providing contact information, and explaining that their participation was voluntary. The cover letter also provided an email address and toll-free number if the designated respondent did not feel qualified to provide content. When a more knowledgeable contact was provided for a coalition, a new survey packet was mailed.

The questionnaire included a screening question to determine the coalition’s sustainability status, 66 questions for sustained coalitions, and 52 questions for not sustained coalitions. The survey included questions regarding the coalition’s structure, mission and goals, funding sources, activities, evaluation methods, sustainability plans, and overall impact. Non-responders received follow-up prompting via postcard, phone, and emails at two week intervals throughout the field period that included a link to a web-based version of the questionnaire. Exhibit 4 provides a breakdown of the responses received and the mode of completion. During the fielding period, three coalitions were identified as duplicate cases. These coalitions received a second grant, a continuation grant, or had merged with another HCAP coalition and responded as a single case. NORC achieved a 63 percent response rate. All analyses are conducted on complete cases only (n=165).

Exhibit 4: Coalition Survey Response by Mode of Administration
Sustainability of Community Coalitions Number Percent
Paper SAQ 65 25
Web SAQ 99 38
Phone Administration 1 0
Partial Complete 18 7
No Respondent with Knowledge of HCAP Grant 5 2
Coalition Disbanded and No Respondent with Knowledge of HCAP Grant 2 1
Nonresponders 67 26
Duplicate Cases 3 1
Total 260 100

The responding HCAP coalitions showed variability on a number of key characteristics. There was considerable variation in the size of the population served by the HCAP coalitions. The average size of the population served by the sustained coalitions was 5.7 million (median=380,800), with a range of [5,000, 300,000,000]. The average size of the population served by the not sustained coalitions was 7.4 million (median=307,896), with a range of [1,000, 50,000,000]. There was no statistically significant difference in the size of community served by sustained versus not sustained coalitions.

As shown in Exhibit 5, both sustained and not sustained coalitions served diverse populations in term of racial and ethnic composition, urbanicity, insurance status, and income. Additionally, sustained coalitions were significantly more likely than not sustained coalitions to serve urban (p<.01), White (p<.01), Hispanic/Latino (p<.10), Asian American/Pacific Islander (p<.01), Native American (p<.01), and mixed race respondents (p<.01). Not sustained coalitions were more likely to report serving underinsured/underserved populations (p<.05).

Exhibit 5: Characteristics of Community Served by Coalition
Demographics of Population Served by Coalition % Sustained Coalitions (n=112) % Not Sustained Coalitions (n=52)
*p<.10; **p<.05; ***p<.01, two-tailed difference of proportions test
Urban*** 81 52
Rural 66 63
Suburban 44 33
Uninsured 95 94
Underinsured/underserved** 89 98
White*** 92 79
African American 85 83
Hispanic/Latino* 89 79
Asian American/Pacific Islander*** 63 33
Native American*** 48 25
Mixed race or other racial or ethnic group (others included Latino mixed race, Hmong, immigrants, refugees, and Haitians)*** 55 27
Low-income 100 100
Middle-income 38 42
High-income 12 6
Other characteristics (others include undocumented workers, seniors, frontier communities, specific conditions like asthma) 15 12

Interviews with sustained and not sustained coalitions. From the pool of grantees that completed the survey (113 (68%) sustained and 52 (32%) not sustained), a subset of 25 (15%) coalitions including 16 sustained and 9 not sustained was selected to participate in telephone interviews. Telephone interviews with key informants of these coalitions were conducted in August and September 2011. The purpose of the interviews was to gather more detailed information and confirm survey responses regarding the coalitions’ experience and strategies for trying to sustain the coalition after federal funding ended, coalition outcomes, and future plans. The key informant interviews allowed for an evenly distributed sampling of cohort, region, urban versus rural, and sustained versus not sustained. HCAP cohorts I-VII were all represented and each cohort had grantees from different regions except those in cohort V where all coalitions were from the Northeast region. All cohorts also had a mix of sustained and not sustained grantees that were interviewed.

Site visits with sustained coalitions. From the group of key informant interviews, we invited six diverse coalitions (sustained and expanded) to participate in a site visit. These coalitions represented high-performing grantees from across the nation. Two grantees from each of the Western, Middle, and Eastern Regions of the United States were chosen. Two members from the NORC team traveled to each location to conduct site visits. All site visits occurred during October 2011. The site visits involved in-depth interviews with the coalition directors and lead staff and representatives from the coalition’s partner organizations. NORC staff facilitated the interviews using a semi-structured interview protocol that focused on facilitators and obstacles of sustainability, the coalition’s structure and dynamics, and lessons learned. Case study reports on the site visits are provided in Appendix A.

While valuable, findings from the key informant interviews and case studies should be considered from a qualitative frame of reference. Results are not generalizable to the entire population of HCAP coalitions. The key informant and case study data are integrated with the quantitative findings throughout the report. The qualitative data are most often utilized to provide additional in-depth explanation, context, and nuance to support the survey findings.

View full report


"rpt.pdf" (pdf, 942.2Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®