The assessment of potential community-level impacts of the CRRI grants was always dependent on data elements to be collected by the grantees. Union City provided youth and adult survey data, as well as community data elements, across several points in time. Lorain needed some additional support from Westat to get its community survey launched, but eventually did so and was able to report both survey and some limited community data elements. Fall River did not report any community-level data.
Other place-based initiatives have noted the importance of conducting community surveys to assess the impact of the initiative. The Annie E. Casey Foundation, for example, funded a 10-year neighborhood improvement initiative called Making Connections that aimed to enhance various community protective factors (e.g., parental economic stability, community ties) for vulnerable youth. In a post hoc assessment of the initiative,11 the survey was explained as follows:
Making Connections presented several reasons to rely on surveys as a primary evaluation tool: the Foundation hoped to measure changes in civic participation, social ties, and other key outcomes that aren't reflected in administrative records. Moreover, local administrative data weren't defined and collected in the same ways in all Making Connections sites, making it hard to pool or compare data.
To ensure consistency across the grantee sites (i.e., comparable frames and sampling methodology and data collection procedures), the Foundation hired a survey research company to conduct this piece of the evaluation.
Although other recent literature reviewing place-based initiatives indicates that measuring the effectiveness of these efforts remains problematic,12 there is continued work to develop tools for measuring resilience. These include the Conjoint Community Resilience Assessment Measure, authored by Aharnonson-Daniel and Lahad 2012,13 which consists of 32 items on a five point Likert scale that measures several key factors, such as residents' faith in community leaders, a sense of community efficacy ("I believe that my community has the ability to overcome crisis"), the respondent's sense that the community is prepared to deal with emergency situations, the respondent's attachment to the community ("I feel that I belong to the place where I live"), and two items on the quality of relationships between community members.
The Prevention Institute has developed a Toolkit for Health and Resilience in Vulnerable Environments,14 an assessment tool that measures certain dimensions of community resilience, including residents' mental health, substance abuse, and other aspects of emotional well-being. Communities Advancing Resilience Toolkit (CART), discussed in Pfefferbaum et al. 2013,15 is both an intervention and a measurement that contains 21 items rated on a Likert scale. Examples include "People in my community are committed to the well-being of the community" and "My community looks at its successes and failures so it can learn from the past." These and other similar tools might be considered by SAMHSA for incorporating into the evaluation of any future place-based initiatives.
Collecting meaningful community-level indicators by using existing datasets has also been pointed out as particularly challenging. For example, in a "best-practices" manual published subsequent to the evaluation of First 5 LA's Best Start initiative, the authors noted the difficulty of identifying public datasets that provide the same information (measured in the same way) across multiple sites with sufficient geographic precision. In addition, and perhaps more importantly, they point out that place-based initiatives are focused on low-income neighborhoods, but cities are comprised of both low-income and high-income residential areas. Even city-specific data may not reveal initiative-induced changes because local-level changes may not prove statistically significant when viewed within the context of a larger, mixed-income population.16
To address this problem, the Urban Institute has established a National Neighborhood Indicators Partnership (http://neighborhoodindicators.org/) in which they work with local organizations to develop locally relevant datasets. With funding from John D. and Catherine T. MacArthur Foundation and the McKnight Foundation, they (like the evaluators in Making Connections) are working to create shared indicators with common measures that will be collected by all of the locales participating in an initiative. Another source of data that may be helpful for future place-based projects is the National Neighborhood Indicators Partnership to determine if existing (or even future) grantees might be able to capitalize on this work. Examination of this resource during the program design stage may enable the identification of existing standard data across communities.
Finally, the CRRI grants were unexpectedly short-lived. The literature on place-based initiatives (e.g., Making Connections) clearly indicates that they require substantial investments of time to effect any community-wide change. This is because community initiatives are complex and are part of a social change process that is larger than a single agency or even a coalition of agencies. In the CRRI projects, the shortened timeframe for planning, launching, and implementing the programs created a split focus for the sites. On top of that considerable effort, they did not receive expected continuation funding and had to begin planning for sustainability just as they were getting established. The planning committees at each site thus had to figure out how to salvage what had been developed and rework their plans because of the unanticipated ending. Future place-based initiatives should account for the timeframe needed to implement community-wide programs. Grantees need an opportunity to work out an early implementation challenges, including staffing, determining if selected interventions need to be reconsidered (e.g., the demise of the Strengthening Families efforts in all three sites), learning GPRA reporting requirements, and the like. Only once these challenges have been addressed can they put their full efforts into project implementation. And it is only once these programs have been fully established that one can reasonably assess their impacts on the local communities.